Human motion health monitoring method and device based on dual reference representation

Abstract

The invention discloses a human motion health monitoring method and device based on dual reference representation, a storage medium and electronic equipment, belongs to the field of artificial intelligence and time sequence classification, and aims to solve the technical problem of how to extract various complex features of human motion signals from different angles so as to obtain accurate and rich feature representation and improve the human motion health monitoring effect. According to the technical scheme, the method comprises the steps that S1, a human body movement health monitoring data set is obtained, wherein the human body movement health monitoring data set disclosed on the network is downloaded; s2, constructing a human motion health monitoring model: constructing the human motion health monitoring model based on dual reference representation; and S3, training the human motion health monitoring model: training the human motion health monitoring model constructed in the step S2 on the human motion health monitoring data set obtained in the step S1.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and time series classification technology, and in particular to a method and device for monitoring human motion health based on dual reference representation. Background Art

[0002] Human motion health monitoring is a technology that monitors individual health by monitoring and analyzing motion sensor data on wearable devices. These movements include but are not limited to daily movements such as walking, running, sitting, standing, climbing stairs, and descending stairs, as well as more specific movements such as cooking, cleaning, and using electronic devices. Human motion health monitoring usually relies on time series data collected from wearable devices, smartphones, or environmental sensors, such as the output of sensors such as accelerometers, gyroscopes, magnetometers, and heart rate monitors. As a cutting-edge scientific and technological application, the value and importance of human motion health monitoring technology in the medical field are self-evident. For example, human motion health monitoring technology can provide safety protection for the elderly or patients with chronic diseases by real-time tracking and analyzing individuals' daily movements, and timely detect abnormal behaviors such as falls, so as to intervene in time, which is of great significance to improving the quality of life of patients and reducing medical costs.

[0003] A core difficulty of the human motion health monitoring task lies in the complexity of motion signals and the sensitivity to subtle fluctuations. Motion signals are usually composed of time series data in multiple dimensions, which reflect the dynamic behavior of individuals when performing specific movements. However, due to the diversity and individual differences in human motion, even seemingly simple movements can have very complex signal characteristics. First, subtle fluctuations in motion signals can lead to changes in motion patterns. For example, between walking and jogging, an individual's movement speed and rhythm may only differ slightly, but these differences may appear as small changes in acceleration or angular velocity in sensor data. If the recognition model cannot accurately capture and distinguish these subtle fluctuations, it may lead to erroneous monitoring results.

[0004] In order to solve the above problems, the present invention introduces a method and device for human motion health monitoring based on dual reference representation. Based on dual reference representation, the method and device enable the model to extract multiple complex features of human motion signals from different angles, thereby obtaining accurate and rich feature representation, thereby improving the accuracy of human motion health monitoring. Summary of the invention

[0005] The technical task of the present invention is to provide a human motion health monitoring method and device, storage medium, and electronic device based on dual reference representation, so as to learn the complex characteristics of human motion signals, thereby improving the accuracy of human motion health monitoring.

[0006] The present invention adopts the following technical means to achieve the invention purpose:

[0007] A human motion health monitoring method based on dual reference representation is characterized in that the method comprises the following steps:

[0008] S1. Obtaining a human motion health monitoring dataset: Downloading a human motion health monitoring dataset publicly available on the Internet;

[0009] S2. Constructing a human motion health monitoring model: Constructing a human motion health monitoring model based on dual reference representation;

[0010] S3. Training a human motion health monitoring model: training the human motion health monitoring model constructed in step S2 on the human motion health monitoring data set obtained in step S1.

[0011] As a further limitation of the present technical solution, step S2 is used to construct a pre-trained input module, construct a lightweight temporal encoder, construct a dual reference representation generation module, construct a dual reference representation comparison module, construct a fine-tuning input module and construct a fine-tuning module, thereby constructing a human motion health monitoring model; the aforementioned modules correspond to sub-steps, respectively, recorded as S201, S202, S203, S204, S205, and S206.

[0012] As a further limitation of the present technical solution, the S201 constructs a pre-training input module, extracts a piece of data from the human motion health monitoring data set obtained in step S1, the data includes a piece of human motion signal data collected from the sensor, thereby obtaining a piece of unlabeled human motion signal data, denoted as T = {x1, x2, x3, ..., x T}, where x i Represents the human motion signal data at the i-th time point.

[0013] As a further limitation of the present technical solution, the S202 constructs a lightweight time series encoder, uses a convolutional network as a component, and encodes the unlabeled human motion signal data constructed in S201 to obtain the unlabeled original time series features, which are recorded as The specific implementation is shown in the following formula:

[0014]

[0015] Among them, T represents the unlabeled human motion signal data, from S201, Encoder lite (·) indicates a lightweight temporal encoder.

[0016] As a further limitation of the technical solution, the S203 constructs a dual reference representation generation module, receives the unlabeled human motion signal data constructed in S201, converts it into image representations using two imaging time series algorithms, and encodes the two image representations using a temporal image coding network, thereby obtaining reference representation 1 and reference representation 2;

[0017] Specifically, the implementation process of the dual reference representation generation module is as follows:

[0018] S20301, using the data enhancement technology Jitter to add Gaussian noise to the unlabeled human motion signal data constructed in S201, and then using the Markov transfer field imaging time series algorithm to transform it, so as to obtain the Markov imaging representation, denoted as The specific implementation is shown in the following formula:

[0019]

[0020] Among them, {x1, x2, x3, …, x T} represents the unlabeled human motion signal data constructed by S201, {ε1, ε2, ε3, …, ε T} represents standard Gaussian noise, MTF(·) represents the Markov transition field imaging time series algorithm;

[0021] S20302. Use the data enhancement technology Time-warping to perform nonlinear distortion processing on the unlabeled human motion signal data constructed in S201, and then use the Gram angular field imaging time series algorithm to transform it to obtain the Gram imaging representation, which is recorded as The specific implementation is shown in the following formula:

[0022]

[0023] Among them, {x1, x2, x3, …, x T} represents the unlabeled human motion signal data constructed by S201, τ(·) represents the warping function for warping all time points based on a smooth curve, and GAF(·) represents the Gram angular field imaging time series algorithm;

[0024] S20303. Use a temporal image coding network to extract features from the Markov imaging representation and the Gram imaging representation, respectively, to obtain reference representation 1 and reference representation 2. Specifically, the implementation process of the temporal image coding network is as follows: first, use the Vision Transformer encoder to perform feature encoding on the Markov imaging representation and the Gram imaging representation, respectively, and then use the Transformer encoder to extract features from them, respectively, to obtain reference representation 1 and reference representation 2, denoted as and The specific implementation is shown in the following formula:

[0025]

[0026] Where ViT(·) represents the Vision Transformer encoder, represents the Markov imaging representation, from formula (2) of S20301, represents the Gram imaging representation, from formula (3) of S20302.

[0027] As a further limitation of the present technical solution, the S204 constructs a dual reference representation comparison module, receives the original temporal coding representation output from the lightweight temporal encoder S202 and the reference representation 1 and the reference representation 2 output from the dual reference representation generation module S203, and then uses the dual reference representation comparison strategy to calculate the dual reference representation comparison loss, and uses it to optimize the model parameters;

[0028] Specifically, the implementation process of the dual reference representation comparison module is as follows:

[0029] S20401. Use a multi-layer perceptron network to map the original temporal coding representation, reference representation 1, and reference representation 2, thereby obtaining the original temporal mapping representation, reference mapping representation 1, and reference mapping representation 2, which are recorded as and The specific implementation is shown in the following formula:

[0030]

[0031] in, represents the original temporal coding representation, from formula (1) of S202, Reference representation 1, formula (4) from S20303, represents reference representation 2, from formula (5) of S20303, MLP(·) represents multi-layer perceptron network;

[0032] S20402. For the i-th original time series mapping representation in the current input batch, calculate the distance between it and all other original time series mapping representations in the current batch input, and record the minimum distance as The specific implementation is shown in the following formula:

[0033]

[0034] in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, Indicates that the current batch input does not contain The k-th original time series mapping representation is from formula (6) of S20401, where k∈[1, i)∪(i, batch], batch represents the current input batch size, Distance(·, ·) represents the distance formula, and min() represents obtaining the minimum value of the input list in the brackets;

[0035] S20403. Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 1, denoted as Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 2, denoted as Calculate the distance between the i-th reference mapping representation 1 and the i-th reference mapping representation 2 in the current input batch, denoted as The specific implementation is shown in the following formula:

[0036]

[0037] in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, representing 1, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, represents 2, formula (6) from S20401, Distance(·,·) represents the distance formula;

[0038] S20404, based on formula (8) from S20403 and Calculate the relative loss, denoted as L rel , then based on formula (8) from S20403 and Calculate the absolute loss, denoted as L abs, and then add the relative loss to the absolute loss to get the dual reference representation contrast loss, denoted as L; the specific implementation is shown in the following formula:

[0039]

[0040]

[0041] L=L rel +L abs (11)

[0042] Here, max(·,·) means obtaining the maximum value of the two inputs.

[0043] As a further limitation of the present technical solution, the S205 constructs a fine-tuning input module, extracts a piece of data from the human motion health monitoring data set obtained in step S1, the data including a piece of human motion signal data collected from the sensor and a corresponding motion label, thereby obtaining a piece of labeled human motion signal data, denoted as T f = {T, label}, where T = {x1, x2, x3, …, x T}, and x i Represents the human motion signal data at the i-th time point;

[0044] The S206 constructs a fine-tuning module, and uses a pre-trained lightweight temporal encoder to encode T in the aforementioned labeled human motion signal data, thereby obtaining a labeled original temporal feature, which is recorded as Then it is passed to the output network layer; the output network layer maps it, and then obtains the probability that the labeled human motion signal data belongs to different human motion categories, denoted as P; the specific implementation is shown in the following formula:

[0045]

[0046] Among them, Encoder lite (·) indicates a lightweight temporal encoder.

[0047] A human motion health monitoring device based on dual reference representation, characterized in that the device comprises:

[0048] A human motion health monitoring data set acquisition unit is used to download a human motion health monitoring data set publicly available on the Internet;

[0049] A human motion health monitoring model construction unit is used to construct a pre-training input module, a lightweight temporal encoder, a dual reference representation generation module, a dual reference representation comparison module, a fine-tuning input module, and a fine-tuning module, thereby constructing a human motion health monitoring model;

[0050] The human motion health monitoring model training unit is used to construct a pre-training loss function, a pre-training optimization function, a fine-tuning loss function, and a fine-tuning optimization function, thereby completing the pre-training, fine-tuning, and prediction of the model.

[0051] An electronic device, characterized in that it comprises: a memory and at least one processor;

[0052] Wherein, the memory stores a computer program;

[0053] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the human motion health monitoring method based on dual reference representation as described above.

[0054] A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and the computer program can be executed by a processor to implement the human motion health monitoring method based on dual reference representation as described above.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The present invention can efficiently extract key timing features in human motion signals through a lightweight timing encoder module, while maintaining the computational efficiency of the model, thereby achieving rapid processing and analysis of human motion signals;

[0057] (2) The present invention can extract complex features of human motion signals from different angles through a dual reference representation generation module, thereby obtaining accurate and rich feature representations, thereby improving the accuracy and reliability of the model for human motion health monitoring;

[0058] (3) The present invention uses a dual reference representation comparison module to enhance the model's sensitivity and recognition ability to abnormal changes, thereby improving the accuracy and reliability of the model's monitoring of human movement health;

[0059] (4) The present invention can optimize the model structure according to specific data through the fine-tuning module, thereby improving the generalization ability and applicability of the model;

[0060] (5) The method and device proposed in the present invention, combined with dual reference representation, can learn the complex characteristics of human motion signals, thereby improving the accuracy of human motion health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flow chart of a method for human motion health monitoring based on dual reference representation

[0062] Figure 2 Flowchart for constructing a human motion health monitoring model

[0063] Figure 3 Flowchart for training the human motion health monitoring model

[0064] Figure 4 Flow chart of a human motion health monitoring device based on dual reference representation

[0065] Figure 5 Structural diagram for dual reference representation

[0066] Figure 6 Schematic diagram of the framework of the human motion health monitoring model based on dual reference representation. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Embodiment 1:

[0069] The overall model framework structure of the present invention is as follows Figure 6 As shown. Figure 6It can be seen that the main framework of the present invention includes two stages, namely the pre-training stage and the fine-tuning stage. The framework structure of the pre-training stage includes a lightweight temporal encoder, a dual reference representation generation module and a dual reference representation comparison module, and the framework structure of the fine-tuning stage includes a lightweight temporal encoder and a fine-tuning module; wherein the lightweight temporal encoder is shared by the pre-training stage and the fine-tuning stage framework. In the pre-training stage, the lightweight temporal encoder receives unlabeled human motion signal data and encodes it to obtain unlabeled original temporal features, which are then passed to the dual reference representation comparison module; the dual reference representation generation module receives unlabeled human motion signal data and converts it into image representation using two imaging time series algorithms, and encodes the two image representations using a temporal image coding network to obtain reference representation 1 and reference representation 2, which are then passed to the dual reference representation comparison module; the dual reference representation comparison module receives the original temporal coding representation output by the lightweight temporal encoder and the reference representation 1 and reference representation 2 output by the dual reference representation generation module, and then uses the dual reference representation comparison strategy to calculate the dual reference representation comparison loss, which is used to optimize the model parameters. In the fine-tuning stage, the pre-trained lightweight time series encoder receives labeled human motion signal data and encodes it to obtain labeled original time series features, which are then passed to the fine-tuning module; the fine-tuning module receives the labeled original time series features, maps them, and then obtains the probability that the human motion signal data belongs to different human motion categories, and feeds back the prediction results to the relevant personnel.

[0070] Embodiment 2:

[0071] The dual reference representation generates a module such as Figure 5 The dual reference representation generation module receives the unlabeled human motion signal data constructed by S201, and converts it into image representations using two imaging time series algorithms, and uses a temporal image coding network to encode the two image representations, thereby obtaining reference representation 1 and reference representation 2;

[0072] Specifically, the implementation process of the dual reference representation generation module is as follows:

[0073] S20301, using the data enhancement technology Jitter to add Gaussian noise to the unlabeled human motion signal data constructed in S201, and then using the Markov transfer field imaging time series algorithm to transform it, so as to obtain the Markov imaging representation, denoted as The specific implementation is shown in the following formula:

[0074]

[0075] Among them, {x1, x2, x3, …, x T} represents the unlabeled human motion signal data constructed by S201, {ε1, ε2, ε3, …, ε T} represents standard Gaussian noise, MTF(·) represents the Markov transition field imaging time series algorithm;

[0076] For example, in pytorch, the code described above is implemented as follows:

[0077]

[0078]

[0079] Among them, sample represents human motion signal data, and markov_images represents Markov imaging representation.

[0080] S20302. Use the data enhancement technology Time-warping to perform nonlinear distortion processing on the unlabeled human motion signal data constructed in S201, and then use the Gram angular field imaging time series algorithm to transform it to obtain the Gram imaging representation, which is recorded as The specific implementation is shown in the following formula:

[0081]

[0082] Among them, {x1, x2, x3, …, x T} represents the unlabeled human motion signal data constructed by S201, τ(·) represents the warping function for warping all time points based on a smooth curve, and GAF(·) represents the Gram angular field imaging time series algorithm;

[0083] For example, in pytorch, the code described above is implemented as follows:

[0084]

[0085]

[0086] Among them, sample represents human motion signal data, and gramian_images represents Gram imaging representation.

[0087] S20303. Use a temporal image coding network to extract features from the Markov imaging representation and the Gram imaging representation, respectively, to obtain reference representation 1 and reference representation 2. Specifically, the implementation process of the temporal image coding network is as follows: first, use the Vision Transformer encoder to perform feature encoding on the Markov imaging representation and the Gram imaging representation, respectively, and then use the Transformer encoder to extract features from them, respectively, to obtain reference representation 1 and reference representation 2, denoted as and The specific implementation is shown in the following formula:

[0088]

[0089] Where ViT(·) represents the Vision Transformer encoder, represents the Markov imaging representation, from formula (2) of S20301, represents the Gram imaging representation, from formula (3) of S20302.

[0090] For example, in pytorch, the code described above is implemented as follows:

[0091]

[0092]

[0093]

[0094] Among them, sample represents human motion signal data, Markov_Images represents Markov imaging representation, Gramian_Images represents Gram imaging representation, feature_Markov represents reference representation 1, and feature_Gramian represents reference representation 2.

[0095] Embodiment 3:

[0096] The dual reference representation comparison module receives the original temporal coding representation output from the lightweight temporal encoder S202 and the reference representation 1 and the reference representation 2 output from the dual reference representation generation module S203, and then uses the dual reference representation comparison strategy to calculate the dual reference representation comparison loss, and uses it to optimize the model parameters;

[0097] Specifically, the implementation process of the dual reference representation comparison module is as follows:

[0098] S20401. Use a multi-layer perceptron network to map the original temporal coding representation, reference representation 1, and reference representation 2, thereby obtaining the original temporal mapping representation, reference mapping representation 1, and reference mapping representation 2, which are recorded as and The specific implementation is shown in the following formula:

[0099]

[0100] in, represents the original temporal coding representation, from formula (1) of S202, Reference representation 1, formula (4) from S20303, represents reference representation 2, from formula (5) of S20303, MLP(·) represents multi-layer perceptron network;

[0101] S20402. For the i-th original time series mapping representation in the current input batch, calculate the distance between it and all other original time series mapping representations in the current batch input, and record the minimum distance as The specific implementation is shown in the following formula:

[0102]

[0103] in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, Indicates that the current batch input does not contain The k-th original time series mapping representation is from formula (6) of S20401, where k∈[1, i)∪(i, batch], batch represents the current input batch size, Distance(·, ·) represents the distance formula, and min() represents obtaining the minimum value of the input list in the brackets;

[0104] S20403. Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 1, denoted as Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 2, denoted as Calculate the distance between the i-th reference mapping representation 1 and the i-th reference mapping representation 2 in the current input batch, denoted as The specific implementation is shown in the following formula:

[0105]

[0106] in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, representing 1, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, represents 2, formula (6) from S20401, Distance(·,·) represents the distance formula;

[0107] S20404, based on formula (8) from S20403 and Calculate the relative loss, denoted as L rel , then based on formula (8) from S20403 and Calculate the absolute loss, denoted as L abs , and then add the relative loss to the absolute loss to get the dual reference representation contrast loss, denoted as L; the specific implementation is shown in the following formula:

[0108]

[0109] L=L rel +L abs (11)

[0110] Here, max(·,·) means obtaining the maximum value of the two inputs.

[0111] For example, the distance formula Distance(·,·) used in the present invention is the Euclidean distance. In pytorch, the code implementation described above is as follows:

[0112]

[0113]

[0114]

[0115] Among them, self.feature_ts represents the unlabeled raw time series features, feature_image1 represents reference representation 1, feature_image2 represents reference representation 2, and triplet_loss represents the dual reference representation contrast loss.

[0116] Embodiment 4:

[0117] As attached Figure 1 As shown, the present invention provides a human motion health monitoring method based on dual reference representation, the method comprising the following steps:

[0118] S1. Obtaining a human motion health monitoring dataset: Downloading a human motion health monitoring dataset publicly available on the Internet;

[0119] S2. Constructing a human motion health monitoring model: Constructing a human motion health monitoring model based on dual reference representation;

[0120] S3. Training a human motion health monitoring model: training the human motion health monitoring model constructed in step S2 on the human motion health monitoring data set obtained in step S1.

[0121] S 1. Obtain human motion health monitoring data set

[0122] Download the publicly available human motion health monitoring dataset from the Internet.

[0123] For example: There are many publicly available human motion health monitoring datasets on the Internet, such as the UCI-HAR dataset. The UCI-HAR dataset is based on smartphone sensor data from 30 volunteers aged 19 to 48, and includes 6 basic movements: walking, going upstairs, going downstairs, sitting, standing, and lying down. The data is collected through customized mobile phone software, including linear acceleration and angular velocity information, with a sampling rate of 50Hz. Each participant repeated the exercise twice, once with the device fixed to the left waist and the other time freely placed. At the same time, the video was recorded and the corresponding movement type was manually labeled. This dataset is often used in the research and development of human motion health monitoring algorithms. The UCI-HAR dataset contains the following types of data and information:

[0124] Sensor signal data: Each sample contains 128 data points, representing sensor data within a 2.56 second time window;

[0125] Motion labels: 6 motion labels corresponding to sensor signals, used to train and verify human motion health monitoring models.

[0126] S2. Constructing a human motion health monitoring model

[0127] The process of building a human motion health monitoring model is as follows: Figure 2 As shown, the main operations are to build a pre-trained input module, build a lightweight temporal encoder, build a dual reference representation generation module, build a dual reference representation comparison module, build a fine-tuning input module and build a fine-tuning module.

[0128] S201. Build a pre-trained input module

[0129] A piece of data is extracted from the human motion health monitoring data set obtained in step S1, which contains a piece of human motion signal data collected from the sensor, thereby obtaining a piece of unlabeled human motion signal data, denoted as T = {x1, x2, x3, ..., x T}, where x i Represents the human motion signal data at the i-th time point.

[0130] S202. Building a lightweight timing encoder

[0131] The lightweight temporal encoder uses a convolutional network as a component to encode the unlabeled human motion signal data constructed by S201, thereby obtaining the unlabeled original temporal features, which are recorded as The specific implementation is shown in the following formula:

[0132]

[0133] Among them, T represents the unlabeled human motion signal data, from S201, Encoder lite (·) indicates a lightweight temporal encoder;

[0134] Example: When the present invention is implemented, the lightweight timing encoder Encoder lit e(·) uses a three-layer convolutional network. In pytorch, the code implementation described above is as follows:

[0135]

[0136]

[0137] Among them, x in represents the unlabeled human motion signal data, x feature represents the unlabeled original time series feature, and model ts represents the lightweight time series encoder.

[0138] S203: Constructing a dual reference representation generation module

[0139] The dual reference representation generation module receives the unlabeled human motion signal data constructed in S201, converts it into image representations using two imaging time series algorithms, and encodes the two image representations using a temporal image coding network, thereby obtaining reference representation 1 and reference representation 2;

[0140] S204: Constructing a dual reference representation comparison module

[0141] The dual reference representation comparison module receives the original temporal encoding representation output from the S202 lightweight temporal encoder and the reference representation 1 and reference representation 2 output from the S203 dual reference representation generation module, and then uses the dual reference representation comparison strategy to calculate the dual reference representation comparison loss, which is used to optimize the model parameters.

[0142] S205. Build fine-tuning input module

[0143] A piece of data is extracted from the human motion health monitoring data set obtained in step S1, which contains a human motion signal data collected from the sensor and a corresponding motion label, thereby obtaining a labeled human motion signal data, which is recorded as T f = {T, label}, where T = {x1, x2, x3, ..., x T}, and x i Represents the human motion signal data at the i-th time point.

[0144] S206 build fine-tuning module

[0145] Use the pre-trained lightweight temporal encoder to encode T in the aforementioned labeled human motion signal data to obtain the labeled original temporal features, recorded as Then it is passed to the output network layer; the output network layer maps it, and then obtains the probability that the labeled human motion signal data belongs to different human motion categories, denoted as P; the specific implementation is shown in the following formula:

[0146]

[0147] Among them, Encoder lite (·) indicates a lightweight temporal encoder.

[0148] S3. Training human motion health monitoring model

[0149] The human motion health monitoring model constructed in step S2 is trained on the human motion health monitoring data set obtained in step S1. The process is as follows: Figure 3 shown.

[0150] S301. Constructing pre-training contrast loss function

[0151] The dual reference constructed in step S20404 is used to represent the contrast loss function, and the formula has been described in step S20404.

[0152] S302. Constructing pre-training optimization function

[0153] After testing a variety of optimization functions, the model finally chose to use the Adam optimization function as the optimization function in the pre-training stage. Except for its learning rate set to 3e-4, the other hyperparameters of Adam are set to the default values ​​in pytorch.

[0154] For example, in pytorch, the code described above is implemented as follows:

[0155] optimizer=Adam(optimizer_grouped_parameters, lr=3e-4)

[0156] Among them, optimizer_grouped_parameters are the parameters to be optimized, and the default is all the parameters in the human motion health monitoring method.

[0157] S303. Construct fine-tuning loss function

[0158] The present invention adopts cross entropy as the loss function in the fine-tuning stage, and the formula is as follows.

[0159]

[0160] Among them, y true is the true label, y pred The prediction result output by the model.

[0161] For example, in pytorch, the code described above is implemented as follows:

[0162] # Calculate the error between the predicted value and the label through the cross entropy loss function

[0163] loss_fct = CrossEntropyLoss()

[0164] loss=loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

[0165] Among them, labels are the real labels and logits are the predicted results output by the model.

[0166] S304. Construct fine-tuning optimization function

[0167] After testing a variety of optimization functions, the model finally chose to use the Adam optimization function as the optimization function in the fine-tuning stage. Except for its learning rate set to 3e-5, the other hyperparameters of Adam were set to the default values ​​in pytorch.

[0168] For example, in pytorch, the code described above is implemented as follows:

[0169] #Optimize model parameters through Adam optimizer

[0170] optimizer=Adam(optimizer_grouped_parameters, lr=3e-5)

[0171] Among them, optimizer_grouped_parameters are the parameters to be optimized, and the default is all the parameters in the human motion health monitoring method.

[0172] When the model has not been pre-trained, it is necessary to further perform steps S301 and S302 for pre-training to optimize the parameters of the model; when the model pre-training is completed, steps S303 and S304 can be performed for fine-tuning and prediction.

[0173] Embodiment 5:

[0174] This device mainly consists of three units, namely, the human motion health monitoring data set acquisition unit, the human motion health monitoring model construction unit and the human motion health monitoring model training unit. Figure 4 The specific functions of each unit are described as follows:

[0175] The human motion health monitoring data set acquisition unit is used to download the human motion health monitoring data set publicly available on the Internet.

[0176] The human motion health monitoring model construction unit is used to construct a pre-training input module, a lightweight temporal encoder, a dual reference representation generation module, a dual reference representation comparison module, a fine-tuning input module and a fine-tuning module, thereby constructing a human motion health monitoring model.

[0177] The human motion health monitoring model training unit is used to construct a pre-training loss function, a pre-training optimization function, a fine-tuning loss function, and a fine-tuning optimization function, thereby completing the pre-training, fine-tuning, and prediction of the model.

[0178] Furthermore, the human body movement health monitoring model construction unit also includes:

[0179] Construct a pre-training input module unit, which is responsible for constructing the human motion signal data required for model pre-training, thereby obtaining unlabeled human motion signal data;

[0180] Construct a lightweight time series encoder unit to encode the human motion signal data, thereby obtaining unlabeled original time series features or labeled original time series features;

[0181] Constructing a dual reference representation generation module unit, which is responsible for transforming and processing the human motion signal data using an imaging time series algorithm, and then extracting its features to obtain reference representation 1 and reference representation 2;

[0182] Construct a dual reference representation comparison module unit, which is responsible for using the dual reference representation comparison strategy to calculate the dual reference representation comparison loss based on the unlabeled original time series features, reference representation 1 and reference representation 2, and use it to optimize the model parameters;

[0183] Construct a fine-tuning input module unit, which is responsible for constructing the human motion signal data required for model fine-tuning, thereby obtaining labeled human motion signal data;

[0184] Construct a fine-tuning module unit, which is responsible for receiving labeled human motion signal data, encoding it using a lightweight temporal encoder, and then mapping it to obtain the probability that the human motion signal data belongs to different human motion categories, and feed it back to relevant personnel as a prediction result.

[0185] Furthermore, the human body movement health monitoring model training unit also includes:

[0186] Construct a pre-trained contrastive loss function unit, which is responsible for calculating the contrastive loss error using a dual reference representation contrastive loss function;

[0187] Construct a pre-training optimization function unit, which is responsible for pre-training and adjusting the parameters in model training to reduce the contrast loss error;

[0188] Construct a fine-tuning loss function unit, which is responsible for calculating the error between the predicted result and the true label using the cross entropy loss function;

[0189] Construct a fine-tuning optimization function unit, which is responsible for fine-tuning the parameters in the model to reduce the prediction error.

[0190] Embodiment 6:

[0191] The present disclosure provides an electronic device, comprising: a memory and at least one processor;

[0192] Wherein, the memory stores a computer program;

[0193] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the human motion health monitoring method based on dual reference representation as described above.

[0194] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0195] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0196] Embodiment 6:

[0197] The present disclosure provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be executed by a processor to implement the human motion health monitoring method based on dual reference representation as described above. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0198] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0199] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0200] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A human motion health monitoring method based on dual reference representation, characterized in that: The method comprises the following steps: S1. Obtaining a human motion health monitoring dataset: Downloading a human motion health monitoring dataset publicly available on the Internet; S2. Constructing a human motion health monitoring model: Constructing a human motion health monitoring model based on dual reference representation; S3. Training a human motion health monitoring model: training the human motion health monitoring model constructed in step S2 on the human motion health monitoring data set obtained in step S1.

2. The method for monitoring human motion health based on dual reference representation according to claim 1, characterized in that: The step S2 is used to construct a pre-training input module, construct a lightweight temporal encoder, construct a dual reference representation generation module, construct a dual reference representation comparison module, construct a fine-tuning input module and construct a fine-tuning module, and then construct a human motion health monitoring model; the aforementioned modules correspond to sub-steps, respectively, recorded as S201, S202, S203, S204, S205, and S206.

3. The method for monitoring human motion health based on dual reference representation according to claim 2, characterized in that: The S201 constructs a pre-training input module, extracts a piece of data from the human motion health monitoring data set obtained in step S1, the data includes a piece of human motion signal data collected from the sensor, thereby obtaining a piece of unlabeled human motion signal data, denoted as T = {x1, x2, x3, ..., x T }, where x i Represents the human motion signal data at the i-th time point; The S202 constructs a lightweight time series encoder, uses a convolutional network as a component, and encodes the unlabeled human motion signal data constructed in S201, thereby obtaining the unlabeled original time series features, which are recorded as The specific implementation is shown in the following formula: Among them, T represents the unlabeled human motion signal data, from S201, Encoder lite (·) indicates a lightweight temporal encoder.

4. The method for monitoring human motion health based on dual reference representation according to claim 2, characterized in that: The S203 constructs a dual reference representation generation module, receives the unlabeled human motion signal data constructed in S201, converts it into image representations using two imaging time series algorithms, and uses a temporal image coding network to encode the two image representations, thereby obtaining reference representation 1 and reference representation 2; Specifically, the implementation process of the dual reference representation generation module is as follows: S20301, using the data enhancement technology Jitter to add Gaussian noise to the unlabeled human motion signal data constructed in S201, and then using the Markov transfer field imaging time series algorithm to transform it, so as to obtain the Markov imaging representation, denoted as The specific implementation is shown in the following formula: Among them, {x1, x2, x3, …, x T } represents the unlabeled human motion signal data constructed by S201, {ε1, ε2, ε3, …, ε T } represents standard Gaussian noise, MTF(·) represents the Markov transfer field imaging time series algorithm; S20302. Use the data enhancement technology Time-warping to perform nonlinear distortion processing on the unlabeled human motion signal data constructed in S201, and then use the Gram angular field imaging time series algorithm to transform it to obtain the Gram imaging representation, which is recorded as The specific implementation is shown in the following formula: Among them, {x1, x2, x3, …, x T } represents the unlabeled human motion signal data constructed by S201, τ(·) represents the warping function for warping all time points based on a smooth curve, and GAF(·) represents the Gram angular field imaging time series algorithm; S20303. Use a temporal image coding network to extract features from the Markov imaging representation and the Gram imaging representation, respectively, to obtain reference representation 1 and reference representation 2. Specifically, the implementation process of the temporal image coding network is as follows: first, use the Vision Transformer encoder to perform feature encoding on the Markov imaging representation and the Gram imaging representation, respectively, and then use the Transformer encoder to extract features from them, respectively, to obtain reference representation 1 and reference representation 2, denoted as and The specific implementation is shown in the following formula: Among them, ViT(.) represents the Vision Transformer encoder, represents the Markov imaging representation, from formula (2) of S20301, represents the Gram imaging representation, from formula (3) of S20302.

5. The method for monitoring human motion health based on dual reference representation according to claim 2, characterized in that: The S204 constructs a dual reference representation comparison module, receives the original temporal coding representation output from the lightweight temporal encoder S202 and the reference representation 1 and the reference representation 2 output from the dual reference representation generation module S203, and then uses the dual reference representation comparison strategy to calculate the dual reference representation comparison loss, and uses it to optimize the model parameters; Specifically, the implementation process of the dual reference representation comparison module is as follows: S20401. Use a multi-layer perceptron network to map the original temporal coding representation, reference representation 1, and reference representation 2, thereby obtaining the original temporal mapping representation, reference mapping representation 1, and reference mapping representation 2, which are recorded as and The specific implementation is shown in the following formula: in, represents the original temporal coding representation, from formula (1) of S202, Reference representation 1, formula (4) from S20303, represents reference representation 2, from formula (5) of S20303, MLP(·) represents multi-layer perceptron network; S20402. For the i-th original time series mapping representation in the current input batch, calculate the distance between it and all other original time series mapping representations in the current batch input, and record the minimum distance as The specific implementation is shown in the following formula: in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, Indicates that the current batch input does not contain The k-th original time series mapping representation is from formula (6) of S20401, where k∈[1, i)∪(i, batch], batch represents the current input batch size, Distance(·, ·) represents the distance formula, and min() represents obtaining the minimum value of the input list in the brackets; S20403. Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 1, denoted as Calculate the distance between the i-th original temporal mapping representation in the current input batch and the i-th reference mapping representation 2, denoted as Calculate the distance between the i-th reference mapping representation 1 and the i-th reference mapping representation 2 in the current input batch, denoted as The specific implementation is shown in the following formula: in, represents the i-th original time series mapping representation in the current input batch, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, representing 1, from formula (6) of S20401, represents the i-th reference mapping in the current input batch, represents 2, formula (6) from S20401, Distance(·,·) represents the distance formula; S20404, based on formula (8) from S20403 and Calculate the relative loss, denoted as L rel , then based on formula (8) from S20403 and Calculate the absolute loss, denoted as L abs , and then add the relative loss to the absolute loss to get the dual reference representation contrast loss, denoted as L; the specific implementation is shown in the following formula: L=L rel +L abs (11) Here, max(·,·) means obtaining the maximum value of the two inputs.

6. The method for monitoring human motion health based on dual reference representation according to claim 2, characterized in that: The S205 constructs a fine-tuning input module, extracts a piece of data from the human motion health monitoring data set obtained in step S1, the data including a piece of human motion signal data collected from the sensor and a corresponding motion label, thereby obtaining a piece of labeled human motion signal data, denoted as T f = {T, label}, where T = {x1, x2, x3, ..., x T }, and x i Represents the human motion signal data at the i-th time point; The S206 constructs a fine-tuning module, and uses a pre-trained lightweight temporal encoder to encode T in the aforementioned labeled human motion signal data, thereby obtaining a labeled original temporal feature, which is recorded as Then it is passed to the output network layer; the output network layer maps it, and then obtains the probability that the labeled human motion signal data belongs to different human motion categories, denoted as P; the specific implementation is shown in the following formula: Among them, Encoder lite (·) indicates a lightweight temporal encoder.

7. The method for monitoring human motion health based on dual reference representation according to claim 1, characterized in that: The step S3 is used to construct the loss function of the pre-training stage, the optimization function of the pre-training stage, the loss function of the fine-tuning stage, and the optimization function of the fine-tuning stage, and then train the dual reference representation model; the above modules correspond to sub-steps, recorded as S301, S302, S303, S304, which are described in sequence as follows: S301. Constructing the contrast loss function in the pre-training phase The dual reference representation contrast loss function constructed in step S20404 is used, and the formula has been described in step S20404; S302. Constructing optimization function in pre-training stage After testing various optimization functions, the model finally chose to use the Adam optimization function as the optimization function in the pre-training stage. Except for the learning rate being set to 3e-4, the other hyperparameters of Adam were set to the default values ​​in pytorch. S303. Construct the loss function of the fine-tuning stage Cross entropy is used as the loss function in the fine-tuning stage, and the formula is as follows: Among them, ytrue is the true label, and ypred is the predicted result output by the model; S304. Constructing the optimization function for the fine-tuning phase The model was tested with a variety of optimization functions, and finally the Adam optimization function was selected as the optimization function in the fine-tuning stage. Except for the learning rate being set to 3e-5, the other hyperparameters of Adam were set to the default values ​​in pytorch. When the model has not been pre-trained, it is necessary to further perform steps S301 and S302 for pre-training to optimize the parameters of the model; when the model pre-training is completed, steps S303 and S304 can be performed for fine-tuning and prediction.

8. A human motion health monitoring device based on dual reference representation using claim 1, characterized in that: The device includes: A human motion health monitoring data set acquisition unit is used to download a human motion health monitoring data set publicly available on the Internet; A human motion health monitoring model construction unit is used to construct a pre-training input module, a lightweight temporal encoder, a dual reference representation generation module, a dual reference representation comparison module, a fine-tuning input module, and a fine-tuning module, thereby constructing a human motion health monitoring model; The human motion health monitoring model training unit is used to construct a pre-training loss function, a pre-training optimization function, a fine-tuning loss function, and a fine-tuning optimization function, thereby completing the pre-training, fine-tuning, and prediction of the model.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the method for human motion health monitoring based on dual reference representation as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the method for human motion health monitoring based on dual reference representation as described in any one of claims 1 to 8.

Images