Assisted decision-making system based on multi-mode historical action data

Abstract

The invention relates to the field of artificial intelligence, and discloses an assisted decision making system based on multi-modal historical action data, and the system comprises a multi-modal data collection module which collects the multi-modal data of a user through an RGB camera, a depth camera and a sensor group; the data preprocessing module is used for preprocessing the collected data, including data cleaning, labeling and feature extraction; the behavior prediction model module is used for carrying out characteristic value identification on the behavior mode of the user through an SN-GCN model according to the historical action data of the user and establishing a model; motion prediction is realized through a multi-attention mechanism of a Transform model; and the early warning decision module is used for calculating a user abnormal risk coefficient according to the sensor data, judging a user behavior state by integrating a prediction result of the behavior prediction model, immediately starting self decision if the user behavior state is judged to be abnormal, and sending a decision signal to the wearable equipment to give an alarm.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an assisted decision-making system based on multimodal historical action data. Background Art

[0002] As the aging of my country's population structure becomes increasingly prominent, we must face up to an indisputable fact: the thinking speed and agility of the elderly are gradually slowing down. This phenomenon not only makes the daily life of the elderly more difficult, but also brings a heavy burden to their family members and the entire society.

[0003] However, looking at the current market, most products are still focused on basic areas such as health products and heart rate monitors. Although these products meet the basic needs of the elderly to a certain extent, at the same time, because the health and behavior monitoring of the elderly requires the integration of multiple types of data, such as various data from wearable devices, medical records, home environment sensor data, etc., the current integration of data from multiple sources is not comprehensive enough, and it is impossible to provide accurate and proactive services for the elderly. Summary of the invention

[0004] The purpose of the present invention is to provide an assisted decision-making system based on multimodal historical action data to solve the above technical problems.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] An assisted decision-making system based on multimodal historical action data, comprising:

[0007] The multimodal data acquisition module collects the user's multimodal data through an RGB camera, a depth camera, and a sensor group, including body motion coordinates, facial motion coordinates, motion sequence data, sensor data, and behavior log data; the sensor data includes heart rate, number of steps, and body temperature;

[0008] Data preprocessing module, which preprocesses the collected data, including data cleaning, labeling and feature extraction;

[0009] The behavior prediction model module uses the SN-GCN model to identify the feature values ​​of the user's behavior pattern and build a model based on the user's historical action data;

[0010] Action prediction is achieved through the multi-attention mechanism of the Transformer model;

[0011] The early warning decision module is used to calculate the user's abnormal risk coefficient based on the sensor data, and judge the user's behavior status based on the prediction results of the comprehensive behavior prediction model. If the user's behavior status is judged to be abnormal, self-decision is immediately initiated and the decision signal is sent to the wearable device for alarm.

[0012] As a further technical solution, the working process of the data preprocessing module is:

[0013] Preliminary image preprocessing: adjust the input image to the size required by the model, and perform data augmentation operations such as random cropping, rotation, flipping, and brightness adjustment of the image to increase the diversity and robustness of the training data;

[0014] Annotate the data and ensure that the annotation data format of the input image meets the requirements of the model. The annotation data includes the coordinates and visibility labels of each key point.

[0015] As a further technical solution, the working process of the data preprocessing module also includes:

[0016] After the data is labeled, the labeled image is input into the OpenPose or HRNet model to extract key points and output the coordinates and confidence of the key points of the human body;

[0017] The specific steps are as follows:

[0018] S1. Coordinate restoration: for scaled or cropped input images, restore the coordinates to the original image size;

[0019] S2, confidence threshold: filter the key points according to the confidence of the model output and remove the key points with low confidence;

[0020] S3, time series smoothing: for consecutive frames in the video, the coordinates of the key points are smoothed in time series;

[0021] S4, Normalization: Normalize the key point coordinates to the standard coordinate system.

[0022] As a further technical solution, the process of obtaining the abnormal behavior risk coefficient is:

[0023] Substitute the heart rate, steps and body temperature obtained by the sensor group into the following formula:

[0024]

[0025] Calculate the user's abnormal behavior risk coefficient F;

[0026] Where ΔP i is the heart rate change per unit time, ΔB i is the change in the number of steps per unit time, ΔT i is the change in body temperature per unit time, ΔP i0 , ΔB i0 , ΔT i0is the standard value; n is the number of unit time, i is the i-th unit time, and g is the judgment coefficient.

[0027] As a further technical solution, the process of judging the user behavior status is as follows:

[0028] The calculated abnormal behavior risk coefficient F is compared with the preset abnormal behavior risk interval (F min , F max ) for comparison:

[0029] If F ≥ F max , the user's behavior status is judged to be abnormal, and self-decision-making is initiated immediately;

[0030] If F∈(F min , F max ), the user's behavior is considered suspicious, and auxiliary decision making is immediately initiated, sending a voice message to the user through the wearable device to ask the user to confirm whether he needs help;

[0031] If F≤F min , then the user behavior is judged to be normal.

[0032] As a further technical solution, the process of obtaining the judgment coefficient is:

[0033] If the prediction result of the behavior prediction model is inconsistent with the user's historical behavior, the judgment coefficient is taken as 1 / 2;

[0034] If the prediction results of the behavior prediction model are consistent with the user's historical behavior, the judgment coefficient is 1 / 3.

[0035] As a further technical solution, the process of obtaining the heart rate change, step number change and body temperature change within a unit time is:

[0036] The time when the prediction result of the prediction model is inconsistent with the user's historical behavior is obtained as the end time t, and the start time t-Δt is determined according to the preset unit time length Δt;

[0037] Fit and draw the curves p(t), b(t), and T(t) of the user's heart rate, number of steps, and body temperature changing with time between t-Δt and t;

[0038] By formula:

[0039]

[0040] The heart rate change per unit time ΔP is calculated respectively i , step number change ΔB i and body temperature change ΔT i ;

[0041] Among them, r i is the comprehensive fluctuation factor per unit time, and p0(t), b0(t), and T0(t) are the reference curves of heart rate, step number, and body temperature changing with time.

[0042] As a further technical solution, the comprehensive fluctuation factor r per unit time i The calculation method is:

[0043] Among them, α, β, γ are exponential coefficients, S is the total number of samples in a unit time, and p j 、b j 、T j are the real-time heart rate, number of steps, and body temperature of the jth sampling point, respectively. They are the average heart rate, number of steps and body temperature respectively.

[0044] Beneficial effects of the present invention:

[0045] The present invention comprehensively considers the user's heart rate, number of steps and temperature, thereby monitoring the user's health status in real time, and then obtains the changes in the user's heart rate, number of steps and temperature within multiple unit times. Obviously, if the change is smaller, it means that the user's state in another monitoring period is more stable, and the user's behavior state is obviously better. If the change in one of the parameters is smaller, or remains unchanged, it means that the probability of an abnormality of a certain parameter of the user gradually decreasing or increasing is higher. Specifically, the formula is used to calculate in a cumulative multiplication manner, so as to amplify the small changes within a unit time so that they can be recognized by the system, so that the calculated abnormal behavior risk coefficient F is more accurate, which provides favorable support for subsequent decision-making and helps users get timely rescue or other treatments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below in conjunction with the accompanying drawings.

[0047] Figure 1 is a system structure block diagram of the present invention;

[0048] Figure 2 Drive data processing flow for user action behavior in the present invention;

[0049] Figure 3 This is the spatiotemporal feature neural network decision process in the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1-Figure 3 As shown, the present invention is an assisted decision-making system based on multimodal historical action data, comprising:

[0052] The multimodal data acquisition module collects the user's multimodal data through an RGB camera, a depth camera, and a sensor group, including body motion coordinates, facial motion coordinates, motion sequence data, sensor data, and behavior log data; the sensor data includes heart rate, number of steps, and body temperature;

[0053] Data preprocessing module, which preprocesses the collected data, including data cleaning, labeling and feature extraction;

[0054] The behavior prediction model module uses the SN-GCN model to identify the feature values ​​of the user's behavior pattern and build a model based on the user's historical action data;

[0055] Action prediction is achieved through the multi-attention mechanism of the Transformer model;

[0056] The early warning decision module is used to calculate the user's abnormal risk coefficient based on the sensor data, and judge the user's behavior status based on the prediction results of the comprehensive behavior prediction model. If the user's behavior status is judged to be abnormal, self-decision is immediately initiated and the decision signal is sent to the wearable device for alarm.

[0057] The working process of the data preprocessing module is as follows:

[0058] Preliminary image preprocessing: adjust the input image to the size required by the model, and perform data augmentation operations such as random cropping, rotation, flipping, and brightness adjustment of the image to increase the diversity and robustness of the training data;

[0059] Annotate the data and ensure that the annotation data format of the input image meets the requirements of the model. The annotation data includes the coordinates and visibility labels of each key point.

[0060] The working process of the data preprocessing module also includes:

[0061] After the data is labeled, the labeled image is input into the OpenPose or HRNet model to extract key points and output the coordinates and confidence of the key points of the human body;

[0062] The specific steps are as follows:

[0063] S1. Coordinate restoration: for scaled or cropped input images, restore the coordinates to the original image size;

[0064] S2, confidence threshold: filter the key points according to the confidence of the model output and remove the key points with low confidence;

[0065] S3, time series smoothing: for consecutive frames in the video, the coordinates of the key points are smoothed in time series;

[0066] S4. Normalization: Normalize the key point coordinates to the standard coordinate system.

[0067] In this embodiment, human motion data is obtained through a variety of sensors, and a large motion model is used to predict human behavior based on the continuous accumulation of historical motion data, and conversational or action-based assisted decision-making is provided. This method achieves a qualitative improvement in the active human-computer interaction of current smart products. The behavior of the elderly is analyzed through large motion model training, and timely predictions and assistance are given to turn the human-computer relationship into an interactive dependency relationship. In addition, based on the user's daily behavior patterns, possible activities are predicted and suggestions are provided, such as diet, rest, exercise, and emotions, as well as dangerous behaviors that may occur in the elderly, such as getting up at night, going out, etc., and early warnings are issued in advance, and lights, curtains, home appliances and other equipment can be automatically controlled according to the user's personal usage habits. In order to realize the above technical solution, it is achieved through the following main process steps:

[0068] Collect data through multiple sensors, and use visual sensor RGB cameras or depth cameras to obtain image data, including the human head, shoulders, elbows, wrists, hips, knees, ankles, face, etc.;

[0069] Use open source frameworks such as OpenPose and HRNet to extract key points of the human body, and convert images or videos into key point coordinates or skeleton information of the human body; use OpenPose to output the coordinate sequence of 18 key points of the human body (x, y coordinates of each key point), and the specific 18 key points include: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, left hip, right hip, right knee, right ankle, left knee, right eye, left eye, right ear, and left ear. These 18 key points cover the main joints and some key parts of the human body, and can effectively describe the posture and movement of the human body;

[0070] Output the human key point sequence in the video frame recorded by the camera, and arrange the 18 key point coordinates of each human body in each frame image into a continuous key point data sequence according to the time sequence; such as the key point coordinates of a human body [(x1, y1), (x2, y2), (x3, y3)... (x18, y18)];

[0071] The key point coordinate sequence is input into the ST-GCN deep learning model, which accepts a key point coordinate sequence of shape (N, T, V, C), where N is the batch size, T is the number of time steps, V is the number of key points (18), and C is the coordinate dimension (usually 2 or 3). In addition, 68 human facial coordinates can be obtained, combined with human motion coordinates to form a comprehensive human motion prediction model;

[0072] The graph convolutional layer (GCN) is used to perform convolution operations in the spatial dimension to capture the spatial dependencies between key points; convolution operations are performed in the temporal dimension to capture the dependencies of key points over time; finally, the multi-head attention mechanism of Transformer is used to parallelly calculate multiple different attention distributions, thereby capturing the complex correlations between different positions in the input sequence; this helps the model focus on more important information when processing long sequences;

[0073] The Softmax function is used in the last layer to output the action classification and behavior prediction results, convert the output into the probability distribution of each category, and select the category with the largest probability as the final action classification result.

[0074] The specific implementation process is as follows:

[0075] (1) Sensor working module:

[0076] Step 1: Deploy RGB cameras or depth cameras in the daily activity space of the elderly, or deploy them on wearable devices. They can measure the changes in human motion, facial expressions and other information to infer the human body's motion state. For example, the action time series reflects the changes in human motion in different states, which can be used to identify actions such as walking, running, and jumping. The spatial coordinate information of the coordinates can help analyze the rotation and twisting of the human body in three-dimensional space.

[0077] Step 2: Deploy the data collection system on the cloud service and partition it for processing. This facilitates the real-time collection and storage of data from sensors in the later stage. The specific data types include: 18 key coordinates of the human body and 68 key coordinates of the face. These data are used as the standard for the sensor to collect raw data. (The 18 key points of the human body are nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, left hip, right hip, right knee, right ankle, left knee, right eye, left eye, right ear, and left ear. The 68 key points of the face are divided into 9 intervals, namely, facial contour (points 1-17), left eyebrow (points 18-22), right eyebrow (points 23-27), nose bridge (points 28-31), nose wing (points 32-36), left eye (points 37-42), right eye (points 43-48), outer contour of the mouth (points 49-60), inner contour of the mouth (points 61-68)).

[0078] (2) Data processing module:

[0079] Step 3: Preprocess the extracted data. The specific processing contents include:

[0080] 1) Image preprocessing: adjust the input image to the size required by the model. For example, OpenPose and HRNet usually require the input image to be of a specific resolution, such as 256x256 or 368x368. Perform data augmentation operations such as random cropping, rotation, flipping, and brightness adjustment of the image to increase the diversity and robustness of the training data.

[0081] 2) Label the data to ensure that the label data format of the input image meets the requirements of the model; the label data usually includes the coordinates and visibility labels of each key point;

[0082] Step 4: After data preprocessing, the processed image is input into the OpenPose or HRNet model for key point extraction. The model outputs the coordinates and confidence of the key points of the human body. Specifically, it includes:

[0083] 1) Coordinate restoration: The coordinates of the key points output by the model are usually relative to the size of the input image; if the input image is scaled or cropped, the coordinates need to be restored to the original image size;

[0084] 2) Confidence threshold: filter the key points according to the confidence of the model output and remove the key points with low confidence to improve the accuracy of key point detection;

[0085] 3) Time series smoothing: For continuous frames in the video, the key point coordinates can be smoothed in time series to reduce jitter and noise;

[0086] 4) Normalization: Normalize the coordinates of key points to a standard coordinate system, such as normalizing them around the nose or hip, to facilitate subsequent motion analysis and recognition;

[0087] (3) ST-GCN model module:

[0088] Step 5: Arrange the processed data key points into the shape of (N, T, V, C); N is the number of data samples input into the model for training or inference at one time, T refers to the time step in the time series data, indicating the length of the input data in the time dimension, V refers to the number of key points contained in each sample, usually the number of feature points of the human body or face, and C is the coordinate dimension of each key point, usually 2D (x, y) or 3D (x, y, z); before entering the model, the data is usually rearranged into (N, C, T, V) to adapt to the input format of the convolution operation; the specific graph convolution formula is as follows:

[0089]

[0090] H(l): The input feature tensor of the lth layer; you can think of it as a table containing information about all nodes (e.g., key points of the human body), with a shape of (N, C, T, V);

[0091] A k : Adjacency matrix; This matrix represents the connection relationship between nodes (key points). It can be understood as a graph that describes which points are connected to each other, and the shape is (V, V);

[0092] W k : The weights of the convolution kernel. They are the parameters that the model needs to learn to transform input features into output features; the shape is (C, C'), where C' is the number of output channels.

[0093] σ: Activation function (e.g. ReLU); it is applied after each layer to introduce non-linearity, enabling the model to learn more complex features.

[0094] H (l+1) : The output feature tensor of the l+1th layer, the result after graph convolution and activation function processing;

[0095] The formula uses graph convolution to integrate and propagate the information of each node while taking into account the connection relationship between nodes. This helps capture the spatial dependencies and interactions of key points of the human body. For example, if we want to recognize a person's walking movements, the key points of the knee, hip, and ankle will be important features; first, the adjacency matrix A k Define the connection relationship between the knee, hip and ankle; second, input feature H (l) Contains the coordinate information of these key points at each time step; third, the convolution operation Wk Aggregating and transforming this information enables the model to recognize that the movement of the knee is part of walking. Fourth, the activation function σ adds nonlinearity to help the model recognize the walking action more accurately.

[0096] In step 6, the temporal convolutional layer (TCN) performs convolution operations on the time dimension to extract temporal features. Assume that the shape of the input data is (N, C', T, V), where C' is the number of channels output from the graph convolutional layer. The specific formula is as follows:

[0097]

[0098] H (t) : The input feature matrix of the tth time step, with a shape of (N, C, V).

[0099] H (t+1) : The output feature matrix of the t+1th time step, with a shape of (N, C', V).

[0100] σ: activation function (such as ReLU), used to introduce nonlinearity;

[0101] The sum operation from 1 to K

[0102] Wk: The weight of the k-th convolution kernel, with a shape of (C', C, 1, 1), indicating the convolution operation used for the k-th time step;

[0103] H (t-k+1) : The input feature matrix of the t-k+1th time step;

[0104] Step 7: In the convolution operation, Wk and H (t-k+1) Perform matrix multiplication, the result is to aggregate the features of the input channel and convert them into the features of the output channel, and finally sum the results of all convolution operations in the time window, and then apply the activation function σ;

[0105] The formula of temporal convolution aggregates and transforms the features of time series data by applying convolution operations in the time dimension, thereby capturing the temporal dependencies in the sequence; in ST-GCN, temporal convolution is used to extract the dynamic change features of human key points in the time dimension, providing a basis for action recognition and prediction.

[0106] (4) Transformer model module:

[0107] In step 8, the spatiotemporal features extracted by ST-GCN are input into the Transformer to further process the temporal features and make the final prediction. This combined model takes advantage of the ability of ST-GCN to capture spatial dependencies and the ability of Transformer to capture long-term dependencies, thereby improving the accuracy and robustness of action recognition. In addition, in the Transformer stage, the spatiotemporal features extracted by ST-GCN are used as input. The dependencies between different positions in the sequence are first captured through the self-attention mechanism, and then the features are nonlinearly transformed through the feedforward neural network, and finally the classification or regression task is performed through the fully connected layer. The specific operations are as follows:

[0108] First, ST-GCN is used to extract the spatiotemporal features of human key points. Assume that the obtained features are represented as H ST-GCN , whose shape is (N,C,T,V), where N is the number of samples, C is the number of channels, T is the time step, and V is the number of key points.

[0109] Second, H ST-GCN Rearrange to a shape suitable for Transformer input. Transformer usually processes input shapes of (N, T, D), where D is the feature dimension. The V and C dimensions can be merged, that is, D = V*C.

[0110]

[0111] Third, use Transformer to process the rearranged features (Shape is ((N, T, D))), the specific steps are as follows (W Q , W K , W V is a trainable weight matrix):

[0112]

[0113] Fourth, implement the self-attention mechanism (where dk is the dimension of the key vector):

[0114]

[0115] Fifth, implement the multi-head attention mechanism:

[0116] textMultiHead(Q,K,V)=Concat(head1,head2,...,head h )W O

[0117] The calculation for each head is:

[0118]

[0119] (5) Transformer prediction module:

[0120] Step 9: Transformer output H Tansformer Processing and classification via feed-forward neural networks:

[0121] FFN(x)=max(0,xW1+b1)W2+b2

[0122] In step 10, the final action recognition prediction result is passed through the Softmax layer to achieve the probability prediction of possible actions:

[0123] Output = Softmax(FFN(H Transformer ))

[0124] Finally, all the processes are combined with the innovative methods of ST-GCN and Transformer models. In the first step, I used ST-GCN to deeply explore the spatiotemporal characteristics of key points of the human body, accurately capturing the spatial correlation between key points and the dynamic changes in time; then, in order to meet the input requirements of Transformer, we carefully arranged these extracted complex features to form a new data structure; then, through the unique self-attention mechanism and multi-head attention mechanism of Transformer, we further analyzed and refined these spatiotemporal features, especially capturing the dependencies over a longer time span;

[0125] At the end of the process, an efficient feedforward neural network is used to accurately classify or regress the Transformer output, thereby achieving precise control and enhanced robustness of action recognition.

[0126] Through the above technical solution, through the combination of RGB camera and inertial sensor, various detailed information of human body movements can be fully captured. RGB camera provides the coordinates of key visual points, and inertial sensor provides the data of motion status. The combination of the two can more accurately identify and analyze human body movements.

[0127] It combines the respective advantages of ST-GCN and Transformer models: ST-GCN's graph convolutional neural network is used to effectively capture the spatial dependencies and interactions of key points of the human body, and the dynamic change characteristics of key points in the time dimension are extracted through the temporal convolutional network; then, the Transformer's self-attention mechanism further processes these spatiotemporal features to capture long-term dependencies and global information, thereby achieving more accurate and robust action recognition and prediction. This combination not only improves the model's ability to recognize complex action patterns, but also enhances its robustness to noise and abnormal data, making it suitable for the analysis of long time series data.

[0128] Increased independence in life: The system can detect the daily activities of the elderly at home, ensuring their safety when living independently, while reducing their dependence on external intervention, improving their quality of life and independence.

[0129] The process of obtaining the abnormal behavior risk coefficient is:

[0130] Substitute the heart rate, steps and body temperature obtained by the sensor group into the following formula:

[0131]

[0132] Calculate the user's abnormal behavior risk coefficient F;

[0133] Where ΔP i is the heart rate change per unit time, ΔB i is the change in the number of steps per unit time, ΔT i is the change in body temperature per unit time, ΔP i0 , ΔB i0 , ΔT i0 is the standard value; n is the number of unit time, i is the i-th unit time, and g is the judgment coefficient.

[0134] The process of judging the user's behavior status is as follows:

[0135] The calculated abnormal behavior risk coefficient F is compared with the preset abnormal behavior risk interval (F min , F max ) for comparison:

[0136] If F ≥ F max , the user's behavior status is judged to be abnormal, and self-decision-making is initiated immediately;

[0137] If F∈(F min , F max ), the user's behavior is considered suspicious, and auxiliary decision making is immediately initiated, sending a voice message to the user through the wearable device to ask the user to confirm whether he needs help;

[0138] If F≤F min , then the user behavior is judged to be normal.

[0139] In this embodiment, the user's heart rate, number of steps and temperature are comprehensively considered to monitor the user's health status in real time, and then the changes in the user's heart rate, number of steps and temperature in multiple unit times are obtained. Obviously, if the change is smaller, it means that the state of the user in another monitoring period is more stable, and the user's behavior state is obviously better. If the change in one of the parameters is smaller, or remains unchanged, it means that the probability of an abnormality of a certain parameter of the user gradually decreasing or increasing is higher. Specifically, through the formula The calculation is performed in a cumulative multiplication manner, thereby magnifying small changes within a certain unit of time so that they can be recognized by the system, making the calculated abnormal behavior risk coefficient F more accurate, providing favorable support for subsequent decision-making, and helping users get timely assistance or other treatment.

[0140] The process of obtaining the judgment coefficient is:

[0141] If the prediction result of the behavior prediction model is inconsistent with the user's historical behavior, the judgment coefficient is taken as 1 / 2;

[0142] If the prediction results of the behavior prediction model are consistent with the user's historical behavior, the judgment coefficient is 1 / 3.

[0143] In this embodiment, the user's behavior prediction and the user's historical behavior are compared by combining the behavior prediction model. Obviously, if there is a significant gap, it means that the user's behavior abnormality risk is higher, and it is more likely to have an accident or need assistance. Therefore, a more accurate abnormal behavior risk assessment model is established by combining the accurate prediction results of the behavior prediction model and the user's various parameters to achieve the accuracy of quick decision-making or auxiliary decision-making.

[0144] The process of obtaining the heart rate change, step number change and body temperature change within a unit time is as follows:

[0145] The time when the prediction result of the prediction model is inconsistent with the user's historical behavior is obtained as the end time t, and the start time t-Δt is determined according to the preset unit time length Δt;

[0146] Fit and draw the curves p(t), b(t), and T(t) of the user's heart rate, number of steps, and body temperature changing with time between t-Δt and t;

[0147] By formula:

[0148]

[0149] The heart rate change per unit time ΔP is calculated respectively i , step number change ΔB i and body temperature change ΔTi ;

[0150] Among them, r i is the comprehensive fluctuation factor per unit time, and p0(t), b0(t), and T0(t) are the reference curves of heart rate, step number, and body temperature changing with time.

[0151] The comprehensive fluctuation factor r per unit time i The calculation method is:

[0152] Among them, α, β, and γ are exponential coefficients, which are comprehensively formulated based on historical data and empirical data. S is the total number of samples in a unit time, and p j 、b j , T j are the real-time heart rate, number of steps, and body temperature of the jth sampling point, respectively. They are the average heart rate, number of steps and body temperature respectively.

[0153] In this embodiment, a calculation method for obtaining the change amount of each parameter in a unit time is provided. Specifically, by comparing the change curve of each parameter over time in a unit time with the reference change curve, the cumulative change amount of each parameter in a unit time is obtained. Obviously, the larger the cumulative change amount, the greater the deviation of the user's health state from the normal health state, that is, the higher the probability of abnormal behavior risk. At the same time, in order to consider the cumulative change alone, the parameter fluctuation amount in the entire unit time is ignored, so the standard deviation is calculated. The comprehensive fluctuation of each parameter in a unit time is represented. Obviously, the larger the fluctuation coefficient, the more unstable the user's health status in the unit time. The greater the impact of abnormal fluctuations on the overall abnormal behavior risk coefficient, so a more accurate abnormal behavior risk assessment can be obtained.

[0154] It should be noted that the calculation formulas and various parameters involved in the calculation in the present invention have been dimensionally processed in advance, and the process of dimensionless processing is well known in the industry and will not be described here.

[0155] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. An assisted decision-making system based on multimodal historical action data, characterized in that: include: The multimodal data acquisition module collects the user's multimodal data through an RGB camera, a depth camera, and a sensor group, including body motion coordinates, facial motion coordinates, motion sequence data, sensor data, and behavior log data; the sensor data includes heart rate, number of steps, and body temperature; Data preprocessing module, which preprocesses the collected data, including data cleaning, labeling and feature extraction; The behavior prediction model module uses the SN-GCN model to identify the feature values ​​of the user's behavior pattern and build a model based on the user's historical action data; Action prediction is achieved through the multi-attention mechanism of the Transformer model; The early warning decision module is used to calculate the user's abnormal risk coefficient based on the sensor data, and judge the user's behavior status based on the prediction results of the comprehensive behavior prediction model. If the user's behavior status is judged to be abnormal, self-decision is immediately initiated and the decision signal is sent to the wearable device for alarm.

2. The assisted decision-making system based on multimodal historical action data according to claim 1, characterized in that: The working process of the data preprocessing module is as follows: Preliminary image preprocessing: adjust the input image to the size required by the model, and perform data augmentation operations such as random cropping, rotation, flipping, and brightness adjustment of the image to increase the diversity and robustness of the training data; Annotate the data and ensure that the annotation data format of the input image meets the requirements of the model. The annotation data includes the coordinates and visibility labels of each key point.

3. The assisted decision-making system based on multimodal historical action data according to claim 2, characterized in that: The working process of the data preprocessing module also includes: After the data is labeled, the labeled image is input into the OpenPose or HRNet model to extract key points and output the coordinates and confidence of the key points of the human body; The specific steps are as follows: S1. Coordinate restoration: for scaled or cropped input images, restore the coordinates to the original image size; S2, confidence threshold: filter the key points according to the confidence of the model output and remove the key points with low confidence; S3, time series smoothing: for consecutive frames in the video, the coordinates of the key points are smoothed in time series; S4, Normalization: Normalize the key point coordinates to the standard coordinate system.

4. The assisted decision-making system based on multimodal historical action data according to claim 1, characterized in that: The process of obtaining the abnormal behavior risk coefficient is: Substitute the heart rate, steps and body temperature obtained by the sensor group into the following formula: Calculate the user's abnormal behavior risk coefficient F; Among them, ΔP i is the change in heart rate per unit time, ΔB i is the change in the number of steps per unit time, ΔT i is the change in body temperature per unit time, ΔP i0 , ΔB i0 , ΔT i0 is the standard value; n is the number of unit time, i is the i-th unit time, and g is the judgment coefficient.

5. The assisted decision-making system based on multimodal historical action data according to claim 4, characterized in that: The process of judging the user's behavior status is as follows: The calculated abnormal behavior risk coefficient F is compared with the preset abnormal behavior risk interval (F min , F max ) for comparison: If F ≥ F max , the user's behavior status is judged to be abnormal, and self-decision-making is initiated immediately; If F∈(F min , F max ), the user's behavior is considered suspicious, and auxiliary decision making is immediately initiated, sending a voice message to the user through the wearable device to ask the user to confirm whether he needs help; If F≤F min , then the user behavior is judged to be normal.

6. The assisted decision-making system based on multimodal historical action data according to claim 5, characterized in that: The process of obtaining the judgment coefficient is: If the prediction result of the behavior prediction model is inconsistent with the user's historical behavior, the judgment coefficient is taken as 1 / 2; If the prediction results of the behavior prediction model are consistent with the user's historical behavior, the judgment coefficient is 1 / 3.

7. The assisted decision-making system based on multimodal historical action data according to claim 6, characterized in that: The process of obtaining the heart rate change, step number change and body temperature change within a unit time is as follows: The time when the prediction result of the prediction model is inconsistent with the user's historical behavior is obtained as the end time t, and the start time t-Δt is determined according to the preset unit time length Δt; Fit and draw the curves p(t), b(t), and T(t) of the user's heart rate, number of steps, and body temperature changing with time between t-Δt and t; By formula: The heart rate change per unit time ΔP is calculated respectively i , step number change ΔB i and body temperature change ΔT i ; Among them, r i is the comprehensive fluctuation factor per unit time, and p0(t), b0(t), and T0(t) are the reference curves of heart rate, step number, and body temperature changing with time.

8. The assisted decision-making system based on multimodal historical action data according to claim 7, characterized in that: The comprehensive fluctuation factor r per unit time i The calculation method is: Among them, α, β, γ are exponential coefficients, S is the total number of samples in a unit time, and p j 、b j 、T j are the real-time heart rate, number of steps, and body temperature of the jth sampling point, respectively. They are the average heart rate, number of steps and body temperature respectively.

Images