AI-based platform user behavior identification monitoring system

Abstract

The invention relates to the technical field of behavior recognition, and discloses an AI-based platform user behavior recognition monitoring system, which comprises a data acquisition module, a data processing module, a multi-modal sensing fusion module, a monitoring module and an abnormal situation deduction module, according to the system, accurate data alignment is realized by constructing a multi-modal sensing network and combining an AI time synchronization algorithm, the limitation of a single sensor in light and view angles is broken through, dynamic and static characteristics of RGB video data and infrared thermal imaging data are decoupled by adopting a two-channel 3D convolutional network, a Transform architecture is fused with radar point cloud and pressure data to generate a space-time incidence matrix, and the accuracy of the system is improved. The method comprises the following steps: establishing an adaptive behavior pattern library through an LSTM-GRU hybrid network, verifying joint constraints, energy metabolism and risk levels in a layered manner based on a causal reasoning graph, and particularly, accurately identifying behaviors of the elderly in a static state through historical feature comparison and a three-level reasoning mechanism for static behaviors, thereby ensuring the monitoring effect of the elderly.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition technology, and more specifically to an AI-based platform user behavior recognition and monitoring system. Background Art

[0002] With the rapid development of Internet platforms, the behavioral data of users on the platforms has exploded. These data contain a wealth of key information such as user preferences, needs, and potential risks. In the current elderly care environment, many elderly people will enter nursing homes for their retirement. Since the number of caregivers is usually small and the number of elderly people is large, and for risk considerations, most nursing homes have installed surveillance and use surveillance to monitor the behavior of the elderly in real time to ensure that the elderly can be informed in time and take measures to deal with any abnormalities.

[0003] With the rapid development of artificial intelligence (AI), user behavior recognition and monitoring systems have been widely applied in numerous scenarios. This is particularly true in places like nursing homes. By collecting data through devices like high-definition cameras, thermal imagers, millimeter-wave radar, and pressure sensors, AI technology enables real-time monitoring and analysis of user behavior. These systems, through continuous data processing and pattern recognition, are gradually improving the accuracy and efficiency of behavior recognition.

[0004] Although the current monitoring platform has been connected to AI technology, it often relies on video motion capture to identify the behavior of the elderly. These motion captures are often performed during the elderly's daily behavior, which means that the recognition is more accurate in dynamic situations. However, the elderly often have poor recognition ability when they are static, resulting in poor monitoring results. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, the present invention provides an AI-based platform user behavior recognition and monitoring system to solve the problems existing in the above-mentioned background technology.

[0006] The present invention provides the following technical solutions:

[0007] An AI-based platform user behavior recognition and monitoring system, including a data acquisition module, a data processing module, a multimodal perception fusion module, a monitoring module, and an abnormal situation deduction module;

[0008] The data acquisition module is used to collect RGB video data, infrared thermal imaging data, millimeter-wave radar point cloud data, and pressure distribution data in real time through high-definition cameras, thermal imagers, millimeter-wave radars, and pressure sensors deployed in the nursing home. It also generates a multimodal aligned dataset through timestamp alignment based on an AI-based time synchronization algorithm.

[0009] The data processing module uses an AI-driven dual-channel 3D convolutional neural network to extract dynamic features (motion trajectory) and static features (micro-expressions, body distribution) based on the RGB video data and infrared thermal imaging data in the multimodal alignment dataset. It then generates a dynamic feature matrix and a static feature tensor using a feature decoupling algorithm.

[0010] The multimodal perception fusion module uses the Transformer architecture to perform cross-modal feature alignment based on the millimeter-wave radar point cloud data and pressure distribution data in the multimodal alignment dataset, generating a spatiotemporal correlation matrix that represents the spatiotemporal relationship.

[0011] The monitoring module, based on the spatiotemporal correlation matrix of the multimodal perception fusion module and combined with environmental data (light intensity, temperature and humidity), establishes a behavior pattern library through an LSTM-GRU hybrid network. When it detects that the duration of the static feature tensor exceeds the threshold, it calls the dynamic feature matrix of the data processing module for historical comparison and verification, and dynamically updates the threshold through the adaptive threshold regulator, ultimately outputting a new behavior pattern library and dynamically adjusted threshold.

[0012] The abnormal situation deduction module is based on the new behavioral pattern library and dynamically adjusted thresholds in the monitoring module, combined with the static feature tensor of the data processing module, and uses the causal inference graph network (CIGN) to build a three-layer reasoning engine: the first layer verifies limb constraint conditions (such as abnormal joint angles), the second layer analyzes energy metabolism characteristics, and the third layer assesses the danger level. When the static characteristics continue to meet the above conditions, it is judged as abnormal and an alarm is issued.

[0013] Preferably, in the data acquisition module, for the AI-based time synchronization algorithm, a multimodal alignment dataset is generated by timestamp alignment. The generation process is as follows:

[0014] Set the RGB video data to V = {v1, v2, ..., v n}, the timestamp set of high-definition camera data is τ V ={τ v1 , τ v2 ,…,τ vn};

[0015] The infrared thermal imaging data collected by the thermal imager is T = {t1, t2, ..., t m}, the timestamp set of infrared thermal imaging data is τ T ={τ t1 , τ t2 ,…,τ tm};

[0016] The point cloud data collected by the millimeter wave radar is P = {p1, p2, ..., p k}, the timestamp set of point cloud data is τP ={τ p1 , τ p2 ,...,τ pk};

[0017] The pressure distribution data collected by the pressure sensor is S = {s1, s2, ..., s l}, the timestamp set of pressure distribution data is τ S ={τ s1 , τ s2 ,…,τ sl};

[0018] Where, v n Represents the nth data point, n represents the number of elements of RGB video data V, τ vn Indicates the vth n The timestamp corresponding to the RGB video data point, t m represents the mth infrared thermal imaging data point, m represents the number of elements of infrared thermal imaging data T, τ tm Indicates the tth m The timestamp corresponding to the thermal imaging data point, p k Represents the kth point cloud data point, k represents the number of elements of the point cloud data P, τ pk Indicates the pth k The timestamp corresponding to the point cloud data point, s l represents the lth pressure distribution data point, l represents the number of elements in the pressure distribution data S, τ sl Indicates the sth l The timestamp corresponding to each pressure distribution data point;

[0019] Introduce the time synchronization function f sy , to achieve time alignment, the alignment formula is as follows:

[0020] f sy (D,τ D )={d i ∣τd i ∈[τ ref -δ,τ ref +δ]};

[0021] Where D can be any one of RGB video data V, infrared thermal imaging data T, point cloud data P or pressure distribution data S, τ D Represents the corresponding timestamp set, τ ref Indicates the reference timestamp, which can be any starting timestamp of RGB video stream data, infrared thermal imaging data, point cloud data or pressure distribution data. δ indicates the allowable time deviation range, d i represents the i-th data point in D, τd i Represents data di timestamp;

[0022] For τ D Calculation is performed, and the calculation formula is as follows;

[0023] τ D ={τ V , τ T , τ P , τ S};

[0024] For RGB video data V, infrared thermal imaging data T, point cloud data P and pressure distribution data S, traverse their timestamp collection τ respectively V , τ T , τ P , τ S , find all the i ∈[τ ref -δ,τ ref +δ] condition, and thus obtain the corresponding subset f sy (V,τ V ),f sy (T,τ T ),f sy (P,τ P ),f sy (S,τ S );

[0025] Based on the above subsets, generate a multimodal alignment dataset D al , the generation formula is as follows:

[0026] D al ={f sy (V,τ V ), f sy (T,τ T ), fsy(P,τ P ), fsy(S,τ S )}.

[0027] Preferably, in the data processing module, the AI-driven dual-channel 3D convolutional neural network extracts dynamic features and static features respectively. The extraction process is as follows:

[0028] The convolution kernel of the AI-driven dual-channel 3D convolutional neural network is set to B, with a size of b×b×b, a stride of c, and a padding of r.

[0029] Extract dynamic features of RGB video data V The extraction formula is as follows;

[0030]

[0031] Where, 3D-CNN dn represents a three-dimensional convolutional neural network for extracting dynamic features;

[0032] Extracting static features of infrared thermal imaging data T The extraction formula is as follows;

[0033]

[0034] Where, 3D-CNN st Represents the three-dimensional convolutional neural network for extracting static features, B′ represents the convolution kernel size of the 3D convolutional neural network for static feature extraction, c′ represents the step size of the 3D convolutional neural network for static feature extraction, and r′ represents the padding method of the 3D convolutional neural network for static feature extraction.

[0035] Preferably, in the data processing module, the dynamic feature matrix and static feature tensor are generated by the feature decoupling algorithm, and the generation process is as follows:

[0036] For dynamic features The decoupling formula is:

[0037]

[0038] Where, Decoupling function of dynamic features extracted from RGB video data V, W dn represents the weight parameter in dynamic feature decoupling, M dn represents the dynamic feature matrix;

[0039] For static features The decoupling formula is:

[0040]

[0041] Where, Decoupling function of the static features extracted from infrared thermal imaging data T, W st represents the weight parameter in static feature decoupling, M st Represents a static feature tensor.

[0042] Preferably, in the multimodal perception fusion module, the Transformer architecture is used to perform cross-modal feature alignment and generate a spatiotemporal correlation matrix that represents the spatiotemporal relationship. The generation process is as follows:

[0043] The point cloud data P and pressure distribution data S are encoded separately by the encoder Enr. The encoding formula is as follows:

[0044] E P =Enr P (P; WnrP );

[0045] E S =Enr S (S; Wnr S );

[0046] Where, E P and E S Represent the encoding features of point cloud data P and pressure distribution data S respectively, Enr P and Enr S Represents the encoders of point cloud data P and pressure distribution data S, Wnr P and Wnr S Represents encoder Enr P and Enr S The weight parameter of

[0047] In the Transformer architecture, let the multi-head attention function be Muad(QA, QK, QV), set QA as the query vector, QK as the key vector, QV as the value vector, and use E P As query and key, E S As the value, calculate the cross-modal attention Acs, the calculation formula is as follows:

[0048] Acs=Muad(E P , E S ;W QA , W QK , W QV );

[0049] Where W QA 、W QK and W QV are the weight matrices in the multi-head attention mechanism;

[0050] Through the decoder structure of Transformer, the aligned features are fused to generate the spatiotemporal correlation matrix MZ that represents the spatiotemporal relationship. st , the generation formula is as follows:

[0051] MZ st =DZe st (Acs; W dec );

[0052] Where, DZe st Represents the matrix used to generate the spatiotemporal correlation matrix MZ st The decoder in the Transformer architecture, W dec Decoder DZe st The weight parameter of .

[0053] Preferably, in the monitoring module, the behavior pattern library is established through the LSTM-GRU hybrid network, and the establishment process is as follows:

[0054] Set the environmental data to light intensity Le, temperature Te and humidity He;

[0055] Set the hidden state of the LSTM-GRU hybrid network to hL LSTM , the hidden state of GRU is hL GRU , the environmental data feature vector is F env =[Le,Te,He];

[0056] The spatiotemporal correlation matrix MZ st and environmental data feature vector F env For splicing, the splicing formula is as follows:

[0057] X iut =[vec(MZ st );F env ];

[0058] Where, X iut Represents the input of the LSTM-GRU hybrid network, vec(MZ st ) represents the vectorization of the spatiotemporal correlation matrix;

[0059] In the LSTM-GRU hybrid network, the hidden state of each time step is updated, and the update formula is as follows:

[0060]

[0061]

[0062] Where, and Represent the hidden states of LSTM and GRU at time step tm, represents the input of the LSTM-GRU hybrid network at time step tm, and Represent the hidden states of LSTM and GRU at the previous time step tm, W LSTM and W GRU are the weight parameters corresponding to LSTM and GRU respectively;

[0063] right and Fusion is performed to obtain the behavior pattern feature F behv .

[0064] Better, about and Fusion is performed to obtain the behavior pattern feature F behv, the formula is as follows:

[0065]

[0066] Where σ represents the activation function, W fuse and b fuse Represent the weight and bias of the fusion layer, tm end represents the last time step of the sequence;

[0067] The behavioral pattern characteristics F under different sequences behv Store them and build a behavior pattern library BOO.

[0068] Preferably, in the monitoring module, when it is detected that the duration of the static feature tensor exceeds a threshold, the dynamic feature matrix of the data processing module is called to perform historical comparison verification. The verification process is as follows:

[0069] Set the static feature tensor duration threshold to

[0070] When a static feature tensor duration is detected Call history dynamic feature matrix For comparative verification, the verification calculation formula is:

[0071]

[0072] Where sim represents similarity, which is used to measure the similarity between current dynamic features and historical dynamic features;

[0073] When a static feature tensor duration is detected No comparison and verification is required, it is directly stored in the pattern library BOO and updated to the new behavior pattern library BOO new ;

[0074] Set the abnormal behavior tendency threshold to θ. If sim < θ, store the behavior in the new behavior pattern library BOO. new and marked as abnormal;

[0075] If sim≥θ, the behavior is stored in the new behavior pattern library BOO new , no marking is performed;

[0076] Dynamically update the static feature tensor duration threshold via an adaptive threshold regulator The update formula is as follows:

[0077]

[0078] Where, represents the updated static feature tensor duration threshold, α represents the learning rate, and NER represents the number of samples, that is, the number of samples used to calculate the threshold update. Represents the static feature duration of the ner-th sample.

[0079] Preferably, in the abnormal situation deduction module, a causal reasoning graph network is used to construct a three-layer reasoning engine. The construction process is as follows:

[0080] The first layer verifies the limb constraints. The verification process is as follows:

[0081] Set the joint angle eigenvector to is the angle of the kcaeth joint;

[0082] The normal angle range is Calculate the joint angle abnormality score using the following formula:

[0083]

[0084] Where s jint represents the joint angle abnormality score and is a non-negative value, kcae represents the total number of joints and is a positive integer, Represents the angle of the ikcaeth joint and is a real number, ikcae=[1,2,…,kcae], represents the mean of the normal joint angle and is a real number, represents the standard deviation of the normal joint angle and is a non-negative real number, z thj represents the abnormal threshold, which is used to determine whether the degree to which the joint angle deviates from the normal range is abnormal, and is a non-negative real number;

[0085] The second level analyzes energy metabolism characteristics:

[0086] Calculate the energy metabolic rate METRQ using the following formula:

[0087]

[0088] In the formula, V02 represents the oxygen consumption of the elderly during activities. It indicates the oxygen consumption of the elderly at rest;

[0089] Calculation of abnormal energy metabolism score S mmet , the calculation formula is as follows:

[0090] If METRQ>METRQ nr ,but

[0091] If METRQ ≤ METRQnr , then S mmet =0;

[0092] Where, METRQ nr Normal METRQ, METRQ max is the maximum METRQ;

[0093] The third level assesses the risk level. The assessment process is as follows:

[0094] Calculate the comprehensive risk level RiLe using the following formula:

[0095]

[0096] Where, and are the weight coefficients corresponding to joint abnormalities, energy metabolism abnormalities and static feature duration, respectively.

[0097] Preferably, in the abnormal situation deduction module, when the static features continue to meet the above conditions, it is determined to be abnormal and an alarm is issued. The determination process is as follows:

[0098] Set the comprehensive risk level threshold to JFXAA;

[0099] If RiLe>JFXAA, it is considered abnormal and an alarm is issued;

[0100] If RiLe≤JFXAA, it is judged as normal.

[0101] Technical effects and advantages of the present invention:

[0102] By deploying high-definition cameras, thermal imagers, millimeter-wave radars, and pressure sensors in nursing homes, we have achieved comprehensive, multi-dimensional data collection on the elderly's behavior. The AI ​​time synchronization algorithm ensures the accuracy of time alignment of data in different modalities, avoiding the impact of data misalignment on analysis results. The fusion of multimodal data makes up for the shortcomings of single sensor data. Especially in low light conditions or limited camera viewing angles, data from thermal imagers and millimeter-wave radars provide supplementary information, ensuring the continuity and integrity of static behavior monitoring, and providing a comprehensive and reliable data foundation for subsequent behavior recognition and monitoring.

[0103] Through the data processing module's dual-channel 3D convolutional neural network, dynamic and static features are effectively extracted from RGB video data and infrared thermal imaging data. The dynamic feature matrix reflects the user's movement pattern and helps to distinguish different types of activities. The static feature tensor captures the user's emotional state and body posture, providing a basis for semantic understanding of behavior and abnormal judgment. The feature decoupling algorithm further optimizes feature expression, allowing dynamic and static features to better play their respective advantages in subsequent fusion processing, thereby improving the system's recognition accuracy and overall performance of the elderly's static behavior.

[0104] Through the Transformer architecture in the multimodal perception fusion model, cross-modal feature alignment and fusion of millimeter-wave radar point cloud data and pressure distribution data are achieved. The generated spatiotemporal correlation matrix comprehensively represents the spatiotemporal relationship of the data. This fusion method fully leverages the advantages of Transformer in processing sequential data and capturing long-range dependencies, allowing data from different modalities to interact and fuse deeply at the feature level. Among them, millimeter-wave radar point cloud data provides the user's motion trajectory and spatial position information, and pressure distribution data reflects the user's contact with the ground or objects. The fusion of the two can more accurately describe the user's static and dynamic behavioral status, improve the system's understanding and analysis capabilities of complex behavioral scenarios, and especially accurately capture the behavioral characteristics of the elderly when they are stationary.

[0105] A comprehensive behavioral pattern library is established by using the LSTM-GRU hybrid network of the monitoring module in combination with the spatiotemporal correlation matrix and environmental data. This network structure can simultaneously capture the long-term dependencies and short-term changes of the data, so that the behavioral pattern library accurately reflects the behavioral characteristics and regularities of users under different environmental conditions. When it is detected that the duration of the static feature tensor exceeds the threshold, the dynamic feature matrix is ​​called for historical comparison and verification, which can timely detect potential abnormal behaviors. The adaptive threshold regulator dynamically updates the threshold, improving the system's adaptability and robustness to behavioral changes. Especially under different ambient lighting and temperature and humidity conditions, the behavioral pattern library established by combining environmental data can better adapt to these changes, improve the accuracy and reliability of behavior monitoring, and avoid the misjudgment or missed judgment problems that may be caused by fixed thresholds.

[0106] The three-layer reasoning constructed through the causal reasoning graph network of the abnormal situation deduction module can comprehensively and in-depth analyze abnormal situations of user behavior. The first layer verifies limb restraint conditions, pays attention to abnormal changes in joint angles, and promptly detects possible limb injuries and improper posture problems in the elderly, especially when the elderly are in a static posture. The second layer analyzes energy metabolism characteristics. By calculating the energy metabolism rate and abnormal energy metabolism score, it determines whether the elderly's energy consumption during activities is abnormal, which helps to discover potential health problems or excessive activity. The third layer assesses the risk level, comprehensively considering factors such as joint abnormalities, energy metabolism abnormalities, and the duration of static characteristics to derive a comprehensive risk level, more accurately judging the degree of danger of the behavior. This layered reasoning method makes anomaly detection more systematic and comprehensive, analyzing and judging behavior from multiple angles, improving the accuracy and reliability of anomaly detection. The mechanism that issues an alarm when static characteristics continue to meet abnormal conditions can promptly remind relevant personnel to pay attention to the elderly's abnormal behavior and take corresponding measures to ensure their safety and health. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0108] The present invention will be further described below in conjunction with specific embodiments. However, people familiar with the art should understand that the detailed description given here in conjunction with the drawings is for better explanation, and the structure of the present invention necessarily exceeds these limited embodiments. For some equivalent replacement solutions or common means, they will not be described in detail herein, but they still fall within the scope of protection of this application.

[0109] Figure 1 The best embodiment of the present invention is shown below in conjunction with the attached Figure 1 The present invention is further described.

[0110] Example 1

[0111] An AI-based platform user behavior recognition and monitoring system, including a data acquisition module, a data processing module, a multimodal perception fusion module, a monitoring module, and an abnormal situation deduction module;

[0112] The data acquisition module is used to collect RGB video data, infrared thermal imaging data, millimeter-wave radar point cloud data, and pressure distribution data in real time through high-definition cameras, thermal imagers, millimeter-wave radars, and pressure sensors deployed in the nursing home. It also generates a multimodal aligned dataset through timestamp alignment based on an AI-based time synchronization algorithm.

[0113] The data processing module uses an AI-driven dual-channel 3D convolutional neural network to extract dynamic and static features based on the RGB video data and infrared thermal imaging data in the multimodal alignment dataset, and generates a dynamic feature matrix and a static feature tensor through a feature decoupling algorithm.

[0114] The multimodal perception fusion module uses the Transformer architecture to perform cross-modal feature alignment based on the millimeter-wave radar point cloud data and pressure distribution data in the multimodal alignment dataset, generating a spatiotemporal correlation matrix that represents the spatiotemporal relationship.

[0115] The monitoring module, based on the spatiotemporal correlation matrix of the multimodal perception fusion module and combined with environmental data, establishes a behavior pattern library through an LSTM-GRU hybrid network. When it detects that the duration of the static feature tensor exceeds the threshold, it calls the dynamic feature matrix of the data processing module for historical comparison and verification, and dynamically updates the threshold through the adaptive threshold regulator, ultimately outputting a new behavior pattern library and dynamically adjusted threshold.

[0116] The abnormal situation deduction module is based on the new behavioral pattern library and dynamically adjusted thresholds in the monitoring module, combined with the static feature tensor of the data processing module, and uses a causal reasoning graph network to build a three-layer reasoning engine: the first layer verifies limb restraint conditions, the second layer analyzes energy metabolism characteristics, and the third layer assesses the danger level. When the static characteristics continue to meet the above conditions, it is judged as abnormal and an alarm is issued.

[0117] As a further aspect of the present invention, in the data acquisition module, a multimodal alignment dataset is generated by timestamp alignment for an AI-based time synchronization algorithm. The generation process is as follows:

[0118] Set the RGB video data to V = {v1, v2, ..., v n}, the timestamp set of high-definition camera data is τ V ={τ v1 , τ v2 ,…,τ vn};

[0119] The infrared thermal imaging data collected by the thermal imager is T = {t1, t2, ..., t m}, the timestamp set of infrared thermal imaging data is τ T ={τ t1 , τ t2 ,…,τ tm};

[0120] The point cloud data collected by the millimeter wave radar is P = {p1, p2, ..., p k}, the timestamp set of point cloud data is τ P ={τ p1 , τp2 ,...,τ pk};

[0121] The pressure distribution data collected by the pressure sensor is S = {s1, s2, ..., s l}, the timestamp set of pressure distribution data is τ S ={τ s1 , τ s2 ,…,τ sl};

[0122] Where, v n Represents the nth data point, n represents the number of elements of RGB video data V, τ vn Indicates the vth n The timestamp corresponding to the RGB video data point, t m represents the mth infrared thermal imaging data point, m represents the number of elements of infrared thermal imaging data T, τ tm Indicates the tth m The timestamp corresponding to the thermal imaging data point, p k Represents the kth point cloud data point, k represents the number of elements of the point cloud data P, τ pk Indicates the pth k The timestamp corresponding to the point cloud data point, s l represents the lth pressure distribution data point, l represents the number of elements in the pressure distribution data S, τ sl Indicates the sth l The timestamp corresponding to each pressure distribution data point;

[0123] Introduce the time synchronization function f sy , to achieve time alignment, the alignment formula is as follows:

[0124] f sy (D,τ D )={d i ∣τd i ∈[τ ref -δ,τ ref +δ]};

[0125] Where D can be any one of RGB video data V, infrared thermal imaging data T, point cloud data P or pressure distribution data S, τ D Represents the corresponding timestamp set, τ ref Indicates the reference timestamp, which can be any starting timestamp of RGB video stream data, infrared thermal imaging data, point cloud data or pressure distribution data. δ indicates the allowable time deviation range, d i represents the i-th data point in D, τd i Represents data d i timestamp;

[0126] For τ D Calculation is performed, and the calculation formula is as follows;

[0127] τ D ={τ V , τ T , τ P , τ S};

[0128] For RGB video data V, infrared thermal imaging data T, point cloud data P and pressure distribution data S, traverse their timestamp collection τ respectively V , τ T , τ P , τ S , find all the i ∈[τ ref -δ,τ ref +δ] condition, and thus obtain the corresponding subset f sy (V,τ V ),f sy (T,τ T ),f sy (P,τ P ),f sy (S,τ S );

[0129] Based on the above subsets, generate a multimodal alignment dataset D al , the generation formula is as follows:

[0130] D al ={f sy (V,τ V ), f sy (T,τ T ), fsy(P,τ P ), fsy(S,τ S )}.

[0131] As a further aspect of the present invention, in the data processing module, the AI-driven dual-channel 3D convolutional neural network extracts dynamic features and static features respectively. The extraction process is as follows:

[0132] The convolution kernel of the AI-driven dual-channel 3D convolutional neural network is set to B, with a size of b×b×b, a stride of c, and a padding of r.

[0133] Extract dynamic features of RGB video data V The extraction formula is as follows;

[0134]

[0135] Where, 3D-CNN dnrepresents a three-dimensional convolutional neural network for extracting dynamic features;

[0136] Extracting static features of infrared thermal imaging data T The extraction formula is as follows;

[0137]

[0138] Where, 3D-CNN st Represents the three-dimensional convolutional neural network for extracting static features, B′ represents the convolution kernel size of the 3D convolutional neural network for static feature extraction, c′ represents the step size of the 3D convolutional neural network for static feature extraction, and r′ represents the padding method of the 3D convolutional neural network for static feature extraction.

[0139] As a further aspect of the present invention, in the data processing module, the dynamic feature matrix and the static feature tensor are generated by the feature decoupling algorithm. The generation process is as follows:

[0140] For dynamic features The decoupling formula is:

[0141]

[0142] Where, Decoupling function of dynamic features extracted from RGB video data V, W dn represents the weight parameter in dynamic feature decoupling, M dn represents the dynamic feature matrix;

[0143] For static features The decoupling formula is:

[0144]

[0145] Where, Decoupling function of the static features extracted from infrared thermal imaging data T, W st represents the weight parameter in static feature decoupling, M st Represents a static feature tensor.

[0146] As a further aspect of the present invention, in the multimodal perception fusion module, the Transformer architecture is used to perform cross-modal feature alignment and generate a spatiotemporal correlation matrix that represents the spatiotemporal relationship. The generation process is as follows:

[0147] The point cloud data P and pressure distribution data S are encoded separately by the encoder Enr. The encoding formula is as follows:

[0148] E P =Enr P (P; Wnr P);

[0149] E S =Enr S (S; Wnr S );

[0150] Where, E P and E S Represent the encoding features of point cloud data P and pressure distribution data S respectively, Enr P and Enr S Represents the encoders of point cloud data P and pressure distribution data S, Wnr P and Wnr S Represents encoder Enr P and Enr S The weight parameter of

[0151] In the Transformer architecture, let the multi-head attention function be Muad(QA, QK, QV), set QA as the query vector, QK as the key vector, QV as the value vector, and use E P As query and key, E S As the value, calculate the cross-modal attention Acs, the calculation formula is as follows:

[0152] Acs=Muad(E P , E S ;W QA , W QK , W QV );

[0153] Where W QA 、W QK and W QV are the weight matrices in the multi-head attention mechanism;

[0154] Through the decoder structure of Transformer, the aligned features are fused to generate the spatiotemporal correlation matrix MZ that represents the spatiotemporal relationship. st , the generation formula is as follows:

[0155] MZ st =DZe st (Acs; W dec );

[0156] Where, DZe st Represents the matrix used to generate the spatiotemporal correlation matrix MZ st The decoder in the Transformer architecture, W dec Decoder DZe st The weight parameter of .

[0157] Example 2

[0158] As a further aspect of the present invention, in the monitoring module, a behavior pattern library is established through an LSTM-GRU hybrid network, and the establishment process is as follows:

[0159] Set the environmental data to light intensity Le, temperature Te and humidity He;

[0160] Set the hidden state of the LSTM-GRU hybrid network to hL LSTM , the hidden state of GRU is hL GRU , the environmental data feature vector is F env =[Le,Te,He];

[0161] The spatiotemporal correlation matrix MZ st and environmental data feature vector F env For splicing, the splicing formula is as follows:

[0162] X iut =[vec(MZ st );F env ];

[0163] Where, X iut Represents the input of the LSTM-GRU hybrid network, vec(MZ st ) represents the vectorization of the spatiotemporal correlation matrix;

[0164] In the LSTM-GRU hybrid network, the hidden state of each time step is updated, and the update formula is as follows:

[0165]

[0166] Where, and Represent the hidden states of LSTM and GRU at time step tm, represents the input of the LSTM-GRU hybrid network at time step tm, and Represent the hidden states of LSTM and GRU at the previous time step tm, W LSTM and W GRU are the weight parameters corresponding to LSTM and GRU respectively;

[0167] right and Fusion is performed to obtain the behavior pattern feature F behv .

[0168] As a further aspect of the present invention, and Fusion is performed to obtain the behavior pattern feature F behv , the formula is as follows:

[0169]

[0170] Where σ represents the activation function, W fuse and b fuse Represent the weight and bias of the fusion layer, tm end represents the last time step of the sequence;

[0171] The behavioral pattern characteristics F under different sequences behv Store them and build a behavior pattern library BOO.

[0172] As a further feature of the present invention, in the monitoring module, when it is detected that the duration of the static feature tensor exceeds the threshold, the dynamic feature matrix of the data processing module is called to perform historical comparison verification. The verification process is as follows:

[0173] Set the static feature tensor duration threshold to

[0174] When a static feature tensor duration is detected Call history dynamic feature matrix For comparative verification, the verification calculation formula is:

[0175]

[0176] Where sim represents similarity, which is used to measure the similarity between current dynamic features and historical dynamic features;

[0177] When a static feature tensor duration is detected No comparison and verification is required, it is directly stored in the pattern library BOO and updated to the new behavior pattern library BOO new ;

[0178] Set the abnormal behavior tendency threshold to θ. If sim < θ, store the behavior in the new behavior pattern library BOO. new and marked as abnormal;

[0179] If sim≥θ, the behavior is stored in the new behavior pattern library BOO new , no marking is performed;

[0180] Dynamically update the static feature tensor duration threshold via an adaptive threshold regulator The update formula is as follows:

[0181]

[0182] Where, represents the updated static feature tensor duration threshold, α represents the learning rate, and NER represents the number of samples, that is, the number of samples used to calculate the threshold update. Represents the static feature duration of the ner-th sample.

[0183] As a further aspect of the present invention, in the abnormal situation deduction module, a causal reasoning graph network is used to construct a three-layer reasoning engine, and the construction process is as follows:

[0184] The first layer verifies the limb constraints. The verification process is as follows:

[0185] Set the joint angle eigenvector to is the angle of the kcaeth joint;

[0186] The normal angle range is Calculate the joint angle abnormality score using the following formula:

[0187]

[0188] Where s jint represents the joint angle abnormality score and is a non-negative value, kcae represents the total number of joints and is a positive integer, Represents the angle of the ikcaeth joint and is a real number, ikcae=[1,2,…,kcae], represents the mean of the normal joint angle and is a real number, represents the standard deviation of the normal joint angle and is a non-negative real number, z thj represents the abnormal threshold, which is used to determine whether the degree to which the joint angle deviates from the normal range is abnormal, and is a non-negative real number;

[0189] The second level analyzes energy metabolism characteristics:

[0190] Calculate the energy metabolic rate METRQ using the following formula:

[0191]

[0192] In the formula, V02 represents the oxygen consumption of the elderly during activities. It indicates the oxygen consumption of the elderly at rest;

[0193] Calculation of abnormal energy metabolism score S mmet , the calculation formula is as follows:

[0194] If METRQ>METRQ nr ,but

[0195] If METRQ ≤ METRQnr , then S mmet =0;

[0196] Where, METRQ nr Normal METRQ, METRQ max is the maximum METRQ;

[0197] The third level assesses the risk level. The assessment process is as follows:

[0198] Calculate the comprehensive risk level RiLe using the following formula:

[0199]

[0200] Where, and are the weight coefficients corresponding to joint abnormalities, energy metabolism abnormalities and static feature duration, respectively.

[0201] As a further feature of the present invention, in the abnormal situation deduction module, when the static features continue to meet the above conditions, it is determined to be abnormal and an alarm is issued. The determination process is as follows:

[0202] Set the comprehensive risk level threshold to JFXAA;

[0203] If RiLe>JFXAA, it is considered abnormal and an alarm is issued;

[0204] If RiLe≤JFXAA, it is judged as normal.

[0205] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. An AI-based platform user behavior recognition and monitoring system, characterized by: It includes data acquisition module, data processing module, multimodal perception fusion module, monitoring module and abnormal situation deduction module; The data acquisition module is used to collect RGB video data, infrared thermal imaging data, millimeter-wave radar point cloud data, and pressure distribution data in real time through high-definition cameras, thermal imagers, millimeter-wave radars, and pressure sensors deployed in the nursing home. It also generates a multimodal aligned dataset through timestamp alignment based on an AI-based time synchronization algorithm. The data processing module uses an AI-driven dual-channel 3D convolutional neural network to extract dynamic and static features based on the RGB video data and infrared thermal imaging data in the multimodal alignment dataset, and generates a dynamic feature matrix and a static feature tensor through a feature decoupling algorithm. The multimodal perception fusion module uses the Transformer architecture to perform cross-modal feature alignment based on the millimeter-wave radar point cloud data and pressure distribution data in the multimodal alignment dataset, generating a spatiotemporal correlation matrix that represents the spatiotemporal relationship. The monitoring module, based on the spatiotemporal correlation matrix of the multimodal perception fusion module and combined with environmental data, establishes a behavior pattern library through an LSTM-GRU hybrid network. When it detects that the duration of the static feature tensor exceeds the threshold, it calls the dynamic feature matrix of the data processing module for historical comparison and verification, and dynamically updates the threshold through the adaptive threshold regulator, ultimately outputting a new behavior pattern library and dynamically adjusted threshold. The abnormal situation deduction module is based on the new behavioral pattern library and dynamically adjusted thresholds in the monitoring module, combined with the static feature tensor of the data processing module, and uses a causal reasoning graph network to build a three-layer reasoning engine: the first layer verifies limb restraint conditions, the second layer analyzes energy metabolism characteristics, and the third layer assesses the danger level. When the static characteristics continue to meet the above conditions, it is judged as abnormal and an alarm is issued.

2. The AI-based platform user behavior recognition and monitoring system according to claim 1 is characterized in that: In the data acquisition module, for the AI-based time synchronization algorithm, a multimodal alignment dataset is generated by timestamp alignment. The generation process is as follows: Set the RGB video data to V = {v1, v2, ..., v n }, the timestamp set of high-definition camera data is τ V ={τ v1 , τ v2 ,…,τ vn }; The infrared thermal imaging data collected by the thermal imager is T = {t1, t2, ..., t m }, the timestamp set of infrared thermal imaging data is τ T ={τ t1 , τ t2 ,…,τ tm }; The point cloud data collected by the millimeter wave radar is P = {p1, p2, ..., p k }, the timestamp set of point cloud data is τ P ={τ p1 , τ p2 ,...,τ pk }; The pressure distribution data collected by the pressure sensor is S = {s1, s2, ..., s l }, the timestamp set of pressure distribution data is τ S ={τ s1 , τ s2 ,…,τ sl }; Where, v n Represents the nth data point, n represents the number of elements of RGB video data V, τ vn Indicates the vth n The timestamp corresponding to the RGB video data point, t m represents the mth infrared thermal imaging data point, m represents the number of elements of infrared thermal imaging data T, τ tm Indicates the tth m The timestamp corresponding to each thermal imaging data point, p k Represents the kth point cloud data point, k represents the number of elements of the point cloud data P, τ pk Indicates the pth k The timestamp corresponding to the point cloud data point, s l represents the lth pressure distribution data point, l represents the number of elements in the pressure distribution data S, τ sl Indicates the sth l The timestamp corresponding to each pressure distribution data point; Introduce the time synchronization function f sy , to achieve time alignment, the alignment formula is as follows: f sy (D,t D )={d i ∣τd i ∈[τ ref -d,t ref +d]}; Where D can be any one of RGB video data V, infrared thermal imaging data T, point cloud data P or pressure distribution data S, τ D Represents the corresponding timestamp set, τ ref Indicates the reference timestamp, which can be any starting timestamp of RGB video stream data, infrared thermal imaging data, point cloud data or pressure distribution data. δ indicates the allowable time deviation range, d i represents the i-th data point in D, τd i Represents data d i timestamp; For τ D Calculation is performed, and the calculation formula is as follows; t D ={τ V ,t T ,t P ,t S }; For RGB video data V, infrared thermal imaging data T, point cloud data P and pressure distribution data S, traverse their timestamp collection τ respectively V , τ T , τ P , τ S , find all the i ∈[τ ref -δ,τ ref +δ] condition, and thus obtain the corresponding subset f sy (V,τ V ),f sy (T,τ T ),f sy (P,τ P ),f sy (S,τ S ); Based on the above subsets, generate a multimodal alignment dataset D al , the generation formula is as follows: D al ={f sy (V,t V ),f sy (T,τ T ),fsy(P,τ P ),fsy(S,τ S )}。 3. The AI-based platform user behavior recognition and monitoring system according to claim 2 is characterized in that: In the data processing module, the AI-driven dual-channel 3D convolutional neural network extracts dynamic features and static features respectively. The extraction process is as follows: The convolution kernel of the AI-driven dual-channel 3D convolutional neural network is set to B, with a size of b×b×b, a stride of c, and a padding of r. Extract dynamic features of RGB video data V The extraction formula is as follows; Where, 3D-CNN dn represents a three-dimensional convolutional neural network for extracting dynamic features; Extracting static features of infrared thermal imaging data T The extraction formula is as follows; Where, 3D-CNN st Represents the three-dimensional convolutional neural network for extracting static features, B′ represents the convolution kernel size of the 3D convolutional neural network for static feature extraction, c′ represents the step size of the 3D convolutional neural network for static feature extraction, and r′ represents the padding method of the 3D convolutional neural network for static feature extraction.

4. The AI-based platform user behavior recognition and monitoring system according to claim 3 is characterized in that: Regarding the data processing module, the generation process of dynamic feature matrix and static feature tensor through feature decoupling algorithm is as follows: For dynamic features The decoupling formula is: Where, Decoupling function of dynamic features extracted from RGB video data V, W dn represents the weight parameter in dynamic feature decoupling, M dn represents the dynamic feature matrix; For static features The decoupling formula is: Where, Decoupling function of the static features extracted from infrared thermal imaging data T, W st represents the weight parameter in static feature decoupling, M st Represents a static feature tensor.

5. The AI-based platform user behavior recognition and monitoring system according to claim 1 is characterized in that: In the multimodal perception fusion module, the Transformer architecture is used to align cross-modal features and generate a spatiotemporal correlation matrix that represents spatiotemporal relationships. The generation process is as follows: The point cloud data P and pressure distribution data S are encoded separately by the encoder Enr. The encoding formula is as follows: E P =Enr P (P;Wnr P ); E S =Enr S (S;Wnr S ); Where, E P and E S Represent the encoding features of point cloud data P and pressure distribution data S respectively, Enr P and Enr S Represents the encoders of point cloud data P and pressure distribution data S, Wnr P and Wnr S Represents encoder Enr P and Enr S The weight parameter of In the Transformer architecture, let the multi-head attention function be Muad(QA, QK, QV), set QA as the query vector, QK as the key vector, QV as the value vector, and use E P As query and key, E S As the value, calculate the cross-modal attention Acs, the calculation formula is as follows: Acs=Muad(E P ,HAVE BEEN S ;W QA ,W QK ,W QV ); Where W QA 、W QK and W QV are the weight matrices in the multi-head attention mechanism; Through the decoder structure of Transformer, the aligned features are fused to generate the spatiotemporal correlation matrix MZ that represents the spatiotemporal relationship. st , the generation formula is as follows: MZ st =DZe st (Acs;W dec ); Where, DZe st Represents the matrix used to generate the spatiotemporal correlation matrix MZ st The decoder in the Transformer architecture, W dec Decoder DZe st The weight parameter of .

6. The AI-based platform user behavior recognition and monitoring system according to claim 5, characterized in that: Regarding the monitoring module, the behavior pattern library is established through the LSTM-GRU hybrid network. The establishment process is as follows: Set the environmental data to light intensity Le, temperature Te and humidity He; Set the hidden state of the LSTM-GRU hybrid network to hL LSTM , the hidden state of GRU is hL GRU , the environmental data feature vector is F env =[Le,Te,He]; The spatiotemporal correlation matrix MZ st and environmental data feature vector F env For splicing, the splicing formula is as follows: X iut =[vec(MZ st );F env ]; Where, X iut Represents the input of the LSTM-GRU hybrid network, vec(MZ st ) represents the vectorization of the spatiotemporal correlation matrix; In the LSTM-GRU hybrid network, the hidden state of each time step is updated, and the update formula is as follows: Where, and Represent the hidden states of LSTM and GRU at time step tm, represents the input of the LSTM-GRU hybrid network at time step tm, and Represent the hidden states of LSTM and GRU at the previous time step tm, W LSTM and W GRU are the weight parameters corresponding to LSTM and GRU respectively; right and Fusion is performed to obtain the behavior pattern feature F behv .

7. The AI-based platform user behavior recognition and monitoring system according to claim 6 is characterized in that: About and Fusion is performed to obtain the behavior pattern feature F behv , the formula is as follows: Where σ represents the activation function, W fuse and b fuse Represent the weight and bias of the fusion layer, tm end represents the last time step of the sequence; The behavioral pattern features F under different sequences behv Store them and build a behavior pattern library BOO.

8. The AI-based platform user behavior recognition and monitoring system according to claim 4 is characterized in that: In the monitoring module, when it is detected that the duration of the static feature tensor exceeds the threshold, the dynamic feature matrix of the data processing module is called for historical comparison verification. The verification process is as follows: Set the static feature tensor duration threshold to When a static feature tensor duration is detected Call history dynamic feature matrix For comparative verification, the verification calculation formula is: Where sim represents similarity, which is used to measure the similarity between current dynamic features and historical dynamic features; When a static feature tensor duration is detected No comparison and verification is required, it is directly stored in the pattern library BOO and updated to the new behavior pattern library BOO new ; Set the abnormal behavior tendency threshold to θ. If sim < θ, store the behavior in the new behavior pattern library BOO. new and marked as abnormal; If sim≥θ, the behavior is stored in the new behavior pattern library BOO new , no marking is performed; Dynamically update the static feature tensor duration threshold via an adaptive threshold regulator The update formula is as follows: Where, represents the updated static feature tensor duration threshold, α represents the learning rate, and NER represents the number of samples, that is, the number of samples used to calculate the threshold update. Represents the static feature duration of the ner-th sample.

9. The AI-based platform user behavior recognition and monitoring system according to claim 8, characterized in that: In the abnormal situation deduction module, a causal reasoning graph network is used to build a three-layer reasoning engine. The construction process is as follows: The first layer verifies the limb constraints. The verification process is as follows: Set the joint angle eigenvector to is the angle of the kcaeth joint; The normal angle range is Calculate the joint angle abnormality score using the following formula: Where s jint represents the joint angle abnormality score and is a non-negative value, kcae represents the total number of joints and is a positive integer, Represents the angle of the ikcaeth joint and is a real number, ikcae=[1,2,…,kcae], represents the mean of the normal joint angle and is a real number, represents the standard deviation of the normal joint angle and is a non-negative real number, z thj represents the abnormal threshold, which is used to determine whether the degree to which the joint angle deviates from the normal range is abnormal, and is a non-negative real number; The second level analyzes energy metabolism characteristics: Calculate the energy metabolic rate METRQ using the following formula: In the formula, V02 represents the oxygen consumption of the elderly during activities. Indicates the oxygen consumption of the elderly at rest; Calculation of abnormal energy metabolism score S mmet , the calculation formula is as follows: If METRQ>METRQ nr ,but If METRQ ≤ METRQ nr , then S mmet =0; Where, METRQ nr Normal METRQ, METRQ max is the maximum METRQ; The third level assesses the risk level. The assessment process is as follows: Calculate the comprehensive risk level RiLe using the following formula: Where, and are the weight coefficients corresponding to joint abnormalities, energy metabolism abnormalities and static feature duration, respectively.

10. The AI-based platform user behavior recognition and monitoring system according to claim 9, characterized in that: Regarding the abnormal situation deduction module, when the static characteristics continue to meet the above conditions, it is judged as abnormal and an alarm is issued. The judgment process is as follows: Set the comprehensive risk level threshold to JFXAA; If RiLe>JFXAA, it is considered abnormal and an alarm is issued; If RiLe≤JFXAA, it is judged as normal.

Images