An old person abnormal behavior recognition and system based on a smart endowment terminal device

Abstract

This invention relates to a system for identifying and managing abnormal behavior in the elderly based on smart elderly care terminal devices, specifically in the field of abnormal behavior recognition. By effectively fusing visual and radar information through cross-modal attention alignment, it overcomes the limitations of single sensors in complex home environments such as obstruction and low light, thereby improving the robustness of data acquisition. The adopted joint training paradigm of mask reconstruction and contrastive learning drives the model to learn general behavioral representations invariant to environmental disturbances, enhancing the model's generalization ability. Utilizing spatiotemporal graph modeling and prototype contrastive quantitative learning, it can accurately capture and distinguish subtle differences in movements, achieving high-precision, fine-grained abnormal behavior recognition. Through a teacher-student framework and a personalized adaptive update mechanism, the model can continuously adapt to the user's unique behavioral patterns and mitigate concept drift, ultimately significantly reducing false alarms and false negatives, providing reliable, long-term, and personalized safety monitoring for the elderly.

Description

Technical Field

[0001] This invention relates to the field of abnormal behavior recognition in the elderly, and more specifically, to a system for recognizing and recognizing abnormal behavior in the elderly based on a smart elderly care terminal device. Background Technology

[0002] With the accelerating aging of society, the need for safety and health monitoring of elderly people living alone is becoming increasingly prominent. Smart elderly care terminal devices based on visual perception, which deploy cameras and other sensors in the home environment, can achieve non-contact, continuous monitoring of the daily activities of the elderly and provide timely warnings when abnormal behavior occurs. This has become an important technological solution. Typical home environments are often unstructured, with highly diverse and random layouts of interior spaces, furniture placement, and the arrangement of personal belongings, creating complex visual backgrounds. At the same time, the behavioral patterns of the elderly are unique, with generally slower movements and some indicators of health. Abnormal behaviors indicating a decline in condition, such as slowly sliding down a wall due to dizziness and weakness, or aimless wandering at night due to cognitive impairment, often differ visually from normal daily activities, such as bending down to pick up an object or pacing around the room in contemplation. In addition, real-world applications face numerous dynamic interferences, including drastic changes in lighting conditions caused by day-night cycles, switching on and off lights, and opening and closing curtains; partial obstruction of the subject by furniture and plants; and interference introduced by moving objects such as pets and visitors. These internal and external factors together constitute a complex, dynamic, and uncertain monitoring environment, posing a fundamental challenge to the accurate identification of behaviors.

[0003] Currently, computer vision-based technologies for recognizing abnormal behaviors in the elderly, especially in areas such as fall detection, have made some progress. However, existing solutions largely rely on standardized datasets collected in controlled environments for model training. These datasets often feature simple, clean backgrounds and lack sufficient coverage of the complex, long-tailed situations in real-world home environments. This results in a significant decrease in the robustness and generalization ability of the trained behavior recognition models when faced with sudden changes in lighting, partial occlusion, and dynamic background clutter. They struggle to reliably extract discriminative spatiotemporal behavioral features from video streams. More importantly, for fine-grained abnormal behaviors, such as slow slips and falls… The two patterns of slowly sitting down are highly similar in terms of movement trajectory and posture. Existing models, due to their limited feature representation capabilities, struggle to learn the subtle discriminative differences between them. This leads to frequent false alarms and missed detections in practical applications. Missed detections will delay the discovery and rescue of sudden health risks in the elderly, while frequent false alarms will trigger a crisis of trust among users, potentially leading to the abandonment of the system. Therefore, how to enable behavior recognition algorithms to resist various interferences and accurately distinguish between normal and abnormal subtle behavioral boundaries in complex, dynamic, and unstructured home environments has become a core technical problem that urgently needs to be solved to improve the practicality and reliability of smart elderly care systems. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a system for identifying and managing abnormal behaviors of the elderly based on smart elderly care terminal devices. The system solves the problems mentioned in the background section by employing a data acquisition and alignment module, a general representation learning module, a fine-grained discrimination module, and a model update module.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: Specifically, it includes: a data acquisition and alignment module, a general representation learning module, a fine-grained discrimination module, and a model update module connected in sequence, wherein; Data Acquisition and Alignment Module: Upon startup, it synchronously acquires the visual video stream from the RGB camera and the point cloud sequence from the millimeter-wave radar, and aligns the visual video stream and the point cloud sequence in time and space. Then, using its internal cross-modal attention alignment unit, it performs mutual querying and feature weighted fusion on the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputs the first joint multimodal feature vector formed after fusion. The general representation learning module receives the first joint multimodal feature vector, arranges the first joint multimodal feature vector in time sequence to form a continuous first joint multimodal feature vector sequence, randomly masks the first joint multimodal feature vector sequence in the spatiotemporal dimension, reconstructs the masked part based on the unmasked part using its internal spatiotemporal Transformer encoder-decoder structure, and applies contrastive learning constraints to visual spatial features and radar spatiotemporal features from the same time period. After joint training, it outputs a general behavior representation vector. Fine-grained discrimination module: Receives a general behavior representation vector, first decodes the sequence of three-dimensional key points of the human body in consecutive frames from the general behavior representation vector, and constructs a spatiotemporal graph with joints as nodes and skeletal connections and temporal connections as edges based on the sequence of three-dimensional key points of the human body. Then, a temporal graph neural network is used to extract features from the spatiotemporal graph to obtain graph features. The extracted graph features are input into the prototype contrast ratio learning unit inside the unit. Fine-grained behavior classification is achieved by calculating the distance with the preset prototype vectors of various behaviors, and the current behavior is judged to be abnormal. Model update module: After deployment in a specific user environment, the trained general representation learning module and fine-grained discrimination module are used together as the teacher model. A student model with the same structure as the teacher model is deployed on the terminal for initialization. During operation, the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavior representation vectors and graph features are stored. The stored historical general behavior representation vectors and graph features are used periodically to replay and train the student model. At the same time, the model update strategy is adjusted according to the degree of difference between the predictions of the student model and the teacher model for the same input sample, so as to realize the personalized adaptation of the model. In a preferred embodiment, the data acquisition and alignment module synchronously acquires the visual video stream from the RGB camera and the point cloud sequence from the millimeter-wave radar, and aligns the visual video stream and the point cloud sequence in time and space as follows: First, a time-stamp-aligned visual video stream consisting of T consecutive RGB images is obtained through hardware synchronization signals, along with a point cloud sequence consisting of T consecutive 3D spatial point sets containing distance, velocity, and angle information. Then, based on pre-completion joint calibration parameters, the 3D spatial points in each frame of the point cloud sequence are projected onto the 2D pixel positions in the corresponding timestamp's RGB image coordinate system, achieving spatial alignment of visual and radar information on the 2D image plane. Next, a pre-trained 2D convolutional neural network is used to process the time-aligned visual video stream frame by frame to extract visual spatial features. Simultaneously, a 3D point cloud feature encoding network is used to process the spatially aligned point cloud sequence frame by frame to extract radar spatiotemporal features.

[0006] In a preferred embodiment, the specific process of performing mutual querying and feature weighted fusion of the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputting the first joint multimodal feature vector after fusion, is as follows: The first basis matrix is ​​defined as the two-dimensional feature matrix obtained by reshaping the visual spatial features in the dimensions of time, feature map height, and feature map width. The second basis matrix is ​​defined as the two-dimensional feature matrix obtained by reshaping the radar spatiotemporal features in the dimensions of time and the number of abstracted point sets. The first basis matrix is ​​projected into a first query matrix through a first learnable linear transformation matrix. The second basis matrix is ​​projected into a first key matrix through a second learnable linear transformation matrix. The second basis matrix is ​​projected into a first value matrix through a third learnable linear transformation matrix. The product between the transpose of the first query matrix and the first key matrix is ​​calculated to obtain the first similarity matrix. A normalized exponential function is applied to each row of the first similarity matrix to obtain the first attention weight matrix. The first attention weight matrix is ​​multiplied by the first value matrix to obtain the first weighted value matrix. The first weighted value matrix is ​​projected through a fourth learnable linear transformation matrix, and the projection result is added to the first basis matrix. The added result is then subjected to layer normalization to generate the enhanced visual feature matrix. The second basis matrix is ​​projected into the second query matrix using the fifth learnable linear transformation matrix; the first basis matrix is ​​projected into the second key matrix using the sixth learnable linear transformation matrix; the first basis matrix is ​​projected into the second value matrix using the seventh learnable linear transformation matrix; the product between the transpose of the second query matrix and the second key matrix is ​​calculated to obtain the second similarity matrix; a normalized exponential function is applied to each row of the second similarity matrix to obtain the second attention weight matrix; the second attention weight matrix is ​​multiplied by the second value matrix to obtain the second weighted matrix; the second weighted matrix is ​​projected using the eighth learnable linear transformation matrix, and the projection result is added to the second basis matrix. The added result is then subjected to layer normalization to generate the enhanced radar feature matrix. The enhanced visual feature matrix and the enhanced radar feature matrix are concatenated along the feature channel dimension to form a concatenated feature matrix. This concatenated feature matrix is ​​then input into a multilayer perceptron for nonlinear transformation and dimensionality reduction. The feature matrix output by the multilayer perceptron is the first joint multimodal feature vector.

[0007] In a preferred embodiment, the specific process of randomly masking the sequence of first joint multimodal feature vectors arranged in time in the general representation learning module in the spatiotemporal dimension is as follows: For the first joint multimodal feature vector sequence, a binary mask indicator matrix with the same length as the total number of first joint multimodal feature vectors in the sequence is generated. The masked positions in the binary mask indicator matrix are represented by a first value, and the unmasked positions are represented by a second value. Then, a learnable mask marker vector is created. Then, a mask replacement operation is performed according to the binary mask indicator matrix, that is, all first joint multimodal feature vectors in the first joint multimodal feature vector sequence corresponding to the first value are replaced with the learnable mask marker vector, while all first joint multimodal feature vectors corresponding to the second value are retained, thus generating a masked feature sequence.

[0008] In a preferred embodiment, the specific process of outputting the general behavioral representation vector after joint training is as follows: The masked feature sequence is input into a spatiotemporal Transformer encoder, which consists of multiple sequentially connected multi-head self-attention layers and feedforward neural network layers. The encoder outputs a sequence of encoder hidden representations containing the global spatiotemporal context. This sequence of encoder hidden representations and positional encoding information are input into a spatiotemporal Transformer decoder. The spatiotemporal Transformer decoder outputs a reconstructed feature sequence for the entire input. Based on the reconstructed feature sequence and the original first joint multimodal feature vector sequence, a reconstruction loss is calculated. At the same time, visual spatial features and radar spatiotemporal features extracted by the data acquisition and alignment module at the same time period as the first joint multimodal feature vector sequence are obtained. The visual spatial features and radar spatiotemporal features are mapped to a shared contrast feature space through a visual modality projection network and a radar modality projection network, respectively, to obtain visual projection features and radar projection features. Based on visual projection features and radar projection features, a cross-modal contrast loss is calculated. This loss function is constructed based on the principle of noise contrast estimation. The reconstruction loss and cross-modal contrast loss are weighted and summed with a pre-defined balanced weight between zero and one to obtain a joint training loss. The parameters of the spatiotemporal Transformer encoder-decoder structure, the visual modality projection network, and the radar modality projection network are simultaneously optimized through the backpropagation algorithm to minimize the joint training loss. After training, the complete, unmasked first joint multimodal feature vector sequence is input into the trained spatiotemporal Transformer encoder, and the encoder hidden representation sequence output by the encoder is used as the general behavior representation vector output.

[0009] In a preferred embodiment, the specific process of decoding the sequence of three-dimensional human body key points from the general behavior representation vector in the fine-grained discrimination module is as follows: The system receives the general behavior representation vector from the general representation learning module, performs a temporal pooling operation on the general behavior representation vector to convert it into a fixed-length, time-ordered sequence of general behavior representation vectors, and then inputs the general behavior representation vectors obtained at each time step after pooling into a lightweight keypoint decoding network. This lightweight keypoint decoding network consists of one or more fully connected layers, and the number of output neurons in its last layer is equal to the preset number of human joints multiplied by three. This network is used to map each high-dimensional general behavior representation vector into the three-dimensional spatial coordinates of all human joints in the corresponding frame, thereby outputting a sequence of three-dimensional human keypoints consisting of multiple consecutive frames, each frame containing the three-dimensional coordinates of multiple joints.

[0010] In a preferred embodiment, the specific process of constructing a spatiotemporal graph based on the human body's three-dimensional keypoint sequence, with joints as nodes and skeletal connections and temporal connections as edges, and using a temporal graph neural network to extract features from the spatiotemporal graph to obtain graph features is as follows: Each joint in each frame of the human body 3D keypoint sequence is defined as a graph node. A node feature vector is constructed for each node. The elements of the node feature vector consist of the 3D coordinates of the joint in the current frame, the first-order motion velocity formed by the difference between the 3D coordinates of the joint in the current frame and the previous frame, and the second-order motion acceleration formed by the difference between the first-order motion velocity of the joint in the current frame and the first-order motion velocity in the previous frame. The edges of the graph are divided into two types: the first type is spatial edges, which connect pairs of joints with physical connections in the same frame according to the standard human skeleton topology; the second type is temporal edges, which connect pairs of joints representing the same joint in two adjacent frames. Two nodes are connected by edges to construct a dynamic spatiotemporal graph. This dynamic spatiotemporal graph is then input into a spatiotemporal graph neural network, which consists of alternating stacked spatial graph convolutional units and temporal convolutional units. The spatial graph convolutional units aggregate features of neighboring nodes along the spatial edges to learn the cooperative relationships between joints, while the temporal convolutional units aggregate features of the same joint in adjacent frames along the temporal edges to learn motion trajectory patterns. After multiple layers of information transmission, global graph pooling is performed on the node feature vectors of all nodes in the final output. By calculating the mean or maximum value of all node feature vectors, a fixed-dimensional graph feature is obtained to represent the entire behavioral segment.

[0011] In a preferred embodiment, the specific process of inputting the extracted graph features into its internal prototype contrast learning unit, and calculating the distance with preset behavioral prototype vectors to achieve fine-grained behavior classification and determine whether the current behavior is abnormal is as follows: The prototype contrast ratio learning unit internally maintains a learnable parameter vector as the behavior prototype vector for each behavior category to be identified, including multiple normal behavior categories and multiple abnormal behavior categories. During the training phase of the prototype contrast ratio learning unit, for each graph feature of the input training sample, the distance between the graph feature and the behavior prototype vector of its own labeled category is calculated, and the distance between the graph feature and the behavior prototype vectors of all other categories is also calculated. The prototype contrast ratio learning unit is optimized by minimizing a joint loss function. For each graph feature of the training sample, the joint loss function calculates the distance between the graph feature and the behavior prototype vector corresponding to the labeled category of the training sample as the first term. Simultaneously, for each other category, a learnable marginal parameter corresponding to that category is calculated, minus the distance between the graph feature and the behavior prototype vector of the labeled category. The distance between the graph feature and the behavior prototype vector of other categories is compared with zero, and the maximum value is taken. Then, it is multiplied by a hard sample mining weight related to the other categories. The maximum value of this weighted sum for all other categories is summed as the second term. Finally, the first term and the second term are added to obtain the loss value of the training sample. The total loss function is obtained by summing the loss values ​​of all training samples. In the inference stage after the prototype contrast quantitative learning unit completes training, for the training sample to be judged, the distance between its graph feature and the behavior prototype vector of all behavior categories is calculated. It is classified into the behavior category corresponding to the behavior prototype vector with the smallest distance. Abnormal behavior is judged by determining whether the predicted category belongs to a set of abnormal behavior categories predefined before deployment. If the minimum distance exceeds the set confidence threshold, the sample is judged as a low confidence prediction or an unknown anomaly.

[0012] In a preferred embodiment, the specific process of initializing a student model with the same structure as the teacher model on the terminal in the model update module, and storing the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavioral representation vectors and graph features is as follows: When deploying in the target user environment, the pre-trained and parameter-fixed general representation learning module and fine-grained discrimination module are used together as the teacher model. Then, a model with the exact same structure as the teacher model is created on the terminal device as the student model, and all parameters of the teacher model are copied to the student model for initialization. During operation, for each input sample consisting of a visual video stream and a point cloud sequence synchronously acquired by the data acquisition and alignment module, inference is performed synchronously through both the teacher model and the student model. This yields the first predicted probability distribution containing predicted class probabilities, the first general behavioral representation vector, and the first graph feature generated by the teacher model, and the second predicted probability distribution containing predicted class probabilities and the second general behavioral representation vector generated by the student model, respectively. The input sample is processed by analyzing the quantity and second graph features, and then filtering the recognition results with high confidence. The filtering conditions are as follows: First, the largest probability value in the first predicted probability distribution output by the teacher model is greater than a preset first high confidence threshold; Second, the Jensen-Shannon divergence value between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model is less than one minus a preset consistency threshold. For input samples that meet both filtering conditions, the corresponding visual video stream and point cloud sequence, the corresponding first general behavior representation vector and first graph features output by the teacher model, and the pseudo-labels given by the teacher model are collectively stored as a sample tuple in a fixed-capacity first-in-first-out circular memory, wherein the pseudo-labels are determined by the category index with the highest probability in the first predicted probability distribution output by the teacher model.

[0013] In a preferred embodiment, the specific process of periodically replaying the training of the student model using stored historical general behavioral representation vectors and graph features, and adjusting the model update strategy based on the degree of discrepancy between the student model and the teacher model in predicting the same input samples, is as follows: Periodically and uniformly sample a batch of historical sample tuples from the circular memory and mix them with the latest generated high-confidence sample tuples to form a training batch. Use the data from this training batch to update the parameters of the student model. Furthermore, during operation, for each new input sample, an online learning step is performed, specifically: calculating the difference in predictive cognitive uncertainty between the teacher model and the student model for the new input sample; based on this difference in predictive cognitive uncertainty, calculating an adaptive distillation weight value, which is obtained by subtracting a preset offset from the difference in predictive cognitive uncertainty, multiplying by a preset slope, and then performing a nonlinear mapping through a sigmoid function; subsequently, calculating the online distillation loss value for the new input sample, which is the product of the adaptive distillation weight value and the Kullback-Leibler divergence between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model; and optimizing the student model using a gradient descent algorithm to minimize the online distillation loss value. Simultaneously, the prediction divergence ratio between the student model and the teacher model within a sliding time window is continuously monitored. The prediction divergence ratio is the proportion of samples where the teacher model and the student model predict different categories for the same input sample within the sliding time window, relative to the total number of samples in that window. Based on different preset intervals in which the prediction divergence ratio falls, the replay training trigger frequency and the online learning rate of the student model are dynamically adjusted. The preset intervals are divided by a first divergence threshold and a second divergence threshold. When the prediction divergence ratio is consistently lower than the first divergence threshold, the replay training frequency and online learning rate are reduced. When the prediction divergence ratio is consistently higher than the second divergence threshold, the replay training frequency and online learning rate are increased, and a reinforcement training based on more historical sample tuples in the recurrent memory is triggered. When the prediction divergence ratio is between the first divergence threshold and the second divergence threshold, the replay training frequency and online learning rate are maintained at their default values. Furthermore, the L2 norm of the difference between the current parameters of the student model and the parameters of the teacher model used as the initialization benchmark is calculated periodically. If the L2 norm exceeds a preset difference threshold, a sparsity projection operation is performed on the parameters of the student model to adjust their parameter values ​​to a spatial range centered on the teacher model parameters used as the initialization benchmark and with the preset difference threshold as the radius.

[0014] The beneficial effects of this invention are as follows: By effectively fusing visual and radar information through cross-modal attention alignment, the limitations of a single sensor in complex home environments such as obstruction and low light are overcome, thereby improving the robustness of data acquisition. The joint training paradigm of mask reconstruction and contrastive learning is used to drive the model to learn general behavioral representations that are invariant to environmental interference, thereby enhancing the model's generalization ability. By utilizing spatiotemporal graph modeling and prototype contrast quantity learning, subtle differences in actions can be accurately captured and identified, achieving high-precision fine-grained abnormal behavior recognition. Through the teacher-student framework and personalized adaptive update mechanism, the model can continuously adapt to the user's unique behavioral patterns and alleviate concept drift, ultimately significantly reducing false alarms and false negatives, providing reliable, long-term, and personalized safety monitoring for the elderly. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0019] Example 1 This embodiment provides, for example Figure 1-2 The system for identifying and managing abnormal behaviors of the elderly based on a smart elderly care terminal device includes: a data acquisition and alignment module, a general representation learning module, a fine-grained discrimination module, and a model update module connected in sequence. Data Acquisition and Alignment Module: Upon startup, it synchronously acquires the visual video stream from the RGB camera and the point cloud sequence from the millimeter-wave radar, and aligns the visual video stream and the point cloud sequence in time and space. Then, using its internal cross-modal attention alignment unit, it performs mutual querying and feature weighted fusion on the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputs the first joint multimodal feature vector formed after fusion. The general representation learning module receives the first joint multimodal feature vector, arranges the first joint multimodal feature vector in time sequence to form a continuous first joint multimodal feature vector sequence, randomly masks the first joint multimodal feature vector sequence in the spatiotemporal dimension, reconstructs the masked part based on the unmasked part using its internal spatiotemporal Transformer encoder-decoder structure, and applies contrastive learning constraints to visual spatial features and radar spatiotemporal features from the same time period. After joint training, it outputs a general behavior representation vector. Fine-grained discrimination module: Receives a general behavior representation vector, first decodes the sequence of three-dimensional key points of the human body in consecutive frames from the general behavior representation vector, and constructs a spatiotemporal graph with joints as nodes and skeletal connections and temporal connections as edges based on the sequence of three-dimensional key points of the human body. Then, a temporal graph neural network is used to extract features from the spatiotemporal graph to obtain graph features. The extracted graph features are input into the prototype contrast ratio learning unit inside the unit. Fine-grained behavior classification is achieved by calculating the distance with the preset prototype vectors of various behaviors, and the current behavior is judged to be abnormal. Model update module: After deployment in a specific user environment, the trained general representation learning module and fine-grained discrimination module are used together as the teacher model. A student model with the same structure as the teacher model is deployed on the terminal for initialization. During operation, the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavior representation vectors and graph features are stored. The stored historical general behavior representation vectors and graph features are periodically used to replay and train the student model. At the same time, the model update strategy is adjusted according to the degree of difference between the predictions of the student model and the teacher model for the same input samples, so as to realize the personalized adaptation of the model.

[0020] In this embodiment, it is specifically necessary to explain the process in the data acquisition and alignment module of simultaneously acquiring the visual video stream acquired by the RGB camera and the point cloud sequence acquired by the millimeter-wave radar, and aligning the visual video stream and the point cloud sequence in time and space as follows: First, hardware synchronization signals ensure that the RGB camera and millimeter-wave radar start up under the same time reference, thereby obtaining a visual video stream consisting of T consecutive RGB images with strictly aligned timestamps, and a point cloud sequence consisting of T consecutive 3D spatial point sets containing distance, velocity, and angle information, where T is a preset time window length, for example, T ranging from 30 to 60, corresponding to an observation duration of 1 to 2 seconds, to capture a complete action segment. Then, based on pre-completed joint calibration parameters, the 3D spatial points in each frame of the point cloud sequence are projected to the 2D pixel positions in the corresponding timestamp RGB image coordinate system, achieving spatial alignment of visual and radar information in the 2D image plane. The joint calibration parameters include the rotation and translation matrix from the millimeter-wave radar coordinate system to the RGB camera coordinate system and the intrinsic parameter matrix of the RGB camera. Next, a pre-trained 2D convolutional neural network is used to process the time-aligned visual video stream frame by frame to extract visual spatial features. The visual spatial features are represented as a four-dimensional tensor, corresponding to the time length T, feature map height, feature map width, and number of visual feature channels, respectively. A pre-trained two-dimensional convolutional neural network (2D convolutional neural network) can be a variant of ResNet or VGG, removing its fully connected layers and retaining only convolutional and pooling layers to extract visual features with a spatial hierarchy. The number of visual feature channels is, for example, 512. Simultaneously, a three-dimensional point cloud feature encoding network is used to process the spatially aligned point cloud sequence frame by frame to extract radar spatiotemporal features. These features are represented as a three-dimensional tensor, corresponding to the time length T, the number of abstracted point sets, and the number of radar feature channels, respectively. The three-dimensional point cloud feature encoding network can use a PointNet or PointNet++ structure for feature learning from the unordered point cloud. The number of abstracted point sets is fixed at N through farthest point sampling or random sampling, for example, N=128, and the number of radar feature channels is, for example, 256. The specific process of performing mutual querying and weighted feature fusion on the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputting the first joint multimodal feature vector after fusion, is as follows: The first basis matrix is ​​defined as the two-dimensional feature matrix obtained by reshaping the visual spatial features in the dimensions of time, feature map height, and feature map width, with dimensions [T × feature map height × feature map width, number of visual feature channels]. The second basis matrix is ​​defined as the two-dimensional feature matrix obtained by reshaping the radar spatiotemporal features in the dimensions of time and the number of abstracted point sets, with dimensions [T × number of abstracted point sets, number of radar feature channels]. The first basis matrix is ​​projected into the first query matrix through the first learnable linear transformation matrix (whose weights are learned through backpropagation during model training). The second basis matrix is ​​projected into the first key matrix through the second learnable linear transformation matrix. The third learnable linear transformation matrix projects the second basis matrix into a first value matrix; the product between the first query matrix and the transpose of the first key matrix is ​​calculated to obtain the first similarity matrix; a normalized exponential function (i.e., the Softmax function) is applied to each row of the first similarity matrix to obtain the first attention weight matrix; the first attention weight matrix is ​​multiplied by the first value matrix to obtain the first weighted matrix; the first weighted matrix is ​​projected through the fourth learnable linear transformation matrix, and the projection result is added to the first basis matrix; the added result is then subjected to layer normalization to generate the enhanced visual feature matrix; The second basis matrix is ​​projected into the second query matrix using the fifth learnable linear transformation matrix; the first basis matrix is ​​projected into the second key matrix using the sixth learnable linear transformation matrix; the first basis matrix is ​​projected into the second value matrix using the seventh learnable linear transformation matrix; the product between the transpose of the second query matrix and the second key matrix is ​​calculated to obtain the second similarity matrix; a normalized exponential function is applied to each row of the second similarity matrix to obtain the second attention weight matrix; the second attention weight matrix is ​​multiplied by the second value matrix to obtain the second weighted matrix; the second weighted matrix is ​​projected using the eighth learnable linear transformation matrix, and the projection result is added to the second basis matrix. The added result is then subjected to layer normalization to generate the enhanced radar feature matrix. The enhanced visual feature matrix and the enhanced radar feature matrix are concatenated along the feature channel dimension to form a concatenated feature matrix. This concatenated feature matrix is ​​then input into a multilayer perceptron for nonlinear transformation and dimensionality reduction. The multilayer perceptron contains at least one fully connected layer and one nonlinear activation function for feature fusion and dimensionality reduction. The feature matrix output by the multilayer perceptron is the first joint multimodal feature vector, with a dimension of [T × feature map height × feature map width, D], where D is the unified feature dimension.

[0021] In this embodiment, it is specifically necessary to explain the process of randomly masking the sequence of first joint multimodal feature vectors arranged in time in the general representation learning module in the spatiotemporal dimension as follows: For the first joint multimodal feature vector sequence, a binary mask indicator matrix with the same length as the total number of first joint multimodal feature vectors in the first joint multimodal feature vector sequence is generated. The ratio of the first value used to indicate the mask position to the second value used to indicate the unmasked position in the binary mask indicator matrix is ​​set to a high ratio between 60% and 80%, and the ratio can be specifically set to 70% to achieve a balance between preserving sufficient reconstruction context and creating a sufficiently challenging training task. The mask position in the binary mask indicator matrix is ​​represented by the first value and the unmasked position is represented by the second value. Then, a learnable mask marker vector is created, and then a mask replacement operation is performed according to the binary mask indicator matrix, that is, all first joint multimodal feature vectors in the first joint multimodal feature vector sequence corresponding to the first value position are replaced with the learnable mask marker vector, while all first joint multimodal feature vectors corresponding to the second value position are retained to generate a masked feature sequence. The specific process of outputting a general behavior representation vector after joint training is as follows: The masked feature sequence is input into a spatiotemporal Transformer encoder, which consists of multiple sequentially connected multi-head self-attention layers and feedforward neural network layers. Each multi-head self-attention layer aggregates contextual information from all positions in the masked feature sequence through a self-attention mechanism. The encoder outputs a hidden representation sequence containing the global spatiotemporal context. This hidden representation sequence, along with positional encoding information, is input into a spatiotemporal Transformer decoder. The spatiotemporal Transformer decoder outputs a reconstructed feature sequence for the entire input. Based on the reconstructed feature sequence and the original first joint multimodal feature vector sequence, a reconstruction loss is calculated. The calculation process for this loss function involves obtaining the first joint multimodal feature vector in the original first joint multimodal feature vector sequence for each position indicated by the binary mask indication matrix, using it as the original feature vector, and then extracting the first joint multimodal feature vector from the reconstructed feature sequence. The first joint multimodal feature vector at the corresponding position is used as the reconstructed feature vector. The cosine similarity between the original feature vector and the reconstructed feature vector is calculated. The cosine similarity is divided by a first temperature parameter and the exponent is taken. Then, it is divided by the sum of the exponents of the cosine similarities between the reconstructed feature vector and the original feature vector of all samples in a batch, divided by the first temperature parameter. Then, the negative logarithm is taken to obtain the loss value at that position. The loss values ​​of all mask positions are summed to obtain the reconstruction loss. The first temperature parameter is used to adjust the sharpness of the similarity distribution and can be set to a scalar value between 0.07 and 0.1, for example, 0.07. At the same time, the visual spatial features and radar spatiotemporal features extracted by the data acquisition and alignment module at the same time period as the first joint multimodal feature vector sequence are obtained. The visual spatial features and radar spatiotemporal features are mapped to a shared contrast feature space through a visual modality projection network and a radar modality projection network, respectively, to obtain the visual projection features and radar projection features. Based on visual projection features and radar projection features, a cross-modal contrastive loss is calculated. This loss function is constructed based on the principle of noise contrastive estimation. The calculation process is as follows: Visual projection features and radar projection features from the same time period are considered as a positive sample pair. Combinations of visual projection features from different time periods and combinations of radar projection features from different time periods in the current positive sample pair are considered as negative sample pairs. First, the first cosine similarity between the two features in the positive sample pair is calculated, and then the second cosine similarity set between the visual projection feature and each radar projection feature that is considered a negative sample is calculated. The first cosine similarity is divided by a second temperature parameter, and the exponent is taken to obtain the first numerator. The first numerator is then divided by the first numerator and divided by all the second cosine similarities. The first loss term is obtained by summing the sums of the exponents of the second temperature parameter and the summation of the results. Symmetrically, the third cosine similarity between two features in a positive sample pair is calculated, and the fourth cosine similarity set between the radar projection feature and each visual projection feature that is a negative sample is calculated. The third cosine similarity is divided by the second temperature parameter and the exponent is taken to obtain the second numerator. The second numerator is divided by the sum of the sums of the second numerator and all the fourth cosine similarities, which are divided by the second temperature parameter and the summation of the results. The negative logarithm of the results is taken to obtain the second loss term. Finally, the first loss term and the second loss term are added together and divided by two to obtain the cross-modal contrast loss. The second temperature parameter is used to adjust the sharpness of the similarity distribution in the contrast task and is usually set to a scalar value between 0.05 and 0.1, for example, 0.07. The reconstruction loss and cross-modal contrast loss are weighted and summed with a preset balanced weight between zero and one to obtain a joint training loss. The balanced weight can be set to 0.7 to give the mask reconstruction task a relatively higher weight, ensuring that a robust spatiotemporal context representation is learned. The parameters of the spatiotemporal Transformer encoder-decoder structure, the visual modality projection network, and the radar modality projection network are optimized simultaneously through the backpropagation algorithm to minimize the joint training loss. After training, the complete, unmasked first joint multimodal feature vector sequence is input into the trained spatiotemporal Transformer encoder, and the encoder hidden representation sequence output by the encoder is used as the general behavior representation vector output.

[0022] In this embodiment, the specific process of decoding the sequence of three-dimensional human body key points from the general behavior representation vector in the fine-grained discrimination module is as follows: The system receives a general behavior representation vector from the general representation learning module. It performs a temporal pooling operation on this vector, transforming it into a fixed-length, time-ordered sequence of general behavior representation vectors. The temporal pooling operation employs either global average pooling or attention-weighted pooling. Attention-weighted pooling calculates the weights of each spatiotemporal feature using a learnable attention network and then performs a weighted sum, focusing more on information-rich regions. Finally, the pooled general behavior representation vectors at each time step are input into a lightweight keypoint decoding network. This lightweight keypoint decoding network consists of one or more fully connected layers. The number of output neurons in the last layer is equal to the preset number of human joints multiplied by three. It is used to map each high-dimensional general behavior representation vector to the three-dimensional spatial coordinates of all human joints in the corresponding frame, thereby outputting a sequence of human three-dimensional keypoints composed of multiple consecutive frames, each frame containing the three-dimensional coordinates of multiple joints. The sequence of human three-dimensional keypoints is represented in data as a three-dimensional tensor. The three dimensions of the three-dimensional tensor correspond to the number of time frames, the preset total number of human joints, and the number of three-dimensional coordinate components of each joint, respectively. The specific process of constructing a spatiotemporal graph with joints as nodes and skeletal and temporal connections as edges based on the sequence of 3D key points of the human body, and extracting graph features from this spatiotemporal graph using a temporal graph neural network is as follows: Each joint in each frame of the human 3D keypoint sequence is defined as a graph node. A node feature vector is constructed for each node. The elements of the node feature vector consist of the joint's 3D coordinates in the current frame, the first-order velocity formed by the difference between the joint's 3D coordinates in the current frame and the previous frame, and the second-order acceleration formed by the difference between the first-order velocity in the current frame and the first-order velocity in the previous frame. The graph edges are divided into two types: the first is spatial edges, which connect pairs of joints with physical connections within the same frame according to the standard human skeleton topology and following the connection relationships defined by the COCO format for 25 joints or the MPII format for 17 joints; the second is temporal edges, which connect pairs of joints representing the same joint in adjacent frames. Two nodes of a joint are connected by an edge, and a dynamic spatiotemporal graph is constructed in the above way. This dynamic spatiotemporal graph is input into a spatiotemporal graph neural network, which consists of alternating stacked spatial graph convolutional units and temporal convolutional units. The spatial graph convolutional units use graph attention network (GAT) or spectral graph convolution, and the temporal convolutional units use one-dimensional convolution. The spatial graph convolutional units aggregate the features of neighboring nodes along the spatial edges to learn the cooperative relationship between joints, and the temporal convolutional units aggregate the features of the same joint in adjacent frames along the temporal edges to learn the motion trajectory pattern. After multiple layers of information transmission, global graph pooling is performed on the node feature vectors of all nodes in the final output. By calculating the mean or maximum value of all node feature vectors, a fixed-dimensional graph feature is obtained to represent the entire behavior segment. The extracted graph features are input into the prototype contrast learning unit, and fine-grained behavior classification is achieved by calculating the distance with the preset prototype vectors of various behaviors. The specific process of determining whether the current behavior is abnormal is as follows: The prototype contrast ratio learning unit internally maintains a learnable parameter vector as the behavior prototype vector for each behavior category to be identified, including multiple normal behavior categories and multiple abnormal behavior categories. During the training phase of the prototype contrast ratio learning unit, for each graph feature of the input training sample, the distance between the graph feature and the behavior prototype vector of its own labeled category is calculated, and the distance between the graph feature and the behavior prototype vectors of all other categories is also calculated. The prototype contrast ratio learning unit achieves optimization by minimizing a joint loss function, which calculates the distance between the graph feature and the labeled category of each training sample. The distance between behavioral prototype vectors is used as the first term. Simultaneously, for each other category besides the labeled category, a learnable marginal parameter corresponding to that category is calculated, minus the distance between the graph feature and the behavioral prototype vector of that other category. The resulting difference is compared with zero, and the maximum value is taken. This maximum value is then multiplied by a hard sample mining weight associated with that other category. The sum of this weighted maximum value for all other categories is used as the second term. Finally, the first and second terms are added to obtain the loss value for the training sample. The total loss function is obtained by summing the loss values ​​of all training samples. This loss function drives the features of samples of the same type to cluster in space and forces features of different types to cluster in space. This feature separation focuses particularly on easily confused categories. The learnable marginal parameter is learned as a relatively smaller positive value for easily confused pairs of behavior categories in the training data. For easily confused pairs like "slowly sit down" and "slowly slip," the marginal parameter might be learned to be 0.3, while for significantly different category pairs, such as "walking" and "sitting," the marginal parameter might be learned to be 1.2. The weights for hard sample mining are determined by whether the distance between the graph feature and the corresponding behavior prototype vector of other categories is less than a preset threshold. Categories with a distance less than the preset threshold are given greater weight. The preset threshold can be set to 6% of the average distance between all category pairs. 10. The weight of mining difficult samples that meet the conditions is set to 2.0, otherwise it is 1.0. In the inference stage after the prototype contrast learning unit completes the training, for the training sample to be judged, the distance between its graph features and the behavior prototype vectors of all behavior categories is calculated, and it is classified into the behavior category corresponding to the behavior prototype vector with the smallest distance. Abnormal behavior is judged by judging whether the predicted category belongs to a set of abnormal behavior categories predefined before deployment. If the minimum distance exceeds the set confidence threshold, the confidence threshold can be set according to the distance distribution of the correctly classified samples on the validation set, taking its 95th percentile. Then the sample is judged as a low confidence prediction or an unknown abnormality.

[0023] In this embodiment, the specific process of initializing a student model with the same structure as the teacher model on the terminal in the model update module, and storing the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavioral representation vectors and graph features is as follows: When deploying in the target user environment, the pre-trained and fixed-parameter general representation learning module and fine-grained discrimination module are used together as the teacher model. Then, a model with the exact same structure as the teacher model is created on the terminal device as the student model, and all parameters of the teacher model are copied to the student model for initialization. During operation, for each input sample consisting of a visual video stream and a point cloud sequence synchronously acquired by the data acquisition and alignment module, inference is performed simultaneously through both the teacher and student models. This yields a first predicted probability distribution containing predicted class probabilities, a first general behavioral representation vector, and a first graph feature generated by the teacher model; and a second predicted probability distribution containing predicted class probabilities, a second general behavioral representation vector, and a second graph feature generated by the student model. Then, high-confidence recognition results are filtered. The filtering criteria are: first, the largest probability value in the first predicted probability distribution output by the teacher model is greater than a preset first high-confidence threshold, which can be set to 0.95, to ensure that the filtered samples... It has extremely high predictive certainty; secondly, the Jensen-Shannon divergence value between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model is less than one minus a preset consistency threshold. This consistency threshold can be set to 0.9, and the condition is that the Jensen-Shannon divergence value is less than 0.1, ensuring that the teacher and student models have a high degree of consistency in their cognition of the sample. The consistency threshold is used to define the upper limit of the maximum allowable difference measure when the first predicted probability distribution and the second predicted probability distribution are considered to be sufficiently consistent. For input samples that meet both of these screening conditions, their corresponding visual video stream and point cloud sequence, the first general behavior representation vector and the first graph feature output by the teacher model, and the pseudo-label given by the teacher model are combined into a sample tuple and stored in a fixed-capacity first-in-first-out circular memory. The capacity of the memory can be set to store 1,000 to 5,000 sample tuples to balance storage overhead and historical data coverage. The pseudo-label is determined by the category index with the highest probability in the first predicted probability distribution output by the teacher model. The specific process of periodically replaying and training the student model using stored historical general behavioral representation vectors and graph features, and adjusting the model update strategy based on the degree of discrepancy between the student model and the teacher model for the same input samples, is as follows: A batch of historical sample tuples is periodically and uniformly sampled from the circular memory and mixed with the latest high-confidence sample tuples to form a training batch. The student model's parameters are updated using this training batch data. The loss function for replay training is obtained by weighting and summing three parts according to preset weight coefficients. The first part is the knowledge distillation loss, calculated by the Kullback-Leibler divergence between the second predicted probability distribution output by the student model and the first predicted probability distribution output by the teacher model. The second part is the feature consistency loss, calculated by the mean squared error between the second general behavioral representation vector output by the student model and the first general behavioral representation vector output by the teacher model, and the mean squared error between the second graph feature output by the student model and the first graph feature output by the teacher model, and weighted and summed according to preset weights. The weight ratio of the mean squared error of the general behavioral representation vector to the graph feature can be set to 1:0.5. The third part is the memory feature contrast loss, calculated as follows: a sample is selected from the current training batch, and the second general behavioral representation vector output by the student model for that sample is compared with the first graph feature output by the teacher model. The behavioral representation vector is used as the query vector. The first general behavioral representation vector output by the teacher model for the same sample is used as the positive sample vector, and the first general behavioral representation vector output by the teacher model for the remaining samples in the current training batch is used as the negative sample vector set. The first cosine similarity between the query vector and the positive sample vector is calculated, and the second cosine similarity set between the query vector and each negative sample vector in the negative sample vector set is calculated. The first cosine similarity is divided by a preset temperature parameter and the exponent is taken to obtain the numerator. The temperature parameter can be set to 0.07. The numerator is divided by the sum of the sum of the numerator and all the second cosine similarities divided by the preset temperature parameter, the exponent is taken, and the negative logarithm of the result is taken to obtain the contrast loss value of the sample. The contrast loss values ​​of all selected samples are averaged to obtain the memory feature contrast loss. The above three calculation parts are weighted and summed according to preset weight coefficients. The weight coefficients of knowledge distillation loss, feature consistency loss, and memory feature contrast loss can be preset to 1.0, 0.5, and 0.2, respectively, to obtain the total replay loss. The student model is optimized by gradient descent algorithm to minimize the total replay loss. Furthermore, during operation, for each new input sample, an online learning step is performed, specifically: calculating the difference in predictive cognitive uncertainty between the teacher model and the student model for that new input sample. This difference in predictive cognitive uncertainty is obtained as follows: performing multiple forward inference calculations on both the teacher and student models, with 5 to 10 forward propagations for each model, randomly discarding some network connections during each calculation to obtain multiple predicted probability distributions. The entropy values ​​of the average distribution of these multiple predicted probability distributions for the teacher model and the student model are calculated separately. The absolute value of the difference between these two entropy values ​​is taken as the difference in predictive cognitive uncertainty. Based on this difference in predictive cognitive uncertainty... An adaptive distillation weight is calculated as follows: The predicted cognitive uncertainty difference is subtracted by a preset offset (e.g., 0.05), multiplied by a preset slope (e.g., 10), and then non-linearly mapped using a sigmoid function to obtain an adaptive distillation weight value between zero and one. Subsequently, the online distillation loss for the new input sample is calculated. This online distillation loss is the product of the adaptive distillation weight and the Kullback-Leibler divergence between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model. The student model is then optimized using gradient descent to minimize this online distillation loss. Simultaneously, the prediction divergence ratio between the student model and the teacher model within a sliding time window is continuously monitored. The prediction divergence ratio is the proportion of samples in the sliding time window (which can be set to the 100 most recent samples) where the teacher model and the student model predict different categories for the same input sample, relative to the total number of samples in that window. Based on different preset intervals for this prediction divergence ratio, the replay training trigger frequency and the online learning rate of the student model are dynamically adjusted. The preset intervals are divided by a first divergence threshold and a second divergence threshold. The first divergence threshold can be set to 0.1, and the second divergence threshold can be set to 0.3. When the prediction divergence ratio is consistently lower than the first divergence threshold, the replay training frequency and online learning rate are reduced. When the prediction divergence ratio is consistently higher than the second divergence threshold, the replay training frequency and online learning rate are increased, and a reinforcement training based on more historical sample tuples in the recurrent memory is triggered. When the prediction divergence ratio is between the first and second divergence thresholds, the replay training frequency and online learning rate remain at their default values. Furthermore, the L2 norm of the difference between the current parameters of the student model and the parameters of the teacher model used as the initialization benchmark is calculated periodically. If the L2 norm exceeds a preset difference threshold, which can be set as a percentage of the L2 norm of the teacher model parameters, such as 5%, then the parameters of the student model are subjected to a sparsity projection operation to adjust their parameter values ​​to a spatial range centered on the teacher model parameters of the initialization benchmark and with the preset difference threshold as the radius, so as to prevent the student model from deviating too much from the initial general knowledge.

[0024] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0030] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A system for identifying and managing abnormal behavior of the elderly based on smart elderly care terminal devices, characterized in that, Specifically, it includes: The data acquisition and alignment module, the general representation learning module, the fine-grained discrimination module, and the model update module are connected in sequence, among which; Data Acquisition and Alignment Module: Upon startup, it synchronously acquires the visual video stream from the RGB camera and the point cloud sequence from the millimeter-wave radar, and aligns the visual video stream and the point cloud sequence in time and space. Then, using its internal cross-modal attention alignment unit, it performs mutual querying and feature weighted fusion on the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputs the first joint multimodal feature vector formed after fusion. The general representation learning module receives the first joint multimodal feature vector, arranges the first joint multimodal feature vector in time sequence to form a continuous first joint multimodal feature vector sequence, randomly masks the first joint multimodal feature vector sequence in the spatiotemporal dimension, reconstructs the masked part based on the unmasked part using its internal spatiotemporal Transformer encoder-decoder structure, and applies contrastive learning constraints to visual spatial features and radar spatiotemporal features from the same time period. After joint training, it outputs a general behavior representation vector. Fine-grained discrimination module: Receives a general behavior representation vector, first decodes the sequence of three-dimensional key points of the human body in consecutive frames from the general behavior representation vector, and constructs a spatiotemporal graph with joints as nodes and skeletal connections and temporal connections as edges based on the sequence of three-dimensional key points of the human body. Then, a temporal graph neural network is used to extract features from the spatiotemporal graph to obtain graph features. The extracted graph features are input into the prototype contrast ratio learning unit inside the unit. Fine-grained behavior classification is achieved by calculating the distance with the preset prototype vectors of various behaviors, and the current behavior is judged to be abnormal. Model update module: After deployment in a specific user environment, the trained general representation learning module and fine-grained discrimination module are used together as the teacher model. A student model with the same structure as the teacher model is deployed on the terminal for initialization. During operation, the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavior representation vectors and graph features are stored. The stored historical general behavior representation vectors and graph features are periodically used to replay and train the student model. At the same time, the model update strategy is adjusted according to the degree of difference between the predictions of the student model and the teacher model for the same input samples, so as to realize the personalized adaptation of the model.

2. The system for identifying and treating abnormal behavior of the elderly based on a smart elderly care terminal device according to claim 1, characterized in that: The data acquisition and alignment module synchronously acquires the visual video stream from the RGB camera and the point cloud sequence from the millimeter-wave radar, and aligns the visual video stream and the point cloud sequence in time and space. The specific process is as follows: First, a time-stamp-aligned visual video stream consisting of T consecutive RGB images is obtained through hardware synchronization signals, along with a point cloud sequence consisting of T consecutive 3D spatial point sets containing distance, velocity, and angle information. Then, based on pre-completion joint calibration parameters, the 3D spatial points in each frame of the point cloud sequence are projected onto the 2D pixel positions in the corresponding timestamp's RGB image coordinate system, achieving spatial alignment of visual and radar information on the 2D image plane. Next, a pre-trained 2D convolutional neural network is used to process the time-aligned visual video stream frame by frame to extract visual spatial features. Simultaneously, a 3D point cloud feature encoding network is used to process the spatially aligned point cloud sequence frame by frame to extract radar spatiotemporal features.

3. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 2, characterized in that: The specific process of performing mutual querying and weighted feature fusion of the visual spatial features extracted from the visual video stream and the radar spatiotemporal features extracted from the point cloud sequence, and outputting the first joint multimodal feature vector after fusion, is as follows: The two-dimensional feature matrix obtained by reshaping the visual spatial features in the dimensions of time, feature map height, and feature map width is defined as the first basis matrix; the two-dimensional feature matrix obtained by reshaping the radar spatiotemporal features in the dimensions of time and the number of abstracted point sets is defined as the second basis matrix; the first basis matrix is ​​projected into a first query matrix through a first learnable linear transformation matrix; the second basis matrix is ​​projected into a first key matrix through a second learnable linear transformation matrix; the second basis matrix is ​​projected into a first value matrix through a third learnable linear transformation matrix; the product between the first query matrix and the transpose of the first key matrix is ​​calculated to obtain the first similarity matrix; Apply the normalized exponential function to each row of the first similarity matrix to obtain the first attention weight matrix; multiply the first attention weight matrix with the first value matrix to obtain the first weighted value matrix; The first weighted matrix is ​​projected through the fourth learnable linear transformation matrix, and the projection result is added to the first basis matrix. The added result is then subjected to layer normalization to generate the enhanced visual feature matrix. The second basis matrix is ​​projected into the second query matrix using the fifth learnable linear transformation matrix; the first basis matrix is ​​projected into the second key matrix using the sixth learnable linear transformation matrix; and the first basis matrix is ​​projected into the second value matrix using the seventh learnable linear transformation matrix. Calculate the product between the second query matrix and the transpose of the second key matrix to obtain the second similarity matrix; Apply the normalized exponential function to each row of the second similarity matrix to obtain the second attention weight matrix; multiply the second attention weight matrix with the second value matrix to obtain the second weighted value matrix; The second weighted matrix is ​​projected through the eighth learnable linear transformation matrix, and the projection result is added to the second basis matrix. The added result is then subjected to layer normalization to generate the enhanced radar feature matrix. The enhanced visual feature matrix and the enhanced radar feature matrix are concatenated along the feature channel dimension to form a concatenated feature matrix. This concatenated feature matrix is ​​then input into a multilayer perceptron for nonlinear transformation and dimensionality reduction. The feature matrix output by the multilayer perceptron is the first joint multimodal feature vector.

4. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 3, characterized in that: In the general representation learning module, the specific process of randomly masking the sequence of first joint multimodal feature vectors arranged in time sequence in the spatiotemporal dimension is as follows: For the first joint multimodal feature vector sequence, a binary mask indicator matrix with the same length as the total number of first joint multimodal feature vectors in the sequence is generated. The masked positions in the binary mask indicator matrix are represented by a first value, and the unmasked positions are represented by a second value. Then, a learnable mask marker vector is created. Then, a mask replacement operation is performed according to the binary mask indicator matrix, that is, all first joint multimodal feature vectors in the first joint multimodal feature vector sequence corresponding to the first value are replaced with the learnable mask marker vector, while all first joint multimodal feature vectors corresponding to the second value are retained, thus generating a masked feature sequence.

5. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 4, characterized in that: The specific process of outputting a general behavioral representation vector after joint training is as follows: The masked feature sequence is input into a spatiotemporal Transformer encoder, which consists of multiple sequentially connected multi-head self-attention layers and feedforward neural network layers. The encoder outputs a sequence of encoder hidden representations containing the global spatiotemporal context. This sequence of encoder hidden representations and positional encoding information are input into a spatiotemporal Transformer decoder. The spatiotemporal Transformer decoder outputs a reconstructed feature sequence for the entire input. Based on the reconstructed feature sequence and the original first joint multimodal feature vector sequence, a reconstruction loss is calculated. At the same time, visual spatial features and radar spatiotemporal features extracted by the data acquisition and alignment module at the same time period as the first joint multimodal feature vector sequence are obtained. The visual spatial features and radar spatiotemporal features are mapped to a shared contrast feature space through a visual modality projection network and a radar modality projection network, respectively, to obtain visual projection features and radar projection features. Based on visual projection features and radar projection features, a cross-modal contrast loss is calculated. This loss function is constructed based on the principle of noise contrast estimation. The reconstruction loss and cross-modal contrast loss are weighted and summed with a pre-defined balanced weight between zero and one to obtain a joint training loss. The parameters of the spatiotemporal Transformer encoder-decoder structure, the visual modality projection network, and the radar modality projection network are simultaneously optimized through the backpropagation algorithm to minimize the joint training loss. After training, the complete, unmasked first joint multimodal feature vector sequence is input into the trained spatiotemporal Transformer encoder, and the encoder hidden representation sequence output by the encoder is used as the general behavior representation vector output.

6. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 5, characterized in that: In the fine-grained discrimination module, the specific process of decoding the sequence of three-dimensional human body key points from the general behavior representation vector is as follows: The system receives a general behavior representation vector from the general representation learning module, performs a temporal pooling operation on the general behavior representation vector, and converts it into a fixed-length, time-ordered sequence of general behavior representation vectors. Then, the general behavior representation vectors obtained at each time step after pooling are input into a lightweight keypoint decoding network. This lightweight keypoint decoding network consists of one or more fully connected layers. The number of output neurons in its last layer is equal to the preset number of human joints multiplied by three. This network is used to map each high-dimensional general behavior representation vector to the three-dimensional spatial coordinates of all human joints in the corresponding frame, thereby outputting a sequence of human three-dimensional keypoints composed of multiple consecutive frames, each frame containing the three-dimensional coordinates of multiple joints.

7. The system for identifying and treating abnormal behavior of the elderly based on a smart elderly care terminal device according to claim 6, characterized in that: The specific process of constructing a spatiotemporal graph based on the human body's three-dimensional keypoint sequence, with joints as nodes and skeletal and temporal connections as edges, and then using a temporal graph neural network to extract features from this spatiotemporal graph to obtain graph features is as follows: Each joint in each frame of the human 3D keypoint sequence is defined as a graph node. A node feature vector is constructed for each node. The elements of the node feature vector consist of the 3D coordinates of the joint in the current frame, the first-order velocity formed by the difference between the 3D coordinates of the joint in the current frame and the previous frame, and the second-order acceleration formed by the difference between the first-order velocity of the joint in the current frame and the first-order velocity of the joint in the previous frame. The edges of the graph are divided into two types: the first is spatial edges, which connect pairs of joints with physical connections in the same frame according to the standard human skeleton topology; the second is temporal edges, which connect pairs of joints representing the same joint in two adjacent frames. Two nodes at a joint are connected by an edge, and a dynamic spatiotemporal graph is constructed in this way. This dynamic spatiotemporal graph is then input into a spatiotemporal graph neural network, which consists of alternating stacked spatial graph convolutional units and temporal convolutional units. The spatial graph convolutional units aggregate the features of neighboring nodes along the spatial edges to learn the cooperative relationship between joints, while the temporal convolutional units aggregate the features of the same joint in adjacent frames along the temporal edges to learn the motion trajectory pattern. After multiple layers of information transmission, global graph pooling is performed on the node feature vectors of all nodes in the final output. By calculating the mean or maximum value of all node feature vectors, a fixed-dimensional graph feature is obtained to represent the entire behavior segment.

8. The system for identifying and managing abnormal behavior of the elderly based on a smart elderly care terminal device according to claim 7, characterized in that: The extracted graph features are input into the prototype contrast learning unit, and fine-grained behavior classification is achieved by calculating the distance with the preset prototype vectors of various behaviors. The specific process of determining whether the current behavior is abnormal is as follows: The prototype contrast ratio learning unit maintains a learnable parameter vector as the behavior prototype vector for each behavior category to be identified, including multiple normal behavior categories and multiple abnormal behavior categories. During the training phase of the prototype contrast ratio learning unit, for each graph feature of the input training sample, the distance between the graph feature and the behavior prototype vector of its own labeled category is calculated, and the distance between the graph feature and the behavior prototype vectors of all other categories is also calculated. The prototype contrast ratio learning unit optimizes by minimizing a joint loss function. For each graph feature of a training sample, the first term of the joint loss function is the distance between the graph feature and the behavior prototype vector corresponding to the labeled category of that training sample. Simultaneously, for each other category, a learnable marginal parameter corresponding to that category is calculated, minus the distance between the graph feature and the behavior prototype vector of that other category. The distance between the behavior prototype vectors of each category is calculated by comparing the difference with zero and taking the maximum value. This maximum value is then multiplied by a hard sample mining weight related to other categories. The sum of this weighted maximum value for all other categories is used as the second term. Finally, the first term is added to the second term to obtain the loss value of the training sample. The total loss function is obtained by summing the loss values ​​of all training samples. In the inference stage after the prototype contrast ratio learning unit completes training, for the training sample to be judged, the distance between its graph features and the behavior prototype vectors of all behavior categories is calculated. The sample is then classified into the behavior category corresponding to the behavior prototype vector with the smallest distance. Abnormal behavior discrimination is completed by determining whether the predicted category belongs to a predefined set of abnormal behavior categories before deployment. If the minimum distance exceeds the set confidence threshold, the sample is judged as a low-confidence prediction or an unknown anomaly.

9. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 8, characterized in that: In the model update module, the specific process of initializing a student model with the same structure as the teacher model on the terminal, and storing the high-confidence recognition results output by the fine-grained discrimination module and their corresponding general behavioral representation vectors and graph features is as follows: When deploying in the target user environment, the pre-trained and fixed-parameter general representation learning module and fine-grained discrimination module are used together as the teacher model. Then, a model with the exact same structure as the teacher model is created on the terminal device as the student model, and all parameters of the teacher model are copied to the student model for initialization. During operation, for each input sample consisting of a visual video stream and a point cloud sequence synchronously acquired by the data acquisition and alignment module, inference is performed synchronously through both the teacher and student models. This yields the first predicted probability distribution containing predicted class probabilities, the first general behavioral representation vector, and the first graph feature generated by the teacher model, and the second predicted probability distribution containing predicted class probabilities, the second general behavioral representation vector, and the second graph feature generated by the student model. The graph features are then used to filter high-confidence recognition results. The filtering conditions are as follows: First, the largest probability value in the first predicted probability distribution output by the teacher model is greater than a preset first high-confidence threshold; Second, the Jensen-Shannon divergence value between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model is less than one minus a preset consistency threshold. For input samples that meet both filtering conditions, their corresponding visual video stream and point cloud sequence, the corresponding first general behavior representation vector and first graph features output by the teacher model, and the pseudo-labels given by the teacher model are combined into a sample tuple and stored in a fixed-capacity first-in-first-out circular memory. The pseudo-labels are determined by the category index with the highest probability in the first predicted probability distribution output by the teacher model.

10. The elderly abnormal behavior identification and system based on smart elderly care terminal equipment according to claim 9, characterized in that: The specific process of periodically replaying and training the student model using stored historical general behavioral representation vectors and graph features, and adjusting the model update strategy based on the degree of discrepancy between the student model and the teacher model for the same input samples, is as follows: Periodically sample a batch of historical sample tuples from the circular memory and mix them with the latest high-confidence sample tuples to form a training batch. Use the data from this training batch to update the parameters of the student model. Furthermore, during operation, for each new input sample, an online learning step is performed, specifically: calculating the difference in predictive cognitive uncertainty between the teacher model and the student model for the new input sample; based on this difference in predictive cognitive uncertainty, calculating an adaptive distillation weight value, which is obtained by subtracting a preset offset from the difference in predictive cognitive uncertainty, multiplying by a preset slope, and then performing a nonlinear mapping through a sigmoid function; subsequently, calculating the online distillation loss value for the new input sample, which is the product of the adaptive distillation weight value and the Kullback-Leibler divergence between the first predicted probability distribution output by the teacher model and the second predicted probability distribution output by the student model; and optimizing the student model using a gradient descent algorithm to minimize this online distillation loss value. Simultaneously, the prediction divergence ratio between the student model and the teacher model within the sliding time window is continuously monitored. The prediction divergence ratio is the proportion of samples where the teacher model and the student model predict different categories for the same input sample within the sliding time window, relative to the total number of samples in that window. Based on different preset intervals in which the prediction divergence ratio falls, the replay training trigger frequency and the online learning rate of the student model are dynamically adjusted. The preset intervals are divided by a first divergence threshold and a second divergence threshold. When the prediction divergence ratio is consistently lower than the first divergence threshold, the replay training frequency and online learning rate are reduced; when the prediction divergence ratio is consistently higher than the second divergence threshold, the replay training frequency and online learning rate are increased, and a reinforcement training based on more historical sample tuples in the recurrent memory is triggered; when the prediction divergence ratio is between the first and second divergence thresholds, the replay training frequency and online learning rate remain at their default values. Furthermore, the L2 norm of the difference between the current parameters of the student model and the parameters of the teacher model used as the initialization benchmark is calculated periodically. If the L2 norm exceeds a preset difference threshold, the parameters of the student model are subjected to a sparsification projection operation to adjust their parameter values ​​to a spatial range centered on the teacher model parameters used as the initialization benchmark and with the preset difference threshold as the radius.

Images