Video monitoring abnormal behavior system for smart home for the aged based on visual intelligence

Abstract

The application relates to the field of computer vision and discloses a video monitoring abnormal behavior system for smart home and old-age care based on visual intelligence, which comprises the following modules: a data acquisition and cleaning module which acquires and cleanses skeleton sequence data; a joint grouping module which divides joints into three groups of upper limbs, lower limbs and trunks according to kinematics priori; an adjacency matrix construction module which constructs a subset adjacency matrix for each group; a spatial feature fusion module which uses a learnable weight prediction network to output group weights, weights the sum of group features and generates a spatial fine-grained feature map; a gate weight generation module which inputs a light network with the average of motion acceleration and outputs multi-scale convolution kernel selection weights; a dynamic convolution generation and time series convolution module which generates a dynamic kernel by weighting and combining convolution kernels according to the selection weights, performs time series convolution on the spatial fine-grained feature map and outputs a space-time joint feature map; and a behavior classification module which inputs the space-time joint feature map into a classifier and outputs a behavior label. The application realizes local fine-grained modeling and dynamic adaptation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence. Background Technology

[0002] With the accelerating aging of the population, the safety of elderly people living alone is becoming increasingly prominent. Video-based automatic abnormal behavior detection technology plays a crucial role in elderly care. Compared to depth image data and RGB video data, skeleton data is more robust to factors such as camera viewpoint changes, background occlusion, and lighting conditions. Therefore, skeleton-based behavior recognition research can be widely applied in fields such as human-computer interaction and smart healthcare. Human skeleton-based action recognition is an important branch of computer vision intelligence. Skeleton-based human behavior recognition and abnormal gait analysis technologies can provide technical support for elderly care services, movement assessment, and medical diagnosis.

[0003] Models based on Graph Convolutional Networks (GCNs) process skeleton sequences into a skeleton topological graph as input, thereby modeling the dependencies between skeleton nodes. Among existing technologies, researchers have proposed a behavior recognition model using Selectable Multi-Scale Graph Convolutional Networks (SMS-GCNs). This model, based on the spatial features extracted by multi-scale graph convolutional modules, constructs selectable multi-scale temporal convolutional modules, achieving excellent recognition accuracy on large skeleton datasets.

[0004] However, the SMS-GCN model still has the following shortcomings: First, existing skeleton behavior recognition methods neglect the multi-scale dependencies of skeleton joints. While SMS-GCN partially addresses this issue by constructing a multi-scale channel topology to refine the adjacency matrix, its spatial graph topology construction is still based on globally unified joint connections, lacking refined modeling of local joint functional groupings. In practical applications, the SMS-GCN model exhibits low accuracy when recognizing specific actions such as "eating" and "pointing," which only involve local joint changes, thus limiting the model's application scenarios.

[0005] Second, the selectable multi-scale temporal convolution module of SMS-GCN performs temporal modeling based on a fixed large kernel network structure. Its temporal convolution kernel selection strategy lacks the ability to dynamically adapt to changes in action speed and cannot adaptively adjust the temporal receptive field according to the dynamic characteristics of the current action.

[0006] To address this, the present invention proposes a video monitoring system for abnormal behavior in smart home elderly care based on visual intelligence, aiming to solve the technical problems in the prior art, namely, the lack of refined local joint grouping in spatial topology modeling and the lack of adaptive dynamic selection of action in temporal convolution kernels. Summary of the Invention

[0007] One objective of this invention is to propose a video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence. This invention fully integrates graph topology adaptive reconstruction technology with functional grouping of local joints, dynamic gating multi-scale temporal convolution technology with motion acceleration perception, learnable cross-group feature fusion weight prediction technology, and real-time weight generation technology of lightweight gating prediction network. It describes in detail the entire process from human skeleton sequence data acquisition, data cleaning and null value imputation, functional joint subset partitioning, subset adjacency matrix construction, learnable fusion weight prediction, spatial fine-grained feature map generation, motion acceleration mean calculation, temporal convolution kernel selection and weight generation, dynamic temporal convolution kernel weighted combination to behavior classification label output. It has the advantages of high local action recognition accuracy, strong dynamic adaptability of action speed, strong ability to distinguish similar actions, and low edge deployment and maintenance costs.

[0008] According to an embodiment of the present invention, a video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence includes: The data acquisition and cleaning module acquires human skeleton sequence data and performs null imputation and noise filtering on the data. The SMS-GCN joint grouping module divides the joints in the cleaned human skeleton sequence data into three functional joint subsets: upper limb movement group, lower limb movement group, and trunk movement group, according to human kinematic priors. The SMS-GCN adjacency matrix construction module constructs a subset adjacency matrix for each functional joint subset. The SMS-GCN spatial feature fusion module inputs the adjacency matrices of each subset into a learnable weight prediction network, which outputs the fusion weights corresponding to each subset. The joint features within each subset are weighted and summed to generate a refined spatial feature map. The SMS-GCN gating weight generation module inputs the spatially refined feature map into a lightweight gating prediction network. The lightweight gating prediction network takes the mean of motion acceleration as input and outputs selection weights for several temporal convolutional kernels. The SMS-GCN dynamic convolutional kernel generation and temporal convolution module weights several temporal convolutional kernels based on the selection weights to generate dynamic temporal convolutional kernels. The dynamic temporal convolutional kernels are then used to perform temporal convolution on the spatially refined feature map to output a spatiotemporal joint feature map. The behavior classification module inputs the spatiotemporal joint feature map into a classifier and outputs behavior category labels.

[0009] Optionally, the skeleton sequence data specifically includes: The joint identifiers of each joint in the human skeleton, the two-dimensional coordinates of each joint in the image frame, the depth values ​​of each joint, the frame number corresponding to each joint, and the detection confidence scores of each joint.

[0010] Optionally, the division into functional joint subsets specifically includes: the upper limb movement group includes the left wrist joint, left elbow joint, left shoulder joint, right wrist joint, right elbow joint and right shoulder joint; the lower limb movement group includes the left ankle joint, left knee joint, left hip joint, right ankle joint, right knee joint and right hip joint; and the trunk movement group includes the neck joint, chest joint and hip center joint.

[0011] Optionally, the construction of the subset adjacency matrix specifically includes: for each functional joint subset, calculating the average distance between any two joint coordinates within the subset within a time window, using the average distance as the initial value of the adjacency matrix element corresponding to the two joints to obtain an initial adjacency matrix, and inputting the initial adjacency matrix into the graph convolutional layer of SMS-GCN for end-to-end learning, with the graph convolutional layer of SMS-GCN updating the value of each element in the adjacency matrix during training.

[0012] Optionally, the learnable weight prediction network specifically includes: It consists of an input flattening layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected sequentially. The input flattening layer expands the matrix elements of the adjacency matrix of each subset into a one-dimensional feature vector in row-major order. The first fully connected layer maps the one-dimensional feature vector to an intermediate feature vector, the dimension of which is a predefined dimension. The batch normalization layer normalizes the mean and variance of each dimension of the intermediate feature vector, and outputs the normalized intermediate feature vector; The modified linear unit activation layer performs a non-linear mapping on the normalized intermediate feature vector, setting the negative values ​​in the normalized intermediate feature vector to zero and leaving the positive values ​​unchanged, and outputs the activation feature vector; The second fully connected layer maps the activation feature vectors to output feature vectors, and the dimension of the output feature vectors is the same as the number of functional joint subsets; The softmax normalization layer performs exponential normalization on each component of the output feature vector to obtain the fusion weights of each functional joint subset. Each fusion weight is a positive number between 0 and 1, and the sum of all fusion weights is equal to 1.

[0013] Optionally, the generation of the spatially refined feature map specifically includes: Within each subset, the joint features are the feature vectors corresponding to each joint output from the previous layer of the SMS-GCN network. The feature vectors of all joints in the upper limb movement group are multiplied by the first fusion weight to obtain the first weighted joint feature tensor. The feature vectors of all joints in the lower limb movement group are multiplied by the second fusion weight to obtain the second weighted joint feature tensor. The feature vectors of all joints in the trunk movement group are multiplied by the third fusion weight to obtain the third weighted joint feature tensor. The first, second, and third weighted joint feature tensors are concatenated along the joint dimension to form a concatenated joint feature tensor. The concatenated joint feature tensor is then summed element-wise along the joint dimension to obtain a spatially refined feature map.

[0014] Optionally, the selection weights of the output of the gated weight generation module of the SMS-GCN specifically include: The mean motion acceleration is the average displacement change rate of each joint between adjacent frames in the spatially refined feature map. The mean motion acceleration values ​​of the three subsets—upper limb action group, lower limb action group, and trunk action group—are concatenated to form a three-dimensional acceleration feature vector. This three-dimensional acceleration feature vector is then input into a lightweight gating prediction network. The lightweight gating prediction network consists of one fully connected layer and one Softmax normalization layer. The fully connected layer maps the three-dimensional acceleration feature vector to a four-dimensional output vector, and the Softmax normalization layer normalizes each component of the four-dimensional output vector into four selection weights.

[0015] Optionally, the step of weighting and combining several temporal convolutional kernels based on selection weights to generate dynamic temporal convolutional kernels specifically includes: The number of selected weights is the same as the number of temporal convolution kernels, and each selected weight corresponds one-to-one with each temporal convolution kernel; Several temporal convolution kernels have different receptive field ranges along the time dimension; Multiply the kernel parameters of each temporal convolution kernel by the corresponding selection weights to obtain each weighted convolution kernel; then add each weighted convolution kernel element by element in the parameter space to generate a dynamic temporal convolution kernel.

[0016] Optionally, the output spatiotemporal joint feature map specifically includes: The spatially refined feature map is a three-dimensional tensor, with the three dimensions being the temporal frame dimension, the joint dimension, and the channel dimension. A dynamic temporal convolution kernel performs a sliding window convolution operation along the temporal frame dimension on the spatially refined feature map. The window size of the dynamic temporal convolution kernel is equal to the receptive field of the kernel. At each sliding window position, the kernel parameters of the dynamic temporal convolution kernel are multiplied element-wise with the feature tensor within the corresponding time window in the spatially refined feature map, and then summed to obtain an output value. After the sliding window traverses all time frames, a spatiotemporal joint feature map is obtained. The temporal frame dimension of the spatiotemporal joint feature map is smaller than that of the spatially refined feature map, while the joint and channel dimensions of the spatiotemporal joint feature map are the same as those of the spatially refined feature map.

[0017] Optionally, the output behavior category label specifically includes: The classifier consists of a global average pooling layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected in sequence. The global average pooling layer performs global average pooling operations on the spatiotemporal joint feature map in the joint dimension and time frame dimension to obtain a one-dimensional global feature vector. The dimension of the one-dimensional global feature vector is the same as the number of channels of the spatiotemporal joint feature map. The first fully connected layer maps the one-dimensional global feature vector to an intermediate classification feature vector, and the dimension of the intermediate classification feature vector is the predefined classification dimension; The batch normalization layer normalizes the mean and variance of each dimension of the intermediate classification feature vector and outputs the normalized intermediate classification feature vector. The modified linear unit activation layer performs a non-linear mapping on the normalized intermediate classification feature vector, setting the negative values ​​in the normalized intermediate classification feature vector to zero, while keeping the positive values ​​unchanged, and outputting the activated classification feature vector. The second fully connected layer maps the activation classification feature vector to the output classification feature vector, and the dimension of the output classification feature vector is equal to the total number of predefined behavior categories. The Softmax normalization layer performs exponential normalization on each component of the output classification feature vector to obtain the probability distribution of each behavior category, and outputs the label corresponding to the behavior category with the highest probability value as the behavior category label.

[0018] The beneficial effects of this invention are: (1) High accuracy in local action recognition. This invention divides the skeletal joints into three functional joint subsets—upper limb action group, lower limb action group, and trunk action group—according to human kinematic priors. Adjacency matrices are independently constructed for each subset and adaptively fused through a learnable weight prediction network. This solves the problem that the existing SMS-GCN model ignores the fine-grained dependencies of local joints due to global unified topology modeling, and significantly improves the recognition accuracy of local joint actions such as "eating", "pointing", and "waving".

[0019] (2) Strong dynamic adaptability of movement speed. This invention calculates the average motion acceleration of each joint as a dynamic representation of movement speed, inputs it into a lightweight gating prediction network to generate selection weights for temporal convolution kernels in real time, and dynamically weights and combines temporal convolution kernels with different receptive fields based on these weights, so that the receptive field of temporal modeling can be adaptively adjusted according to the speed of the current movement, while taking into account the instantaneous feature capture of rapid falling movements and the long-range dependency modeling of slow sitting movements.

[0020] (3) Strong ability to distinguish similar actions. This invention uses the synergistic effect of local joint functional grouping fusion in the spatial dimension and dynamic convolution of motion acceleration perception in the temporal dimension to simultaneously perform refined modeling of behavioral features in two orthogonal dimensions of space and time. It can effectively distinguish similar actions such as "sitting and dozing off" from "fainting and stillness", "slowly getting up" from "staggering and falling down", and reduce the false alarm rate.

[0021] (4) Significant advantages in edge deployment and privacy protection. This invention performs behavior recognition based on skeleton sequence data. The skeleton data only contains joint coordinate information and does not contain any original image or video data, thus avoiding the risk of privacy leakage from the data source. At the same time, SMS-GCN is a lightweight graph convolutional network with only 2.00M parameters, which is suitable for deployment on smart home edge computing devices. There is no need to upload monitoring data to the cloud, which meets the dual requirements of real-time performance and privacy protection in home-based elderly care scenarios. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the video monitoring system for abnormal behavior in smart home elderly care based on visual intelligence, as proposed in this invention. Figure 2 This is a schematic diagram of the graph topology adaptive reconstruction process of the video surveillance abnormal behavior system for smart home elderly care based on visual intelligence proposed in this invention.

[0023] Figure 3This is a schematic diagram of the multi-scale temporal convolution process of the video surveillance abnormal behavior system for smart home elderly care based on visual intelligence proposed in this invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0025] refer to Figures 1-3 A video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence includes: The data acquisition and cleaning module acquires human skeleton sequence data and performs null imputation and noise filtering on the data. The SMS-GCN joint grouping module divides the joints in the cleaned human skeleton sequence data into three functional joint subsets: upper limb movement group, lower limb movement group, and trunk movement group, according to human kinematic priors. The SMS-GCN adjacency matrix construction module constructs a subset adjacency matrix for each functional joint subset. The SMS-GCN spatial feature fusion module inputs the adjacency matrices of each subset into a learnable weight prediction network, which outputs the fusion weights corresponding to each subset. The joint features within each subset are weighted and summed to generate a refined spatial feature map. The SMS-GCN gating weight generation module inputs the spatially refined feature map into a lightweight gating prediction network. The lightweight gating prediction network takes the mean of motion acceleration as input and outputs selection weights for several temporal convolutional kernels. The SMS-GCN dynamic convolutional kernel generation and temporal convolution module weights several temporal convolutional kernels based on the selection weights to generate dynamic temporal convolutional kernels. The dynamic temporal convolutional kernels are then used to perform temporal convolution on the spatially refined feature map to output a spatiotemporal joint feature map. The behavior classification module inputs the spatiotemporal joint feature map into a classifier and outputs behavior category labels.

[0026] In this embodiment, the skeleton sequence data specifically includes: The joint identifiers of each joint in the human skeleton, the two-dimensional coordinates of each joint in the image frame, the depth values ​​of each joint, the frame number corresponding to each joint, and the detection confidence scores of each joint.

[0027] Specifically, the joint detection confidence score includes: The joint detection confidence score is a measure of the confidence of the pose estimation algorithm for the prediction results of each joint coordinate. The confidence score is a floating-point number between 0 and 1. The higher the confidence score, the higher the confidence of the joint coordinate prediction result.

[0028] The specific depth values ​​for each joint are as follows: The depth coordinates of each joint in the human skeleton in three-dimensional space, that is, the front-to-back distance (Z-axis coordinate) of the joint relative to the camera plane.

[0029] In this embodiment, the functional joint subset specifically includes: the upper limb movement group includes the left wrist joint, left elbow joint, left shoulder joint, right wrist joint, right elbow joint and right shoulder joint; the lower limb movement group includes the left ankle joint, left knee joint, left hip joint, right ankle joint, right knee joint and right hip joint; and the trunk movement group includes the neck joint, chest joint and hip center joint.

[0030] In this embodiment, constructing the subset adjacency matrix specifically includes: For each functional joint subset, calculating the average distance between any two joint coordinates within the subset within a time window, using the average distance as the initial value of the adjacency matrix element corresponding to the two joints to obtain the initial adjacency matrix, and inputting the initial adjacency matrix into the graph convolutional layer of SMS-GCN for end-to-end learning. The graph convolutional layer of SMS-GCN updates the value of each element in the adjacency matrix during training.

[0031] Specifically, the graph convolutional layers of SMS-GCN include: The convolutional layer for channel topology refinement expands the single shared adjacency matrix in standard graph convolution into a set of multi-scale adjacency matrices in the channel dimension, enabling independent modeling of topological relationships for each channel feature; The input to the graph convolutional layer is the skeleton feature map output from the previous layer. This feature map contains three dimensions: the number of channels, the number of time frames, and the number of joints. Graph convolutional layers perform channel topology refinement graph convolution operations. Standard graph convolution uses the same adjacency matrix for all input channels, while channel topology refinement graph convolution builds an adjacency matrix independently for each channel or group of channels. By processing each channel independently, it models the input features in the feature space, achieving more refined topological relationship modeling. Graph convolutional layers construct two types of adjacency matrices: pairwise adjacency matrices and multi-joint adjacency matrices. Pairwise adjacency matrices are used to model the local physical connections between adjacent joints, while multi-joint adjacency matrices are used to model the long-range dependencies between non-adjacent joints. The two types of adjacency matrices are fused to generate a multi-scale channel topology-refined adjacency matrix. Graph convolutional layers capture joint dependencies at different scales within the same layer through a multi-scale adjacency matrix set. The graph convolutional network consists of multiple layers, with the Tanh function used as the activation function between each layer and a preset dropout rate set. After graph convolution operations and activation function processing, the graph convolutional layer outputs an updated skeleton feature map. The output feature map has three dimensions: the number of output channels of the graph convolutional layer, the number of time frames, and the number of joints.

[0032] The preset discard rate follows these guidelines: The commonly used starting range is between 0.1 and 0.5, with 0.5 often used as the starting reference value for fully connected layers. Different dropout rates are suitable for different network layers. Input layers are usually set to lower values, such as 0.1 to 0.2, to prevent the loss of too much original information. Fully connected layers are prone to overfitting due to their large number of parameters, so higher values, such as around 0.5, can be set. Convolutional layers have a relatively low risk of overfitting and are usually set between 0.1 and 0.25. If the model is severely overfitted, the preset dropout rate should be increased; if the model is underfitted, the preset dropout rate should be decreased. The final accurate value needs to be determined by comparative experiments on the validation set.

[0033] End-to-end learning specifically includes: The elements of the adjacency matrix are used as trainable network parameters and jointly optimized with other parameters of the SMS-GCN network through the backpropagation algorithm, without the need for manual design or fixing of the adjacency matrix. For each functional joint subset, calculate the average distance between any two joint coordinates within the subset within the time window, and use this average distance as the initial value of the corresponding element in the adjacency matrix to obtain the initial adjacency matrix. Each element in the initial adjacency matrix is ​​used as a trainable network parameter and given an initial value during the initialization of the SMS-GCN network. During training, these parameters, along with the convolutional kernel weights, fully connected layer weights, etc., in the network participate in gradient updates. The parameterized adjacency matrix is ​​used as the input to the graph convolutional layer. The adjacency matrix defines the spatial connection weights between joints within the subset. The graph convolution operation calculates the updated joint features based on the adjacency matrix and the joint feature map, thus completing the forward propagation process. The output of the graph convolutional layer is passed through subsequent network layers in sequence to finally obtain the behavior classification prediction result. The prediction result is compared with the labeled real behavior category and the classification loss is calculated. The gradient of the classification loss with respect to each element in the adjacency matrix is ​​calculated through the backpropagation algorithm, and the value of each element in the adjacency matrix is ​​updated according to the gradient direction. Repeat the above process of forward propagation, loss calculation, backpropagation and parameter update until the classification loss converges. After training, the values ​​of each element in the adjacency matrix are automatically adjusted to the optimal values ​​that best reflect the true connection relationship of the joints in the subset. End-to-end learning does not rely on manually defined physical connection priors or fixed distance thresholds. The adjacency matrix is ​​optimized entirely by training data, enabling it to adaptively learn the optimal connection strength between joints under different action categories.

[0034] In this embodiment, the learnable weight prediction network specifically includes: It consists of an input flattening layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected sequentially. The input flattening layer expands the matrix elements of the adjacency matrix of each subset into a one-dimensional feature vector in row-major order. The first fully connected layer maps the one-dimensional feature vector to an intermediate feature vector, the dimension of which is a predefined dimension. The batch normalization layer normalizes the mean and variance of each dimension of the intermediate feature vector, and outputs the normalized intermediate feature vector; The modified linear unit activation layer performs a non-linear mapping on the normalized intermediate feature vector, setting the negative values ​​in the normalized intermediate feature vector to zero and leaving the positive values ​​unchanged, and outputs the activation feature vector; The second fully connected layer maps the activation feature vectors to output feature vectors, and the dimension of the output feature vectors is the same as the number of functional joint subsets; The softmax normalization layer performs exponential normalization on each component of the output feature vector to obtain the fusion weights of each functional joint subset. Each fusion weight is a positive number between 0 and 1, and the sum of all fusion weights is equal to 1.

[0035] The dimensions of the intermediate feature vector are predefined dimensions, specifically including: Empirical estimation methods estimate the number of neurons in the input and output layers to make the hidden layer dimension fall between the two, or refer to commonly used empirical formulas to determine the initial value. Based on the task requirement setting method, the intermediate layer dimension is set with reference to the number of target categories or the feature representation requirements of the classification task, so that it has sufficient feature transformation capability. The experimental adjustment method involves starting with a baseline value and conducting comparative experiments. The dimensions are gradually adjusted based on the model's performance on the validation set, increasing the dimensions when underfitting and decreasing the dimensions when overfitting. The automated search method uses hyperparameter optimization tools to automatically search for the optimal dimension configuration within a specified range. In the learnable weight prediction network of SMS-GCN, the predefined dimension is used as a hyperparameter of the network structure to control the output dimension of the first fully connected layer.

[0036] Specifically, the nonlinear mapping of the normalized intermediate feature vector by the modified linear unit activation layer includes: For each element in the normalized intermediate feature vector, determine whether the value of the element is greater than zero. Keep the elements with values ​​greater than zero unchanged, and assign the value of elements less than or equal to zero to zero. Output the activation feature vector. The input dimension and output dimension of the corrected linear unit activation layer are the same, both of which are predefined dimensions. The corrected linear unit activation layer does not contain learnable parameters, and the same mapping operation is performed on all elements of the normalized intermediate feature vector.

[0037] Specifically, the second fully connected layer maps the activation feature vector to the output feature vector, including: The second fully connected layer maps the activation feature vector to the output feature vector. The activation feature vector is a fixed-dimensional feature representation of the output of the previous modified linear unit activation layer. The second fully connected layer performs a linear transformation operation on the activation feature vector. The second fully connected layer contains a learnable weight matrix and bias vectors. The number of rows in the weight matrix is ​​equal to the dimension of the output feature vector, the number of columns in the weight matrix is ​​equal to the dimension of the activation feature vector, and the dimension of the bias vector is equal to the dimension of the output feature vector. The linear transformation operation is as follows: perform matrix multiplication on the weight matrix and the activation feature vector, add the result to the bias vector element by element to obtain the output feature vector; The dimension of the output feature vector is the same as the number of functional joint subsets.

[0038] In this embodiment, generating a spatially refined feature map specifically includes: Within each subset, the joint features are the feature vectors corresponding to each joint output from the previous layer of the SMS-GCN network. The feature vectors of all joints in the upper limb movement group are multiplied by the first fusion weight to obtain the first weighted joint feature tensor. The feature vectors of all joints in the lower limb movement group are multiplied by the second fusion weight to obtain the second weighted joint feature tensor. The feature vectors of all joints in the trunk movement group are multiplied by the third fusion weight to obtain the third weighted joint feature tensor. The first, second, and third weighted joint feature tensors are concatenated along the joint dimension to form a concatenated joint feature tensor. The concatenated joint feature tensor is then summed element-wise along the joint dimension to obtain a spatially refined feature map.

[0039] In this embodiment, the selection weights of the several temporal convolution kernels output by the gated weight generation module of SMS-GCN specifically include: The mean motion acceleration is the average displacement change rate of each joint between adjacent frames in the spatially refined feature map. The mean motion acceleration values ​​of the three subsets—upper limb action group, lower limb action group, and trunk action group—are concatenated to form a three-dimensional acceleration feature vector. This three-dimensional acceleration feature vector is then input into a lightweight gating prediction network. The lightweight gating prediction network consists of one fully connected layer and one Softmax normalization layer. The fully connected layer maps the three-dimensional acceleration feature vector to a four-dimensional output vector, and the Softmax normalization layer normalizes each component of the four-dimensional output vector into four selection weights.

[0040] Specifically, the fully connected layer maps the three-dimensional acceleration feature vector to a four-dimensional output vector. The fully connected layer contains a learnable weight matrix and a bias vector. The number of rows in the weight matrix is ​​equal to the dimension of the output vector, which is 4. The number of columns in the weight matrix is ​​equal to the dimension of the input vector, which is 3. The dimension of the bias vector is equal to the dimension of the output vector, which is 4. The fully connected layer performs a linear transformation on the three-dimensional acceleration feature vector. Specifically, the linear transformation involves performing matrix multiplication on the weight matrix and the three-dimensional acceleration feature vector to obtain a four-dimensional intermediate vector; and then adding the four-dimensional intermediate vector to the bias vector element by element to obtain a four-dimensional output vector. Each component of the four-dimensional output vector corresponds to an unnormalized selection weight of a temporal convolution kernel, and the four temporal convolution kernels have different receptive field ranges along the time dimension.

[0041] In this embodiment, generating a dynamic temporal convolution kernel by weighted combination of several temporal convolution kernels based on selection weights specifically includes: The number of selected weights is the same as the number of temporal convolution kernels, and each selected weight corresponds one-to-one with each temporal convolution kernel; Several temporal convolution kernels have different receptive field ranges along the time dimension; Multiply the kernel parameters of each temporal convolution kernel by the corresponding selection weights to obtain each weighted convolution kernel; then add each weighted convolution kernel element by element in the parameter space to generate a dynamic temporal convolution kernel.

[0042] Specifically, the receptive field range is: Corresponding to four timing window sizes, the four timing window sizes are sequentially increased; The temporal convolutional kernel with the smallest window size is used to capture the instantaneous features of fast actions, while the temporal convolutional kernel with the largest window size is used to capture the long-range dependency features of slow actions. The four temporal convolutional kernels cover different frame ranges in the time dimension, and the window size of each temporal convolutional kernel is an odd number of time frames.

[0043] Specifically, the kernel parameters of each temporal convolution kernel include: The kernel parameters of a temporal convolution kernel are the set of weight values ​​at each position of the temporal convolution kernel in the time dimension, and each weight value is a learnable floating-point parameter; The number of kernel parameters for each temporal convolution kernel is equal to the temporal window size; Multiply the kernel parameters of each temporal convolution kernel by the corresponding selection weights, and multiply each weight value in the kernel parameter set of each temporal convolution kernel by the corresponding selection weight to obtain the weighted kernel parameter set of each weighted convolution kernel. When a temporal convolution kernel with weights greater than zero is selected, the weight values ​​in the weighted kernel parameter set are amplified; when a temporal convolution kernel with weights less than one is selected, the weight values ​​in the weighted kernel parameter set are reduced; and when a temporal convolution kernel with weights of zero is selected, the weight values ​​in the weighted kernel parameter set are set to zero.

[0044] In this embodiment, the output spatiotemporal joint feature map specifically includes: The spatially refined feature map is a three-dimensional tensor, with the three dimensions being the temporal frame dimension, the joint dimension, and the channel dimension. A dynamic temporal convolution kernel performs a sliding window convolution operation along the temporal frame dimension on the spatially refined feature map. The window size of the dynamic temporal convolution kernel is equal to the receptive field of the kernel. At each sliding window position, the kernel parameters of the dynamic temporal convolution kernel are multiplied element-wise with the feature tensor within the corresponding time window in the spatially refined feature map, and then summed to obtain an output value. After the sliding window traverses all time frames, a spatiotemporal joint feature map is obtained. The temporal frame dimension of the spatiotemporal joint feature map is smaller than that of the spatially refined feature map, while the joint and channel dimensions of the spatiotemporal joint feature map are the same as those of the spatially refined feature map.

[0045] Specifically, the dynamic temporal convolution kernel performs sliding window convolution operations along the temporal frame dimension on the spatially refined feature map, including: A dynamic temporal convolution kernel moves along the time frame dimension with a preset stride. At each window position, the kernel parameters of the dynamic temporal convolution kernel are multiplied element-wise with the feature values ​​within the corresponding time window in the spatial refinement feature map. All product results are summed to obtain a scalar value as the output feature value for that window position. The window size of the dynamic temporal convolution kernel is equal to the number of frames corresponding to the receptive field of the temporal convolution kernel. The dynamic temporal convolution kernel remains fixed along the channel direction of the time frame dimension and does not participate in sliding, but only slides along the time direction. Each time the sliding window has traversed all window positions, a time frame dimension feature value is output. After the sliding window has traversed all window positions, a time frame dimension compressed spatiotemporal joint feature map is obtained. The number of time frames in the spatiotemporal joint feature map is equal to the number of input time frames minus the window size plus the preset stride and divided by the preset stride. The joint dimension and channel dimension of the spatiotemporal joint feature map are consistent with the joint dimension and channel dimension of the spatial refinement feature map.

[0046] In this embodiment, the output behavior category labels specifically include: The classifier consists of a global average pooling layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected in sequence. The global average pooling layer performs global average pooling operations on the spatiotemporal joint feature map in the joint dimension and time frame dimension to obtain a one-dimensional global feature vector. The dimension of the one-dimensional global feature vector is the same as the number of channels of the spatiotemporal joint feature map. The first fully connected layer maps the one-dimensional global feature vector to an intermediate classification feature vector, and the dimension of the intermediate classification feature vector is the predefined classification dimension; The batch normalization layer normalizes the mean and variance of each dimension of the intermediate classification feature vector and outputs the normalized intermediate classification feature vector. The modified linear unit activation layer performs a non-linear mapping on the normalized intermediate classification feature vector, setting the negative values ​​in the normalized intermediate classification feature vector to zero, while keeping the positive values ​​unchanged, and outputting the activated classification feature vector. The second fully connected layer maps the activation classification feature vector to the output classification feature vector, and the dimension of the output classification feature vector is equal to the total number of predefined behavior categories. The Softmax normalization layer performs exponential normalization on each component of the output classification feature vector to obtain the probability distribution of each behavior category, and outputs the label corresponding to the behavior category with the highest probability value as the behavior category label.

[0047] The intermediate classification feature vector has a predefined classification dimension, which specifically includes: The estimation is based on the multiple relationship of the total number of behavior categories. The set value is usually greater than the total number of behavior categories to retain sufficient feature transformation capability, while being less than the dimension of the input feature vector to achieve feature compression. Experimental adjustments were made based on the classification accuracy on the validation set. When underfitting, the predefined classification dimension was increased to enhance the model's expressive power, and when overfitting, the predefined classification dimension was decreased to limit the model's complexity. The system adapts to the computing resource constraints of edge devices, selecting dimension values ​​that meet real-time requirements while ensuring classification accuracy.

[0048] The data acquisition and cleaning module corresponds to the data input stage of "video surveillance," used to extract human skeleton sequences from surveillance videos. The joint grouping module, adjacency matrix construction module, and spatial feature fusion module of SMS-GCN correspond to spatial feature extraction in "visual intelligence," achieving refined modeling of local joint movements through functional grouping and learnable weighted fusion of the human skeleton. The gating weight generation module and dynamic convolution kernel generation and temporal convolution module of SMS-GCN correspond to temporal dynamic analysis in "abnormal behavior," adaptively adjusting the temporal receptive field through a motion acceleration perception gating mechanism to capture changes in movement at different speeds. The behavior classification module ultimately outputs behavior category labels, enabling automatic identification and early warning of abnormal behaviors such as falls and fainting. The entire claim revolves around the smart home elderly care scenario, using skeleton data acquired from video surveillance as input and abnormal behavior discrimination as output, constructing a complete end-to-end technical solution.

[0049] Example 1: The present invention was deployed in a home-based elderly care intelligent monitoring system to verify its effectiveness in a real-life environment. The system was installed in the residence of an elderly person living alone, whose daily activities were concentrated in the living room, bedroom, and bathroom. Three depth vision sensors were deployed in the residence, located above the TV cabinet in the living room, in a corner of the bedroom ceiling, and in the bathroom ceiling, respectively, to collect human skeletal sequence data. All data processing was completed on a local edge device.

[0050] The elderly man lived his daily life as usual without receiving any special instructions, and the system ran continuously for 45 days. During this period, the system recorded a large amount of real behavioral data: the elderly man got up at around 6:30 am every day, spent most of his day watching TV or napping on the sofa in the living room, took a short walk on the balcony after lunch, and went to bed around 10 pm. Behind this seemingly regular daily routine, the system captured many subtle anomalies through skeleton sequence data that would be impossible to see with ordinary monitoring.

[0051] The system first preprocesses the raw skeleton data through a data acquisition and cleaning module, filling in missing joint coordinates and filtering out jitter noise. Then, a joint grouping module divides all 17 joints into upper limb, lower limb, and trunk movement groups based on human kinematic priors. An adjacency matrix construction module builds a subset adjacency matrix for each group, and a spatial feature fusion module automatically learns the fusion weights for each group through a learnable weight prediction network, generating a refined spatial feature map. A gating weight generation module calculates the mean motion acceleration and inputs it into a lightweight gating prediction network, outputting selection weights for four temporal convolution kernels at different scales. A dynamic convolution kernel generation and temporal convolution module combines these weights to create dynamic temporal convolution kernels and completes temporal convolution. Finally, a behavior classification module outputs behavior category labels.

[0052] To more clearly illustrate the superiority of this invention in real-world scenarios, we conducted a 45-day parallel comparative test between this system and several existing technical solutions in the same elderly person's residence. The comparative solutions included a traditional single-modal behavior recognition method, a pure skeleton graph convolutional method, the original selectable multi-scale graph convolutional network, and a simplified version with the attention mechanism removed. Table 1 records the comparison results of key performance indicators of different methods in a real-world home-based elderly care scenario.

[0053] Table 1: Performance Comparison of Different Behavior Recognition Methods in Elderly Care Scenarios

[0054] Table 1 shows some interesting patterns. First, let's look at the fall detection rate. This invention achieves 96.8%, which is 4.5 percentage points higher than the original SMS-GCN method's 92.3%, and significantly higher than the traditional single-modal method's 78.2% by 18.6 percentage points. What does this difference mean in actual monitoring? The traditional method misses nearly 22 falls out of every 100 detected falls, while this invention misses only about 3 out of every 100, effectively reducing the risk of missed falls by nearly 85%. Next, let's look at the local joint movement recognition rate. The difference is even more pronounced. This invention's 93.2% is an 11.6 percentage point improvement over the original SMS-GCN's 81.6%, mainly due to the joint grouping module's decoupling and separate modeling of the topological structures of the upper limbs, lower limbs, and trunk. In daily activities that primarily involve arm movements, such as reaching for objects or drinking water, traditional methods often mistakenly attribute the relative movement between the shoulder and elbow to trunk offset. However, this invention, by constructing an adjacency matrix for the upper limb movement group separately, can clearly capture the fine spatial dependencies between the wrist, elbow, and shoulder joints, which is the fundamental reason for the significant improvement in recognition rate.

[0055] The accuracy rate for distinguishing similar actions best reflects the comprehensive advantages of this invention. This metric measures whether the system can accurately distinguish actions that appear similar but are fundamentally different. For example, an elderly person leaning forward on a sofa to reach for a remote control on a coffee table, and an elderly person suddenly falling forward due to discomfort, have very similar initial posture changes, but their subsequent acceleration curves and skeletal angle changes are completely different. The 89.6% accuracy rate of this invention is 15.1 percentage points higher than the original SMS-GCN's 74.5%. This improvement comes from the synergistic effect of two dimensions: spatially, functional grouping reveals the fine movement patterns of local joints, and temporally, dynamic gating convolution kernels adaptively match the velocity characteristics of the action.

[0056] The difference in false alarm and false negative frequencies is also very telling. This invention generates only 0.6 false alarms and 0.2 false negatives per week, while the original SMS-GCN generates 1.7 and 1.0 false negatives respectively. This means that over a month, the original method might generate approximately 7 false alarms and 4 false negatives, while this invention generates only 2.4 and 0.8 false negatives respectively. For monitoring elderly people living alone, too many false alarms can cause fatigue for family members, and too many false negatives can lead to safety accidents. This invention reduces both of these indicators to a practically acceptable range. Regarding response latency, although the invention's 67 milliseconds is only 9 milliseconds faster than the original SMS-GCN, in actual deployment, this difference is enough to ensure that alarm notifications reach family members' phones almost one-tenth of a second earlier.

[0057] Under complex lighting and partial occlusion conditions, the recognition rates of this invention were 94.1% and 91.7%, respectively, representing improvements of 6.9 and 8.1 percentage points compared to the original SMS-GCN's 87.2% and 83.6%. Skeleton data itself is not sensitive to lighting, so all skeleton-based methods perform relatively stably under varying lighting conditions. However, this invention demonstrates a more significant advantage under partial occlusion. This is because when a joint is occluded, other related joints within the same functional group can compensate for the loss by passing information through a subset adjacency matrix, a mechanism lacking in globally unified topology methods. Regarding model parameters, this invention has 2.1 million, only 0.1 million more than the original SMS-GCN's 2.0 million. This increase mainly comes from the additional parameters of the learnable weight prediction network and the gated prediction network, but this slight increase is acceptable compared to the significant improvement in recognition performance. In summary, this invention outperforms existing methods in almost all evaluation dimensions, particularly excelling in the three core requirements of elderly care monitoring: local action recognition, similar action differentiation, and environmental robustness.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence, characterized in that: include: The data acquisition and cleaning module acquires human skeleton sequence data and performs null padding and noise filtering on the human skeleton sequence data. SMS-GCN's joint grouping module divides the joints in the cleaned human skeleton sequence data into three functional joint subsets: upper limb movement group, lower limb movement group, and trunk movement group, according to human kinematic priors. The adjacency matrix construction module of SMS-GCN constructs a subset adjacency matrix for each functional joint subset. The spatial feature fusion module of SMS-GCN inputs the adjacency matrix of each subset into the learnable weight prediction network. The learnable weight prediction network outputs the fusion weights corresponding to each subset and performs weighted summation on the joint features within each subset to generate a spatially refined feature map. The SMS-GCN gating weight generation module inputs the spatially refined feature map into the lightweight gating prediction network. The lightweight gating prediction network takes the mean of motion acceleration as input and outputs the selection weights of several temporal convolutional kernels. The SMS-GCN dynamic convolutional kernel generation and temporal convolution module weights several temporal convolutional kernels based on the selection weights to generate dynamic temporal convolutional kernels. The dynamic temporal convolutional kernels are used to perform temporal convolution on the spatially refined feature map to output a spatiotemporal joint feature map. The behavior classification module inputs the spatiotemporal joint feature map into the classifier and outputs behavior category labels.

2. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The skeleton sequence data specifically includes: The joint identifiers of each joint in the human skeleton, the two-dimensional coordinates of each joint in the image frame, the depth values ​​of each joint, the frame number corresponding to each joint, and the detection confidence scores of each joint.

3. The intelligent home-based elderly care video monitoring system for abnormal behavior based on visual intelligence according to claim 1, characterized in that, The division into functional joint subsets specifically includes: the upper limb movement group, which includes the left wrist joint, left elbow joint, left shoulder joint, right wrist joint, right elbow joint, and right shoulder joint; the lower limb movement group, which includes the left ankle joint, left knee joint, left hip joint, right ankle joint, right knee joint, and right hip joint; and the trunk movement group, which includes the neck joint, chest joint, and central hip joint.

4. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The construction of the subset adjacency matrix specifically includes: For each functional joint subset, calculating the average distance between any two joint coordinates within the subset within a time window, using the average distance as the initial value of the adjacency matrix element corresponding to the two joints to obtain the initial adjacency matrix, and inputting the initial adjacency matrix into the graph convolutional layer of SMS-GCN for end-to-end learning. The graph convolutional layer of SMS-GCN updates the value of each element in the adjacency matrix during training.

5. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The learnable weight prediction network specifically includes: It consists of an input flattening layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected in sequence; the input flattening layer expands the matrix elements of the adjacency matrix of each subset into a one-dimensional feature vector in row-major order; the first fully connected layer maps the one-dimensional feature vector to an intermediate feature vector, the dimension of which is a predefined dimension; The batch normalization layer normalizes the mean and variance of each dimension of the intermediate feature vector and outputs the normalized intermediate feature vector. The modified linear unit activation layer performs a nonlinear mapping on the normalized intermediate feature vector, setting the negative values ​​in the normalized intermediate feature vector to zero, while keeping the positive values ​​unchanged, and outputting the activation feature vector. The second fully connected layer maps the activation feature vectors to output feature vectors, and the dimension of the output feature vectors is the same as the number of functional joint subsets. The softmax normalization layer performs exponential normalization on each component of the output feature vector to obtain the fusion weights of each functional joint subset. Each fusion weight is a positive number between 0 and 1, and the sum of all fusion weights is equal to 1.

6. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The generation of the spatially refined feature map specifically includes: The fusion weights output by the Softmax normalization layer contain three weight values, which correspond to the upper limb action group, lower limb action group, and trunk action group in the order of functional joint subsets. The first weight value corresponds to the upper limb action group, the second weight value corresponds to the lower limb action group, and the third weight value corresponds to the trunk action group. Multiplying all joint feature vectors of the upper limb action group by the first weight value yields the first weighted joint feature tensor. Multiply all joint feature vectors of the lower limb movement group by the second weight value to obtain the second weighted joint feature tensor; Multiply all joint feature vectors of the trunk motion group by the third weight value to obtain the third weighted joint feature tensor; The first weighted joint feature tensor, the second weighted joint feature tensor, and the third weighted joint feature tensor are concatenated along the joint dimension to form a concatenated joint feature tensor. The spliced ​​joint feature tensor is summed element-wise along the joint dimension to obtain a spatially refined feature map.

7. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The selection weights of several temporal convolutional kernels output by the gated weight generation module of the SMS-GCN specifically include: The mean motion acceleration is the average displacement change rate of each joint between adjacent frames in the spatially refined feature map. The mean motion acceleration values ​​of the three subsets—upper limb action group, lower limb action group, and trunk action group—are concatenated to form a three-dimensional acceleration feature vector. This three-dimensional acceleration feature vector is then input into a lightweight gating prediction network. The lightweight gating prediction network consists of one fully connected layer and one Softmax normalization layer. The fully connected layer maps the three-dimensional acceleration feature vector into a four-dimensional output vector, and the Softmax normalization layer normalizes each component of the four-dimensional output vector into four selection weights.

8. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The step of generating a dynamic temporal convolutional kernel by weighting several temporal convolutional kernels based on selection weights specifically includes: The number of selected weights is the same as the number of temporal convolution kernels, and each selected weight corresponds one-to-one with each temporal convolution kernel; Several temporal convolution kernels have different receptive field ranges along the time dimension; Multiply the kernel parameters of each temporal convolution kernel by the corresponding selection weights to obtain each weighted convolution kernel; then add each weighted convolution kernel element by element in the parameter space to generate a dynamic temporal convolution kernel.

9. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The output spatiotemporal joint feature map specifically includes: The spatially refined feature map is a three-dimensional tensor, with the three dimensions being the temporal frame dimension, the joint dimension, and the channel dimension. A dynamic temporal convolution kernel performs a sliding window convolution operation along the temporal frame dimension on the spatially refined feature map. The window size of the dynamic temporal convolution kernel is equal to the receptive field of the kernel. At each sliding window position, the kernel parameters of the dynamic temporal convolution kernel are multiplied element-wise with the feature tensor within the corresponding time window in the spatially refined feature map, and then summed to obtain an output value. After the sliding window traverses all time frames, a spatiotemporal joint feature map is obtained. The temporal frame dimension of the spatiotemporal joint feature map is smaller than that of the spatially refined feature map, while the joint and channel dimensions of the spatiotemporal joint feature map are the same as those of the spatially refined feature map.

10. The video surveillance system for abnormal behavior in smart home elderly care based on visual intelligence according to claim 1, characterized in that, The output behavior category labels specifically include: The classifier consists of a global average pooling layer, a first fully connected layer, a batch normalization layer, a modified linear unit activation layer, a second fully connected layer, and a Softmax normalization layer connected in sequence. The global average pooling layer performs global average pooling operations on the spatiotemporal joint feature map in the joint dimension and time frame dimension to obtain a one-dimensional global feature vector. The dimension of the one-dimensional global feature vector is the same as the number of channels of the spatiotemporal joint feature map. The first fully connected layer maps the one-dimensional global feature vector to an intermediate classification feature vector, and the dimension of the intermediate classification feature vector is the predefined classification dimension; The batch normalization layer normalizes the mean and variance of each dimension of the intermediate classification feature vector and outputs the normalized intermediate classification feature vector. The modified linear unit activation layer performs a non-linear mapping on the normalized intermediate classification feature vector, setting the negative values ​​in the normalized intermediate classification feature vector to zero, while keeping the positive values ​​unchanged, and outputting the activated classification feature vector. The second fully connected layer maps the activation classification feature vector to the output classification feature vector, and the dimension of the output classification feature vector is equal to the total number of predefined behavior categories. The Softmax normalization layer performs exponential normalization on each component of the output classification feature vector to obtain the probability distribution of each behavior category, and outputs the label corresponding to the behavior category with the highest probability value as the behavior category label.

Images