Multi-modal real-time action recognition method and system based on lightweight deep learning

Abstract

The application relates to the technical field of action recognition and discloses a multi-modal real-time action recognition method and system based on light-weight deep learning. The method synchronously collects motion trajectory data, skeletal key point sequences and multi-modal sensing information of an environment state of a target object from multiple source sensing devices; the motion trajectory data is subjected to space-time feature alignment, a motion continuity index is generated to determine an action response period; behavior pattern information is extracted and subjected to multi-scale feature analysis, and combined with the multi-modal sensing information to determine an abnormal behavior grade; and the identification blind area in a real-time action stream is dynamically corrected based on the action response period and the abnormal behavior grade. The method improves the comprehensiveness, accuracy and adaptability of action recognition through multi-source data fusion, dynamic response period adjustment, multi-scale feature analysis and dynamic blind area correction, and light-weight design makes the method applicable to resource-limited devices, and the method has strong practical value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, in particular to a multi-modal real-time action recognition method and system based on lightweight deep learning. BACKGROUND

[0002] In the current field of action recognition, with the continuous expansion of application scenarios such as intelligent monitoring, human-computer interaction, and motion analysis, higher and higher requirements are put forward for the accuracy and efficiency of real-time action recognition. Traditional action recognition methods mostly rely on data collected by single sensing devices, such as obtaining image sequences only through visual cameras or relying only on acceleration sensors to capture motion data. This single-modal data collection method is difficult to fully reflect the complex characteristics of actions and is easily disturbed by environmental factors. For example, in an environment with poor lighting conditions, the quality of images obtained by visual sensors decreases, which greatly reduces the accuracy of action recognition; while relying only on motion sensors, it is difficult to distinguish subtle differences in the execution process of actions, making the recognition results lack robustness.

[0003] Existing action recognition methods often face the problem of excessive consumption of computing resources when processing real-time data. Although deep learning models can improve recognition accuracy, the complex network structure requires a large amount of computing support, making it difficult to achieve real-time response in mobile devices or embedded systems. In addition, during the processing of action sequences, due to the differences in data collection frequency of different sensing devices, the phenomenon of spatio-temporal feature misalignment easily occurs, leading to inaccurate action state tracking and further affecting the reliability of the recognition results. When there is an action recognition blind area, traditional methods usually use fixed correction strategies, which cannot be dynamically adjusted according to actual conditions, further reducing the adaptability and accuracy of the recognition system. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal real-time action recognition method and system based on lightweight deep learning to solve the problems raised in the background.

[0005] To achieve the above purpose, the present application provides a multi-modal real-time action recognition method based on lightweight deep learning, which comprises:

[0006] Synchronously collecting motion trajectory data and skeletal key point sequences of a target object from multiple source sensing devices, and real-time acquiring multi-modal sensing information of the environment state;

[0007] Aligning the spatio-temporal features of the motion trajectory data, generating a motion continuity index, and determining the action response period for state tracking of the target object in the action sequence according to the motion continuity index;

[0008] extracting behavior pattern information in a target action execution process, performing multi-scale feature analysis on the behavior pattern information to obtain spatio-temporal constraint features in action execution, and determining an abnormal behavior level in an action recognition process according to the spatio-temporal constraint features and the multi-modal sensing information;

[0009] dynamically correcting a recognition blind area in real-time action flow based on the action response period and the abnormal behavior level.

[0010] Preferably, the motion trajectory data includes time series data of displacement coordinates, motion speed, acceleration and motion direction of the target object in three-dimensional space.

[0011] Preferably, the multi-modal sensing information includes ambient light intensity, background noise spectrum, temperature field distribution and spatial obstacle topology information.

[0012] Preferably, the action response period in which the target object is tracked in the action sequence according to the motion continuity index specifically includes:

[0013] constructing a spatio-temporal continuity model of the action sequence according to the motion continuity index;

[0014] outputting a tracking time window of real-time action tracking from the spatio-temporal continuity model;

[0015] determining the action response period in which the target object is tracked in the action sequence through the tracking time window.

[0016] Preferably, extracting behavior pattern information in a target action execution process specifically includes:

[0017] real-time capturing of pose transformation features of the action segment;

[0018] generating core behavior patterns of action execution according to the pose transformation features;

[0019] determining behavior pattern information in a target action execution process through the core behavior patterns.

[0020] Preferably, performing multi-scale feature analysis on the behavior pattern information specifically includes:

[0021] constructing a spatio-temporal association model of the action segment and historical abnormal action based on the behavior pattern information;

[0022] outputting feature fusion strength in action execution through the spatio-temporal association model;

[0023] determining dynamic partition boundaries of each action segment in the behavior pattern according to the feature fusion strength;

[0024] A spatio-temporal constraint feature of a time when the dynamic partition boundary generation action is performed.

[0025] Preferably, determining an abnormal behavior level in the action recognition process according to the spatio-temporal constraint feature and the multi-modal sensing information specifically includes:

[0026] Extracting a spatio-temporal consistency parameter of the action sequence according to the spatio-temporal constraint feature;

[0027] Calculating a coupling strength of an environmental interference factor through the multi-modal sensing information;

[0028] Generating an abnormal behavior level in the action recognition process based on the spatio-temporal consistency parameter and the coupling strength of the environmental interference factor.

[0029] Preferably, dynamically correcting a recognition blind area in the real-time action stream based on the action response period and the abnormal behavior level specifically includes:

[0030] Determining a recognition sampling density of the real-time action stream according to the action response period;

[0031] Calculating a confidence offset of the recognition blind area according to the abnormal behavior level;

[0032] Inputting the recognition sampling density and the confidence offset into a lightweight deep learning model to generate a dynamically updated action recognition decision tree;

[0033] The spatio-temporal feature alignment of the motion trajectory data specifically includes:

[0034] Dividing a spatio-temporal unit of the action sequence according to the motion trajectory data;

[0035] Interpolating and compensating for the missing spatio-temporal unit to generate a complete action state sequence;

[0036] Extracting a multi-scale feature vector of the action state sequence;

[0037] Generating a motion continuity index through the multi-scale feature vector.

[0038] Preferably, the construction method of the lightweight deep learning model includes:

[0039] Inputting the motion trajectory data and the skeletal key point sequence into a spatio-temporal feature fusion backbone network to generate a spatio-temporal joint feature vector;

[0040] Performing a dynamic pruning operation on the spatio-temporal joint feature vector to remove redundant feature channels and compress the feature dimension to generate a pruned lightweight feature tensor;

[0041] The multi-modal sensing information is used for cross-modal attention weighting on the pruned lightweight feature tensor to generate a multi-modal fusion feature matrix.

[0042] The multi-modal fusion feature matrix is multi-task jointly optimized by the action response period and the abnormal behavior level to output lightweight deep learning model parameters.

[0043] Preferably, the application also includes a multi-modal real-time action recognition system based on lightweight deep learning, which is used for executing the multi-modal real-time action recognition method based on lightweight deep learning as described above, and the system includes:

[0044] A multi-source sensing module is configured to synchronously collect motion trajectory data and skeletal key point sequences of a target object from a multi-source sensing device and to real-time acquire multi-modal sensing information of an environment state.

[0045] A feature fusion module is configured to perform spatiotemporal feature alignment on the motion trajectory data to generate a motion continuity index, and to determine an action response period for state tracking of the target object in an action sequence according to the motion continuity index.

[0046] A behavior analysis module is configured to extract behavior pattern information in a target action execution process, to perform multi-scale feature analysis on the behavior pattern information to obtain spatiotemporal constraint features in the action execution, and to determine an abnormal behavior level in an action recognition process according to the spatiotemporal constraint features and the multi-modal sensing information.

[0047] A dynamic correction module is configured to dynamically correct a recognition blind area in a real-time action stream based on the action response period and the abnormal behavior level.

[0048] Compared with the prior art, the application has the following beneficial effects:

[0049] By synchronously collecting motion trajectory data and skeletal key point sequences of a target object from a multi-source sensing device and combining multi-modal sensing information of an environment state, all-around capturing of action features is achieved. Fusion of multi-source data can make up for the limitations of single-modal data in information expression, so that the description of action features is more comprehensive and accurate, which helps to improve the recognition ability of complex actions.

[0050] In the processing of motion trajectory data, the motion continuity index is generated by spatiotemporal feature alignment, and the action response period is determined accordingly, which can effectively solve the spatiotemporal misalignment problem caused by the difference in data collection frequency of different sensing devices, and ensure the continuity and accuracy of action state tracking. This way of dynamically adjusting the response period based on the characteristics of the data itself makes the action recognition system better adapt to different action rhythms and changing speeds, and enhances the perception ability of the system to action sequences.

[0051] The multi-scale feature analysis of the behavior pattern information obtains the spatio-temporal constraint features during action execution, and determines the abnormal behavior level in combination with the multi-modal sensing information, which can more meticulously mine the internal laws and abnormal features of the action. The multi-scale analysis helps to capture action details at different levels, from macro action trends to micro action changes, which can effectively express the features, thereby improving the accuracy of abnormal behavior judgment and reducing misjudgment and missed judgment.

[0052] The dynamic correction of the recognition blind area in the real-time action stream based on the action response cycle and the abnormal behavior level gets rid of the limitations of the traditional fixed correction strategy. The dynamic correction can take targeted correction measures according to the specific situation and action features of the recognition blind area, so that the correction process is more flexible and effective, which significantly improves the adaptability and stability of the action recognition system under complex environment and variable action, and ensures the reliability of the recognition result.

[0053] The application of lightweight deep learning reduces the demand for computing resources while ensuring recognition accuracy, so that the method can realize real-time action recognition on resource-constrained devices, expand its application range in mobile terminals, embedded systems and other scenarios, and enhance the practicality and popularity of the technology. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The timing diagram of the multi-modal real-time action recognition method based on lightweight deep learning described in the present application;

[0055] Figure 2 The flowchart for determining the action response cycle;

[0056] Figure 3 The flowchart for multi-scale feature analysis;

[0057] Figure 4 The flowchart for dynamic correction of the recognition blind area;

[0058] Figure 5 The flowchart for constructing the lightweight deep learning model. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0060] Please refer to Figure 1The application provides a multi-modal real-time action recognition method based on lightweight deep learning, which comprises the following steps:

[0061] The motion trajectory data and the skeletal key point sequence of a target object are synchronously collected by a multi-source sensing device, and multi-modal sensing information of an environment state is simultaneously acquired. The motion trajectory data is processed by spatiotemporal feature alignment to generate a motion continuity index, which is used to determine an action response period for state tracking in an action sequence. During the action execution process, behavior pattern information is extracted and analyzed by multi-scale features to obtain spatiotemporal constraint features. In combination with the multi-modal sensing information, the system calculates an abnormal behavior level. Finally, based on the action response period and the abnormal behavior level, a dynamic correction is made to a recognition blind area in a real-time action stream to achieve high-precision real-time action recognition.

[0062] Embodiment 1: The motion trajectory data collection system comprises an array of inertial measurement units distributed at key parts of the body of the target object, each unit integrating a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer. These units synchronously collect data at a frequency of no less than 100 Hz. Displacement coordinates are generated in a world coordinate system by fusing double integration of the accelerometer, angular changes of the gyroscope and directional references of the magnetometer. The coordinate system is set as a right-handed system, with the X-axis pointing to the direction of travel of the target object, the Y-axis pointing to the left horizontally and the Z-axis pointing upward. The time resolution of the displacement data is no less than 10 ms, and the built-in sensor fusion algorithm of the system comprises a Kalman filter module for real-time elimination of sensor noise and drift error.

[0063] The change value of the displacement coordinates at consecutive time points is sent to a velocity calculation module. The module uses the central difference method to calculate the instantaneous velocity vector, and the calculation formula runs in an embedded processor with a time window width of 20 ms. Acceleration data is obtained by combining the original readings of the accelerometer with the rotational transformation of the attitude transformation matrix, which is generated by real-time solving of quaternions. The motion direction is directly obtained by the attitude quaternion, and the four components (q0, q1, q2, q3) of the quaternion are estimated by fusing the accelerometer and magnetometer data from the gyroscope data, with the same update frequency as the displacement coordinates. The entire motion trajectory data preprocessing is completed on an edge computing node. After the original analog signals are digitized by a 24-bit analog-to-digital converter, a dedicated digital signal processor performs filtering, solving and fusion calculation.

[0064] The time series data management adopts a timestamp-based streaming approach. The system maintains a continuously updated data buffer with a length of 0.5 seconds, equivalent to 50 consecutive frames of displacement, velocity, acceleration, and orientation vector data. The update strategy of the buffer is a sliding window mechanism, with the window sliding forward by 0.25 seconds each time, retaining the previous 25 frames of data and reading in the new 25 frames of data. This 50% overlap rate helps maintain the continuity of the motion trajectory. The data within the window is organized into a two-dimensional matrix in chronological order: the rows represent the time frames, and the columns correspond to the displacement values, velocity components, acceleration components, and motion direction quaternion components of the X, Y, and Z axes, respectively. When new sensor data arrives, the system inserts the corresponding row and column according to the timestamp. In the case of unpredictable data delays or losses, the system automatically starts a linear interpolation program to compensate for the missing data using adjacent valid frame data, maintaining the integrity of the time series. The window data after data filling will enter the next processing stage.

[0065] The acquisition of multi-modal sensory information relies on an environmental sensor network deployed around the target object. Ambient light intensity monitoring uses NIST-traceable illuminance sensor nodes. These nodes are placed in multiple positions in the identification area to avoid the influence of a single node's shadow. The illuminance sensor measures a range of 0 to 100 kilo lux (klx) with an accuracy better than ±5%. The system polls all the light sensor nodes every 100 milliseconds, converts the raw voltage readings to lux values using a linear calibration formula, and performs a weighted average processing in the spatial dimension to output the overall light intensity reading. This reading is aligned with the motion trajectory data stream according to the timestamp.

[0066] The spectral features of background noise are captured by a microphone array installed in the four corners of the target environment. The microphones are omnidirectional capacitive models with a frequency response range of 20 Hz to 20 kHz. The acoustic data is digitized at a sampling rate of 44.1 kHz with a resolution of 16 bits per channel. The audio data stream is divided into short-time frames with a unit of 100 milliseconds, and each frame is multiplied by a Hanning window function to suppress spectral leakage. The windowed data is fed into a Fast Fourier Transform processor to calculate the amplitude spectrum. The final output spectral features are represented as spectral band energies in one-third octave bands, divided into 24 band points, and each band's energy value is log-normalized. The spectral data of all microphone channels are fused through a spatial beamforming algorithm to enhance the directionality of the sound source in the target area.

[0067] The temperature field distribution data is sourced from a non-cooled infrared thermal imager suspended from the top. The thermal imager works in the wavelength range of 8 to 14 microns, and the detector resolution is 640x480 pixels. The actual physical size represented by each pixel is 2.3 mm x 2.3 mm when the system preset detection distance is 3 meters. The thermal imager outputs 10 complete infrared images per second, and each image contains a two-dimensional temperature value matrix. After the matrix is corrected by the emissivity compensation algorithm (based on the preset target material emissivity), it is transmitted to the central processing unit through the Ethernet interface. The central processing unit performs two-dimensional spatial interpolation on the temperature matrix to improve the spatial resolution, and calculates the average temperature, temperature gradient distribution of the entire identification area, and the local average temperature of the identified target area.

[0068] The detection of the spatial obstacle topology combines a 77GHz frequency band millimeter wave radar module and a depth vision system. The millimeter wave radar scans the identified environment with a vertical field of view angle of 30 degrees and a horizontal field of view angle of 120 degrees, and the detection distance range is 0.2 meters to 20 meters, with a distance resolution better than 4 centimeters. The radar data contains the distance, azimuth angle and radial velocity information of each detection point, and the generated point cloud data per second exceeds 2000 points. The depth vision system uses a structured light or time of flight principle to provide a VGA resolution (640x480) depth map with a frame rate of 30 Hz. The system is equipped with a synchronous triggering mechanism to ensure that the radar scanning and depth image acquisition time are strictly aligned.

[0069] The spatial point cloud data obtained by the two sensors is fused through a registration algorithm. First, the coordinate system is transformed to map all points to the global world coordinate system. Then, the iterative closest point algorithm is applied to optimize the registration accuracy. The fused three-dimensional point cloud data enters the obstacle model generation process. The point cloud is first filtered by the radius filtering algorithm to remove noise outliers, and then a density-based spatial clustering algorithm is used to identify different obstacle objects. For each identified obstacle cluster, the system calculates its minimum bounding rectangle and extracts the position coordinates, size information (length, width and height) and pointing vector. The spatial relationship of all obstacles is organized in a graph data structure, with each node representing an obstacle entity and each edge storing the spatial distance, angle relationship and connectivity properties between adjacent obstacles. This dynamically updated obstacle topology graph is refreshed 10 times per second, providing physical constraint information for action recognition environment modeling. All environment information streams are synchronized with the action trajectory data according to a unified time reference.

[0070] Example 2: see Figure 2The calculation process of the motion continuity index relies on a neural network architecture. The network is designed as a three-layer long short-term memory structure, with 128 memory cells in each layer. The input data comes from the multi-dimensional features generated by the motion trajectory processing module, including the norm of displacement vector, the rate of change of velocity vector, the root mean square value of acceleration, and the differential module length of direction quaternion in the last 20 time frames. These feature values are uniformly normalized to the range of 0 to 1 before entering the network. The activation function inside the network uses a rectified linear unit, and the output layer generates a single scalar continuity score through full connection. The closer the score value is to 1, the higher the motion continuity, and the closer it is to 0, the more motion mutations or interruptions there are. The mean square error is used as the loss function during network training, and the adaptive moment estimation algorithm is used for parameter update. The initial learning rate is set to 0.001, and the gradient clipping threshold is set to 1.0. The training data comes from a labeled dataset containing multiple types of daily actions, and the performance is evaluated on an independent validation set after each training period.

[0071] The adjustment mechanism of the tracking time window is based on the dynamic calculation of the continuity score. The system sets two decision thresholds: when the continuity score exceeds 0.75, it is judged as a high continuity state, and a 1.0 second large window is enabled; when the score is less than 0.25, a 0.2 second small window is started; in the intermediate state, the window length is calculated linearly, continuously changing between 0.2 and 1.0 seconds. The actual application of the window uses an overlapping time slicing strategy, with a 50% overlap between the end time of the current window and the start time of the next window. The window position slides along the time axis, with a sliding step equal to 50% of the window length. The start and end timestamps of each window are recorded, which form the basic time framework for state tracking.

[0072] The calculation of the action response period is based on the actual window processing delay. The time from data preparation to analysis result output for each tracking time window is accurately measured, and the system records the time difference between the data entering the processing queue and the result returning through a high-precision timer. The system maintains a ring buffer containing the processing times of the last 10 windows, and the period value is the arithmetic mean of all time differences in the buffer. When the average processing time exceeds the preset upper limit of 10 milliseconds, the system automatically activates the resource optimization program, including reducing the priority of non-critical tasks and enabling the simplified feature calculation mode. The final output action response period value is in milliseconds, accurate to one decimal place.

[0073] The capture of the pose transformation features relies on a dedicated convolutional architecture processor. The processor input is a sequence of continuous skeletal keypoint data, each frame containing 17 three-dimensional joint coordinates. The network structure contains five layers of two-dimensional convolution operations, the first two layers with a 3x3 convolution kernel size and 32 filters; the middle two layers with a 3x3 kernel size and 64 filters; and the last layer with a 1x1 convolution and 16 output channels. All convolution layers are followed by batch normalization and parameterized modification units. The network output is not the coordinates themselves, but the rate of change of the angle between the keypoint vectors. Specifically, the system selects the shoulder-elbow-wrist, hip-knee-ankle, and other key limb triangular regions, calculates the spatial angle change of each connecting vector between adjacent frames in the joint coordinate system, and outputs a time sequence of 12 sets of joint angle change features.

[0074] The extraction process of the core behavior pattern uses an improved clustering analysis method. The input pose transformation feature sequence is first standardized, and then compressed to a five-dimensional feature space using principal component dimensionality reduction technology. The clustering process uses a density-based spatial clustering algorithm, with a neighborhood radius of 0.3 in the Euclidean distance of the feature space and a minimum sample size of 15 consecutive action frames. The algorithm automatically identifies the dense areas in the action segment and outputs the number of recognized core behavior patterns and their feature barycenter positions. Each recognized core pattern contains the mean vector and covariance matrix description of the standardized pose features. The behavior pattern information during the execution of the target action is constructed as a feature space distance sequence from the current action frame to the nearest mode barycenter point, which is smoothed by a 0.1 second wide moving average and then output.

[0075] The environmental interference data processing flow has multi-source spatio-temporal alignment capability. The point cloud data generated by the millimeter wave radar is refreshed at a period of 40 milliseconds, and the depth map of the depth camera is updated at a period of 33 milliseconds. The system ensures that every fourth frame of radar data and the fifth frame of depth map have a common time reference point through hardware synchronization signals. The conversion of the radar coordinate system to the global coordinate system uses an extrinsic parameter matrix transformation, which is determined during the initial calibration of the system. Before point cloud fusion, spatial filtering is performed to remove detection points more than 10 meters away from the target object. The point cloud conversion of the depth map uses a pinhole camera model to calculate the inverse projection, and each pixel generates a corresponding three-dimensional point coordinate according to the depth value. After the calculation of the two kinds of point clouds is completed, the data registration uses a matching strategy based on normal vectors, which preferentially aligns the feature points in areas with high surface curvature. The final fused point cloud is down-sampled to a spatial resolution of 2 centimeters through a voxel grid.

[0076] The obstacle topology modeling process continuously updates the dynamic database. Each object region detected by the spatial clustering records its minimum bounding box parameters (length, width, height), location centroid and principal direction vector. These objects are classified into three categories: static structures (such as walls), quasi-static objects (such as furniture) and dynamic obstacles (such as moving human bodies). Each object instance is assigned a unique identifier and a topology node data record is created, which includes the current bounding box size, location coordinates and direction angle information. The topology edges between nodes store properties such as the distance between the centroids of the two objects, the included angle of the principal directions and the spatial connectivity state. The system establishes an object motion predictor to calculate the displacement vector of objects of the dynamic obstacle type between two consecutive frames, thereby predicting the possible location interval in the subsequent time period (0.5 seconds). All environment modeling data is updated at a frequency of 30 frames per second, and the complete spatial topology graph is reconstructed after each update.

[0077] Example 3: refer to Figure 3 The construction of the spatio-temporal correlation model adopts a graph-based network processing architecture. The network accepts two types of input feature vectors: the behavior pattern information vector of the current action segment (dimension 64) and the representative feature vector of the historical abnormal action database (dimension 64). The network structure contains two consecutive feature transformation stages: the first stage applies graph convolution operations to process the spatial relationships between action segments, and the adjacency matrix is initialized according to the anatomical connections of the joint nodes; the second stage performs cross-time feature aggregation, and the processed features of the last 10 frames are fed into the graph pooling layer. The graph convolution layer is configured with 32 spatial filters, with a kernel size of 3x3 and a sliding step size of 1; the pooling layer uses the maximum pooling strategy, and the output is compressed to 1 / 4 of the dimension. The final fully connected layer of the network produces a spatio-temporal correlation matrix, which is a 16x16 real symmetric positive definite matrix, recording the correlation strength between all input feature vectors.

[0078] The calculation of feature fusion strength is derived from the spectral analysis process of the correlation matrix. The system performs eigenvalue decomposition on the generated correlation matrix, and the orthogonal basis composed of the eigenvectors is taken as the principal direction of the discriminant space. The normalized result of the largest eigenvalue is taken as the core index of the feature fusion strength:

[0079] ;

[0080] In this formula, the symbol represents the largest eigenvalue in the correlation matrix, is the th eigenvalue of the matrix. The calculation result ranges from 0 to 1, and the higher the value, the stronger the feature fusion effectiveness. In the real-time processing process, the system independently generates a feature fusion strength value for each new action segment.

[0081] The determination of dynamic partition boundary relies on adaptive threshold mechanism. The basic boundary threshold is initially set to 0.5, and the actual working threshold is adjusted in real time according to the feature fusion strength. The adjustment rule establishes three-level condition judgment: when , the boundary threshold is automatically increased by 20%; when , the boundary threshold is correspondingly decreased by 15%; and when in the intermediate interval, the basic threshold is maintained unchanged. The boundary value is directly used to divide the attribution area of the action segment in the behavior mode space. Each attribution area is defined as a hypercube area in the 64-dimensional feature space, and the boundary point is determined by the weighted average value of the core behavior mode feature vector, and the weight coefficient is proportional to .

[0082] The extraction of spatiotemporal consistency parameter relies on the structure of bidirectional recurrent neural network. The input of the network is the spatiotemporal constraint feature vector sequence of the action sequence (sequence length 20 frames, feature dimension 48 per frame), and the output is the spatiotemporal consistency score of each action frame. The network uses a gated recurrent unit to implement, and sets two processing directions: the forward layer processes from the 1st frame to the 20th frame, and the reverse layer backtracks from the 20th frame to the 1st frame. The hidden layer outputs of the two directions are fused at the 10th cycle unit, and finally the consistency score is output through the fully connected layer (value range-1 to 1). The network is configured with 128 hidden units, and the hidden state vector is transmitted after layer normalization processing.

[0083] The coupling strength analysis of environmental interference factors is realized by multiple sensor feature regression. The input data includes environmental light intensity sequence (time resolution 0.1 seconds), noise spectrum energy distribution (24 frequency bands), temperature gradient change vector (size 3x3), and adjacency matrix of obstacle topology graph (40x40). Standardization processing is adopted in the data preprocessing stage, and all features are connected into a unified vector (total dimension 1,728). The linear regression model includes a coefficient vector β (dimension 1,728) and a bias term, which is fitted to the interference intensity label data by least squares method. The final coupling strength is calculated as:

[0084] ;

[0085] In the formula, the symbol represents the preprocessed environmental feature vector; the symbol is the regression coefficient vector; and the symbol is the constant bias term of the regression model. The model parameter pre-training stage collects typical scene samples of environmental interference, including strong light mutation, instantaneous impact noise, temperature sudden change and obstacle position mutation.

[0086] The synthesis strategy of abnormal behavior level adopts a two-factor weighted model. The spatiotemporal consistency parameter produced by the recurrent network and the environmental coupling strength As an input factor. The rank synthesis formula adopts a nonlinear transformation structure:

[0087] ;

[0088] In the formula, the symbol represents the synthesized abnormal behavior rank, with a value range of [-1, 1]; the symbols are weighting coefficients (preset values = 0.6, = 0.4); the symbol represents a hyperbolic tangent activation function. The final rank is discretized and output according to the value: when , it is a severe abnormality rank, , it is a moderate abnormality rank, , it is a mild abnormality rank, , it is determined as normal behavior. All operations are performed in a dedicated digital signal processor, and the single-frame calculation delay does not exceed 3 milliseconds.

[0089] The historical abnormal action library adopts a dynamic cyclic update mechanism. The storage capacity in the library is set to 200 action segment feature vectors, and each segment contains continuous 20 frames of data. The storage standard is to trigger automatic collection when the detected abnormal behavior rank reaches moderate or above. When the storage reaches the upper limit, the new abnormal action data replaces the earliest stored segment in chronological order. The abnormal feature vector calculation module is executed every time it is triggered: a sequence of 20-frame behavior pattern information vectors is extracted from the action stream, and is compressed into a single 64-dimensional feature vector through a two-layer convolutional network (configured with 32 filters). The feature vector is stored in association with the storage timestamp, forming a closed-loop self-optimization mechanism.

[0090] Example 4: refer to Figure 4 , the dynamic adjustment process of the sampling density is based on the real-time monitoring results of the action response period. The system maintains a ring buffer with a length of 10, recording the measurement values of the last 10 action response periods. When more than 60% of the period values in the buffer are lower than 20 milliseconds, the system sets the sampling density to 50 hertz; when more than 60% of the period values are higher than 20 milliseconds, the sampling density is linearly reduced to 30 hertz. The adjustment of the sampling density adopts a smooth transition strategy, with a step size of no more than 5 hertz each time, avoiding data discontinuity caused by sudden changes in the sampling rate. In the 50-hertz sampling mode, the system collects a complete set of motion trajectory data every 20 milliseconds; in the 30-hertz mode, the collection interval is extended to 33 milliseconds. The sampling time is triggered by a high-precision hardware timer, with a time error controlled within ±0.1 milliseconds.

[0091] The calculation process of the confidence offset involves historical data analysis of abnormal behavior levels. The system maintains a sliding window of length 5, storing the last 5 abnormal behavior level assessment results. Each level is converted to a standard score according to the numerical mapping rule: severe abnormality corresponds to -1.0 points, moderate abnormality corresponds to -0.5 points, mild abnormality corresponds to 0 points, and normal behavior corresponds to +0.5 points. The offset calculation uses an exponential weighted moving average algorithm, with the latest data weight set to 0.3 and the historical data weight decay coefficient set to 0.7. The calculation result ranges between -1.0 and +1.0, with negative values indicating reduced recognition result reliability and positive values indicating improved reliability. The system sets three offset interval thresholds: high priority correction is triggered when the offset is below -0.5; regular correction is performed between -0.5 and 0.2; and only monitoring is performed when the offset is above 0.2 without active correction.

[0092] The architecture of the lightweight deep learning model is based on the improved MobileNetV3 structure. The input layer accepts two parallel data streams: sampling density features (dimension 1) and confidence offset (dimension 1), which are expanded to 16 dimensions through respective fully connected layers and then concatenated into 32-dimensional combined features. The network body contains 12 inverted residual blocks, each with an expansion ratio of 4 and a dynamic adjustment of the number of intermediate channels between 16 and 96. The final classification head contains two branches: the action category prediction branch outputs the probability distribution of 20 common action categories; the recognition quality evaluation branch outputs the reliability score of the current recognition result (0-1 range). The model parameter amount is controlled within 1.2MB, and the single inference time on an embedded processor does not exceed 8 milliseconds.

[0093] The spatiotemporal unit segmentation of the action sequence uses a fixed frame number strategy. Each unit contains 10 consecutive frames of motion trajectory data, with a 2-frame overlap between adjacent units to ensure temporal continuity. The unit boundary is strictly aligned with the system clock's 10-millisecond time, ensuring consistency of time markers. The missing data processing uses a three-stage recovery mechanism: when the number of missing frames does not exceed 3, use the cubic spline interpolation algorithm to complete; when 4-6 frames are missing, enable template matching filling based on action type; when more than 6 frames are missing, directly mark the unit as invalid and trigger reacquisition. The data structure output by each valid unit contains 10 sets of timestamp-aligned displacement coordinates, velocity vectors, acceleration values, and direction quaternions.

[0094] The extraction process of multi-scale feature vectors adopts a pyramid processing architecture. The base feature extraction layer processes individual spatio-temporal units, outputting 32-dimensional motion trajectory features and 16-dimensional skeletal keypoint features. The first scale processes the original sampling rate data directly to calculate statistical features (mean, variance, extreme value); the second scale calculates dynamic change rate features after downsampling the data to 50%; and the third scale extracts frequency energy features after Gaussian smoothing processing. The feature vectors of the three scales are finally concatenated into a 120-dimensional comprehensive description vector, which is fed into the subsequent continuity analysis module.

[0095] The generation of motion continuity indicators is based on the distribution similarity analysis of multi-scale feature vectors. The system maintains a reference feature library to store the standard feature distribution of typical continuous actions. The matching process of the current feature vector with the reference library uses Mahalanobis distance measurement, and the distance value is converted to a continuity score in the range of 0-1 through a sigmoid function. The score calculation considers three key factors: the consistency of adjacent frame feature changes (weight 40%), the repeatability of action cycles (weight 30%), and the stability after environmental interference compensation (weight 30%). The score update frequency is synchronized with the spatio-temporal unit segmentation, and a new continuity indicator value is generated for each new unit.

[0096] Table 1: Typical data sampling parameters recorded during system operation.

[0097]

[0098] The execution flow of the dynamic correction mechanism includes three parallel processing threads. The main thread continuously monitors the change trend of the recognition sampling density and confidence offset, and sends an adjustment request to the correction decision module when the parameters exceed the preset range. The second thread is responsible for maintaining the knowledge distillation process of the lightweight model, and regularly updates part of the model weights with the latest collected high-quality action samples. The third thread executes background environment monitoring, aligns the timestamps of interference events such as light mutations and noise peaks with the action data stream, and provides environmental context reference for correction decisions. All correction operations are recorded with detailed log information, including the recognition results before correction, the parameter conditions that trigger correction, the correction strategies adopted, and the output conclusions after correction.

[0099] The abnormal situation handling adopts a hierarchical response strategy. For extreme cases with a continuity score below 0.3, the system initiates a safety protocol to suspend real-time recognition and switches to a diagnostic mode to collect complete motion cycle data. Moderate abnormalities (score 0.3-0.6) trigger an enhanced sampling mode, temporarily increasing the sampling density by 20% and extending the analysis window by 50%. Mild abnormalities only generate a warning log without interrupting the normal process. All abnormal events are associated with specific time period markers, and an abnormality index is established in the data storage layer for subsequent offline analysis. The system recovery mechanism includes an automatic rollback function that automatically restores to the stable parameter configuration before the last correction when five consecutive corrections fail to improve recognition quality.

[0100] Embodiment 5: refer to Figure 5 The spatio-temporal feature fusion backbone network adopts a parallel dual-channel processing architecture. The motion trajectory data input stream is constructed as a one-dimensional convolution processing pipeline. The original input data is a four-dimensional time series tensor of three-dimensional displacement coordinates, velocity components, acceleration components, and direction quaternions. The time dimension length matches the number of frames divided by the motion response period. The first convolution layer is configured with thirty-two time filters, with a kernel length of five frames and a sliding step of one frame. The output feature map is activated by a rectified linear unit. The second convolution kernel length is reduced to three frames, and the number of filters is increased to sixty-four. The feature map is compressed in height by maximum pooling. The third convolution layer uses a single-frame kernel size, and the filter channel is expanded to one hundred and twenty-eight dimensions. The output tensor retains the complete time dimension but significantly increases the feature depth. The skeleton key point input stream is constructed as a graph convolution processing path. The input data is structured as a tensor of time x number of joints x three-dimensional coordinates. The initial graph convolution layer defines the adjacency relationship based on the preset human body skeleton connection topology. Sixty-four spatial filters are used to extract local joint group features. The second graph convolution layer expands the receptive field to the complete body range, maintains the configuration of sixty-four filters, but introduces an attention mechanism to dynamically adjust the joint connectivity weight. After completing three feature transformations respectively, the dual-stream features perform channel splicing. The fused spatio-temporal joint feature vector has the structure characteristics of complete time dimension preservation and feature depth of two hundred and fifty-six dimensions.

[0101] The dynamic pruning operation is implemented after the feature fusion is completed. The feature importance evaluation module receives the spatio-temporal joint feature vector as input, and calculates the importance score of each feature channel through the auxiliary analysis network. The auxiliary network structure contains three layers of full connection, and outputs a single channel score. The scoring algorithm is based on the statistical distribution characteristics of the channel feature on the entire data set, focusing on the quantification contribution of its kurtosis, skewness and variance characteristics to the action discrimination ability. The system sets the lower limit of the score threshold to the thirtieth percentile value, and the feature channel below the threshold is marked as redundant. The pruning process is not completely fixed, and a flexible retention mechanism is configured: when the channel score is in the critical zone, the mutual information value of the feature and the current environmental sensing information is combined for secondary judgment. After completing the channel screening, feature dimension compression is performed, and principal component analysis technology is used to maintain thirty percent of the original information, and finally a lightweight feature tensor with a dimension of seventy-seven is generated.

[0102] The cross-modal attention weighting module realizes the associated fusion of environmental information and motion features. The multi-modal sensing information preprocessing is divided into four parallel paths: the ambient light intensity sequence is constructed into a two-dimensional feature of time x intensity; the background noise spectrum is converted into a twenty-four-dimensional frequency band energy curve; the temperature field distribution data is extracted through the spatial gradient operator to obtain three direction differential features; and the obstacle topology information is processed into an adjacent matrix feature vector. The four features are respectively transformed into one hundred and twenty-eight-dimensional feature representations through independent embedding sub-networks. The attention mechanism query vector is derived from the lightweight feature tensor, and the key vector and value vector are derived from the splicing results of the multi-modal embedding features. The attention head structure is configured with eight independent units, and the query, key and value vector dimensions of each head are set to sixty-four. Each head generates an independent attention weight distribution, which is realized through scaled dot product calculation. The outputs of the eight heads are spliced in the feature dimension and mapped through the projection matrix to generate a two hundred and fifty-six-dimensional feature matrix fused with environmental constraints. The weight calculation is updated in real time, and the attention distribution pattern is regenerated for each action recognition request.

[0103] The multi-task joint optimization framework establishes a decoupled structure of shared representation and task-specific branches. The multi-modal fusion feature matrix is input into the shared feature extraction module, which is composed of three consecutive separate convolution blocks, each block is configured with an expansion ratio of four and a bottleneck dimension of thirty-two. After global average pooling in the time dimension, the feature map enters the double-branch output structure: the action response cycle prediction branch is a single neuron output regression model, and the loss function adopts the form of smooth L1 loss; the abnormal behavior level classification branch is a four-dimensional probability output layer, corresponding to four categories of normal behavior, mild abnormality, moderate abnormality and severe abnormality, and the loss function uses a cross-entropy weighting strategy. The multi-task loss fusion strategy adopts a dynamic weight coefficient mechanism, with an initial cycle prediction loss weight of 0.4 and a classification loss weight of 0.6. The system automatically evaluates the task loss ratio every fifty action samples, and when the loss reduction of a certain task is less than five thousandths for five consecutive times, the loss weight of this task is increased by 0.05 while balancing the weight of the other task. Model parameter updating is performed on an embedded graphics processor using a stochastic optimization algorithm with momentum, with a momentum factor of 0.9. The initial learning rate is set to 0.1, and it is decayed to 95% of the previous value every 100 iterations. The complete model training is completed offline in the system initialization stage, and the trained parameters are fixed in the memory. In the real-time recognition stage, only the forward calculation process is performed.

[0104] The model parameter fixation adopts a modular deployment scheme. The spatiotemporal feature fusion backbone network is compiled into a separate processing unit, the dynamic pruning operation is implemented as a configurable plug-in, and the attention weighting module is packaged as a multi-thread callable service. The system loads in the initialization sequence when starting: first load the sensor driver interface, then initialize the feature extraction module parameters, and finally activate the multi-task prediction components. During operation, three ring-shaped data buffers are maintained: the sensor raw data buffer contains the last two seconds of data stream; the feature tensor intermediate storage area retains the output of eight consecutive inference cycles; the decision history record area stores the last thirty recognition results. The service exception handling module monitors the running delay of each component, and automatically switches to the backup calculation path when the feature fusion stage exceeds ten milliseconds—this path skips the third convolution layer to reduce the calculation amount by 35%. The system maintains a self-checking log, which records in detail the model reconfiguration events, parameter adjustment process and component running state change history.

[0105] It should be noted that, in this text, relational terms such as first and second are used merely to distinguish one entity or action from another, without necessarily requiring or implying any such actual relationship or order between such entities or actions. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0106] While embodiments of the application have been shown and described, it is to be understood that the application is not limited to the details of the embodiments described, since numerous changes, modifications, substitutions and variations can be made thereto without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A multi-modal real-time action recognition method based on lightweight deep learning, characterized in that, The method comprises the following steps: Synchronously collecting motion trajectory data and skeletal key point sequences of a target object from multi-source sensing devices, and acquiring multi-modal sensing information of an environment state in real time; Aligning the spatio-temporal features of the motion trajectory data to generate a motion continuity index, and determining a motion response period for state tracking in a motion sequence of the target object according to the motion continuity index; Extracting behavior pattern information in a target motion execution process, analyzing multi-scale features of the behavior pattern information to obtain spatio-temporal constraint features in the motion execution, and determining an abnormal behavior level in a motion recognition process according to the spatio-temporal constraint features and the multi-modal sensing information; Dynamically correcting a recognition blind area in a real-time motion stream based on the motion response period and the abnormal behavior level; The multi-scale feature analysis of the behavior pattern information specifically comprises: Constructing a spatio-temporal association model of motion segments and historical abnormal motions based on the behavior pattern information; Outputting a feature fusion strength in the motion execution through the spatio-temporal association model; Determining dynamic partition boundaries of each motion segment in the behavior pattern according to the feature fusion strength; Generating spatio-temporal constraint features in the motion execution through the dynamic partition boundaries; The determination of the abnormal behavior level in the motion recognition process according to the spatio-temporal constraint features and the multi-modal sensing information specifically comprises: Extracting a spatio-temporal consistency parameter of the motion sequence according to the spatio-temporal constraint features; Calculating a coupling strength of an environmental interference factor through the multi-modal sensing information; Generating an abnormal behavior level in the motion recognition process based on the spatio-temporal consistency parameter and the coupling strength of the environmental interference factor.

2. The multi-modal real-time action recognition method based on lightweight deep learning according to claim 1, characterized in that, The motion trajectory data includes time series data of displacement coordinates, motion speed, acceleration and motion direction of the target object in a three-dimensional space. 3.The method of claim 1, wherein The multi-modal sensing information includes environmental light intensity, background noise spectrum, temperature field distribution and spatial obstacle topology information. 4.The method of claim 1, wherein The determination of the motion response period for state tracking in the motion sequence of the target object according to the motion continuity index specifically comprises: Constructing a spatio-temporal continuity model of the motion sequence according to the motion continuity index; Outputting a tracking time window of real-time motion tracking from the spatio-temporal continuity model; Determining the motion response period for state tracking in the motion sequence of the target object through the tracking time window. 5.The multi-modal real-time action recognition method based on lightweight deep learning according to claim 1, wherein, The extraction of the behavior pattern information in the target motion execution process specifically comprises: Real-time capturing of posture transformation features of motion segments; Generating core behavior patterns of motion execution according to the posture transformation features; Determining the behavior pattern information in the target motion execution process through the core behavior patterns. 6.The method of claim 1, wherein The dynamic correction of the recognition blind area in the real-time motion stream based on the motion response period and the abnormal behavior level specifically comprises: Determining a recognition sampling density of the real-time motion stream according to the motion response period; Calculating a confidence offset of the recognition blind area according to the abnormal behavior level; Inputting the recognition sampling density and the confidence offset into a lightweight deep learning model to generate a dynamically updated motion recognition decision tree; The spatio-temporal feature alignment of the motion trajectory data specifically comprises: According to the motion trajectory data, a space-time unit of action sequence is segmented; Interpolation compensation is performed on the missing space-time unit to generate a complete action state sequence; A multi-scale feature vector of the action state sequence is extracted; A motion continuity index is generated through the multi-scale feature vector.

7. The lightweight deep learning based multi-modal real-time action recognition method of claim 6, wherein, The construction method of the lightweight deep learning model includes: The motion trajectory data and the sequence of skeletal key points are input into a space-time feature fusion backbone network to generate a space-time joint feature vector; A dynamic pruning operation is performed on the space-time joint feature vector to remove redundant feature channels and compress the feature dimension, generating a pruned lightweight feature tensor; The pruned lightweight feature tensor is cross-modality attention weighted using the multi-modal sensing information to generate a multi-modal fusion feature matrix; The multi-modal fusion feature matrix is multi-task jointly optimized through the action response period and the abnormal behavior level, and the lightweight deep learning model parameters are output.

8. A lightweight deep learning based multi-modal real-time action recognition system for performing a lightweight deep learning based multi-modal real-time action recognition method according to any one of claims 1 to 7, characterized in that, It includes: A multi-source sensing module is used to synchronously collect motion trajectory data and skeletal key point sequences of a target object from a multi-source sensing device, and to real-time acquire multi-modal sensing information of an environment state; A feature fusion module is used to align the space-time features of the motion trajectory data, generate a motion continuity index, and determine an action response period for state tracking of the target object in the action sequence according to the motion continuity index; A behavior analysis module is used to extract behavior pattern information in the execution process of the target action, to perform multi-scale feature analysis on the behavior pattern information, to obtain space-time constraint features during action execution, and to determine an abnormal behavior level in the action recognition process according to the space-time constraint features and the multi-modal sensing information; A dynamic correction module is used to dynamically correct a recognition blind area in a real-time action stream based on the action response period and the abnormal behavior level.

Images