Mobile terminal real-time action monitoring method and system

Abstract

The invention discloses a mobile terminal real-time action monitoring method and system. The method belongs to the technical field of neural network target monitoring. The problem that the application effect of an existing mobile terminal motion monitoring model in real-time scenes such as monitoring APP and AR interaction is seriously restricted due to the fact that the existing mobile terminal motion monitoring model is difficult to balance the monitoring precision, the model size and the reasoning speed is solved. A complete data set construction process of data collection, frame extraction, standardized preprocessing, multi-dimensional enhancement and double-input construction is provided, and the generalization ability of the model is remarkably improved; a 3D convolution and ResNet fused lightweight model is designed, the action spatial-temporal feature capture capability and the mobile terminal deployment requirement are considered, and the model parameter scale is only 60% of that of a traditional 3D CNN model; through double-input data construction and a result optimization mechanism, the monitoring accuracy of the model in a real scene is not lower than 84%, and the actual application requirement is met.

Description

Technical Field

[0001] This invention belongs to the field of neural network target monitoring technology, specifically relating to a real-time motion monitoring method and system for mobile devices. Background Technology

[0002] In the field of mobile motion detection technology, there are fundamental differences between mobile and non-mobile motion detection. Non-mobile devices typically have sufficient computing resources, storage capacity, and continuous power supply, which can support the high computing power consumption requirements of complex models. However, mobile devices are limited by hardware size, battery life, and heat dissipation performance, and are subject to core constraints such as limited computing power, tight storage resources, and strict power consumption control. This requires mobile motion detection models to be lightweight and have low latency while ensuring detection accuracy.

[0003] Currently, the mainstream implementation schemes for mobile motion detection models are still mainly based on pure 2D convolution or simple 3D convolution structures. For example, the paper "Factorized 3D-CNN for Real-Time FallDetection and Action Recognition on Embedded System" published in IEEE Xplore 2024 (authors N Noor and IK Park) points out that in existing motion detection models for embedded / mobile devices, simple 3D convolution structures such as C3D and R(2+1)D account for more than 60%. These models capture spatiotemporal features by directly expanding the temporal dimension of the convolution kernel, but this leads to a surge in parameter size.

[0004] However, these structures have significant drawbacks: pure 2D convolutions can only capture spatial features and cannot effectively correlate temporal dependencies between consecutive frames, resulting in the loss of dynamic temporal information about actions; while simple 3D convolutions take into account both spatiotemporal dimensions, this is achieved by simply expanding the kernel dimension, causing a surge in model parameter size and increasing the computational burden on mobile devices. In practical applications, models using simple 3D convolutional structures such as C3D and R(2+1)D require more than 140ms for inference on 16 frames of video on mobile devices, which is insufficient to meet the needs of real-time interactive scenarios; while pure 2D convolutional models are prone to misjudgments in complex motion monitoring due to incomplete capture of spatiotemporal features. These problems make it difficult for existing mobile motion monitoring models to balance monitoring accuracy, model size, and inference speed, severely limiting their application effectiveness in real-time scenarios such as monitoring apps and AR interactions. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a real-time motion monitoring method for mobile devices. This method aims to solve the problem that existing mobile motion monitoring models struggle to balance monitoring accuracy, model size, and inference speed, severely limiting their application effectiveness in real-time scenarios such as monitoring apps and AR interactions.

[0006] The method includes the following steps: S1. Construct a motion monitoring dataset, perform data preprocessing and enhancement on the original motion videos, and generate a standardized training dataset; S2. Construct a motion detection model that integrates 3D convolution and ResNet, and complete the model training and optimization using the standardized training dataset. S3. Convert the trained motion detection model into ONNX format, optimize the model input / output interface and computation graph structure, and generate an ONNX motion detection model adapted for mobile devices. S4. Collect video streams via mobile devices, extract frames, manage caches and preprocess the video streams to generate tensor data that meets the input requirements of the ONNX motion monitoring model; S5. Input the tensor data into the preloaded ONNX motion monitoring model, and complete the motion feature extraction and classification through multi-threaded inference and hardware acceleration strategies to obtain the original motion monitoring results. S6. Optimize the original monitoring results by using confidence filtering and context consistency verification mechanisms, and output the final action monitoring results.

[0007] Furthermore, when constructing the motion monitoring dataset, continuous frames are extracted from the original motion video, and the extracted video frames are subjected to data augmentation processing. Five key frames are extracted from each 30-frame continuous video segment according to the uniform sampling rule to construct a dual input data pair of 30 complete video frames + 5 key frames.

[0008] Furthermore, a motion monitoring model is built based on the PyTorch framework. The structure of the motion monitoring model is as follows: Input layer: Receives two input tensors, namely a video frame tensor and a keyframe tensor; Feature extraction backbone layer: The first stage consists of two 3D convolutional layers, each followed by a BatchNorm3d layer and a ReLU activation function, used to initially extract the spatiotemporal features of the action; The second stage consists of three ResNet residual blocks, each containing two 3D convolutional layers. Residual connections are used to alleviate the gradient vanishing problem and enhance the ability to extract deep features. The third stage: global average pooling layer, which pools the feature map into a high-dimensional feature vector; The classification head consists of two fully connected layers and a Softmax activation function, which realizes the mapping of high-dimensional features to action categories and probability normalization.

[0009] Furthermore, when converting the trained motion monitoring model to ONNX format, the PyTorch torch.onnx.export function is called to export the model, and the optimization tools provided by ONNX Runtime are used to optimize the computation graph of the exported ONNX format motion monitoring model.

[0010] Furthermore, the cache management in step S4 is as follows: a circular buffer is used to store the acquired video frames, and the buffer capacity is set to N frames. When the number of frames in the buffer reaches N frames, the earliest frame is removed and a new acquired frame is added according to the first-in-first-out principle to ensure that the buffer always maintains a continuous N frames, providing stable input data for model inference. When the number of frames in the buffer is less than N frames, the last frame is copied to supplement the N frames to avoid inference failure due to insufficient frame quantity.

[0011] Furthermore, the multi-threaded inference involves setting the number of mobile CPU cores to 2 to balance the overhead of multi-operator parallelism and thread scheduling. The hardware acceleration strategy involves detecting whether the mobile device supports GPU acceleration. If it does, GPU inference mode is enabled, the GPU device ID is configured as 0, and the model inference computation task is assigned to the GPU for execution. If GPU acceleration is not supported, it automatically downgrades to CPU inference mode.

[0012] Furthermore, the confidence level screening specifically involves setting a confidence level threshold. If the confidence probability of the original monitoring results is lower than This indicates that the model has low confidence in the result; in this case, the monitoring result should not be updated, and the previous valid monitoring result should be retained. If the confidence probability is not lower than [a certain value], the model should be retained. If so, then the result will be stored temporarily. The context consistency verification mechanism specifically involves maintaining a historical queue of the five most recent monitoring results. ,in Indicates the first Based on the temporary monitoring results, calculate the consistency score between the current temporary result and the results in the historical queue. Set consistency threshold ,like This indicates a significant difference between the current result and historical results, potentially indicating a false alarm. In this case, a second inference verification is triggered, re-inputting the current tensor data into the model for inference. The result with the higher confidence probability from the two inference results is taken as the final temporary result. If so, the current temporary result will be retained directly.

[0013] Another aspect of the present invention provides a real-time motion monitoring system for mobile devices, comprising: This module is used to build motion monitoring datasets, preprocess and enhance raw motion videos, and generate standardized training datasets. This module is used to build a motion detection model that integrates 3D convolution and ResNet, and completes model training and optimization using the standardized training dataset. This module is used to convert trained motion detection models into ONNX format, optimize model input / output interfaces and computation graph structure, and generate mobile-compatible ONNX motion detection models. This module is used to capture video streams via mobile devices, extract frames, manage caches and preprocess the video streams, and generate tensor data that meets the input requirements of the ONNX motion monitoring model. This module is used to input the tensor data into the preloaded ONNX motion monitoring model, and to complete motion feature extraction and classification through multi-threaded inference and hardware acceleration strategies to obtain the original motion monitoring results. This module is used to optimize the original monitoring results using confidence filtering and context consistency verification mechanisms, and output the final action monitoring results.

[0014] The beneficial effects of the method described in this invention are as follows: A complete dataset construction process of "data collection - frame extraction - standardized preprocessing - multi-dimensional enhancement - dual-input construction" is proposed, which significantly improves the model's generalization ability. A lightweight model that integrates 3D convolution and ResNet is designed to balance the ability to capture spatiotemporal features of actions with the needs of mobile deployment. The model's parameter scale is only 60% of that of traditional 3D CNN models. Through dual-input data construction and result optimization mechanisms, the model's monitoring accuracy in real-world scenarios is no less than 84%, meeting the needs of practical applications.

[0015] The method described in this invention is applicable to various mobile scenarios such as intelligent interaction, health monitoring, education and entertainment, and security protection, providing developers with an efficient and reliable mobile action monitoring solution. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the mobile terminal real-time motion monitoring method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the network structure of the motion monitoring model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dataset construction and preprocessing process in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process and optimization of converting a PyTorch model to ONNX in an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the complete process of mobile data preprocessing and inference in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0018] Example 1 This embodiment provides a method for real-time motion monitoring on a mobile device, such as... Figure 1 As shown, the method includes the following steps.

[0019] Step S1: Construct a motion monitoring dataset, perform data preprocessing and enhancement on the original motion videos, and generate a standardized training dataset.

[0020] The quality of the motion monitoring dataset directly determines the model's generalization ability. This step involves constructing a high-quality dataset through a standardized data processing workflow, as detailed below. Figure 3 As shown: 1. Data Collection: Collect raw video data covering 35 common movements, including body swaying while crying, head rotation, raising hands, sitting still, etc. For each action, at least 100 videos should be collected, with a video length of 2 to 5 seconds and a resolution of no less than 1920×1080. The collection scenarios include different environments such as indoors and outdoors, and the subjects collected cover people of different ages and genders to ensure the diversity of the dataset.

[0021] 2. Frame Extraction: The OpenCV library is used to extract frames from the original video at a frequency of 30 frames / second to obtain a continuous video frame sequence. Blurry frames are removed by image sharpness evaluation (using the variance method, with the sharpness threshold set to 30), and abnormal frames such as static frames are removed by the frame difference method.

[0022] 3. Standardized preprocessing: The extracted valid frames are centered and cropped to retain the square area in the center of the image (e.g., when the video resolution is 1920×1080, it is cropped to 1080×1080). Then, they are scaled to 224×224 pixels using bilinear interpolation to ensure the frame size consistency of the input model. The RGB channel order of the image is converted to BGR channel order to be consistent with the image reading format of OpenCV and the channel requirements during model training.

[0023] 4. Data Augmentation: To improve the model's generalization ability, multi-dimensional data augmentation is performed on the standardized images: Random horizontal flip: Flip the image horizontally with a 50% probability, while keeping the semantic meaning of the action unchanged; Brightness and contrast adjustment: The brightness adjustment coefficient is randomly selected from [0.8, 1.2], and the contrast adjustment offset is randomly selected from [-20, 20]. Gaussian noise addition: Adds noise to image pixel values ​​that conforms to... Gaussian noise is used to simulate image noise in real-world scenes; Random cropping: A 200×200 region is randomly cropped from the 224×224 image and then scaled back to 224×224 to enhance the model's adaptability to changes in action position.

[0024] 5. Dataset partitioning and pixel normalization: The enhanced dataset is divided into training, validation and test sets in an 8:1:1 ratio. The pixel values ​​of all images are normalized, mapping the value range from [0,255] to [0,1] to avoid unstable model training due to excessively large pixel values. The calculation method is as follows: ,in These are the normalized pixel values. The original pixel value (range 0~255).

[0025] 6. Dual-input data construction: Extract 5 keyframes (sampling indices 0, 6, 12, 18, 24) from each 30 consecutive video frames according to the uniform sampling rule to construct a dual-input data pair of "30 complete video frames + 5 keyframes". The 30 complete video frames are used to capture the temporal continuity features of the action, and the 5 keyframes are used to highlight the key pose features of the action to improve the model's monitoring accuracy.

[0026] Step S2: Construct a motion detection model that integrates 3D convolution and ResNet based on the PyTorch framework, and complete the model training and optimization using the standardized training dataset.

[0027] The motion detection model designed in this embodiment adopts a fusion structure of "3D convolution + ResNet residual blocks", such as Figure 2 As shown, this approach balances the ability to capture spatiotemporal features with the need for a lightweight model: 1. Model structure details: Input layer: Receives two input tensors, namely a video frame tensor (shape: (batch_size, 3, 30, 224, 224)) and a keyframe tensor (shape: (batch_size, 3, 5, 224, 224)). Feature extraction backbone layer: The first stage consists of two 3D convolutional layers (3×3×3 kernel size, 1×2×2 stride, 1×1×1 padding), followed by a BatchNorm3d layer and a ReLU activation function after each convolutional layer, used to initially extract the spatiotemporal features of the action; The second stage consists of three ResNet residual blocks, each containing two 3D convolutional layers (3×3×3 kernel size, 1×1×1 stride, and 1×1×1 padding). The residual connections alleviate the gradient vanishing problem and enhance the deep feature extraction capability. The third stage: global average pooling layer, which pools the feature map from (batch_size, 256, 8, 7, 7) to a high-dimensional feature vector of (batch_size, 256); The calculation expression for a 3D convolutional layer is: ; in, Output feature maps of 3D convolutional layers in channels Time step ,high ,width eigenvalues ​​at that location Input the number of channels. The size of the convolution kernel is the time dimension. , The size of the convolution kernel is the spatial dimension. These are the convolution kernel weight parameters. The input feature map contains the feature values ​​at the corresponding positions. These are bias parameters; The calculation expression for the ResNet residual block is: ; ; in, For residual mapping, It is the ReLU activation function. , These are the weight parameters of the convolutional layers in the residual block. , For bias parameters, This represents the output characteristics of the residual block.

[0028] The core differences and technical logic between the model structure constructed in this embodiment and existing technologies are as follows: First, targeted solutions to the shortcomings of existing technologies: Existing mobile motion detection models are constrained by hardware resources and suffer from two core defects: First, pure 2D convolutional models can only extract single-frame image features through spatial convolution, completely ignoring the temporal dependencies between consecutive frames. This leads to misjudgments of spatiotemporally different actions such as "raising a hand" versus "waving a hand," and "walking" versus "running," resulting in a detection accuracy of less than 80% on the UCF101 dataset. Second, simple 3D convolutional models (such as C3D and R(2+1)D) extract spatiotemporal features by directly extending the temporal dimension of the convolutional kernel. Although they can capture temporal information, the number of parameters grows exponentially (the basic C3D model has more than 13 million parameters), and the inference time on low-end and mid-range mobile devices exceeds 140ms / 16 frames, which cannot meet the real-time interaction requirements.

[0029] The fusion structure of "3D convolution + ResNet residual block" in this embodiment precisely overcomes the above-mentioned technical bottlenecks through dual collaborative design: (1) Optimization of spatiotemporal feature capture capability: The 3D convolution kernel adopts a 3×3×3 three-dimensional structure (3 in time dimension and 3×3 in space dimension). Through single convolution operation, the three-dimensional information of "time-height-width" is synchronously associated, and the spatiotemporal coupling features of the action are directly extracted. Compared with the limitation of pure 2D convolution which can only capture static spatial features, this design can effectively associate the action changes between consecutive frames (such as the temporal process of raising an arm), ensuring that the dynamic information of the action is not lost. (2) Lightweight model optimization: ResNet residual blocks solve the gradient vanishing problem in deep networks without increasing the complexity of additional parameters by using the superposition structure of "identity mapping (X) + residual mapping (F (X))". This allows the model to maintain deep feature extraction capabilities with only 6 million parameters (60% of traditional 3D CNNs). Compared with simple 3D convolutional models, the number of parameters is reduced by 40%, and the inference time on mobile devices is reduced by more than 35%, perfectly balancing the needs of spatiotemporal feature capture and lightweight design.

[0030] Secondly, the technical problems and solutions related to computational latency: Although the "3D convolution + ResNet residual block" structure has addressed the core shortcomings of existing technologies, in order to further reduce computational latency in real-time inference scenarios on mobile devices, this embodiment fixes the time dimension of the 3D convolution kernel to 3 (increasing the time dimension to 5 or 7 would increase inference time by 30% to 50%, exceeding the real-time threshold of mobile devices). This introduces a new technical problem: key pose features in long temporal action sequences (30 frames) are easily diluted by redundant frames, resulting in insufficient recognition of core action features by the model.

[0031] Specifically, 30 consecutive frames of video contain a large number of motion transition frames (such as the arm raising transition frame in the "raising hand" action, and the body leaning forward transition frame in the "bending over" action). When the features of these transition frames are mixed with key pose frames (such as the arm fully raised frame and the body fully bent frame), the spatiotemporal features extracted by 3D convolution will be biased towards "averaging", weakening the uniqueness of the core pose. Especially when the action speed is fast (such as waving quickly) or the background is complex (such as outdoor crowd scenes), this problem will cause the monitoring accuracy to drop by 5% to 8%, becoming a new bottleneck restricting the performance of the model.

[0032] To accurately address the aforementioned new technical challenges, this embodiment employs a "dual-input data construction" technique in step S10, which forms a collaborative optimization loop with the "3D convolution + ResNet residual block" fusion model. (1) The role of 30 complete video frames as input: to provide complete action temporal context information for the fusion model, to ensure that 3D convolution can capture the continuous spatiotemporal changes of the action from start to end, to avoid action logic misjudgment caused by temporal breaks, and to ensure the integrity of action monitoring; (2) The role of 5 key frame inputs: Through uniform sampling rules (sampling indices 0, 6, 12, 18, 24), the five core stages of the action, namely the start, rise, peak, fall, and termination, are accurately located, and the key pose frames are separated and enhanced from the 30 consecutive frames of the sequence. This is equivalent to providing the model with "core feature anchors of the action", guiding 3D convolution and ResNet residual blocks to focus on the feature extraction of key poses, and offsetting the feature dilution problem caused by transition frames. (3) Dual input synergistic effect: During the training and inference process, the fusion model learns the temporal logical correlation of actions through 30 complete frames of input and strengthens the feature weight of core posture through 5 key frames of input. The two complement each other to form a feature extraction mechanism of "temporal integrity + key feature prominence" - ultimately improving the monitoring accuracy by 8%~10% without increasing the model parameters and inference time, completely solving the new technical problems, and further consolidating the solution effect of the original technical defects.

[0033] The classification head consists of two fully connected layers and a Softmax activation function, which maps high-dimensional features to action categories and normalizes their probabilities. The first fully connected layer maps the 256-dimensional high-dimensional feature vector to a 128-dimensional intermediate feature vector, reducing the feature dimension while retaining the core action feature information; The second fully connected layer maps the 128-dimensional intermediate feature vector to a 35-dimensional original score vector (corresponding to 35 action categories); The Softmax activation function performs probability normalization on the original 35-dimensional score vector, converting the values ​​of each dimension into probability values ​​between 0 and 1, with the sum of the probabilities of all dimensions being 1, and finally outputting a logits vector (in probability distribution form).

[0034] The mathematical expression for the Softmax activation function is: ; in, For the first Normalized probability of action class The output of the second fully connected layer Raw scores for each type of action For action category index (1~35); 2. Model training process: Initialization: The weight parameters of the convolutional and fully connected layers are initialized using the He initialization method, and the bias parameters are initialized to 0; Optimizer configuration: The AdamW optimizer is used, and the initial learning rate is set to... The weight decay coefficient is set to This is used to suppress model overfitting; Learning rate scheduling: A cosine annealing learning rate decay strategy is adopted, in which the learning rate decays periodically according to a cosine function with the number of training epochs, and the minimum learning rate is... This improves the model's convergence speed and generalization ability. Loss function: The cross-entropy loss function is used to calculate the difference between the model's prediction and the true label. The expression for the loss function is as follows: ,in For true labels in one-hot encoded format, Predict probabilities for the model.

[0035] Training and Early Stopping: The training set is trained iteratively in batches of 16. After each training round, the model accuracy is evaluated using the validation set. When the validation set accuracy does not improve for 10 consecutive rounds, the early stopping mechanism is triggered to save the current optimal model weights and avoid model overfitting.

[0036] 3. Model Performance Evaluation: The optimal model is evaluated using a test set. Evaluation metrics include accuracy, recall, F1 score, and confusion matrix. The model is guaranteed to achieve an average accuracy of no less than 88.5% and a recall of no less than 80% for each action on the test set, meeting the needs of actual mobile applications.

[0037] Step S3: Convert the trained PyTorch model to ONNX format, optimize the model input / output interface and computation graph structure, and generate an ONNX motion monitoring model adapted for mobile devices.

[0038] To enable mobile deployment of PyTorch models, they need to be converted to ONNX format and optimized accordingly. Specific steps are as follows: Figure 4 As shown: 1. PyTorch model preparation: Load the optimal model weights after training, build the same model structure as during training, call model.eval() to set the model to evaluation mode, and disable the training mode behavior of Dropout layer and BatchNorm3d layer; 2. Virtual Input Construction: Construct virtual input tensors with the same format as the real input. The video frame tensor has a shape of (1, 3, 30, 224, 224) (batch dimension × channel dimension × time dimension × height dimension × width dimension), and the keyframe tensor has a shape of (1, 3, 5, 224, 224). The data type is float32, which is used to determine the input and output dimensions of the model during export. 3. ONNX format export: Call the PyTorch torch.onnx.export function to export the model. Set the export parameters: the input names are "video_frames" (video frame tensors) and "key_frames" (key frame tensors), the output name is "logits", the dynamic axis is set to None (fixed input and output dimensions), the opset version is 12, and the inference logic during training is preserved during export. The specific settings are as follows: model: The PyTorch model to be exported; args: Virtual input tensor tuple (video_input, key_input); f: Output ONNX file path; input_names: A list of input node names ["video_frames", "key_frames"]; output_names: A list of output node names ["logits"]; dynamic_axes: None (fixes input and output dimensions, improving inference speed on mobile devices); opset_version: 12 (compatible with mainstream ONNX Runtime versions); do_constant_folding: True (Perform constant folding optimization on export).

[0039] 4. ONNX Model Optimization: Optimize the computational graph of the exported ONNX model using the optimization tools provided by the ONNX Runtime. Constant folding: Replace constant expressions in the model with constant values ​​to reduce the amount of computation during inference; Redundant node elimination: Remove redundant operator nodes in the model that have no practical effect; Operator fusion: Merging multiple consecutive simple operators (such as Conv+BatchNorm+ReLU) into a composite operator to reduce operator scheduling overhead; Optionally, the optimized ONNX model is subjected to INT8 quantization using a dynamic quantization strategy. This reduces the model storage size and inference computation without significantly sacrificing model accuracy (accuracy loss ≤3%). The mathematical expression for the quantization process is as follows:

[0040] in, The weights are quantized 8-bit integers. The weights are the original 32-bit floating-point numbers. , These are the minimum and maximum values ​​of the weights, respectively. Accuracy verification: After quantization, the model inference accuracy needs to be verified to ensure that the output error with the original PyTorch model does not exceed 1e-3 and the test set accuracy loss does not exceed 3%.

[0041] 5. Model Validation: Load the optimized ONNX model using ONNX Runtime, input test set data, and verify the consistency between the model inference results and the original PyTorch model to ensure that the inference logic has not been changed during the model conversion and optimization process.

[0042] Step S4: Capture video streams via mobile devices, extract frames, manage caches, and preprocess the video streams to generate tensor data that meets the input requirements of the ONNX model.

[0043] Mobile data acquisition and preprocessing must be adapted to the hardware characteristics of mobile devices to ensure processing efficiency and data format correctness. Mobile data acquisition and preprocessing operations include... Figure 5 As shown in (a): 1. Video Stream Acquisition: The video stream is acquired using the native camera framework of the mobile device (CameraX on the Android platform and AVFoundation on the iOS platform). The acquisition resolution is set to 1920×1080, the acquisition frame rate is 30 frames / second, the front camera is selected first to meet the needs of selfie scenarios, and video frame data is obtained in real time through camera preview callback during the acquisition process.

[0044] 2. Frame Buffer Management: A circular buffer (CircularFifoQueue) is used to store the acquired video frames. The buffer capacity is set to 30 frames. When the number of frames in the buffer reaches 30, the oldest frame is removed and a new acquired frame is added according to the first-in-first-out principle to ensure that the buffer always maintains 30 consecutive frames, providing stable input data for model inference. When the number of frames in the buffer is less than 30, the last frame is copied to make up to 30 frames to avoid inference failure due to insufficient frame quantity.

[0045] 3. Mobile Data Preprocessing: Considering the hardware performance limitations of mobile devices, the preprocessing workflow needs to balance efficiency and accuracy. Size normalization: Each video frame is centered and cropped, retaining a square area and then scaled to 224×224 pixels. The image scaling interface with mobile hardware acceleration is used to improve processing speed. Channel conversion: Images captured by mobile cameras are mostly in ARGB format, which needs to be converted to BGR format. The conversion process is implemented through bitwise operations to avoid the overhead of complex image format conversion. Pixel value normalization: Mapping pixel values ​​from [0,255] to [0,1] is achieved through a simple division operation, without the need for complex calculations; Tensor Reassembly: 30 frames of data are reassembled in the order of "channel-time-height-width" (C×T×H×W) to generate a video tensor (shape: (1, 3, 30, 224, 224)). Five keyframes are extracted to generate a keyframe tensor (shape: (1,3, 5, 224, 224)). The sampling indices are 0, 6, 12, 18, and 24. The tensor reassembly process is implemented by array copying to avoid frequent memory allocation.

[0046] Step S5: Input the tensor data into the preloaded ONNX motion monitoring model, and complete the motion feature extraction and classification through multi-threaded inference and hardware acceleration strategies to obtain the original motion monitoring results.

[0047] Mobile ONNX model inference requires optimized inference configuration to fully utilize hardware resources. The mobile ONNX model inference process is as follows: Figure 5 As shown in (b): 1. Model preloading: During the initialization of the mobile application, ONNX model files are read from the application's assets directory using an asynchronous loading method. The loading process is executed in a background thread to avoid blocking the UI thread and causing application lag. During the loading process, information such as model file size and loading time is recorded for troubleshooting.

[0048] 2. Inference Environment Configuration: Initialize the ONNX Runtime environment and configure inference optimization parameters: Thread count configuration: Set IntraOpNumThreads to the number of CPU cores on the mobile device and InterOpNumThreads to 2 to balance multi-operator parallelism and thread scheduling overhead; Enable graph optimization (GraphOptimizationLevel.ORT_ENABLE_ALL); Graph optimization: Set GraphOptimizationLevel to ORT_ENABLE_ALL to enable all available computation graph optimizations; Hardware acceleration: Detect whether the mobile device supports GPU acceleration (Android platform detects OpenCL support, iOS platform detects Metal support). If supported, enable GPU inference mode, configure GPU device ID as 0, and allocate model inference computation tasks to the GPU for execution; if GPU is not supported, automatically downgrade to CPU inference mode.

[0049] 3. Model inference execution: Input tensor construction: The preprocessed video tensors and keyframe tensors are converted into input tensors acceptable to the ONNX model through the OnnxTensor.createTensor interface of ONNX Runtime; Construct a model input mapping table to associate video tensors and keyframe tensors with the input nodes "video_frames" and "key_frames" of the ONNX model, respectively, ensuring that the input names are consistent with the input node names of the ONNX model; Inference call: Call the ONNX Runtime's run interface (OrtSession.run()) to perform model inference and obtain the output logits vector.

[0050] 4. Original Result Analysis: The logits vector is activated by Sigmoid to obtain the confidence probability of each action category. The action category with the highest confidence probability is selected as the original action monitoring result. The original result includes the action category index, Chinese name, English name and confidence probability. The expression for calculating the Sigmoid activation function is:

[0051] in, For the first Confidence probability of a class of actions, The first in the logits vector The original predicted scores for each element.

[0052] Step S6: Optimize the original monitoring results using confidence filtering and context consistency verification mechanisms, and output the final action monitoring results.

[0053] To improve the stability and accuracy of monitoring results, the original monitoring results need to be optimized: 1. Confidence Filtering: Set the confidence threshold If the confidence probability of the original monitoring results is lower than This indicates that the model has low confidence in the result; in this case, the monitoring result should not be updated, and the previous valid monitoring result should be retained. If the confidence probability is not lower than [a certain value], the model should be retained. If so, then the result will be stored temporarily. 2. Context consistency check: Maintain a historical queue of the 5 most recent monitoring results. ,in Indicates the first Based on the temporary monitoring results, calculate the consistency score between the current temporary result and the results in the historical queue. :

[0054] in, For the current temporary monitoring results, For indicator functions, when and When the action categories are consistent ,otherwise ; Set consistency threshold (Default value is 0.3), if This indicates a significant difference between the current result and historical results, potentially indicating a false alarm. In this case, a second inference verification is triggered, re-inputting the current tensor data into the model for inference. The result with the higher confidence probability from the two inference results is taken as the final temporary result. If so, the current temporary result will be retained directly; 3. Results Output: Based on the user's language preference set in the application interface, the final motion monitoring results are output. When the user prefers Chinese, the Chinese name of the motion and the confidence probability (rounded to two decimal places) are displayed. When the user prefers English, the English name of the motion and the confidence probability are displayed. The results are displayed in real time in a designated area of ​​the application interface.

[0055] This embodiment further provides model resource management steps after steps S1-S6: when the mobile application is moved to the background, resources such as the ONNX Runtime environment, model session, and input / output tensors are released to reduce memory usage; when the application returns to the foreground, the model inference environment is reinitialized; the integrity of the model files is checked periodically, and if the files are damaged or missing, the user is prompted to download the model files again or repair the application.

[0056] Furthermore, to achieve the above objectives, this embodiment also provides a mobile terminal real-time motion monitoring system, the system comprising: The dataset construction and preprocessing module is used to collect motion video data, perform data cleaning, standardization, data augmentation and dataset partitioning, generate training set, validation set and test set, and output a standardized dataset in dual-input format. The PyTorch model training module is used to build a 3D convolutional-ResNet fusion action detection model based on the PyTorch framework, configure training hyperparameters, complete model training through the training set, optimize the model and judge early stopping using the validation set, evaluate the model performance through the test set, and output the optimal PyTorch model. The model conversion and optimization module is used to convert the best PyTorch model into ONNX format, perform computation graph optimization and optional INT8 quantization, verify the inference accuracy and consistency of the converted model, and output a mobile-adapted ONNX motion monitoring model. The video capture and caching module is used to capture video streams through the mobile camera, perform frame extraction and loop caching management, and output a continuous 30-frame video sequence to ensure the continuity and integrity of the input data. The mobile data preprocessing module is used to perform size normalization, channel conversion, pixel value standardization and keyframe extraction on continuous video sequences, and generate video tensors and keyframe tensors that meet the input requirements of the ONNX model and are adapted to the processing capabilities of mobile hardware. The ONNX model inference module is used to asynchronously load ONNX models, configure multi-threaded and hardware-accelerated inference parameters, perform tensor inference and output raw motion monitoring results, and support automatic switching between CPU and GPU inference modes. The monitoring result optimization module is used to optimize the original monitoring results through confidence filtering and context consistency verification, thereby improving the stability and accuracy of the results. The results display and resource management module is used to display the final monitoring results according to the user's language preference, manage the loading and release of model inference resources, and ensure the smooth operation of the application.

[0057] This invention also provides a terminal, which includes a memory, a processor, and a PyTorch-ONNX-based mobile real-time motion monitoring program stored in the memory. When executed by the processor, the program implements the steps of the aforementioned mobile real-time motion monitoring method. The terminal is a mobile device such as a smartphone or tablet, supporting Android 8.0 and above, iOS 12.0 and above operating systems, and is compatible with Snapdragon 6 series and above, Dimensity 7 series and above processors, meeting the deployment needs of mid-to-low-end mobile devices.

[0058] Compared with the prior art, this embodiment has the following beneficial effects: 1. Standardized Dataset Construction: A complete dataset construction process is proposed, consisting of "data collection - frame extraction - standardized preprocessing - multi-dimensional enhancement - dual-input construction", which significantly improves the model's generalization ability; 2. Model Structure Optimization: A lightweight model integrating 3D convolution and ResNet was designed, balancing the ability to capture spatiotemporal action features with the requirements of mobile deployment. The model parameter size is only 60% of that of traditional 3D CNN models. 3. Complete deployment chain: Implements an end-to-end technical chain of "PyTorch training - ONNX conversion - computation graph optimization - quantization processing - mobile inference" to ensure consistency between model training and deployment; 4. Excellent inference performance: Through optimization strategies such as multi-threaded inference, GPU acceleration, and model quantization, low-latency inference is achieved on mid-to-low-end mobile devices, with inference time controlled within 10 seconds; 5. High monitoring accuracy: Through dual-input data construction and result optimization mechanisms, the model achieves a monitoring accuracy of no less than 84% in real-world scenarios, meeting the needs of practical applications; 6. Strong compatibility: Based on the ONNX format, it enables cross-framework deployment, supports mainstream mobile operating systems such as Android and iOS, and is adapted to the hardware performance of mid-to-low-end mobile devices.

[0059] Example 2 This embodiment further defines Embodiment 1, and uses specific application examples to further illustrate Embodiment 1.

[0060] Dataset construction and model training: 1. Data collection: Collect video data for 35 types of actions, 150 segments for each action, with a video resolution of 1920×1080. The collection scenes include indoor offices, outdoor parks, etc., and the age range of the subjects is 18-60 years old. 2. Frame extraction and preprocessing: Frames are extracted at 30 frames / second using OpenCV, and blurry frames are removed using the variance method to obtain a valid frame sequence; the valid frames are centered, cropped, and scaled to 224×224, and then converted to BGR format. 3. Data Augmentation: Perform augmentation operations such as random horizontal flipping, brightness / contrast adjustment, Gaussian noise addition, and random cropping to generate an augmented dataset; 4. Dataset partitioning: The dataset is divided into a training set (42,000 segments), a validation set (5,250 segments), and a test set (5,250 segments) in an 8:1:1 ratio, and the pixel values ​​are normalized. 5. Model Training: A 3D convolutional-ResNet fusion model was built based on the PyTorch 1.12 framework, configured with AdamW optimizer (learning rate 1e-4), cosine annealing scheduler, cross-entropy loss function, batch size 16, training for 100 epochs, and early stopping patience=10. 6. Model Evaluation: The model achieved an average accuracy of 88.5% on the test set, and the recall rate for all types of actions was higher than 82%, meeting the requirements for mobile deployment.

[0061] PyTorch Model to ONNX Conversion and Optimization 1. Model Export: Load the optimal PyTorch model weights, construct the virtual input tensor, and call torch.onnx.export to export the ONNX model (opset=12). 2. Computational graph optimization: Using the ONNX Runtime optimization tool, constant folding, redundant node elimination, and operator fusion were performed, reducing the model size from 32MB to 24MB after optimization; 3. Model Quantization: INT8 dynamic quantization was performed, reducing the model size to 6MB. The test set accuracy was 86.3%, with a precision loss of 2.4%. 4. Consistency verification: With the same test data input, the output error of the ONNX model and the PyTorch model after quantization is 8.7e-4, which meets the consistency requirements.

[0062] Android Deployment and Testing: 1. Video capture: Use the CameraX framework to capture video streams (1920×1080, 30 frames / second), with a circular buffer capacity of 30 frames; 2. Data preprocessing: Frame cropping and scaling are performed using the Android native Bitmap API, ARGB to BGR conversion is achieved through bitwise operations, and tensor reconstruction is completed by array copying; 3. Model Loading and Inference: Integrate ONNX Runtime Android SDK (1.16.3) to asynchronously load the quantized ONNX model, configure 4-thread inference, and enable GPU acceleration; 4. Performance Testing: Tested on Redmi 13C (Helio G85 processor): Preprocessing time: Approximately 850ms for 30 frames; Inference time: Approximately 4.2 seconds per inference in GPU mode and approximately 7.8 seconds per inference in CPU mode; Resource consumption: CPU utilization during inference is 35%~40%, and memory usage is approximately 180MB; Monitoring accuracy: The average accuracy rate in real-world testing is 84.5%, which meets the requirements for real-time interaction.

Claims

1. A method for real-time motion monitoring on a mobile device, characterized in that, The method includes the following steps: S1. Construct a motion monitoring dataset, perform data preprocessing and enhancement on the original motion videos, and generate a standardized training dataset; S2. Construct a motion detection model that integrates 3D convolution and ResNet, and complete the model training and optimization using the standardized training dataset. S3. Convert the trained motion detection model into ONNX format, optimize the model input / output interface and computation graph structure, and generate an ONNX motion detection model adapted for mobile devices. S4. Collect video streams via mobile devices, extract frames, manage caches and preprocess the video streams to generate tensor data that meets the input requirements of the ONNX motion monitoring model; S5. Input the tensor data into the preloaded ONNX motion monitoring model, and complete motion feature extraction and classification through multi-threaded inference and hardware acceleration strategies to obtain the original motion monitoring results. S6. Optimize the original monitoring results by using confidence filtering and context consistency verification mechanisms, and output the final action monitoring results.

2. The mobile terminal real-time motion monitoring method according to claim 1, characterized in that, When constructing the motion monitoring dataset, continuous frames are extracted from the original motion video. The extracted video frames are then subjected to data augmentation processing. Five key frames are extracted from each 30-frame continuous video segment according to the uniform sampling rule to construct a dual input data pair of 30 complete video frames + 5 key frames.

3. The mobile terminal real-time motion monitoring method according to claim 2, characterized in that, A motion monitoring model is built based on the PyTorch framework. The structure of the motion monitoring model is as follows: Input layer: Receives two input tensors, namely a video frame tensor and a keyframe tensor; Feature extraction backbone layer: The first stage consists of two 3D convolutional layers, each followed by a BatchNorm3d layer and a ReLU activation function, used to initially extract the spatiotemporal features of the action; The second stage consists of three ResNet residual blocks, each containing two 3D convolutional layers. Residual connections are used to alleviate the gradient vanishing problem and enhance the ability to extract deep features. The third stage: global average pooling layer, which pools the feature map into a high-dimensional feature vector; The classification head consists of two fully connected layers and a Softmax activation function, which realizes the mapping of high-dimensional features to action categories and probability normalization.

4. The mobile terminal real-time motion monitoring method according to claim 3, characterized in that, When converting the trained motion detection model to ONNX format, the PyTorch torch.onnx.export function is called to export the model, and the optimization tools provided by ONNX Runtime are used to optimize the computation graph of the exported ONNX format motion detection model.

5. The mobile terminal real-time motion monitoring method according to claim 4, characterized in that, The cache management in step S4 is as follows: a circular buffer is used to store the acquired video frames. The buffer capacity is set to N frames. When the number of frames in the buffer reaches N frames, the oldest frame is removed and a new acquired frame is added according to the first-in-first-out principle to ensure that the buffer always maintains a continuous N frames, providing stable input data for model inference. When the number of frames in the buffer is less than N frames, the last frame is copied to supplement N frames to avoid inference failure due to insufficient frame quantity.

6. The mobile terminal real-time motion monitoring method according to claim 5, characterized in that, The multi-threaded inference method sets the number of mobile CPU cores to 2 to balance the overhead of multi-operator parallelism and thread scheduling. The hardware acceleration strategy involves detecting whether the mobile device supports GPU acceleration. If it does, GPU inference mode is enabled, the GPU device ID is configured as 0, and the model inference computation task is assigned to the GPU for execution. If GPU acceleration is not supported, it automatically downgrades to CPU inference mode.

7. The mobile terminal real-time motion monitoring method according to claim 6, characterized in that, The confidence level screening specifically involves setting a confidence threshold. If the confidence probability of the original monitoring results is lower than This indicates that the model has low confidence in the result; in this case, the monitoring result should not be updated, and the previous valid monitoring result should be retained. If the confidence probability is not lower than [a certain value], the model should be retained. If so, then the result will be stored temporarily. The context consistency verification mechanism specifically involves maintaining a historical queue of the five most recent monitoring results. ,in Indicates the first Based on the temporary monitoring results, calculate the consistency score between the current temporary result and the results in the historical queue. Set consistency threshold ,like This indicates a significant difference between the current result and historical results, potentially indicating a false alarm. In this case, a second inference verification is triggered, re-inputting the current tensor data into the model for inference. The result with the higher confidence probability from the two inference results is taken as the final temporary result. If so, the current temporary result will be retained directly.

8. A mobile terminal real-time motion monitoring system, characterized in that, The system includes: This module is used to build motion monitoring datasets, preprocess and enhance raw motion videos, and generate standardized training datasets. This module is used to build a motion detection model that integrates 3D convolution and ResNet, and completes model training and optimization using the standardized training dataset. This module is used to convert trained motion detection models into ONNX format, optimize model input / output interfaces and computation graph structure, and generate mobile-compatible ONNX motion detection models. This module is used to capture video streams via mobile devices, extract frames from the video streams, manage caches and preprocess them, and generate tensor data that meets the input requirements of the ONNX motion monitoring model. This module is used to input the tensor data into the preloaded ONNX motion monitoring model, and to complete motion feature extraction and classification through multi-threaded inference and hardware acceleration strategies to obtain the original motion monitoring results. This module is used to optimize the original monitoring results using confidence filtering and context consistency verification mechanisms, and output the final action monitoring results.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-7.

Images