Time action detection method and system based on multi-channel sensor time series data

Abstract

The application relates to the field of Internet of Things sensing technology and discloses a time action detection method and system based on multi-channel sensor time sequence data, which comprises the following steps: preprocessing sensor data, modeling local time patterns and global dependencies through a stacked spatio-temporal convolution module, performing preliminary prediction based on extracted features, calculating correction amounts on the basis of the preliminary prediction, respectively correcting a preliminary start time, a preliminary end time and a preliminary category score, and outputting a final action prediction result. The application solves the problems of sensor signal noise interference and multi-channel correlation feature modeling in the time action detection process, enhances the similar action distinguishing capability, adopts a two-step strategy of preliminary prediction and boundary correction to improve the action positioning boundary precision, reduces the false detection and missed detection, and solves the action segment truncation problem.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) sensing technology, specifically to a time-based motion detection method and system based on multi-channel sensor time-series data. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of computer vision, mobile smart terminals, and IoT sensing technologies, Temporal Action Detection (TAD) has become an important research direction in the field of behavior understanding. TAD locates the time interval of an action from a continuous time series and identifies its category. Early applications primarily focused on video data, achieving action localization and recognition by modeling the spatiotemporal features of image frame sequences. In recent years, with the widespread adoption of mobile phone inertial sensors, wearable devices, and wireless sensing devices, TAD has gradually expanded to application scenarios based on sensor time-series data. Compared to vision-based solutions, sensor signals are less affected by occlusion, viewing angle, and lighting changes, exhibiting strong environmental robustness and deployment flexibility, making it valuable for applications in human-computer interaction, smart elderly care, motion monitoring, and the smart IoT.

[0004] For methods based on sensor time-series data, existing methods still have significant limitations in practical detection processes due to the characteristics of raw signals, which are typically multi-channel, high-noise, strongly temporal, and highly individual variable. On the one hand, sensor data usually consists of multiple channels such as acceleration, angular velocity, and gravity components. Different channels reflect the changes in movement across different physical dimensions. These channels exhibit correlations, amplitude differences, phase differences, and fluctuations due to changes in device placement. Modeling only a single channel or using simple splicing or weighting for feature fusion often fails to fully extract complementary information between channels, resulting in insufficient distinguishability between similar movements. On the other hand, human movement is essentially a dynamic process that changes continuously over time. Differences between different movements are not only reflected in instantaneous amplitude but also in time-dependent features such as rhythmic changes, stage evolution, and duration during the movement. Existing methods are insufficient in modeling the time dimension, or cannot take into account both short-term local changes and long-term global dependencies. This can easily lead to inaccurate identification of the start and end times of actions, resulting in problems such as boundary positioning offset, incomplete action segment truncation, and increased false positives and false negatives. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a time-based motion detection method and system based on multi-channel sensor time-series data. This method solves the problems of sensor signal noise interference and multi-channel correlation feature modeling during time-based motion detection, thereby enhancing the ability to distinguish similar motions. Furthermore, it employs a two-step strategy of preliminary prediction and boundary correction to improve the accuracy of motion localization boundaries and reduce false detections, missed detections, and motion segment truncation issues.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: One or more embodiments provide a time-based motion detection method based on multi-channel sensor timing data, comprising the following steps: Acquire multi-channel sensor timing data and perform preprocessing; Feature extraction is performed on the preprocessed time series data. Local time patterns and global dependencies are modeled by stacked spatiotemporal convolution modules to obtain multi-scale time-frequency features that change over time. Based on multi-scale time-frequency features, the initial start time, initial end time, and initial category score of the action interval are predicted, thus obtaining the initial interval; Within the boundary neighborhood of the initial start time and the initial end time, boundary features are extracted from multi-scale time-frequency features; a category distribution is constructed based on the initial category score, and a category fusion feature is generated based on the category distribution; the boundary features and the category fusion feature are fused to obtain the boundary correction amount; Initially, the initial action region is determined based on the initial interval, and the corresponding regional features are extracted from the multi-scale time-frequency features to obtain the category correction amount. The boundary correction is fused with the initial start time and the initial end time to obtain the final start time and the final end time. The category correction is fused with the initial category score to obtain the final category score, thus outputting the final action prediction result.

[0007] One or more embodiments provide a time-based motion detection system based on multi-channel sensor timing data, including: The acquisition and preprocessing module is configured to acquire multi-channel sensor time-series data and perform preprocessing. The feature extraction module is configured to extract features from the preprocessed time series data. By modeling local time patterns and global dependencies through stacked spatiotemporal convolution modules, multi-scale time-frequency features that change over time are obtained. The preliminary prediction module is configured to predict the preliminary start time, preliminary end time, and preliminary category score of the action interval based on multi-scale time-frequency features, thereby obtaining the preliminary interval. The boundary correction identification module is configured to extract boundary features from multi-scale time-frequency features within the boundary neighborhood of the initial start time and the initial end time; construct a category distribution based on the initial category score, and generate category fusion features based on the category distribution; and fuse the boundary features with the category fusion features to obtain the boundary correction amount. The category correction amount identification module is configured to determine the initial action region based on the initial interval at the beginning, and extract the region features corresponding to the initial action region from the multi-scale time-frequency features to obtain the category correction amount; The prediction output module is configured to fuse the boundary correction amount with the initial start time and the initial end time to obtain the final start time and the final end time, and to fuse the category correction amount with the initial category score to obtain the final category score, thereby outputting the final action prediction result.

[0008] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, the computer instructions perform the steps in the above-described time-motion detection method based on multi-channel sensor timing data.

[0009] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps in the aforementioned time-motion detection method based on multi-channel sensor timing data.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention first acquires and preprocesses multi-channel sensor time-series data to reduce the impact of noise interference and amplitude differences between channels on the detection results. It then uses a stacked spatiotemporal convolution module to jointly model local temporal patterns and global dependencies, thereby fully exploring the complementary information between different sensor channels and the dynamic evolution characteristics of actions in the time dimension. This yields multi-scale time-frequency features that change over time. The combination of preprocessing and deep feature extraction effectively solves the problem of noise interference in sensor signals, enhances the model's ability to model multi-channel correlation features, and improves the representation of multi-channel sensor correlation information and the temporal evolution characteristics of actions, thus improving the ability to distinguish between similar actions. After generating the initial interval, the system does not output directly but uses local feature jumps at the action boundary for secondary correction, thereby solving the positioning deviation caused by phase lag or noise interference in the sensor signal at the moment of action triggering. By employing a strategy of preliminary prediction and boundary correction, the sensor signal can be more sensitively captured at the moment of action triggering. Compared with the single regression scheme, it significantly improves the boundary accuracy of action localization and reduces the problems of false detection, missed detection, and incomplete action segment truncation caused by noise interference, signal mutation, or boundary ambiguity. Especially in scenarios such as smart rehabilitation training where it is necessary to accurately distinguish the start and end points of actions, it can effectively reduce the false detection rate.

[0011] The advantages of the present invention, as well as its additional advantages, will be described in detail in the following specific embodiments. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0013] Figure 1 This is an improved I3D network structure diagram of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the Inception module in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the preliminary prediction module of Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the prediction correction module of Embodiment 1 of the present invention; Figure 5 This is a flowchart of the time-motion detection method based on multi-channel sensor timing data according to Embodiment 1 of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0015] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0016] It should be noted that the terminology used herein is for describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features within those embodiments can be combined with each other. The embodiments will now be described in detail with reference to the accompanying drawings.

[0017] Example 1 In one or more of the technical solutions disclosed in the embodiments, such as Figures 1 to 5 As shown, the time-based motion detection method based on multi-channel sensor time-series data includes the following steps: S1. Acquire multi-channel sensor timing data and perform preprocessing; S2. Feature extraction is performed on the preprocessed time series data. Local time patterns and global dependencies are modeled by stacked spatiotemporal convolution modules to obtain multi-scale time-frequency features that change over time. S3. Based on multi-scale time-frequency features, the initial start time, initial end time, and initial category score of the action interval are predicted, thus obtaining the initial interval; S4. Within the boundary neighborhood of the initial start time and the initial end time, extract boundary features from the multi-scale time-frequency features; construct a category distribution based on the initial category score, and generate category fusion features based on the category distribution; fuse the boundary features with the category fusion features to obtain the boundary correction amount; S5. Determine the initial action region based on the initial interval, and extract the region features corresponding to the initial action region from the multi-scale time-frequency features output in step S2 to obtain the category correction amount; S6. The boundary correction amount is fused with the initial start time and the initial end time to obtain the final start time and the final end time. The category correction amount is fused with the initial category score to obtain the final category score, thereby outputting the final action prediction result.

[0018] To address the technical challenges of insufficient multi-channel information fusion, inadequate modeling of action time dependencies, and difficulty in accurately locating action start and end times in time-based action detection using multi-channel sensor time-series data, this embodiment first acquires and preprocesses multi-channel sensor time-series data to reduce the impact of noise interference and channel amplitude differences in the original signal on the detection results. Then, a stacked spatiotemporal convolution module is used to jointly model local temporal patterns and global dependencies, thereby fully exploring the complementary information between different sensor channels and the dynamic evolution characteristics of actions in the time dimension, obtaining multi-scale time-frequency features that change over time. The combination of preprocessing and deep feature extraction effectively solves the problem of noise interference from sensor signals and enhances the model's ability to model multi-channel correlated features. After generating the initial interval, the system does not output directly but uses local feature transitions at the action boundary for secondary correction, thus resolving the positioning deviation caused by phase lag or noise interference in the sensor signal at the moment of action triggering. By employing a strategy of preliminary prediction and boundary correction, the sensor signal can be more sensitively captured at the moment of action triggering. Compared with the single regression scheme, it significantly improves the boundary accuracy of action localization, especially in scenarios such as smart rehabilitation training where it is necessary to accurately distinguish the start and end points of actions, and can effectively reduce the false detection rate.

[0019] In step S1, the acquired multi-channel sensor time series data is preprocessed, including denoising, normalizing and aligning the sensor time series data to obtain regularized time series data. Optionally, denoising can be performed by filtering the original sensor time-series signal to remove high-frequency random noise and abnormal fluctuations. Specifically, at least one of the following methods can be used: A low-pass filter is used to filter out high-frequency noise signals; Moving average filtering is used to smooth the time series data; Median filtering is used to remove burst impulse noise; In optional cases, a Kalman filter can be used to estimate the state of dynamic signals to improve signal stability.

[0020] The above noise reduction process can effectively reduce the impact of environmental interference on signals, such as equipment jitter and electromagnetic interference.

[0021] Optionally, normalization processing is performed to unify the numerical scale of sensor data from different channels, in order to eliminate the impact of differences in physical units and amplitudes on model training. Specifically: For each channel of data, perform Min-Max Normalization to map the data to a preset interval, such as [0,1]. Alternatively, Z-score standardization can be used to make the data satisfy a zero-mean and unit-variance distribution; The above implementation method uses normalization processing to express the data of each channel at the same scale, which is beneficial to improving the multi-channel feature fusion effect.

[0022] Optionally, alignment processing involves synchronizing and unifying multi-channel sensor data in the time dimension to ensure that the data from each channel corresponds under the same time reference. This includes: resampling sensor data with different sampling frequencies to unify the sampling rate; segmenting the continuous data stream using a time window method and setting a fixed window length and step size; and performing time synchronization correction on multi-channel data with time offsets to align the data from each channel under the same timestamp. The above implementation method ensures the consistency of multi-channel data in the time dimension through time alignment processing, thereby avoiding feature misalignment problems.

[0023] In terms of technical implementation, the data acquisition step S1 collects raw data streams from sensors such as triaxial accelerometers and triaxial gyroscopes via SPI or I2C interfaces. Preprocessing involves median filtering or wavelet denoising to remove high-frequency glitches, and linear interpolation aligns the sampled data at different frequencies to a unified time axis. After Min-max normalization, the input data is regularized.

[0024] In other implementations, the multi-channel sensor can be expanded to include an electromyography (EMG) sensor, a magnetometer, or a plantar pressure sensor. This method is not only applicable to behavior recognition on mobile devices, but can also be extended to applications such as fall detection in elderly care, athlete training posture analysis, and the standardization of assembly line movements in industrial production lines.

[0025] The time-based motion detection method in this embodiment improves the modal consistency and signal-to-noise ratio of multi-source heterogeneous data by performing denoising, normalization, and alignment preprocessing on multi-channel sensor time-series data, providing high-quality standardized input for subsequent feature extraction.

[0026] After preprocessing the mobile phone sensor time series data, such as denoising, normalization, and alignment, the regularized multi-channel time series data is input into the backbone network for feature extraction.

[0027] In step S2, feature extraction is performed on the preprocessed time series data. The feature extraction network adopts the improved I3D network. The Inception module is introduced into the improved I3D network to form a multi-scale time-frequency feature extraction structure. like Figure 1 As shown, the improved I3D network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a third convolutional layer, a second pooling layer, multiple cascaded Inception modules (abbreviated as Inc.), a third pooling layer, a fourth pooling layer, and a feature output layer. The input layer receives the temporal data obtained after preprocessing in step S1. The first convolutional layer uses a 7×7×7 three-dimensional convolutional kernel to perform preliminary feature extraction on the input data. The first pooling layer uses a 1×3×3 max pooling operation to downsample the feature map. The second convolutional layer uses a 1×1×1 three-dimensional convolutional kernel to compress and map the channel features. The third convolutional layer uses a 3×3×3 three-dimensional convolutional kernel to further extract local temporal features. The second pooling layer uses a 1×3×3 max pooling operation to further compress the features. After the second pooling layer, the first and second Inception modules are connected in sequence, and the third, fourth, and fifth Inception modules are connected through the third pooling layer. After the fifth Inception module, the sixth Inception module is connected, and after the fourth pooling layer, the seventh and eighth Inception modules are connected in sequence. Finally, the deep features are output as the feature extraction result, i.e., multi-scale time-frequency temporal features.

[0028] Sensor data typically contains multiple physical dimensions, such as X / Y / Z-axis acceleration, angular velocity, and gravitational acceleration from a mobile phone sensor. These channels of data require collaborative analysis to fully characterize behavior. For example, during running, not only does the X-axis acceleration change drastically, but the Z-axis angular velocity also exhibits a specific pattern. Information from a single channel may lead to misjudgments, such as confusing "running" with "jumping." This embodiment uses an I3D network combined with the Inception module to deeply fuse multi-channel features. This enables the modeling of the correlation, complementarity, and dynamic coupling relationships between different channels within a unified model framework, thereby fully mining the behavioral discrimination information contained in multi-dimensional sensor data.

[0029] In the above embodiments, the improved I3D network introduces the Inception module, which can more effectively process sensor signals with complex time-frequency characteristics. This avoids the problems of information loss when processing long-term actions or insufficient resolution when processing short-term actions caused by single-scale convolutional kernels, thus enhancing the discriminative power of features. Different human actions have different duration periods and frequencies of change in the time dimension. For example, the signal frequency of fast running is higher, while that of slow walking is lower. By capturing local temporal patterns through 3D convolution (I3D), key temporal boundaries of actions can be identified. A further technical solution involves the structure of the Inception module, which includes stacking multiple different types of convolutional kernels together to replace a single convolutional kernel, such as... Figure 2 As shown, this method can extract diverse features.

[0030] The Inception module, through multiple parallel branches and using convolutional kernels or pooling operations of different sizes, can simultaneously capture short-term local changes and long-term global trends in regular time series data, and fuse features of different scales in the channel dimension to generate more representative time series features.

[0031] In other implementations, the internal structure of the Inception module can be adjusted based on computational resources, for example, by using 1×1×1 convolutions for dimensionality reduction to decrease the number of parameters. Furthermore, the feature extraction network can also introduce residual connections on top of I3D to support deeper network training.

[0032] The feature extraction network used in this embodiment progressively models the local temporal patterns and global temporal dependencies of signals through stacked spatiotemporal convolutional modules. At the same time, it enhances the feature response to key action segments by utilizing the interaction mechanism between channels. Finally, it outputs a deep feature vector that can accurately represent the essence of the action, providing a solid feature foundation for subsequent behavior classification and time localization.

[0033] In step S3, based on multi-scale time-frequency features, the initial start time, initial end time, and initial category score of the action interval are predicted, thus obtaining the initial interval, which can be achieved through the initial prediction module.

[0034] like Figure 3 As shown, the preliminary prediction module adopts a dual-branch structure, including a localization branch and a classification branch, which process the multi-scale time-frequency features of the input respectively. Each branch includes an activation layer, a normalization layer, and a convolutional layer connected in sequence, which are used to perform activation, normalization, and convolutional mapping operations on the input features in sequence to obtain the prediction results for the corresponding task.

[0035] The localization branch is used to map multi-scale time-frequency features to the localization embedding space. The system performs a regression task and outputs initial localization results through convolutional layers; the initial localization results include the initial start time and initial end time of the action range. In the preliminary prediction module, the time dimension is used. This represents the total number of time steps. After processing by the temporal convolutional layer, we can obtain the [number of steps]. The initial start time of each time step and preliminary end time As shown in equation (1): (1); in, and It indicates the distance between the start and end points of an action.

[0036] The classification branch is used to map multi-scale time-frequency features to the classification embedding space. The system performs a classification task and outputs initial classification results through convolutional layers; these initial classification results are preliminary category scores corresponding to each action category. Specifically, the temporal convolutional layer integrates multi-scale time-frequency features. As input, new feature representations are generated through convolution operations. These representations are then processed by linear regression, as shown in Equation (2), to obtain preliminary category scores. : (2); like Figure 4 As shown, step S4 is implemented through the constructed prediction correction module. The prediction correction module corrects the preliminary prediction result by combining the output of the preliminary prediction module and the multi-scale time-frequency features extracted in step S2. The prediction correction module includes two main parts: a classification channel and a localization channel. The classification channel is used to correct the prediction of the action category, and the localization channel is used to predict the error value between the preliminary localization result and the actual action position, thereby optimizing and accurately adjusting the preliminary prediction result.

[0037] In step S4, within the initial start time and initial end time obtained in step S3, in the corresponding start boundary neighborhood... and end boundary neighborhood Within this process, boundary features are extracted from the multi-scale time-frequency features output in step S2. Specifically, the boundary pooling method is used to extract boundary features from the multi-scale time-frequency features within the boundary neighborhood between the initial start time and the initial end time.

[0038] Since features are typically concentrated near action boundaries, this embodiment employs a boundary pooling method. This method utilizes the preliminary prediction results from the initial prediction module to extract salient features around the start and end boundaries; these boundary features are represented as follows: and The formula for boundary max pooling is as shown in equation (3): (3); in, The length of the action interval. These two hyperparameters are used to control the proportions of the outer and inner selection regions within the action range, respectively. In step S4, a category distribution is constructed based on the preliminary category scores, and category fusion features are generated based on the category distribution. Specifically, the preliminary category scores are distributed to construct the category distribution at all time points; then, a learnable matrix is ​​used to map the category distribution to obtain the category fusion features. This process is implemented in the localization channel of the prediction and correction module. The category features are combined into a hybrid form, which is determined by the output distribution of the localization channel, as shown in equation (4): (4); in, Represents all time points The category distribution, i.e., the preliminary category score, Number of categories; For learnable matrices, For the number of categories, It represents the dimension of each category.

[0039] This step incorporates the initially predicted category semantic information into the subsequent localization correction process, enabling boundary correction to not only rely on local features within the boundary neighborhood but also to combine action category information for joint judgment, thereby improving the accuracy of boundary correction.

[0040] The boundary correction value is obtained by fusing boundary features with class fusion features. Specifically, the boundary features and class fusion features can be fused and then used to make predictions using a CNN to obtain the boundary correction value. As shown in equation (5): (5); in, This represents the start time boundary correction amount corresponding to the k-th candidate action. This represents the end time boundary correction amount corresponding to the k-th candidate action.

[0041] In the classification channel of the prediction correction module, in order to extract features located in the action region derived from the preliminary prediction based on the input preliminary prediction classification features, this embodiment uses convolutional pooling to extract features and obtain the category correction amount. As shown in equation (6): (6); in, The multi-scale time-frequency features output in step S2; In step S6, the boundary correction amount is fused with the initial start time and the initial end time to obtain the start time corresponding to the final k-th candidate action. End time corresponding to the final k-th candidate action The formula is: (7); The final category score is obtained by combining the category adjustment amount with the preliminary category score. The formula is: (8); in, This indicates the preliminary predicted length of the k-th action; This indicates the scores for the categories to which the actions in the preliminary prediction section and the prediction correction module belong; In one possible implementation, after outputting the final action prediction result, the method further includes using a non-maximum suppression (NMS) algorithm to filter overlapping action prediction positions, retaining the prediction position with the highest confidence. Non-maximum suppression (NMS) is used to remove redundant prediction results, selecting the prediction position with the highest confidence from action prediction positions with high overlap.

[0042] The time-based action detection method proposed in this embodiment can be applied to mobile phone sensor signal processing scenarios, enabling accurate behavior recognition under privacy protection conditions on the device side. First, a feature extraction network is used to deeply encode multi-channel sensor time-series signals, effectively mining the correlation features between different sensor channels and the dynamic change patterns of actions over time. Second, an anchor-free two-step prediction structure is adopted to directly complete action interval localization and action category recognition on the sensor time-series data, without relying on visual acquisition devices such as cameras. This overcomes the limitations of traditional visual modalities in terms of occlusion, lighting, viewing angle, and privacy protection, and can effectively adapt to time-series data output by sensors such as accelerometers, gyroscopes, and magnetometers, thereby improving the practicality and environmental adaptability of time-based action detection in mobile terminals and IoT scenarios.

[0043] Example 2 Based on Embodiment 1, this embodiment provides a time-based motion detection system based on multi-channel sensor time-series data, including: The acquisition and preprocessing module is configured to acquire multi-channel sensor time-series data and perform preprocessing. The feature extraction module is configured to extract features from the preprocessed time series data. By modeling local time patterns and global dependencies through stacked spatiotemporal convolution modules, multi-scale time-frequency features that change over time are obtained. The preliminary prediction module is configured to predict the preliminary start time, preliminary end time, and preliminary category score of the action interval based on multi-scale time-frequency features, thereby obtaining the preliminary interval. The boundary correction identification module is configured to extract boundary features from multi-scale time-frequency features within the boundary neighborhood of the initial start time and the initial end time; construct a category distribution based on the initial category score, and generate category fusion features based on the category distribution; and fuse the boundary features with the category fusion features to obtain the boundary correction amount. The category correction amount identification module is configured to determine the initial action region based on the initial interval at the beginning, and extract the region features corresponding to the initial action region from the multi-scale time-frequency features to obtain the category correction amount; The prediction output module is configured to fuse the boundary correction amount with the initial start time and the initial end time to obtain the final start time and the final end time, and to fuse the category correction amount with the initial category score to obtain the final category score, thereby outputting the final action prediction result.

[0044] It should be noted that each module in this embodiment corresponds one-to-one with each step in embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0045] Example 3 Based on Embodiment 1, this embodiment provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps in the time-motion detection method based on multi-channel sensor timing data described in Embodiment 1.

[0046] The memory stores computer program instructions, and the processor connects to the memory and sensor interface via a bus. When the processor executes computer instructions, the drive device completes a full logical loop from sensor data acquisition, preprocessing, feature extraction, preliminary interval prediction, boundary and category correction to final result output.

[0047] The system utilizes a hardware pipeline to process continuous sensor data streams. The sensor interface communicates with external sensing hardware, reading analog or digital signals into the processor. With instruction support from memory, the processor decomposes complex 3D convolution and boundary pooling operations into parallel processing tasks, using a bus to enable high-speed data transfer between processing and storage units. Hardware-accelerated temporal feature extraction and correction algorithms ensure real-time motion detection, meeting the low-latency interaction requirements of mobile devices.

[0048] In the hardware architecture, the sensor interface can also adapt to multiple communication protocols. The memory not only stores program code but also temporarily stores input normalized data, intermediate timing feature layers, and various correction parameters. The processor can be a general-purpose CPU or a DSP. The bus coordinates the processor's read and write access to the memory, ensuring that data flow is not blocked when performing high-load tasks such as boundary / class correction.

[0049] This embodiment achieves localized processing on the device side through an integrated hardware and software design, eliminating the need to upload raw sensor data to the cloud and thus greatly protecting user privacy and security. Simultaneously, because the algorithm is optimized within the processor, the device can achieve real-time monitoring of high-frequency sensor data with low power consumption, making it suitable for integration into smartphones, smartwatches, or embedded monitoring devices to achieve stable motion recognition functionality in complex environments.

[0050] In other implementations, the hardware architecture of the device can be modified as needed. For example, an FPGA can be used to implement the hardware acceleration logic for the feature extraction network, or the memory and processor can be integrated into the same system-on-a-chip (SoC) to further reduce size. The sensor interface can also support wireless transmission methods such as Bluetooth or Wi-Fi to receive data streams from remote sensor nodes.

[0051] Example 4 Based on Embodiment 1, this embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, they complete the steps in the time-motion detection method based on multi-channel sensor time-series data described in Embodiment 1.

[0052] It should be understood that in this invention, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0053] The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information about the device type.

[0054] In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here. Those skilled in the art will recognize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0055] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0056] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0057] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0059] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A time action detection method based on multi-channel sensor time series data, characterized in that, Includes the following steps: Acquire multi-channel sensor timing data and perform preprocessing; Feature extraction is performed on the preprocessed time series data. Local time patterns and global dependencies are modeled by stacked spatiotemporal convolution modules to obtain multi-scale time-frequency features that change over time. Based on multi-scale time-frequency features, the initial start time, initial end time, and initial category score of the action interval are predicted, thus obtaining the initial interval; Boundary features are extracted from multi-scale time-frequency features within the boundary neighborhood of the initial start time and the initial end time. A category distribution is constructed based on the preliminary category scores, and a category fusion feature is generated based on the category distribution; the boundary features are fused with the category fusion feature to obtain the boundary correction amount; Initially, the initial action region is determined based on the initial interval, and the corresponding regional features are extracted from the multi-scale time-frequency features to obtain the category correction amount. The boundary correction is fused with the initial start time and the initial end time to obtain the final start time and the final end time. The category correction is fused with the initial category score to obtain the final category score, thus outputting the final action prediction result.

2. The time action detection method based on multi-channel sensor time series data of claim 1, wherein, Feature extraction is performed on the preprocessed time-series data. The feature extraction network adopts an improved I3D network, which introduces the Inception module to form a multi-scale time-frequency feature extraction structure. The improved I3D network includes an input layer, a first convolutional layer, a first pooling layer, a second convolutional layer, a third convolutional layer, a second pooling layer, multiple cascaded Inception modules, a third pooling layer, a fourth pooling layer, and a feature output layer.

3. The method for time action detection based on multi-channel sensor timing data of claim 1, wherein, Based on multi-scale time-frequency features, the preliminary start time, preliminary end time, and preliminary category score of the action interval are predicted, thus obtaining the preliminary interval, which is realized through the preliminary prediction module. The preliminary prediction module adopts a dual-branch structure, including a localization branch and a classification branch. The two branches process the input multi-scale time-frequency features respectively. Each branch includes an activation layer, a normalization layer, and a convolutional layer connected in sequence, which are used to perform activation, normalization, and convolutional mapping operations on the input features in sequence to obtain the prediction results of the corresponding task.

4. The method for time action detection based on multi-channel sensor timing data of claim 1, wherein, The boundary pooling method is used to extract boundary features from multi-scale time-frequency features within the boundary neighborhood of the initial start time and the initial end time.

5. The method for time action detection based on multi-channel sensor time series data of claim 1, wherein, Based on the preliminary category scores, a category distribution is constructed, and category fusion features are generated based on the category distribution. Specifically, the preliminary category scores are distributed to construct the category distribution at all time points; then, a learnable matrix is ​​used to map the category distribution to obtain the category fusion features.

6. The method for time action detection based on multi-channel sensor time series data of claim 1, wherein, The boundary features are fused with the category fusion features to obtain the boundary correction value. Specifically, the boundary features are fused with the category fusion features and then a CNN is used for prediction to obtain the boundary correction value.

7. The method for time action detection based on multi-channel sensor time series data of claim 1, wherein, After outputting the final action prediction result, the algorithm also includes using a nonmaximum suppression algorithm to filter overlapping action prediction positions and retain the prediction position with the highest confidence.

8. A time action detection system based on multi-channel sensor time series data, characterized in that, include: The acquisition and preprocessing module is configured to acquire multi-channel sensor time-series data and perform preprocessing. The feature extraction module is configured to extract features from the preprocessed time series data. By modeling local time patterns and global dependencies through stacked spatiotemporal convolution modules, multi-scale time-frequency features that change over time are obtained. The preliminary prediction module is configured to predict the preliminary start time, preliminary end time, and preliminary category score of the action interval based on multi-scale time-frequency features, thereby obtaining the preliminary interval. The boundary correction identification module is configured to extract boundary features from multi-scale time-frequency features within the boundary neighborhood of the initial start time and the initial end time. A category distribution is constructed based on the preliminary category scores, and a category fusion feature is generated based on the category distribution; the boundary features are fused with the category fusion feature to obtain the boundary correction amount; The category correction amount identification module is configured to determine the initial action region based on the initial interval at the beginning, and extract the region features corresponding to the initial action region from the multi-scale time-frequency features to obtain the category correction amount; The prediction output module is configured to fuse the boundary correction amount with the initial start time and the initial end time to obtain the final start time and the final end time, and to fuse the category correction amount with the initial category score to obtain the final category score, thereby outputting the final action prediction result.

9. An electronic device, comprising: It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, complete the steps in the time-motion detection method based on multi-channel sensor timing data as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps in the time-motion detection method based on multi-channel sensor time-series data as described in any one of claims 1-7.

Images