Method, medium and device for action recognition based on multi-sensor data fusion

Abstract

This application provides a method, medium, and device for action recognition based on multi-sensor data fusion, belonging to the field of action detection technology. The method includes: acquiring acceleration and angular velocity data from a smart interactive device; generating upward throw feature data based on the acceleration data within a sliding time window; suppressing sway detection when the upward throw feature data meets the upward throw condition; when the upward throw condition is not met, filtering the validity of the acceleration data based on the angular velocity data, and using the acceleration data corresponding to the angular velocity data with the acquisition time matching the angular velocity sway feature as the data to be processed; performing a mixed filter on the data to be processed, and calculating the variance value between a first number of consecutive mixed-filtered data to be processed; when the variance value exceeds a variance threshold, it is counted as a valid sway frame; when the cumulative number of valid sway frames exceeds a preset sway frame number threshold, a swaying action is determined to have occurred. This application can improve the accuracy of action detection.

Description

Technical Field

[0001] This application relates to the field of motion detection technology, and in particular to a motion recognition method, medium, and device based on multi-sensor data fusion. Background Technology

[0002] Intelligent interactive devices (such as intelligent dolls, intelligent toys, and intelligent ornaments) have been widely used in the consumer electronics field. Their core relies on sensors to collect signals such as motion, posture, touch, and ambient light to realize user action recognition and interactive response.

[0003] In existing technologies, there are generally two approaches to using multi-sensor data: one is to focus on improving the measurement accuracy of individual sensors; the other is to perform simple logical combinations (such as AND and OR operations) on the results at the decision-making level after each sensor has independently completed its measurement and preliminary judgment. However, this "independent measurement, later combination" model has obvious drawbacks. Because different interactive actions (such as throwing, shaking, and flipping) may produce similar data characteristics on the same sensor (such as an accelerometer), and the measurement processes of each sensor are isolated from each other, it is difficult for the system to use cross-sensor data correlation to distinguish actions. For example, the rising phase of a throwing action and a shaking action both produce continuous acceleration changes. Existing technologies rely solely on acceleration thresholds or simply combine them with independent gyroscope thresholds, which can easily misjudge a throwing action as a shaking action, or vice versa.

[0004] Therefore, there is an urgent need for a novel action recognition method based on the fusion of measurement data from multiple sensors to solve the aforementioned technical problems. Summary of the Invention

[0005] The purpose of this application is to provide a motion recognition method, medium, and device based on multi-sensor data fusion to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, this application provides an action recognition method based on multi-sensor data fusion, the method comprising: Continuously acquire sensor data from intelligent interactive devices, wherein the sensor data includes at least acceleration data and angular velocity data; Generate upward projectile feature data based on the acceleration data within the first sliding time window; When the upward throwing feature data meets the upward throwing condition, the shaking detection is suppressed, and the falling feature data continues to be generated based on the acceleration data. When the falling feature data meets the falling condition, it is determined that a throwing action has occurred. When the upward throwing feature data does not meet the upward throwing condition, the validity of the acceleration data is filtered according to the angular velocity data, and the acceleration data corresponding to the angular velocity data that meets the angular velocity swaying feature at the acquisition time is taken as the data to be processed. Perform a mixed filter on the data to be processed, and calculate the variance between a first number of consecutive mixed-filtered data to be processed. When the variance value exceeds the variance threshold, it is counted as a valid shake frame. When the cumulative number of valid shake frames exceeds the preset shake frame number threshold, it is determined that a shake action has occurred.

[0007] Optionally, the angular velocity swaying feature includes whether the angular velocity modulus exceeds a preset angular velocity threshold; the step of filtering the validity of the acceleration data based on the angular velocity data, and taking the acceleration data corresponding to the angular velocity data with the acquisition time matching the angular velocity swaying feature as the data to be processed, includes: Calculate the angular velocity modulus based on the angular velocity data, and determine whether the angular velocity modulus exceeds the angular velocity threshold. If so, the acceleration data corresponding to the acquisition time and angular velocity data will be used as the data to be processed; otherwise, the acceleration data corresponding to the acquisition time and angular velocity data will be discarded and will not be used as the data to be processed.

[0008] Optionally, the variance threshold is a dynamic threshold that is adaptively updated based on the currently calculated variance value.

[0009] Optionally, the formula for calculating the variance threshold is: W(t) = (1-b) × W(t-1) + V(t) × b × c; where W(t) is the current variance threshold, W(t-1) is the base threshold updated last time, V(t) is the currently calculated variance, and b and c are preset coefficients.

[0010] Optionally, after performing mixed filtering on the data to be processed, the method further includes: Calculate the instantaneous changes between the first number of continuously mixed filtered data to be processed; When the instantaneous change exceeds the preset instantaneous change threshold, the number of effective shaking frames is changed to be no less than the shaking frame number threshold, and a shaking action is determined to have occurred.

[0011] Optionally, the method further includes: When the upward throw characteristic data does not meet the upward throw condition, the acceleration modulus is calculated based on the acceleration data; When the acceleration modulus exceeds the preset modulus length threshold, a shaking action is determined to have occurred.

[0012] Optionally, the upward throw characteristic data includes the horizontal amplitude of acceleration data in the horizontal direction, the vertical acceleration trend in the vertical direction, and the acceleration vector amplitude pulse; the upward throw conditions include: the horizontal amplitude exceeds the amplitude threshold, the vertical acceleration trend exceeds the upward throw threshold, and the acceleration vector amplitude pulse exceeds the pulse threshold. The falling feature data includes: the acceleration modulus and / or the absolute value of vertical acceleration of the acceleration data within the second sliding time window after the peak timestamp; the falling conditions include: the number of consecutive falling frames exceeds a preset falling frame number threshold; when the acceleration modulus is less than the first threshold or the absolute value of vertical acceleration is less than the second threshold, the corresponding acceleration data is determined to be a falling frame; the peak timestamp is the timestamp corresponding to the acceleration data of the maximum acceleration modulus within the first sliding time window.

[0013] Optionally, the upward throw feature data includes the proportion of vertical acceleration, and the upward throw conditions include: the proportion of vertical acceleration in N1 consecutive frames exceeds a proportional threshold, or the acceleration data in N2 consecutive frames shows a sudden acceleration change in the vertical direction.

[0014] Optionally, the sensor data further includes touch data; the method further includes: When the upward throw feature data meets the upward throw condition, touch detection is suppressed; When the upward throw feature data does not meet the upward throw condition, touch detection is initiated based on the touch data; When a valid touch action is detected, the intelligent interactive device is controlled to execute a touch response and the inactive timer is reset. When the inactive timer reaches a preset low-power duration, the intelligent interactive device is controlled to enter a low-power mode.

[0015] Optionally, the sensor data further includes light-sensing data; the method further includes: When the upward projection feature data meets the upward projection condition, light-sensitive detection is suppressed; When the upward throw feature data does not meet the upward throw condition, the light difference between the largest and smallest light-sensing data is calculated based on the light-sensing data in the third sliding time window, and the index difference between the index value of the largest light-sensing data and the index value of the smallest light-sensing data is calculated. Based on the light perception difference and the index difference, it is analyzed whether a valid light perception change has occurred.

[0016] Optionally, the method further includes: When the upward throw feature data meets the upward throw condition, tumbling detection is suppressed; When the upward throw characteristic data does not meet the upward throw condition, the roll angle at the corresponding moment is calculated based on the acceleration data and angular velocity data. When the time interval between two roll angles exceeds a preset time interval threshold, and the angle change value between the two roll angles exceeds a preset roll angle change threshold, a rollover operation is determined to have occurred.

[0017] Optionally, the method further includes: When the upward-throwing feature data meets the upward-throwing condition, inversion detection and reversal detection are suppressed. When the upward throw characteristic data does not meet the upward throw condition, the roll angle and pitch angle at the corresponding moment are calculated based on the acceleration data and angular velocity data. When the absolute value of the roll angle is less than the first absolute threshold and the absolute value of the pitch angle is greater than the second absolute threshold, a reversal action is determined to have occurred. When the absolute value of the roll angle is less than the second absolute threshold and the pitch angle is within the preset pitch angle threshold range, it is determined that an inverted action has occurred.

[0018] In a second aspect, this application provides a computer-readable storage medium storing executable instructions that, when executed by a processor, cause the processor to perform the method described in any embodiment of this application.

[0019] A third aspect of this application provides an electronic device, comprising: One or more processors; A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to perform the method as described in any embodiment of this application.

[0020] The action recognition method, medium, and device based on multi-sensor data fusion in this application prioritize the recognition of throwing motions and suppress sway detection when sensor data features match the characteristics of throwing motions. This achieves mutual exclusion between throwing and swaying motions, avoiding misjudgments of throwing and swaying from the process level and solving the confusion problem caused by overlapping action features in traditional solutions. In addition, for sway detection, a combination of angular velocity and acceleration data is used, with angular velocity data as the validity criterion for acceleration data. This enhances anti-interference capabilities, filters out environmental noise that only changes acceleration, such as desktop vibration and collisions, and significantly reduces the false trigger rate of swaying motions. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0022] Figure 1 This is a flowchart illustrating an action recognition method based on multi-sensor data fusion in one embodiment; Figure 2 This is a flowchart illustrating an electronic device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0026] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0027] This application provides an action recognition method based on multi-sensor data fusion, such as Figure 1 As shown, the method includes: Step 105: Continuously acquire sensor data from the smart interactive device, wherein the sensor data includes at least acceleration data and angular velocity data.

[0028] In this embodiment, the intelligent interactive device is an electronic device that incorporates one or more technologies and is capable of responding to user actions. This device can provide corresponding feedback based on one or more non-button inputs, such as user gestures, touch, posture changes, and environmental changes. For example, it can recognize user intentions in real time based on these inputs and automatically execute feedback responses such as lighting, sound effects, vibration, and mechanical movement. Intelligent interactive devices include, but are not limited to: intelligent toys, intelligent wearables, intelligent ornaments, intelligent home control terminals, and portable interactive electronic devices. For example, intelligent toys can be intelligent interactive dolls, intelligent pet toys, intelligent figurines, intelligent robots, etc. Another example is the physical doll involved in patent CN120415746A.

[0029] Sensor data can include data collected by one or more of the following: IMU sensors, touch sensors, and light sensors. Taking an IMU sensor as an example, it can be a six-axis IMU sensor or a nine-axis IMU sensor. A six-axis IMU sensor integrates a three-axis accelerometer and a three-axis gyroscope, capable of collecting acceleration and angular velocity data in three directions respectively. A nine-axis IMU sensor, based on the six-axis IMU sensor, can further integrate a magnetometer, also capable of collecting magnetic data. For example, the acceleration data, collected by the three-axis accelerometer, represents the linear motion intensity of the device in the X, Y, and Z axes (in grams); the angular velocity data, collected by the three-axis gyroscope, represents the rotational speed of the device around the X, Y, and Z axes (in degrees per second).

[0030] The X-axis, also known as the roll axis or left-right direction, is parallel to the horizontal plane of the equipment and points to the right side of the equipment (with the equipment's forward direction as a reference); the Y-axis, also known as the pitch axis or forward-backward direction, is parallel to the horizontal plane of the equipment and points directly in front of the equipment; and the Z-axis, also known as the yaw axis or vertical direction, is perpendicular to the horizontal plane of the equipment and points directly upward, opposite to the direction of gravity.

[0031] For example, when the device is stationary, the acceleration data is approximately (0, 0, 1)g and the angular velocity data is approximately (0, 0, 0)° / s; when the user holds the device and shakes it from side to side, the acceleration data and angular velocity data change significantly in sync.

[0032] By continuously acquiring sensor data at a fixed or dynamically adjustable frequency, a continuous data stream is formed.

[0033] Step 110: Generate upward projectile feature data based on the acceleration data within the first sliding time window.

[0034] In this embodiment, the collected acceleration data is stored in a first sliding time window. The length of the first sliding time window can be a preset fixed length or an adaptively changing suitable length. For example, the length of the first sliding time window is 10 frames, and the window is continuously scrolled and updated to retain the latest 10 frames of acceleration data.

[0035] The upward throw feature data is generated or extracted based on the data within the window. This upward throw feature data characterizes the motion of the device being thrown upwards. Specifically, it is a set of feature quantities extracted / calculated from multi-frame acceleration data that uniquely characterizes the physical motion of "the device being thrown upwards," used to distinguish the throwing motion from ordinary shaking, collisions, vibrations, and other interfering motions. The upward throw feature data is the basis for determining whether the device has entered the upward throw phase. It is composed of one or more dimensions, such as horizontal direction, vertical direction, and amplitude abrupt change. It includes one or more of the following: magnitude in each axis direction, difference between frames, average value, amplitude, etc. For example, it includes one or more of the following: horizontal amplitude (referred to as "horizontal amplitude") change, vertical upward acceleration trend (referred to as "vertical acceleration trend"), acceleration vector amplitude pulse (inter-frame amplitude abrupt change value), etc.

[0036] The horizontal amplitude change can be the difference between the maximum and minimum values ​​of the X-axis or Y-axis acceleration within the window, or the difference in X-axis or Y-axis acceleration between two adjacent frames. A throw is usually not a perfect vertical throw; there will always be some horizontal swaying, but the amplitude will not be too large. If the amplitude is too large, it may resemble a lateral swing rather than a throw. The vertical upward acceleration trend represents the difference in Z-axis acceleration between the end and beginning of the window, or the difference in Z-axis acceleration between two adjacent frames. It reflects the net upward acceleration of the device and is a key indicator of a throw. The acceleration vector amplitude pulse represents the maximum change in the acceleration membrane within the window. At the moment of throw, the total acceleration jumps from 1g to a value much greater than 1g, forming a pulse. The vertical acceleration percentage represents the proportion of the Z-axis acceleration modulus in the total acceleration modulus. In the initial stage of a throw, almost all the energy is concentrated in the vertical direction, so this value is very high.

[0037] Understandably, the data within the window is constantly updated over time, so the resulting upward throw characteristic data will also be updated in real time based on the acceleration data. For example, at the moment a user throws a smart doll upwards, features such as upward acceleration, horizontal jitter, and sudden increase in amplitude can be generated based on the real-time collected acceleration data. These signals are then calculated to form the upward throw characteristic data.

[0038] Step 115: When the upward throwing feature data meets the upward throwing condition, suppress the shaking detection and continue to generate falling feature data based on the acceleration data.

[0039] In this embodiment, the throwing action is divided into an upward throwing phase and a downward falling phase based on its characteristics. Only when both the characteristics of the upward throwing phase and the characteristics of the downward falling phase are met is it determined that the user has initiated a throwing interaction with the smart interactive device. Based on the characteristics of this throwing operation at different phases, upward throwing conditions and downward falling conditions are preset.

[0040] The upward throw condition is a set of rules that determine whether the current motion has entered / is in the upward throw phase. It is used to determine whether the currently collected upward throw feature data is sufficient to confirm that the throw action has begun. The upward throw condition is usually a combination of one or more thresholds. Correspondingly, the falling condition is a set of rules that determine whether the current motion has entered / is in the falling phase. It is the final determination rule to confirm that the device has actually thrown something. The falling condition is a rule used to determine whether the falling feature data constitutes a successful throw event. The falling condition is usually also a combination of one or more thresholds, such as requiring multiple consecutive frames (if the falling frame number threshold is ≥3) to meet the falling frame judgment criteria.

[0041] After extracting acceleration data, upward throwing detection is initiated first. When the upward throwing condition is met, it indicates that the intelligent interactive data is in the upward throwing phase. During this process, swaying detection is suppressed, and further suppression of other interactive actions such as rolling, inversion, reversal, and touch detection can be achieved. This is to avoid confusing the characteristics of the throwing action with swaying, rolling, inversion, reversal, and touch actions, thereby improving the recognition accuracy of various actions.

[0042] Specifically, based on the collected acceleration data, if the upward throwing condition is met, the detection process for shaking, rolling, inverting, reversing, and touching is paused. If the upward throwing condition is not met, the detection of other corresponding actions is initiated.

[0043] For example, if a user throws the device upwards, the device's acceleration changes drastically at the moment of ascent, meeting the conditions for throwing upwards. The system immediately blocks the shaking detection, preventing the upward segment from being mistaken for shaking.

[0044] While suppressing swaying and other motion detection, acceleration data continues to be monitored, and fall feature data is extracted. Fall feature data refers to the characteristic quantities extracted from acceleration data after the peak throw timestamp, used to identify free fall / weightlessness states, and is the basis for confirming the throw action. After the highest point of the throw, the equipment enters a weightless state, and acceleration decreases significantly; this feature is used to capture this physical phenomenon. The peak timestamp refers to the point in time when the equipment is identified as being at its highest point of the throw. Specifically, when a customer detects that the acceleration at the upward edge of the throw reaches its maximum value, this moment is recorded as the peak timestamp. The peak timestamp is used to define the effective time window for free fall verification, avoiding misjudgments due to timeout.

[0045] Step 120: When the falling feature data meets the falling conditions, it is determined that a throwing action has occurred.

[0046] The fall characteristic data may include one or more dimensions of data such as the acceleration modulus and the absolute value of acceleration in the vertical direction within the second sliding time window after the peak timestamp. The second sliding time window is a time window with a preset fixed duration (e.g., 2000ms) starting from the peak timestamp, and free fall / fall verification is only performed within this window.

[0047] The acceleration magnitude (or acceleration vector magnitude, acceleration vector magnitude length, acceleration vector amplitude, total acceleration amplitude) represents the vector magnitude (magnitude) of three-axis acceleration. That is, the acceleration data in the three dimensions of the X, Y, and Z axes are accel.x, accel.y, and accel.z respectively. Therefore, the corresponding acceleration magnitude is calculated as accelMag = sqrt(accel.x...). 2 +accel.y 2 +accel.z 2 Generally speaking, in a stationary state, mag ≈ 1.0g, and in free fall, mag ≈ 0.2g~0.5g. The absolute value of vertical acceleration refers to the absolute value of the Z-axis acceleration, used to determine whether the equipment is in a state of vertical weightlessness. In free fall, the Z-axis acceleration is close to 0g, and its absolute value is much smaller than normal gravity.

[0048] When both the upward throwing condition and the downward falling condition are detected simultaneously, a throwing action is determined to have occurred. This means the user is deemed to have performed a throwing interaction with the smart interactive device. Based on the recognized throwing action and the relevant scenario, the smart interactive device makes an appropriate response, thereby enabling interaction with the user.

[0049] Step 125: When the upward throwing feature data does not meet the upward throwing condition, the validity of the acceleration data is filtered according to the angular velocity data, and the acceleration data corresponding to the angular velocity data that meets the angular velocity swaying feature is taken as the data to be processed.

[0050] In this embodiment, if the upward throwing condition is not met, shaking detection is initiated. Similarly, for shaking detection, in addition to acceleration, rotation is also generated during the shaking motion. Therefore, angular velocity data and acceleration data are comprehensively identified during shaking detection. Angular velocity is used as a gating factor to determine whether the corresponding acceleration data is valid relative to the shaking motion. If valid, it is used as the data to be processed for shaking motion detection.

[0051] Among them, angular velocity swaying characteristics refer to patterns in angular velocity data that can characterize swaying motion. For example, angular velocity data in one or more directions exceeds a corresponding threshold, or the angular velocity magnitude exceeds a corresponding threshold.

[0052] In one embodiment, the angular velocity sway characteristic includes whether the angular velocity magnitude exceeds a preset angular velocity threshold. The angular velocity magnitude gyroMag can be expressed by the formula gyroMag=sqrt(gx 2 +gy 2 +gz 2 ), where gx, gy, and gz represent the angular velocity data in the X, Y, and Z axes, respectively. Shaking, accompanied by rapid back-and-forth rotation of the device, significantly increases the angular velocity modulus, while environmental vibrations (such as desktop shaking) typically do not produce large angular velocities. By setting an angular velocity threshold, environmental vibrations can be filtered out.

[0053] Specifically, the angular velocity modulus is calculated based on the angular velocity data, and it is determined whether the angular velocity modulus exceeds the angular velocity threshold. If it does, the acceleration data corresponding to the acquisition time of the angular velocity data is used as data to be processed; otherwise, the acceleration data corresponding to the acquisition time of the angular velocity data is discarded and not used as data to be processed. The acceleration data corresponding to the acquisition time of the angular velocity data indicates that the timestamps of the two data are the same or similar, that is, the two data were acquired at the same time or within a short period of time.

[0054] The angular velocity model is used to determine whether the equipment is actually rotating. If the angular velocity does not exceed the threshold, the action of the equipment may be environmental noise, thus filtering out environmental noise that only has acceleration changes but no rotation.

[0055] The system iterates through each frame of data in the window. If the angular velocity modulus is detected to be below the angular velocity threshold and no particularly severe acceleration data is generated, it is directly determined that no shaking action has been triggered, and there is no need to continue shaking analysis based on the acceleration data corresponding to the angular velocity data. For example, for a frame with timestamp t, its corresponding angular velocity modulus gyroMag(t) is calculated. If gyroMag(t) exceeds the angular velocity threshold, the acceleration data acc(t) at that moment is considered to be possibly generated by shaking and is retained as data to be processed; if gyroMag(t) does not exceed the angular velocity threshold, the corresponding acc(t) is directly discarded.

[0056] If the angular velocity modulus exceeds the corresponding angular velocity threshold, a shaking analysis is performed on the acceleration data at the same time stamp as the angular velocity data, and this acceleration data is treated as data to be processed. When the upward throw characteristic data does not meet the upward throw condition, it is determined whether the angular velocity modulus exceeds a preset angular velocity threshold, and the acceleration data at the same time stamp as the angular velocity data exceeding the threshold is treated as data to be processed. For angular velocity data that does not exceed the angular velocity threshold, the corresponding acceleration is discarded and not treated as data to be processed.

[0057] For example, the angular velocity threshold is a value set based on practical experience or the actual performance of the device, such as 2.5° / s. When the user gently shakes the doll, the gyroscope data far exceeds 2.5, and the corresponding acceleration data is retained; however, when someone walks by causing a slight vibration in the floor, the accelerometer fluctuates but the doll itself does not rotate (the gyroscope data is close to 0), and the acceleration data is discarded to avoid misjudgment.

[0058] Step 130: Perform mixed filtering on the data to be processed, and calculate the variance value between the first number of consecutive mixed-filtered data to be processed.

[0059] In this embodiment, the hybrid filtering may include both low-pass and high-pass filtering to improve the purity of the jitter signal. The low-pass filter is used to update the baseline to remove long-term drift; the high-pass filter is used to remove slowly changing components to retain the effective signal of rapid jitter.

[0060] Similarly, taking the acceleration data in the three dimensions of the X, Y, and Z axes as accel.x, accel.y, and accel.z respectively, the process of this hybrid filtering can include: baseline.x = a1.x × accel.x + (1 a1.x) × baseline.x'; baseline.y = a1.y × accel.y + (1 a1.y) × baseline.y'; baseline.z = a1.z × accel.z + (1 a1.z) × baseline.z'; filtered.x = (accel.x baseline.x) × a2.x; filtered.y = (accel.y baseline.y) ×a2.y; filtered.z = (accel.z baseline.z) × a2.z baseline.x, baseline.y, and baseline.z are the updated triaxial acceleration baselines used to track slowly changing gravity components and eliminate sensor drift; baseline.x', baseline.y', and baseline.z' are the original triaxial acceleration baselines; a1.x, a1.y, and a1.z represent the corresponding low-pass filter coefficients, for example, all of which can be set to 0.06; a2.x, a2.y, and a2.z represent the corresponding high-pass filter coefficients, for example, all of which can be set to 0.92; filtered.x, filtered.y, and filtered.z represent the effective triaxial jitter signals output by the high-pass filter (i.e., the data to be processed after mixed filtering), used for subsequent variance calculation.

[0061] For each frame of data to be processed, a hybrid filtering process is performed to obtain the corresponding frame of hybrid filtered data to be processed. This hybrid filtered data is then stored in a buffer of length denoted by a first number. This buffer is similar to a sliding window, storing the most recent first number of hybrid filtered data to be processed. This first number can be any suitable number, such as 8.

[0062] Variance calculation is performed on the data in the buffer to obtain the variance values ​​between the first number of consecutively mixed filtered data to be processed. The variance values ​​between the consecutively mixed filtered data to be processed can be the single-axis variance value or the sum of the variance values ​​of three axes in one direction, or they can be the average or maximum value based on the variance values ​​of multiple axes.

[0063] Specifically, for the first number of valid jitter signals filtered.x, filtered.y, and filtered.z calculated above, variance can be calculated independently for one or more dimensions / axis (X-axis, Y-axis, Z-axis directions) to obtain the corresponding single-axis variance value. For example, the X-axis variance can be calculated for the first number of filtered.x signals, and the resulting value can be used as the corresponding X-axis variance value; or the Y-axis or Z-axis variance can be calculated for the first number of filtered.y or filtered.z signals, and the resulting value can be used as the corresponding Y-axis or Z-axis variance value.

[0064] Furthermore, after calculating the variance values ​​of the three axes separately, the maximum or average value of the variance values ​​of the three axes can be taken as the variance value between the first number of continuously mixed filtered data to be processed.

[0065] Alternatively, based on the first number of valid jitter signals filtered.x, filtered.y, and filtered.z, the vector magnitude of filtered.x, filtered.y, and filtered.z for each frame can be calculated to obtain the effective jitter vector magnitude frameMag for the corresponding frame. Then, based on this first number of vector magnitudes, variance is calculated to obtain the three-axis summed variance. Where frameMag = sqrt(filtered.x, filtered.y, and filtered.z). 2 + filtered.y 2 +filtered.z 2 ).

[0066] Taking the calculation of the X-axis variance as an example, there are a first number of valid jitter signals filtered.x: {0.2, 0.5, 0.3, 0.7, 0.4, 0.6, 0.3, 0.5}. Based on the variance calculation formula, the corresponding X-axis variance value can be calculated to be 0.0248.

[0067] Step 135: When the variance value exceeds the variance threshold, it is counted as a valid shake frame.

[0068] When the variance of the filtered acceleration data within a certain time window (e.g., 8 frames) exceeds the current variance threshold, that window is marked as a valid shake frame. It represents that the device's shaking intensity reached the shake level within this short time period.

[0069] Optionally, the variance threshold can be a suitable value set according to the actual situation, such as 0.25, or different variance thresholds can be set according to the variance values ​​calculated in different ways. For example, different variance thresholds can be set for single-axis variance values ​​or three-axis combined variance values. When one or more of the variance values ​​exceed the corresponding variance threshold, the data in the current buffer is counted as a valid shake frame.

[0070] Understandably, the data in the buffer is constantly updated in real time. Each update corresponds to an update of the variance and an identification of whether a new valid shake frame has been generated. Consequently, the count of valid shake frames will also change accordingly. If no valid shake frame has been generated, the cumulative number of valid shake frames remains unchanged or decreases, for example, by subtracting one from the previous cumulative count. If a shake frame has been generated, the cumulative number of valid shake frames increases, for example, by adding one to the previous cumulative count.

[0071] Step 140: When the number of cumulative valid shaking frames exceeds the preset shaking frame count threshold, a shaking action is determined to have occurred.

[0072] In this embodiment, the shaking frame number threshold is a critical value used to determine whether a shaking action has occurred, and it can be any preset suitable value. For example, it can be set to any suitable value greater than the initial value, such as a positive integer greater than the initial value + 1, such as 2, 3, 4, 5, etc.

[0073] A corresponding shake detection period can be set. At the beginning of a period, the number of accumulated valid shake frames is changed to a preset initial value (e.g., 0), and the number of accumulated valid shake frames is adaptively determined to increase, decrease, or remain unchanged as the buffer data is updated. When the number of accumulated valid shake frames reaches the shake frame count threshold, a shake action is determined to have occurred, and the smart interactive device executes the corresponding shake command. If the shake frame count threshold is never reached, a shake action is determined to have occurred within the current period, and the next shake period begins after the current period ends, with the corresponding number of accumulated valid shake frames set to the initial value.

[0074] Optionally, a shake detection judgment period can be set only when no valid shake frames are generated and the cumulative number of valid shake frames remains unchanged. At the beginning of the judgment period, the number of valid shake frames is an initial value (e.g., 0), and it is accumulated from this initial value within the current judgment period until the duration reaches the length of a judgment period. If no valid shake frames are generated, the number remains unchanged at the initial value, and the accumulation starts again from the initial value after entering the next judgment period.

[0075] If no valid shake frames are generated but the cumulative number of valid shake frames decreases, then there is no need to set a shake detection period. In this case, whenever the data in the buffer is updated, the corresponding cumulative number of valid shake frames will also change accordingly. If no valid shake frames are generated, the cumulative number will continue to decrease until it decreases to a preset initial value and remains unchanged.

[0076] The action recognition method based on multi-sensor data fusion in this application prioritizes the recognition of throwing motions. When sensor data features match those of a throwing motion, it suppresses sway detection, achieving mutual exclusion between throwing and swaying motions. This avoids misjudgments of throwing and swaying at the process level and solves the confusion problem caused by overlapping action features in traditional solutions. Furthermore, for sway detection, it combines angular velocity and acceleration data, using angular velocity data as the validity criterion for acceleration data. This enhances anti-interference capabilities, filters out environmental noise such as desktop vibrations and collisions that only change acceleration, and significantly reduces the false trigger rate of swaying motions. After eliminating the possibility of throwing motions and environmental interference, it employs a sliding window + variance + continuous frame recognition method, making sway recognition more stable and accurate, avoiding interference from insensitive actions such as momentary spikes and slight jitters.

[0077] In one embodiment, the variance threshold is a dynamic threshold that is adaptively updated based on the currently calculated variance value.

[0078] In this embodiment, the variance threshold can be dynamically updated, changing based on the previously calculated variance value. Specifically, it is adaptively updated based on the previously updated variance threshold and the currently calculated variance value.

[0079] The initial variance threshold can be a preset fixed value. When the smart interactive device is woken up, the current variance threshold can be adjusted to the corresponding initial fixed value. Thereafter, it will be updated based on this initial fixed value as the interaction with the device progresses.

[0080] Specifically, when the user is actively moving, the variance threshold increases to avoid oversensitivity; when the user is quiet, the variance threshold decreases to maintain the ability to capture subtle movements.

[0081] In one embodiment, the variance threshold is calculated using the following formula: W(t)=(1-b)×W(t-1)+V(t)×b×c.

[0082] Where W(t) is the current variance threshold, W(t-1) is the previously updated variance threshold, V(t) is the currently calculated variance, and b and c are preset coefficients that can be adaptively adjusted according to the device usage scenario. b represents the learning rate, and c represents the energy weight. For example, with b=0.12, c=1.5, W(t-1)=0.8, and V(t)=1.0, the new variance threshold W(t)=0.884.

[0083] As can be seen, the current sharp fluctuation (variance 1.0) slightly increases the dynamic threshold from 0.8 to 0.884, requiring a stronger signal to trigger the shaking next time. If the user's movements become gentler afterward, and the variance V(t) remains at 0.5 for several consecutive times, the threshold will gradually decrease.

[0084] By dynamically updating the variance threshold, when the device experiences severe fluctuations, the dynamic threshold slightly increases from 0.8 to 0.884, requiring a stronger signal to trigger the next shake. If the user's movements become gentler afterward, and the variance V(t) remains at 0.5 for several consecutive times, the threshold will gradually decrease accordingly.

[0085] In one embodiment, after performing mixed filtering on the data to be processed, the method further includes: calculating the instantaneous change between a first number of consecutively mixed filtered data to be processed; when the instantaneous change exceeds a preset instantaneous change threshold, changing the number of effective shaking frames to be no less than the shaking frame number threshold, and determining that a shaking action has occurred.

[0086] Instantaneous change refers to the absolute value of the difference between two adjacent frames of data in a multi-frame (first quantity) dataset after hybrid filtering. It is used to capture sudden and violent shaking in a shaking motion (such as rapidly swinging the device). Instantaneous change focuses on inter-frame abrupt changes, unlike variance (which reflects the overall fluctuation intensity). It can quickly identify sudden and violent shaking motions, enabling rapid triggering of shaking judgment without waiting for the accumulation of consecutive effective shaking frames.

[0087] The instantaneous change threshold is a critical value used to determine whether the instantaneous change has reached the characteristics of sudden shaking. It can be a preset fixed value and can be adjusted according to the sensitivity requirements of the equipment.

[0088] When a momentary change in data between two adjacent frames is detected to exceed a momentary change threshold, a shaking action is directly determined to have occurred. Specifically, the number of valid shaking frames that change in real time can be directly adjusted to a shaking frame count threshold, or a value larger than the shaking frame count threshold, thereby triggering a shaking action.

[0089] The instantaneous change is calculated as the absolute value of the difference between two adjacent frames (frameMag, effective jitter vector magnitude), i.e., the instantaneous change. frameMag = |frameMag_(i) - frameMag_(i-1)|, where frameMag_(i) is the current frame (i.e., the current data to be processed after mixing and filtering), and frameMag_(i-1) is the previous frame relative to the current frame. For example, if the instantaneous change threshold is 0.3, and the frameMag values ​​of two adjacent frames are 0.245 and 0.616 respectively, the instantaneous change is |0.616 - 0.245| = 0.371 > 0.3, triggering a fast judgment.

[0090] Optionally, firstly, for the mixed-filtered data to be processed (data filtered on the X, Y, and Z axes), the three-axis combined modulus value frameMag_i for each frame is calculated according to the above logic (taking 8 frames as an example, i = 1~8); secondly, the instantaneous change ΔframeMag between two adjacent frames in the 8 consecutive frameMag sequences is calculated, resulting in 7 instantaneous change values; thirdly, each instantaneous change value is compared with the corresponding instantaneous change threshold. If any instantaneous change value is greater than the instantaneous change threshold, it indicates that a sudden and severe shaking has occurred, and a fast trigger operation is performed: the current accumulated number of effective shaking frames is forcibly changed to the shaking frame count threshold; since the number of effective shaking frames has reached the shaking frame count threshold, a shaking action is directly determined, without needing to continue the above variance calculation, effective shaking frame counting, and cumulative determination processes, skipping the frame-by-frame accumulation process; if no instantaneous change value is greater than the instantaneous change threshold, other methods are used to detect the shaking action.

[0091] In one embodiment, the method further includes: when the upward throwing feature data does not meet the upward throwing condition, calculating the acceleration modulus based on the acceleration data; when the acceleration modulus exceeds a preset first modulus length threshold, determining that a shaking action has occurred.

[0092] In this embodiment, when the upward throwing condition is not met, in addition to identifying the shaking action by calculating the number of effective shaking frames as described above, the acceleration modulus can also be used to identify whether a shaking action has occurred. The first modulus length threshold is a benchmark value used to determine whether the acceleration modulus reaches an extremely strong shaking characteristic; it can be a preset fixed value, such as 8.0g. When the acceleration modulus exceeds this threshold, it indicates that the device has experienced an extremely strong motion impact.

[0093] Specifically, acceleration modulus calculation can be performed directly on the acceleration data, or the above-mentioned hybrid filtering process can be performed on the acceleration data, and acceleration modulus calculation can be performed on the acceleration data after hybrid filtering. Direct acceleration modulus calculation based on the original acceleration data (without filtering) can quickly capture extremely strong shaking motions (such as forceful swinging or violent shaking), avoiding the omission of extremely strong motions due to filtering, variance calculation and other processes.

[0094] For example, if the upward throw feature data does not meet the upward throw condition (i.e., the current action is not a throw), the sway detection process begins. At this point, the raw acceleration data (accel.x, accel.y, accel.z) of the current frame can be extracted, and the acceleration magnitude of the current frame can be calculated without performing mixed filtering. The calculation formula is: accelMag = sqrt(accel.x... 2 +accel.y 2 +accel.z 2The calculated acceleration modulus is compared with the first modulus length threshold. If the acceleration modulus is greater than the first modulus length threshold, it indicates that the current shaking motion is extremely strong, and the shaking motion is directly determined to have occurred. If the acceleration modulus is less than or equal to the first modulus length threshold, it indicates that the current shaking intensity has not reached an extremely strong level, and the above-mentioned processes of valid screening of acceleration data, dynamic updating of variance threshold, mixed filtering of acceleration data, calculation of instantaneous change, and updating of shaking frame count are continued.

[0095] In one embodiment, the upward throw feature data includes the horizontal amplitude of acceleration data in the horizontal direction, the vertical acceleration trend in the vertical direction, and the acceleration vector amplitude pulse; the upward throw conditions include: the horizontal amplitude exceeding an amplitude threshold, the vertical acceleration trend exceeding an upward throw threshold, and the acceleration vector amplitude pulse exceeding a pulse threshold. The falling feature data includes: the acceleration magnitude and / or the absolute value of vertical acceleration of acceleration data within a second sliding time window after the peak timestamp; the falling conditions include: the number of consecutive falling frames exceeding a preset falling frame number threshold; when the acceleration magnitude is less than a first threshold or the absolute value of vertical acceleration is less than a second threshold, the corresponding acceleration data is determined to be a falling frame, and the peak timestamp is the timestamp corresponding to the acceleration data with the maximum acceleration magnitude within the first sliding time window.

[0096] In this embodiment, the horizontal amplitude of the acceleration data in the horizontal direction refers to the amplitude of acceleration fluctuation of the device moving in the horizontal plane (X-axis or Y-axis), reflecting the intensity of horizontal swaying of the device at the moment of being thrown. Generally speaking, when the device is stationary, the horizontal amplitude is ≈0g; at the moment of being thrown, the acceleration of the device in the X-axis or Y-axis may suddenly change from 0g to 0.25g, at which time the horizontal amplitude is 0.25g.

[0097] The vertical acceleration trend refers to the continuous change in direction and magnitude of the equipment's acceleration in the vertical direction (Z-axis), characterizing whether the equipment has a sustained upward acceleration trend. Vertical upward acceleration is the core characteristic of a tossing motion, manifested as a sustained increase in Z-axis acceleration within a short period. For example, when tossing upwards, the Z-axis acceleration gradually increases from 1.0g to 1.5g and then to 1.8g, forming a clear upward acceleration trend; ordinary shaking is mainly a left-right movement, without a sustained upward acceleration trend.

[0098] The acceleration vector amplitude pulse refers to the instantaneous change in the amplitude of the acceleration vector between two adjacent frames. It characterizes whether the motion intensity suddenly and drastically increases within a frame and is a key feature for identifying projectile impacts. Mag = Current frame acceleration magnitude (current frame acceleration modulus) The magnitude of acceleration in the previous frame (the acceleration modulus in the previous frame). For example, when stationary, Mag≈0g; At the moment of launch: the acceleration jumps from 1.0g to 1.5g. Mag=0.5g, forming a distinct amplitude pulse.

[0099] The upward throw condition can be comprehensively considered from several or all of the following dimensions: X-axis amplitude, Z-axis amplitude, Z-axis amplitude change, and acceleration vector amplitude pulse size. For example, the upward throw condition requires the simultaneous satisfaction of the following three points a1~a3. Among them, condition a1: Horizontal amplitude ampX > horizontal amplitude threshold, or the absolute value of the change in horizontal amplitude | ampX |> Amplitude change threshold; Condition a2: Vertical acceleration trend | ampZ|>the upward throw threshold, and the vertical (Z-axis) amplitude firstZ corresponding to the acceleration data of the previous frame within the first sliding time window>the vertical amplitude threshold; Condition a3: acceleration vector amplitude pulse Mag > pulse threshold (e.g., 0.1g), and the average acceleration modulus meanMag within the first sliding time window > the second modulus length threshold, or the magnitude of the acceleration modulus drop from the peak magDrop within the first sliding time window > the magnitude drop threshold.

[0100] The values ​​of various thresholds, such as the horizontal amplitude threshold, amplitude change threshold, upward throw threshold, vertical amplitude threshold, pulse threshold, second modulus length threshold, and amplitude reduction threshold, can be set according to actual conditions. For example, the horizontal amplitude threshold can be set to 0.2, the amplitude change threshold to 0.1, the upward throw threshold to 0.1, the vertical amplitude threshold to 0.1, the pulse threshold to 0.1, the second modulus length threshold to 1.2, and the amplitude reduction threshold to 0.5.

[0101] The second sliding time window is a preset fixed duration (e.g., 2000ms) starting from the peak timestamp. Data within this time window is subjected to free fall verification to identify whether the device is in a descent state. The size of the second sliding time window can be the same as or different from the size of the first sliding time window.

[0102] When the upward throw condition is met, the acceleration data corresponding to the peak timestamp is identified from the first sliding time window, and the acceleration data corresponding to the peak timestamp is used as the starting frame data of the second sliding time window.

[0103] For the acceleration data within the second sliding time window, the corresponding acceleration magnitude or the corresponding absolute value of vertical acceleration is calculated. If the acceleration magnitude is less than the first threshold, or the absolute value of vertical acceleration is less than the second threshold, the corresponding data frame is determined to be a falling frame. The number of consecutive falling frames identified as falling frames within the second sliding time window is counted. If this number of consecutive falling frames exceeds a preset falling frame count threshold, the falling condition is also satisfied. This indicates that the device's action satisfies both the upward throwing condition and the falling condition, thus determining that a throwing action has occurred.

[0104] The first threshold, the second threshold, and the number of falling frames threshold can all be set to any suitable value, such as setting the first threshold to 0.6, the second threshold to 0.4, and the number of falling frames threshold to 3.

[0105] In one embodiment, the upward-throwing feature data includes the proportion of vertical acceleration, and the upward-throwing conditions include: the proportion of vertical acceleration in N1 consecutive frames exceeds a proportional threshold, or the acceleration data in N2 consecutive frames shows a sudden acceleration change in the vertical direction.

[0106] In this embodiment, the upward throw condition can be set not only to conditions a1~a3 mentioned above, but also to conditions b1 and b2. Condition b1: The proportion of vertical acceleration in N1 consecutive frames exceeds a proportional threshold; Condition b2: Acceleration data in N2 consecutive frames shows a sudden acceleration change in the vertical direction. When conditions b1 or b2 are met, or conditions a1~a3 are met, it can be determined that the upward throw feature data meets the upward throw condition.

[0107] The vertical acceleration percentage refers to the proportion of the absolute value of the vertical (Z-axis) acceleration in the total amplitude of the three-axis acceleration (i.e., the acceleration magnitude), used to determine whether the current motion is predominantly vertically upward. A vertical acceleration abrupt change refers to a sudden and significant increase in Z-axis acceleration between adjacent frames. This acceleration abrupt change can be reflected by the difference in Z-axis acceleration amplitude between adjacent frames. (ampZ) exceeds the preset throw threshold.

[0108] The system identifies data frames that meet the above criteria. If N1 consecutive frames show a vertical acceleration percentage exceeding a threshold, the upward throw condition is satisfied. Similarly, if N2 consecutive frames show a sudden acceleration change in the vertical direction, the upward throw condition is also satisfied. N1, N2, and the threshold can all be set to any suitable value; for example, the threshold could be 70%, and N1 and N2 could both be 3.

[0109] In this embodiment, by setting multiple throwing and falling conditions, and by identifying whether the throwing feature data meets the throwing conditions and the falling feature data meets the falling conditions in stages, the accuracy of the throwing action recognition can be improved.

[0110] In one embodiment, the sensor data further includes touch data; the method further includes: suppressing touch detection when the upward throw feature data meets the upward throw condition; initiating touch detection based on the touch data when the upward throw feature data does not meet the upward throw condition; controlling the smart interactive device to execute a touch response and resetting the no-operation timer when a valid touch action is detected; and controlling the smart interactive device to enter a low-power mode when the duration of the no-operation timer reaches a preset low-power duration.

[0111] In this embodiment, the device also integrates a touch sensor, which can be one or more of a capacitive touch sensor, a resistive touch sensor, etc. Touch data can be user touch operation signals collected by the touch sensor, used to characterize whether the user touched the device, the touch location, and the touch duration. Touch detection refers to the detection of touch actions, which can include one or more specific touch type actions such as single tap, continuous tap, single stroking, continuous stroking, long press, poking, and swiping.

[0112] Specifically, after identifying that the tossing condition is met, touch detection, shaking detection, light sensing detection, rolling detection, reversal detection, and inversion detection are suppressed for a preset time period following the aforementioned peak timestamp. This preset time period is used to prevent sensor data generated by the user or the environment during the tossing action from being misinterpreted as valid touch, shaking, rolling, reversal, or inversion operations. The preset time period can be the duration corresponding to the second sliding time window mentioned above, or any suitable duration, such as 2000ms. That is, after the tossing peak occurs, touch detection, shaking detection, light sensing detection, rolling detection, reversal detection, and inversion detection are not performed within the preset time period.

[0113] When the upward throw feature data does not meet the upward throw condition, or after a preset time after the peak timestamp, touch detection is initiated based on the touch data to identify whether touch actions of the following types have occurred: single tap, continuous tap, single stroking, continuous stroking, long press, poking, sliding, etc.

[0114] A valid touch action refers to a touch operation that meets preset touch conditions, distinct from accidental touches (such as unintentional touches during tossing). It is used to trigger the device's touch response. A no-operation timer is a timing module used to count the cumulative duration of no valid interaction actions on the device (such as no tossing, no shaking, no valid touch, no rolling, no reversing, no inversion, etc.). This is used to trigger the device's low-power mode and reduce energy consumption. Low-power duration refers to the base duration for the device to enter low-power mode when the no-operation timer accumulates to this duration. For example, this duration can be set to 30000ms (30 seconds) or any other suitable value.

[0115] The inactive timer starts at 0. Whenever the device detects a valid touch, toss, or shake action, the timer is reset to 0. If no valid interaction is detected, the timer continues to increment until the preset low-power duration is reached, triggering the low-power mode.

[0116] The device can initiate touch detection in two situations: 1. If the upward throwing feature data does not meet the upward throwing conditions (the current action is not throwing), touch detection is initiated immediately to identify touch data in real time; 2. If the upward throwing feature data meets the upward throwing conditions, but has exceeded the preset time (2000ms) after the peak timestamp, touch detection suppression is lifted and touch detection is initiated.

[0117] After touch detection is initiated, it continuously checks whether the touch data meets the conditions for a valid touch action. If a valid touch action is detected, the smart interactive device is controlled to execute a preset touch response (such as playing voice, turning on an LED, or providing vibration feedback), while the inactivity timer is reset to 0 and restarted. If no valid touch action is detected, the inactivity timer continues to accumulate. It also continuously checks whether the accumulated duration of the inactivity timer has reached a preset low-power duration: ① If it has, the smart interactive device is controlled to enter a low-power mode, reducing the sensor acquisition frequency and disabling unnecessary feedback modules; if it has not reached the preset low-power duration, it continues to maintain normal operation mode, continuously monitoring touch data and other action data.

[0118] Furthermore, when a valid touch action is detected, the historical data buffers for acceleration and angular velocity are cleared, requiring sufficient samples to be re-accumulated before relevant threshold updates can be restored. This multi-source feedback mechanism ensures that the dynamic thresholds comprehensively reflect the overall motion state of the device, rather than relying solely on data from a single sensor.

[0119] In one embodiment, the sensor data further includes light-sensing data; the method further includes: suppressing light-sensing detection when the upward-throwing feature data meets the upward-throwing condition; when the upward-throwing feature data does not meet the upward-throwing condition, calculating the light-sensing difference between the largest and smallest light-sensing data in the light-sensing data within the third sliding time window, and calculating the index difference between the index value of the largest and smallest light-sensing data; and analyzing whether a valid light-sensing change has occurred based on the light-sensing difference and the index difference.

[0120] In this embodiment, the light-sensing data is the ambient light intensity signal collected by the light-sensing sensor (such as a photoresistor, photodiode, etc.) built into the smart interactive device, representing the brightness of the environment in which the device is located. For example, when the device is in natural indoor light, the light-sensing data (brightness) = 600; when moved to a dark room, the light-sensing data = 100; and when moved to strong light, the light-sensing data = 900.

[0121] When the device is in the process of being tossed up, the analysis and recognition of light sensing data is paused, and changes in light sensing data are ignored to avoid changes in illumination caused by the movement of the device position during tossing up being misjudged as valid changes in light sensing.

[0122] The third sliding time window is used to collect light-sensing data and calculate the light-sensing difference and index difference. It is independent of the first sliding time window (throw-up detection) and the second sliding time window (fall detection). For example, it can be set to 10 frames (corresponding to 100ms, consistent with the sensor's acquisition cycle). The third sliding time window continuously scrolls and updates, retaining the light-sensing data of the most recent 10 frames.

[0123] The light sensitivity difference refers to the difference between the maximum and minimum light sensitivity data within the third sliding time window, used to characterize the amplitude of ambient light brightness change within the window. That is, light sensitivity difference = maximum light sensitivity data within the window. The minimum ambient light sensitivity value within the window. A larger difference in ambient light sensitivity indicates a more drastic change in ambient light brightness; a smaller difference in ambient light sensitivity indicates a more stable ambient light with no significant change.

[0124] The index value represents the sequence number of the data within the corresponding sliding time window. For example, the 10 frames of light-sensing data in the third sliding time window are [500, 520, 510, 480, 490, 150, 160, 140, 170, 155]. The index value corresponding to the maximum light-sensing data 520 is 2, and the index value of the minimum light-sensing data 140 is 8. The index difference is 6. The index difference between the maximum and minimum light-sensing data reflects the time interval between the occurrence of maximum and minimum brightness, used to distinguish between rapid light-sensing changes (such as suddenly turning off the lights) and slow light-sensing changes (such as gradually drawing the curtains). An effective light-sensing change refers to a light-sensing change whose light-sensing difference and index difference meet the preset judgment conditions, that is, a significant change in ambient light deliberately triggered by the user (not a change caused by instantaneous interference or throwing movement), used to trigger the corresponding interactive response of the device.

[0125] The intelligent interactive device continuously collects sensor data, including acceleration and angular velocity data, and simultaneously collects light-sensing data output by the light sensor. It then determines in real time whether the upward throwing characteristic data meets the upward throwing conditions: 1. If the upward throwing conditions are met (currently in the process of throwing), light sensing is suppressed, the analysis and recognition of light-sensing data are paused, all changes in light-sensing data are ignored, and light-sensing data no longer participates in action recognition; 2. If the upward throwing conditions are not met (currently not in the process of throwing), light sensing is activated, and the light change analysis process begins.

[0126] The system maintains a light sensor failure flag. When the number of consecutive triggers of the light sensor reaches a preset threshold (e.g., 10 times), and the triggered action ID belongs to the failure list, the light sensor failure flag is set to true. Afterward, the light sensor detection function directly returns false at the entry point, and the light sensor data is no longer used for action recognition.

[0127] After initiating light sensing detection, the real-time collected light sensing data is stored in a third sliding time window (e.g., 10 frames). The window continuously scrolls, always retaining the light sensing data from the most recent 10 frames. The light sensing data within the third sliding time window is analyzed to calculate two key parameters: light sensing difference and index difference. Based on preset judgment conditions, it is analyzed whether a valid light sensing change has occurred. If the conditions are not met (e.g., light sensing difference ≤ 200, or the absolute value of the index difference < 2), it is judged as an invalid light sensing change (e.g., slight light fluctuation, slow light sensing change), and no response is executed; monitoring of light sensing data continues.

[0128] For example, the criteria for determining the light perception difference are: the light perception difference must be greater than the preset light perception threshold (e.g., 200). If the light perception difference exceeds the corresponding light perception threshold and the index difference is negative, then it is determined that a valid light perception change from bright to dark has occurred; if the light perception difference exceeds the corresponding light perception threshold and the index difference is positive, then it is determined that a valid light perception change from dark to bright has occurred.

[0129] In one embodiment, the method further includes: suppressing roll detection when the upward throw feature data meets the upward throw condition; calculating the roll angle at the corresponding moment based on the acceleration data and angular velocity data when the upward throw feature data does not meet the upward throw condition; and determining that a rollover operation has occurred when the time interval between the roll angles at two moments exceeds a preset time interval threshold and the angle change value between the roll angles at two moments exceeds a preset roll angle change threshold.

[0130] In this embodiment, similarly, the device may rotate during the throwing action (such as a slight flip when throwing). This rotation is not a rolling action intentionally triggered by the user. Therefore, it is necessary to suppress the rolling detection to ensure the accuracy of the rolling recognition and to make it mutually exclusive with the throwing detection.

[0131] Roll angle is one of the attitude angles calculated by the IMU through the fusion of acceleration and angular velocity data. It represents the left and right tilt angle of the device around the X-axis (horizontal axis), and the unit is degrees (°). Based on the static gravity tilt angle reference provided by the accelerometer, combined with the dynamic integral correction of the gyroscope angular velocity, the corresponding roll angle is calculated through attitude fusion algorithms (such as complementary filtering).

[0132] The time interval threshold refers to the baseline value that the time interval between two roll angles must exceed. It is used to distinguish between rapid rolls and slow tilts, for example, 500ms. If the time interval between two roll angles exceeds this threshold, it means that the change in roll angle is continuous and stable, rather than an instantaneous fluctuation; if it does not exceed the threshold, it means that it is an instantaneous flip (such as a slight shaking) and is not judged as a roll.

[0133] The roll angle change threshold refers to the baseline value that the difference in roll angle between two moments must exceed. It characterizes the amplitude of a rollover maneuver, for example, set to 50°. The roll angle change threshold is the core criterion for determining a rollover maneuver. Only when the angle change exceeds this threshold is it considered a significant left-right roll (i.e., a rollover maneuver); if the change does not exceed the threshold, it indicates only a slight tilt and is not considered a rollover. Roll angle change value = |Current roll angle The roll angle at the previous moment.

[0134] The intelligent interactive device continuously collects acceleration and angular velocity data, performs real-time roll angle calculation, and determines in real-time whether the upward throw characteristic data meets the upward throw conditions. If the upward throw conditions are met, roll detection is suppressed and the rotational changes of the device are ignored; if the upward throw conditions are not met (the current action is not a throw), roll detection is initiated, and the roll angle calculation and roll judgment process begins. After roll detection is initiated, the roll angle roll(t) at the current moment is calculated based on the acceleration and angular velocity data of the current frame; at the same time, the roll angles roll(t-1) to roll(tn) and the corresponding moments t-1 to tn of the previous 1 to n moments are recorded.

[0135] The system identifies the time intervals between roll angles at multiple time points, extracts two roll angles whose time intervals exceed the corresponding time interval threshold, calculates the roll angle change value of these two roll angles, and determines that a rollover operation has occurred if one of the roll angle change values ​​is greater than the roll angle change threshold. The control device then executes a rollover response (such as playing a rollover sound effect or switching a light mode). If the roll angle change value between any two roll angles exceeding the corresponding time interval threshold is less than the roll angle change threshold, then it is determined that no rollover operation has occurred, and the system continues to calculate the roll angle and monitor the rollover status.

[0136] In one embodiment, the method further includes: suppressing reversal detection when the upward throw feature data meets the upward throw condition; calculating the roll angle and pitch angle at the corresponding moment based on the acceleration data and angular velocity data when the upward throw feature data does not meet the upward throw condition; and determining that a reversal action has occurred when the absolute value of the roll angle is less than a first absolute threshold and the absolute value of the pitch angle is greater than a second absolute threshold.

[0137] Pitch is also one of the attitude angles calculated by the IMU through the fusion of acceleration and angular velocity data. It represents the tilt angle of the device around the Y-axis, and the unit is degrees (°). It can also be calculated based on the static gravity tilt angle reference provided by the accelerometer, combined with the dynamic integral correction of the gyroscope angular velocity, through attitude fusion algorithms (such as complementary filtering).

[0138] The first absolute threshold is used to determine whether the roll angle is close to horizontal (no left or right roll); the second absolute threshold is used to determine whether the pitch angle meets the attitude requirements for reversal. For example, the first absolute threshold can be set to 20° and the second absolute threshold can be set to 170°. When |roll| < 20° and |pitch| > 170° is detected, it is determined that a reversal action has occurred, and the corresponding response is executed.

[0139] In one embodiment, light detection is also suppressed when the device is in an inverted state.

[0140] In one embodiment, when the upward throw feature data meets the upward throw condition, inversion detection is suppressed; when the upward throw feature data does not meet the upward throw condition, the roll angle and pitch angle at the corresponding moment are calculated based on the acceleration data and angular velocity data; when the absolute value of the roll angle is less than a third absolute threshold and the pitch angle is within a preset pitch angle threshold range, an inversion action is determined to have occurred.

[0141] The third absolute threshold and pitch angle threshold range are used to determine whether the roll angle and pitch angle meet the inverted posture requirements, respectively. For example, when the third absolute threshold is greater than the first absolute threshold and less than the second absolute threshold, such as setting the third absolute threshold to 50°, the lower limit of the pitch angle threshold range can be set to -105° and the upper limit to -75°.

[0142] If |roll| < 50° and -105° < pitch < -75°, then an inverted action is determined to have occurred, and the control device executes the corresponding response.

[0143] The action recognition method based on multi-sensor data fusion in this application has the following technical advantages: 1. High-precision motion recognition: Through bidirectional coupling filtering of accelerometer and gyroscope, and unidirectional suppression gating of throwing and shaking, the gyroscope angular velocity is used as the gating condition for the validity of accelerometer data, and the accelerometer filtering variance affects the judgment threshold of subsequent frames through EMA feedback, achieving bidirectional deep coupling between the two sensors at the measurement process level. The intermediate state data generated by the two-stage state machine of throwing detection is used as the pre-gating condition for multiple detection modules such as shaking, rolling, inverting, reversing, and slapping, eliminating feature overlap and misjudgment between actions at the signal processing level, realizing deep fusion at the sensor measurement process level, improving the accuracy of motion recognition, and reducing the confusion rate between throwing and shaking.

[0144] 2. The dynamic threshold is influenced by multiple paths, including direct accelerometer variance control, indirect gyroscope gating, paused updates during launch, and touch event buffer reset. This achieves true multi-source feedback adaptation, ensuring the threshold continuously adjusts to user habits and environmental conditions. During intense use, the threshold automatically increases to avoid over-triggering, while in quiet environments, it automatically decreases to maintain sensitivity.

[0145] 3. Intelligent State Management: Multi-sensor fusion results drive a finite state machine to perform context-aware state switching, reducing accidental operations. IMU motion state suppresses touch event processing, while touch events, in turn, affect IMU-related state transitions and data acquisition switches, forming a two-way influence relationship. For example, after being thrown into the air, the device automatically enters a low-power standby state, and after being woken up by a touch, it automatically resumes normal operating mode, which can extend battery life.

[0146] 4. High-efficiency resource utilization: IMU and light sensor data acquisition are turned off in low power mode, the sliding window mechanism limits memory usage, and the buffer triggered by touch events releases historical data in a timely manner, significantly reducing power consumption.

[0147] Furthermore, embodiments of this application also provide a computer-readable storage medium storing executable instructions thereon, which, when executed by a processor, cause the processor to perform the methods described in the above-described method embodiments.

[0148] Furthermore, this application also provides a computer program product, the computer program product including a non-transitory computer-readable storage medium storing a computer program, which, when connected to a computer device, is executed by one or more processors of the computer device, is capable of performing all or part of the steps of the method described in any embodiment of this application.

[0149] In one embodiment, an electronic device is also provided, including one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the steps in the above-described method embodiments. This electronic device may be, for example, the aforementioned intelligent interactive device.

[0150] In one embodiment, such as Figure 2 The diagram illustrates the structure of an electronic device used to implement an embodiment of this application. The electronic device includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 202 or a program loaded from a storage portion 208 into a random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the electronic device 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0151] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0152] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the instructions are executed by central processing unit (CPU) 201, the various method steps described in this application are performed.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0154] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments. For example, all the embodiments above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A motion recognition method based on multi-sensor data fusion, characterized in that, The method includes: Continuously acquire sensor data from intelligent interactive devices, wherein the sensor data includes at least acceleration data and angular velocity data; Generate upward projectile feature data based on the acceleration data within the first sliding time window; When the upward throwing feature data meets the upward throwing condition, the shaking detection is suppressed, and the falling feature data continues to be generated based on the acceleration data. When the falling feature data meets the falling condition, it is determined that a throwing action has occurred. When the upward throwing feature data does not meet the upward throwing condition, the validity of the acceleration data is filtered according to the angular velocity data, and the acceleration data corresponding to the angular velocity data that meets the angular velocity swaying feature at the acquisition time is taken as the data to be processed. Perform a mixed filter on the data to be processed, and calculate the variance between a first number of consecutive mixed-filtered data to be processed. When the variance value exceeds the variance threshold, it is counted as a valid shake frame. When the cumulative number of valid shake frames exceeds the preset shake frame number threshold, it is determined that a shake action has occurred.

2. The method according to claim 1, characterized in that, The angular velocity swaying feature includes whether the angular velocity modulus exceeds a preset angular velocity threshold; the step of filtering the validity of the acceleration data based on the angular velocity data, and taking the acceleration data corresponding to the angular velocity data with the acquisition time matching the angular velocity swaying feature as the data to be processed, includes: Calculate the angular velocity modulus based on the angular velocity data, and determine whether the angular velocity modulus exceeds the angular velocity threshold. If so, the acceleration data corresponding to the acquisition time and angular velocity data will be used as the data to be processed; otherwise, the acceleration data corresponding to the acquisition time and angular velocity data will be discarded and will not be used as the data to be processed.

3. The method according to claim 1, characterized in that, The upward throw characteristic data includes the horizontal amplitude of acceleration data in the horizontal direction, the vertical acceleration trend in the vertical direction, and the acceleration vector amplitude pulse; the upward throw conditions include: the horizontal amplitude exceeds the amplitude threshold, the vertical acceleration trend exceeds the upward throw threshold, and the acceleration vector amplitude pulse exceeds the pulse threshold. The falling feature data includes: the acceleration modulus and / or the absolute value of vertical acceleration of the acceleration data within the second sliding time window after the peak timestamp; the falling conditions include: the number of consecutive falling frames exceeds a preset falling frame number threshold; when the acceleration modulus is less than the first threshold or the absolute value of vertical acceleration is less than the second threshold, the corresponding acceleration data is determined to be a falling frame; the peak timestamp is the timestamp corresponding to the acceleration data of the maximum acceleration modulus within the first sliding time window.

4. The method according to claim 1, characterized in that, The upward throw feature data includes the proportion of vertical acceleration, and the upward throw conditions include: the proportion of vertical acceleration in N1 consecutive frames exceeds a proportional threshold, or the acceleration data in N2 consecutive frames shows a sudden acceleration change in the vertical direction.

5. The method according to any one of claims 1 to 4, characterized in that, The sensor data also includes touch data; the method further includes: When the upward throw feature data meets the upward throw condition, touch detection is suppressed; When the upward throw feature data does not meet the upward throw condition, touch detection is initiated based on the touch data; When a valid touch action is detected, the intelligent interactive device is controlled to execute a touch response and the inactive timer is reset. When the inactive timer reaches a preset low-power duration, the intelligent interactive device is controlled to enter a low-power mode.

6. The method according to any one of claims 1 to 4, characterized in that, The sensor data also includes light-sensing data; the method further includes: When the upward projection feature data meets the upward projection condition, light-sensitive detection is suppressed; When the upward throw feature data does not meet the upward throw condition, the light difference between the largest and smallest light-sensing data is calculated based on the light-sensing data in the third sliding time window, and the index difference between the index value of the largest light-sensing data and the index value of the smallest light-sensing data is calculated. Based on the light perception difference and the index difference, it is analyzed whether a valid light perception change has occurred.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the upward throw feature data meets the upward throw condition, tumbling detection is suppressed; When the upward throw characteristic data does not meet the upward throw condition, the roll angle at the corresponding moment is calculated based on the acceleration data and angular velocity data. When the time interval between two roll angles exceeds a preset time interval threshold, and the angle change value between the two roll angles exceeds a preset roll angle change threshold, a rollover operation is determined to have occurred.

8. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the upward-throwing feature data meets the upward-throwing condition, inversion detection and reversal detection are suppressed. When the upward throw characteristic data does not meet the upward throw condition, the roll angle and pitch angle at the corresponding moment are calculated based on the acceleration data and angular velocity data. When the absolute value of the roll angle is less than the first absolute threshold and the absolute value of the pitch angle is greater than the second absolute threshold, a reversal action is determined to have occurred. When the absolute value of the roll angle is less than the second absolute threshold and the pitch angle is within the preset pitch angle threshold range, it is determined that an inverted action has occurred.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1 to 8.

Images