An event-triggered audio detection method based on a smart wearable device

Abstract

The present application relates to the field of acoustic sensing and industrial intelligent detection technology, and discloses an event-triggered audio detection method based on intelligent wearable devices, which adopts a wearable device integrated with a six-axis IMU for close-fitting deployment, fuses acceleration and angular velocity data through Kalman filtering to obtain stable attitude angles and accurately determine the start and completion states of industrial operations, and realizes event-triggered audio acquisition; the acquired audio is sequentially subjected to weak signal filtering, Hilbert envelope extraction, adaptive threshold pulse detection, and the pulse intelligibility score is calculated through multi-dimensional acoustic feature weighting to complete automatic discrimination of operation quality; finally, the result is fed back to the wearable device in real time to form a man-machine collaborative quality inspection closed loop. The present application significantly improves the audio acquisition trigger precision and weak feature signal recognition accuracy, has the characteristics of low power consumption, no blind area, quantification, real-time feedback, and is suitable for in-situ, mobile, high-consistency quality inspection in scenes such as industrial assembly, electrical plugging and structural fastening.

Description

Technical Field

[0001] This invention relates to the field of acoustic sensing and industrial intelligent detection technology, specifically to an event-triggered audio detection method based on a smart wearable device. Background Technology

[0002] In industrial assembly, electrical connection, and structural fastening production scenarios, critical operations such as snap-fit ​​engagement, insertion, and bolt tightening are often accompanied by weak, transient characteristic acoustic signals. These signals are crucial for determining whether operations are performed correctly and connections are reliable. Traditional detection methods, relying on human auditory and visual experience, are highly subjective, lack standardized judgment criteria, and are difficult to quantify and trace. Post-event electrical performance testing is significantly delayed, unable to provide real-time guidance for on-site operations, and fails to meet the high-efficiency, high-reliability process control requirements of intelligent manufacturing.

[0003] Existing acoustic monitoring systems mostly employ fixed microphone arrays, handheld devices, or large equipment integrated modules. Their deployment locations are fixed and cannot move with operators, resulting in significant blind spots in multi-station, mobile, and space-constrained scenarios, making it difficult to capture key audio signals in their optimal location. Furthermore, these systems generally use a continuous, all-time acquisition mode without triggering control based on actual operational actions, leading to inaccurate timing of audio acquisition start and stop, large amounts of invalid data, and high device power consumption. These are the core shortcomings that the S2 action-precise trigger acquisition method of this invention aims to address.

[0004] In the signal processing and recognition stages, existing solutions mostly employ simple amplitude thresholds or fixed parameter detection, lacking sophisticated processing mechanisms for weak transient acoustic signals. This makes it difficult to accurately extract "click"-like characteristic pulses, resulting in inaccurate identification of characteristic acoustic signals. Operational compliance judgments rely on subjective experience, leading to low consistency and accuracy in detection. While smart wearable devices offer advantages such as close-fitting, motion-following, and low-intrusion capabilities, they are currently not applied to in-situ acoustic sensing in industry, failing to achieve accurate acquisition, accurate recognition, and real-time feedback of industrial operation audio. Summary of the Invention

[0005] The present invention aims to provide an event-triggered audio detection method based on smart wearable devices to solve the problems of inaccurate audio acquisition triggering and inaccurate identification of characteristic acoustic signals during industrial operations.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: An event-triggered audio detection method based on a smart wearable device includes: S1, the smart wearable device integrating a six-axis IMU is worn on the operator's hand to start the smart wearable device; S2 continuously acquires triaxial acceleration and triaxial angular velocity data at a preset sampling rate, and uses Kalman filtering to fuse the accelerometer and gyroscope data to obtain a stable attitude angle; based on the filtered data, it performs industrial operation start and operation completion judgments to trigger or terminate audio acquisition. S3 sequentially performs signal preprocessing and weak signal filtering, feature pulse detection, pulse clarity quantification and classification on the acquired audio data, and outputs industrial operation quality judgment results based on pulse clarity scores. S4 feeds back the judgment results to smart wearable devices in real time, forming a closed loop of human-machine collaborative quality inspection.

[0007] The principle and advantages of this solution are as follows: In practical applications, it relies on a six-axis IMU and Kalman filtering to achieve precise action triggering, avoiding the high power consumption and data redundancy caused by continuous recording; it achieves in-situ audio acquisition of industrial operations through wearable, close-fitting deployment, overcoming the shortcomings of large blind spots and poor flexibility of fixed equipment deployment; it achieves accurate identification of the start and end of industrial operations through Kalman filter fusion and action threshold determination, ensuring that audio acquisition is only activated during effective action periods, achieving accurate acquisition timing, less redundant data, and lower power consumption; and it achieves accurate detection of weak characteristic acoustic signals and quantitative judgment of operation quality by performing weak signal removal, transient pulse extraction, adaptive threshold detection, and multi-dimensional clarity scoring on audio signals, solving the problems of strong subjectivity, inaccurate identification, and inability to quantify in traditional detection methods, thus achieving "accurate acquisition, precise identification, and stable judgment" in-situ industrial audio detection.

[0008] Preferably, as an improvement, the fusion of accelerometer and gyroscope data using Kalman filtering includes: Initialization noise covariance matrix Q and observation noise covariance matrix R:

[0009]

[0010] in, This represents the variance of the attitude angle process noise. This represents the noise variance during the gyroscope's angular velocity process. This represents the variance of the attitude angle observation noise. The Kalman filter model parameters are updated at preset intervals. The inputs are the pitch angle calculated by the accelerometer and the Y-axis angular velocity of the gyroscope, and the output is the filtered and stable angle.

[0011] Technical effect: By integrating accelerometer and gyroscope data, acceleration jitter and gyroscope drift are eliminated, resulting in a smooth and stable attitude angle output, thus improving the accuracy of motion recognition.

[0012] Preferably, as an improvement, the industrial operation start determination includes: Monitor the filtered pitch angle θ and the gyroscope Y-axis angular velocity. The hand gesture detection window for a preset duration will be activated when the following conditions are met:

[0013]

[0014] Within this window, if the pitch angle symbol is reversed a preset number of times and the interval between adjacent reversals meets the requirements, it is determined to be a valid operation preparation action, triggering audio acquisition.

[0015] Technical benefits: Facilitates accurate identification of the starting intent for typical operations such as plugging and assembly, avoids accidental triggering, and ensures that audio acquisition starts synchronously with the actual action.

[0016] Preferably, as an improvement, the operation completion determination includes: After audio acquisition is started, calculate the resultant acceleration amplitude, resultant angular velocity amplitude, and acceleration deviation:

[0017]

[0018]

[0019] in, For the resultant acceleration amplitude, The resultant angular velocity amplitude, For acceleration deviation, , , These represent the X, Y, and Z axis acceleration components output by the six-axis IMU in the smart wearable device. , , These are the X-axis, Y-axis, and Z-axis angular velocity components output by the six-axis IMU, respectively. when Less than the acceleration threshold When the angular velocity is less than the threshold and continues for a preset duration, and the recorded duration is not less than the preset minimum requirement, the action is considered to have ended, and audio recording is terminated.

[0020] Technical benefits: It facilitates accurate judgment of when the operation stops, automatically ends the acquisition, retains only the valid audio segments, and reduces redundant data and power consumption.

[0021] Preferably, as an improvement, the signal preprocessing and weak signal filtering include: The multi-channel audio retains only the first channel; the maximum absolute amplitude of the signal is calculated, and if the maximum absolute amplitude is less than the preset weak signal threshold, it is directly determined as an invalid signal and the judgment result is returned.

[0022] Technical benefits: It facilitates the removal of empty recording segments without effective sound, avoids invalid calculations, and improves the efficiency and reliability of judgment.

[0023] Preferably, as an improvement, the characteristic pulse detection includes: The audio signal and its envelope were extracted using Hilbert transform, and a smooth envelope was obtained by Savitzky-Golay filtering.

[0024]

[0025]

[0026] in, This is the original audio time-domain signal; The analytic signal is obtained by the Hilbert transform; j is the imaginary unit; { } represents the Hilbert transform operator; To analyze the modulus of the signal, i.e., the instantaneous envelope of the audio signal; For Savitzky-Golay filtering operators; The smoothed envelope is the result of smoothing processing. An adaptive threshold is set based on the absolute deviation of the median. The envelope peak value is detected and the pulse width is calculated to filter out effective pulses with a duration between 0.2ms and 2.0s.

[0027] Technical effects: Accurately extracts transient "click" pulses, suppresses noise, preserves true features, and improves the detection rate of weak and effective signals.

[0028] Preferably, as an improvement, the adaptive threshold is:

[0029] Where T is the adaptive detection threshold; min() is the minimum value function; The maximum value of the smooth envelope; This is a median function; is the smoothed envelope signal after Savitzky-Golay filtering; MAD is the median absolute deviation.

[0030] Technical benefits: It facilitates adaptive threshold adjustment based on real-time noise levels, has strong anti-interference capabilities, and can stably detect target pulses under different operating conditions.

[0031] Preferably, as an improvement, the pulse clarity quantification and classification includes: extracting multi-dimensional acoustic features from effective pulse segments, calculating a comprehensive clarity score by weighting them according to preset weights; if the clarity score of any pulse reaches a preset threshold, it is judged as qualified, otherwise it is judged as unqualified.

[0032] Technical benefits: It facilitates the conversion of auditory "crispness / blurriness" into calculable values, enabling objective quantitative judgment of insertion quality with high consistency.

[0033] Preferably, as an improvement, the comprehensive sharpness scoring formula is as follows:

[0034] in, ~ These are the weighting coefficients for the corresponding features. A comprehensive score for pulse clarity; The proportion of high-frequency energy is ≥4kHz; The 95% energy spectrum roll-off normalized value; is the normalized value of the spectral centroid; brightness is the brightness characteristic; For the sharpness of the envelope rise; Normalized zero-crossing rate; This is the normalized peak factor.

[0035] Technical benefits: Facilitates multi-dimensional and comprehensive characterization of pulse sound quality; scoring results closely match actual assembly quality, improving the accuracy of pass / fail determination.

[0036] Preferably, as an improvement, in step S4, the judgment result is fed back to the operator in real time via a display screen, LED indicator, voice prompt, or vibration motor.

[0037] Technical benefits: Real-time on-site result prompts eliminate the need to check the terminal, creating a closed-loop guidance system and improving operational efficiency and quality stability. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating an event-triggered audio detection method based on a smart wearable device. Detailed Implementation

[0039] The following detailed description illustrates the specific implementation method: The basic implementation examples are as follows: Figure 1As shown, an event-triggered audio detection method based on a smart wearable device is applied to a smart wearable device hardware system for industrial operation detection. This method is used to identify characteristic acoustic signals generated during industrial operations, including automotive wiring harness terminal insertion, bolt tightening, snap-fit ​​assembly, electrical connections, and structural fastening. The system includes a smart wearable terminal and a data processing unit. The smart wearable device includes a smart bracelet, a wrist-worn terminal, etc. The smart wearable terminal integrates a main control unit, an audio acquisition module, a multimodal working condition sensing module, a wireless communication module, and a human-machine interaction module. The main control unit is electrically connected to the audio acquisition module, the multimodal working condition sensing module, the wireless communication module, and the human-machine interaction module. The wireless communication module establishes a bidirectional wireless communication connection with the data processing unit. The main control unit uses a microcontroller; in this embodiment, an ESP32-S3 microcontroller is used for tasks such as module timing control, sensor data fusion, audio acquisition start / stop control, communication scheduling, and local logic judgment.

[0040] The audio acquisition module uses an audio codec chip. In this embodiment, the ES8311 and ES7210 audio codec chips are used to form a heterogeneous audio acquisition front end, which supports the acquisition of dual-channel audio signals with a high signal-to-noise ratio (≥100 dB) to capture weak acoustic features generated during operation.

[0041] The multimodal working condition sensing module adopts a six-axis inertial measurement unit. In this embodiment, the QMI8658 six-axis inertial measurement unit is used, which integrates a three-axis accelerometer (range ±16g) and a three-axis gyroscope (range ±2000° / s), and has sub-millisecond dynamic response capability, which is used to monitor the operator's arm posture changes and motion characteristics in real time.

[0042] The wireless communication module uses a Wi-Fi module (a Wi-Fi4 module supporting IEEE 802.11n) or a Bluetooth module to package and upload audio data and inertial sensing data to the host computer and to download the analysis results from the host computer.

[0043] The human-computer interaction module is used for status display and real-time feedback of operation results. In this embodiment, a 2.06-inch AMOLED screen is used. The display screen serves as a local feedback interface, providing the operator with visual status prompts.

[0044] The data processing unit uses a PC host computer, which runs an audio analysis program developed based on Python to identify and judge characteristic events in the audio (such as the sound of plugging and unplugging), and then sends the analysis results back to the wearable device.

[0045] An event-triggered audio detection method based on smart wearable devices includes: S1 involves wearing a smart wearable device with an integrated six-axis IMU on the operator's hand to activate the device.

[0046] S2, continuously collect triaxial acceleration and triaxial angular velocity data at a preset sampling rate. In this embodiment, a sampling rate of 100Hz is used to continuously collect triaxial acceleration (unit: mg) and triaxial angular velocity (unit: ° / s) output by the six-axis inertial measurement unit.

[0047] To improve the accuracy of angle estimation, Kalman filtering is used to fuse accelerometer and gyroscope data to obtain a stable attitude angle, including: Initialization noise covariance matrix Q and observation noise covariance matrix R:

[0048]

[0049] in, This represents the variance of the attitude angle process noise. This represents the noise variance during the gyroscope's angular velocity process. This represents the variance of attitude angle observation noise; in this embodiment, , , .

[0050] The Kalman filter model parameters are updated at preset intervals. These parameters include the current attitude estimate (i.e., the stable angle), gyroscope bias, and error covariance matrix. In this embodiment, the update occurs every 10 ms. The inputs are the pitch angle calculated by the accelerometer and the gyroscope Y-axis angular velocity. The pitch angle represents the body tilt angle calculated from gravitational acceleration, and the gyroscope Y-axis angular velocity represents the speed of the wrist's forward and backward pitch rotation. The output is the filtered stable angle, which is the stable, smooth, and jerk-free wrist tilt angle for engineering operations (such as plugging actions).

[0051] Industrial operation initiation and completion are determined based on the filtered data to trigger or terminate audio acquisition. Industrial operation initiation (plug-in initiation) determination includes: Monitor the filtered pitch angle θ and the gyroscope Y-axis angular velocity. The hand gesture detection window will be activated for a preset duration (e.g., 500 ms) when the following conditions are met:

[0052]

[0053] Within this window, if the pitch angle symbol is reversed a preset number of times and the interval between adjacent reversals meets the requirements (in this embodiment, the preset number of reversals is 2 times and the interval between adjacent reversals is not less than 100 ms), it is determined to be a valid operation preparation action, triggering audio acquisition.

[0054] Operation completion determination includes: After audio acquisition is initiated, the system enters the "motion → stationary" detection mode. It calculates the resultant acceleration amplitude, resultant angular velocity amplitude, and acceleration deviation.

[0055]

[0056]

[0057] in, For the resultant acceleration amplitude, The resultant angular velocity amplitude, For acceleration deviation, , , These represent the X, Y, and Z axis acceleration components output by the six-axis IMU in the smart wearable device. , , These are the X-axis, Y-axis, and Z-axis angular velocity components output by the six-axis IMU.

[0058] when Less than the acceleration threshold When the angular velocity is less than the threshold value and remains below the preset duration, while the recorded duration is not less than the preset minimum requirement, the action is considered complete, and audio recording is terminated. In this embodiment, , If the recording lasts for 200 ms and the recorded duration is ≥800 ms, the action is considered complete, and audio recording is terminated.

[0059] S3 sequentially performs signal preprocessing and weak signal filtering, characteristic pulse detection, pulse clarity quantification and classification on the acquired audio data. The acquired audio data is in 48kHz, 16-bit PCM format. Based on the pulse clarity score, the industrial operation quality judgment result is output.

[0060] The signal preprocessing and weak signal filtering include: retaining only the first channel of the multi-channel audio; calculating the maximum absolute amplitude of the signal. If the maximum absolute amplitude is less than the preset weak signal threshold, in this embodiment it is If the signal is invalid, it will be directly determined as an invalid signal and the judgment result will be returned.

[0061] The said characteristic pulse detection includes: using Hilbert transform to extract the analytic signal and envelope of the audio signal, and obtaining the smoothed envelope through Savitzky-Golay filtering:

[0062]

[0063]

[0064] Among them, is the original audio time-domain signal; is the analytic signal obtained by Hilbert transform; j is the imaginary unit; { } is the Hilbert transform operator; is the modulus of the analytic signal, that is, the audio instantaneous envelope; is the Savitzky-Golay filtering operator; is the smoothed envelope after smoothing processing.

[0065] Set an adaptive threshold based on the median absolute deviation, detect the envelope peak through find_peaks and calculate the pulse width using peak_widths, and screen out the effective pulses with a duration between 0.2 ms and 2.0 s.

[0066] The said adaptive threshold is:

[0067] Among them, T is the adaptive detection threshold; min() is the function to take the minimum value; is the maximum value of the smoothed envelope; is the function to take the median; is the smoothed envelope signal after Savitzky-Golay filtering; MAD is the median absolute deviation.

[0068] The said pulse clarity quantization and classification includes: extracting multi-dimensional acoustic features from the effective pulse segment The feature categories include high-frequency energy, spectral roll-off, spectral centroid, spectral centroid, rising sharpness, zero-crossing rate & peak factor, and calculate the comprehensive clarity score by weighting according to the preset weights; if the clarity score of any pulse reaches the preset threshold, it is determined to be qualified, otherwise it is determined to be unqualified. The specific indicators and corresponding weights of the multi-dimensional acoustic features are shown in Table 1.

[0069] Table 1

[0070] The formula for the comprehensive clarity score is:

[0071] in, ~ These are the weighting coefficients for the corresponding features, and the specific values ​​are shown in Table 1. A comprehensive score for pulse clarity; The proportion of high-frequency energy is ≥4 kHz; The 95% energy spectrum roll-off normalized value; is the normalized value of the spectral centroid; brightness is the brightness characteristic; For the sharpness of the envelope rise; Normalized zero-crossing rate; This is the normalized peak factor.

[0072] S4. The judgment result is fed back to the smart wearable device in real time, forming a human-machine collaborative quality inspection closed loop. In this embodiment, the judgment result is fed back to the operator in real time through a display screen, LED indicator, voice prompts, or vibration motor, so that the operator can adjust the operation strategy in real time.

[0073] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An event-triggered audio detection method based on a smart wearable device, characterized in that, include: S1, the smart wearable device integrating a six-axis IMU is worn on the operator's hand to start the smart wearable device; S2 continuously collects triaxial acceleration and triaxial angular velocity data at a preset sampling rate, and uses Kalman filtering to fuse the accelerometer and gyroscope data to obtain a stable attitude angle; Industrial operation start-up and operation completion determination are based on filtered data to trigger or terminate audio acquisition. S3 sequentially performs signal preprocessing and weak signal filtering, feature pulse detection, pulse clarity quantification and classification on the acquired audio data, and outputs industrial operation quality judgment results based on pulse clarity scores. S4 feeds back the judgment results to smart wearable devices in real time, forming a closed loop of human-machine collaborative quality inspection.

2. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The fusion of accelerometer and gyroscope data using Kalman filtering includes: Initialization noise covariance matrix Q and observation noise covariance matrix R: in, This represents the variance of the attitude angle process noise. This represents the noise variance during the gyroscope's angular velocity process. This represents the variance of the attitude angle observation noise. The Kalman filter model parameters are updated at preset intervals. The inputs are the pitch angle calculated by the accelerometer and the Y-axis angular velocity of the gyroscope, and the output is the filtered and stable angle.

3. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The industrial operation start determination includes: Monitor the filtered pitch angle θ and the gyroscope Y-axis angular velocity. The hand gesture detection window for a preset duration will be activated when the following conditions are met: Within this window, if the pitch angle symbol is reversed a preset number of times and the interval between adjacent reversals meets the requirements, it is determined to be a valid operation preparation action, triggering audio acquisition.

4. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The operation completion determination includes: After audio acquisition is started, calculate the resultant acceleration amplitude, resultant angular velocity amplitude, and acceleration deviation: in, For the resultant acceleration amplitude, The resultant angular velocity amplitude, For acceleration deviation, , , These represent the X, Y, and Z axis acceleration components output by the six-axis IMU in the smart wearable device. , , These are the X-axis, Y-axis, and Z-axis angular velocity components output by the six-axis IMU, respectively. when Less than the acceleration threshold When the angular velocity is less than the threshold and continues for a preset duration, and the recorded duration is not less than the preset minimum requirement, the action is considered to have ended, and audio recording is terminated.

5. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The signal preprocessing and weak signal filtering include: The multi-channel audio retains only the first channel; the maximum absolute amplitude of the signal is calculated, and if the maximum absolute amplitude is less than the preset weak signal threshold, it is directly determined as an invalid signal and the judgment result is returned.

6. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The characteristic pulse detection includes: The audio signal and its envelope were extracted using Hilbert transform, and a smooth envelope was obtained by Savitzky-Golay filtering. in, This is the original audio time-domain signal; The analytic signal is obtained by the Hilbert transform; j is the imaginary unit; { } represents the Hilbert transform operator; To analyze the modulus of the signal, i.e., the instantaneous envelope of the audio signal; For Savitzky-Golay filtering operators; The smoothed envelope is the result of smoothing processing. An adaptive threshold is set based on the absolute deviation of the median. The envelope peak value is detected and the pulse width is calculated to filter out effective pulses with a duration between 0.2ms and 2.0s.

7. The event-triggered audio detection method based on a smart wearable device according to claim 6, characterized in that, The adaptive threshold is: Where T is the adaptive detection threshold; min() is the minimum value function; The maximum value of the smooth envelope; This is a median function; is the smoothed envelope signal after Savitzky-Golay filtering; MAD is the median absolute deviation.

8. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that, The pulse clarity quantification and classification includes: extracting multi-dimensional acoustic features from effective pulse segments and calculating a comprehensive clarity score by weighting them according to preset weights; if the clarity score of any pulse reaches a preset threshold, it is judged as qualified, otherwise it is judged as unqualified.

9. The event-triggered audio detection method based on a smart wearable device according to claim 8, characterized in that, The formula for overall clarity scoring is: in, ~ These are the weighting coefficients for the corresponding features. A comprehensive score for pulse clarity; The proportion of high-frequency energy is ≥4 kHz; The 95% energy spectrum roll-off normalized value; is the normalized value of the spectral centroid; brightness is the brightness characteristic; For the sharpness of the envelope rise; Normalized zero-crossing rate; This is the normalized peak factor.

10. The event-triggered audio detection method based on a smart wearable device according to claim 1, characterized in that: In S4, the judgment result is fed back to the operator in real time through a display screen, LED indicator, voice prompt, or vibration motor.

Images