An abnormal audio-video event detection and early warning method and device for video glasses

Abstract

This invention discloses a method and apparatus for detecting and warning of abnormal audio and video events in video glasses, belonging to the field of audio and video detection technology. The method and apparatus include: S1, collecting multi-source data from the video glasses, performing preprocessing, feature parsing and extraction, storing the data, and constructing an abnormal event database; S2, constructing a baseline and analyzing external disturbances using image motion features and action nearest neighbor vectors; S3, conducting audio-visual-inertial co-correlation and anomaly credibility assessment based on multimodal temporal synchronization data; and S4, performing early warning decision calculations based on anomaly handling sequence features. This solves the problem of false alarms and missed alarms caused by the inability to distinguish between everyday head movements, running voluntary movements, and external anomalies such as falls and collisions due to similar features in existing video glasses.

Description

Technical Field

[0001] This invention relates to the field of audio and video event detection technology, specifically to a method and device for detecting and warning of abnormal audio and video events in video glasses. Background Technology

[0002] With the widespread adoption of smart wearable devices, video glasses, as a convenient interactive device, have been widely used in various fields such as daily life, sports monitoring, and health monitoring. Especially in the field of security monitoring, video glasses can record the wearer's visual and auditory information in real time, providing crucial data for the detection and early warning of abnormal events. Currently, many edge-side anomaly detection methods classify events based on video motion patterns and audio impact characteristics, attempting to identify abnormal events such as falls and collisions. These methods typically rely on features such as changes in optical flow in video images, impact spectra in audio signals, and abrupt changes in angular velocity in inertial sensors.

[0003] For example, invention patent CN118537761B discloses an obstacle detection method, apparatus, device, and readable storage medium. The obstacle detection method is applied to a head-mounted device and includes: acquiring an environmental image; if an obstacle is detected in the environment based on the environmental image, determining the actual distance between the obstacle and a target user, wherein the target user is a user wearing the head-mounted device; if the actual distance is less than a distance threshold, determining the actual angle between the obstacle and the target user, wherein the actual angle is the angle between the direction from the target user to the obstacle and the estimated direction of movement of the target user; if the actual angle is within a preset danger angle range, outputting a prompt message.

[0004] For example, the invention patent with announcement number CN112052763B discloses a video anomaly event detection method based on bidirectional retrospective generative adversarial network. The steps are as follows: a generative adversarial network consisting of a generator, a frame discriminator, and a sequence discriminator is constructed. During training, a bidirectional retrospective method is adopted, which combines forward and backward prediction with retrospective prediction. The generative adversarial network is trained by alternating updates of the generator, frame discriminator, and sequence discriminator to obtain a generator that can accurately predict future frame images of normal events in the video but cannot accurately predict future frame images of abnormal events in the video. Thus, the abnormal event is detected based on the prediction error.

[0005] However, these features often fail to effectively distinguish between the wearer's voluntary actions and externally triggered abnormalities. Within a short time window, the sudden changes in angular velocity, drastic changes in optical flow, and the sound of glasses frames colliding, generated by activities such as head movements, running, and going up and down stairs, show high similarity in the time-frequency domain to the motion characteristics of abnormal events such as falls and collisions. This leads to the technical bottleneck of high false alarm rates and missed alarms in traditional methods.

[0006] Therefore, in order to address the above problems, there is an urgent need for a method and device for detecting and warning of abnormal audio and video events in video glasses. Summary of the Invention

[0007] Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method and device for detecting and warning of abnormal audio and video events in video glasses, which solves the problem of false alarms and missed alarms caused by the inability to distinguish between daily head movements, running voluntary actions and external anomalies such as falls and collisions due to similar characteristics in existing video glasses.

[0008] Technical solution To achieve the above objectives, the present invention provides the following technical solution: a method and apparatus for detecting and warning of abnormal audio and video events in video glasses, comprising: S1, collecting multi-source data from video glasses and performing preprocessing, performing feature analysis and extraction, storing the data, and constructing an abnormal event database; S2, constructing a baseline and performing external disturbance analysis using image motion features and action nearest neighbor center vectors, and performing disturbance segment identification and autonomous action baseline dynamic update operations based on the external disturbance analysis results; S3, performing audio-visual-inertial-visual collaborative correlation and abnormal credibility assessment based on multimodal temporal synchronization data, and performing pseudo-anomaly screening, abnormal segment marking, and event feature archiving operations based on the anomaly assessment results; S4, performing early warning decision calculation based on abnormal handling sequence features, and performing abnormal early warning, evidence solidification, and scheduling instruction generation based on the calculation results.

[0009] Further, the specific steps for acquiring and preprocessing multi-source data from the video glasses are as follows: Acquiring multi-source data from the video glasses, including video data, audio data, and inertial data; Acquiring video data: Obtaining the video frame sequence within the observation time window through the image acquisition unit; Extracting the optical flow histogram as the image motion feature vector using the optical flow method; Extracting the target region and obtaining the bounding box coordinate sequence as the target region feature vector using the target detection algorithm; Calculating the sharpness of each frame using the Laplacian operator, and taking the p-th percentile within the window as the sharpness retention value; Extracting static background feature points between adjacent frames using the feature point matching method, and estimating the average displacement of the global homography matrix as the background drift value using the RANSAC algorithm; Acquiring audio data: Obtaining the ambient sound sequence and the sound sequence of contact between the frame and the glasses through a dual-microphone array; Extracting the Mel spectrogram and inputting it into a lightweight convolutional neural network, outputting the impact spectrum feature vector; Separating human voice using speech activity detection. Based on short-time energy analysis in the mid-to-high frequency band, the frame friction feature vector is extracted; inertial data is collected: angular velocity, acceleration, and attitude angle sequences are obtained through a six-axis inertial measurement unit; the peak angular velocity within the window is calculated as the angular velocity mutation vector, the acceleration variance is calculated as the acceleration mutation vector, and the attitude angle change rate is calculated as the attitude transition vector; for each observation record, the observation segment identifier, acquisition time, device identifier, and scene identifier are synchronously written, and time alignment and window aggregation are performed according to the acquisition time; deblurring, occlusion observation segment marking, and brightness normalization are performed on the video frame sequence; wind noise suppression, broadband noise suppression, and frame contact sound separation are performed on the environmental sound sequence; abnormal peak correction and normalization are performed on the angular velocity sequence, acceleration sequence, and attitude change sequence; video features, sound features, contact sound features, and inertial features within the same observation time window are integrated into abnormal observation segments, and a unique observation segment sequence number is assigned to each abnormal observation segment.

[0010] Further, the specific steps for feature parsing and extraction, storage, and construction of an abnormal event database are as follows: Motion features, angular velocity change features, acceleration change features, and contact sound features are extracted from a historical autonomous action sample set to construct an individual autonomous action baseline. Cluster analysis is then performed to obtain the nearest neighbor center vector for the autonomous action. Instantaneous collision response features, friction response features, and short-time impulse response features are extracted from the frame contact sound sequence and inertial sequence. Intensity aggregation analysis is then performed on these response features to output the frame contact interference value for the abnormal observation segment. Visual abnormal features are extracted based on target area displacement changes, contour changes, attitude changes, and optical flow mutations in the video frame sequence, and visual abnormal values ​​are output. Sound abnormal features are extracted based on impact spectrum changes, short-time energy changes, and sound source mutations in the environmental sound sequence, and sound abnormal values ​​are output. The similarity value of the autonomous action is calculated based on the distance relationship between the motion features and inertial change features of the current abnormal observation segment and the historical autonomous action reference feature set. The preprocessed multi-source abnormal event data is then stored and an abnormal event database is constructed.

[0011] Furthermore, the specific steps for baseline construction and external disturbance analysis using image motion features and action nearest neighbor center vectors are as follows: Obtain the first... The image motion feature vector, action nearest neighbor center vector, sharpness retention value, background drift value, and frame contact interference value of the first abnormal observation segment; for the first... The difference between the motion feature vector and the nearest neighbor center vector of the first abnormal observation segment is calculated to obtain the offset vector; the offset vector is then subjected to a L2 norm operation to obtain the offset amplitude; the offset amplitude is multiplied by the sharpness retention value to obtain the residual amplification; the background drift value and the frame contact interference value are added together and then one is added to obtain the residual suppression term; the offset amplitude is divided by the residual suppression term to obtain the first abnormal observation segment. External disturbance residuals of an anomalous observation segment.

[0012] Furthermore, the specific steps for performing disturbance segment identification and autonomous action baseline dynamic update based on the external disturbance analysis results are as follows: By comparing the external disturbance residual value with the residual threshold in real time, when the external disturbance residual value is less than the residual threshold, the corresponding abnormal observation segment is marked as an autonomous action-dominant observation segment, and only the observation segment index, posture summary, and sound summary are retained; if the image motion feature vector of the observation segment has a maximum cosine similarity with the autonomous action reference feature set greater than the similarity threshold, the observation segment is written into the pending baseline buffer area and the individual autonomous action baseline is updated to adapt to changes in the wearer's individual habits; when the external disturbance residual value is greater than or equal to the residual threshold, the m consecutive time windows before and after the current observation segment are frozen to construct a forward evidence band. The background drift value is corrected based on the median of the background reference point displacement sequence in both the forward and backward evidence bands. The sharpness retention value is corrected based on the k% quantile of the video frame sharpness sequence within the evidence band. The contact interference value is corrected based on the mean of the short-time energy sequence of the frame contact sound within the evidence band. The external disturbance residual value is recalculated based on the corrected background drift value, sharpness retention value, and contact interference value. If the recalculated external disturbance residual value is still greater than or equal to the residual threshold, the current observation segment is marked as an external trigger candidate segment, and the external disturbance residual value is output to the anomaly correlation and evaluation module. If the recalculated external disturbance residual value is less than the residual threshold, the corresponding abnormal observation segment is marked as an autonomous action-dominated observation segment.

[0013] Furthermore, the specific steps for conducting audio-visual inertial-visual collaborative correlation and anomaly reliability assessment based on multimodal temporal synchronization data are as follows: Using environmental sound sequences, video frame sequences, angular velocity sequences, acceleration sequences, attitude change sequences, and time synchronization records as basic inputs, an audio-visual time difference analysis model is constructed using the audio-visual temporal correlation method. Based on the impact initiation time in the environmental sound sequence and the target area change time in the video frame sequence, the audio-visual time difference value is output, and a reference audio-visual time difference value is output by combining the statistical results of audio-visual time offsets in historical autonomous action observation segments. An inertial-visual collaborative analysis model is constructed using the inertial-visual collaborative analysis method. Based on the optical flow change trend in the video frame sequence and the synchronization response relationship between the angular velocity sequence, acceleration sequence, and attitude change sequence, the inertial-visual collaborative change value is output, and a reference inertial-visual collaborative change value is output by combining the statistical results of inertial changes and image changes in historical autonomous action observation segments. The first step is to obtain the... External disturbance residual values ​​of each anomalous observation segment Visual anomalies Abnormal sound values; for the first The visual and acoustic anomaly values ​​of each anomalous observation segment are squared, summed, and the square root is taken. This sum is then added to the external disturbance residual value to obtain the joint anomaly amplitude. The absolute difference between the audio-visual time difference and the reference time difference is calculated to obtain the time difference offset. The absolute difference between the inertial-visual co-variance value and the reference inertial-visual co-variance value is calculated to obtain the co-variance offset. The external disturbance residual value is added to the joint anomaly amplitude to obtain the anomaly enhancement term. The time difference offset and the co-variance offset are added together and then one is added to obtain the credible suppression term. The anomaly enhancement term is divided by the credible suppression term to obtain the first anomaly. The confidence value of the anomalous trigger for each anomalous observation segment.

[0014] Furthermore, the specific steps for performing pseudo-anomaly screening, anomalous segment marking, and event feature archiving based on the anomaly assessment results are as follows: Real-time comparison of the anomaly trigger confidence value and trigger threshold; when the anomaly trigger confidence value is less than the trigger threshold, it is determined to be a pseudo-anomaly, and the observed segment is moved to the pending baseline buffer area; if the observed segment is in continuous... If the confidence value of an anomaly trigger is less than the trigger threshold within a certain time period, the individual autonomous action baseline is updated again and archived to the anomaly event database. When the confidence value of an anomaly trigger is greater than or equal to the trigger threshold, the observation segment is marked as an anomaly observation segment. An anomaly handling sequence is constructed in descending order based on the confidence value of the anomaly trigger. The continuous window association analysis method is used to perform temporal association and repetition matching on the anomaly observation segments in adjacent observation time windows. If the time overlap rate is >d% and the spatial target area IoU is >T, they are merged into the same event, and the number of consecutive confirmations and the duration of the anomaly are output and archived to the anomaly event database.

[0015] Furthermore, the specific steps for conducting early warning decision-making calculations based on the characteristics of anomaly handling sequences are as follows: obtaining the anomaly trigger confidence value, number of consecutive confirmations, anomaly duration, and autonomous action similarity value of the anomaly handling sequence; for the first... The number of consecutive confirmations for each anomaly observation segment is incremented by one and then subjected to a natural logarithm to obtain the confirmation enhancement term; the duration of the anomaly is incremented by one and then subjected to a natural logarithm to obtain the persistence enhancement term; the anomaly trigger confidence value, the confirmation enhancement term, and the persistence enhancement term are added together to obtain the early warning enhancement term; the similarity value of autonomous actions is incremented by one to obtain the decision suppression term; the early warning enhancement term is divided by the decision suppression term to obtain the [missing term]. Early warning decision value for an abnormal observation segment.

[0016] Furthermore, the specific steps for executing anomaly warning, evidence solidification, and scheduling instruction generation based on the calculation results are as follows: By comparing the warning decision value with the warning threshold in real time, when the warning decision value is less than the warning threshold, the corresponding abnormal observation segment is marked as a false alarm abnormal observation segment, the current sampling rhythm is maintained, and the complete record is converted into keyframe summary, sound envelope summary, and attitude change summary; when the warning decision value is greater than or equal to the warning threshold, the corresponding abnormal observation segment is marked as a warning observation segment, the observation time window is expanded forward to extract f preceding observation segments, and expanded backward to extract b continuing observation segments to start backtracking and supplementary recording; the original video, sound, and inertial records in the cache are solidified into read-only evidence segments, the target area tracking is switched to frame-by-frame trajectory reconstruction, the impact sound detection is switched to short-window fine scanning, the attitude change detection is switched to abrupt point relocation, and the image change, impact sound arrival, and attitude instability are connected into an event time sequence chain according to the chronological order; a triggering end-side reminder instruction is generated, and the event type mark, warning time, number of consecutive confirmations, and batch traceability information are archived to the abnormal event database.

[0017] Furthermore, a second aspect of the present invention provides an abnormal audio and video event detection and early warning device for video glasses, and applies an abnormal audio and video event detection and early warning method for video glasses, comprising: a data acquisition and preprocessing module, used to acquire multi-source data from video glasses and perform preprocessing, perform feature parsing and extraction, store the data, and construct an abnormal event database; a disturbance residual characterization module, used to construct a baseline and perform external disturbance analysis through image motion features and action nearest neighbor center vectors, and perform disturbance segment identification and autonomous action baseline dynamic update operations based on the external disturbance analysis results; an anomaly association and evaluation module, used to perform audio-visual inertial co-association and anomaly credibility evaluation based on multimodal temporal synchronization data, and perform pseudo-anomaly screening, anomaly segment marking, and event feature archiving operations based on the anomaly evaluation results; and an event confirmation and early warning module, used to perform early warning decision calculation based on anomaly handling sequence features, and perform anomaly early warning, evidence solidification, and scheduling instruction generation based on the calculation results.

[0018] Beneficial effects The present invention has the following beneficial effects: (1) This invention constructs an individual autonomous action baseline by collecting daily action samples of the wearer, and calculates the external disturbance residual value by combining the deviation of the image motion features and the baseline center vector. This enables accurate differentiation between the wearer's autonomous actions and external abnormal triggers, effectively solving the technical problem of high-frequency false alarms caused by the similarity between daily head movements, running and other autonomous actions and external abnormal features such as falls and collisions in the prior art.

[0019] (2) In this invention, by introducing the sharpness retention value, background drift value and frame contact interference value as correction factors in the calculation of external disturbance residuals, the sharpness is used to amplify the real deviation response, and the background drift and frame interference are used to suppress the pseudo-abnormalities caused by head movement and wearing friction, thereby realizing the accurate quantification of the degree of behavioral deviation, effectively solving the bottleneck problem of the lack of stable criteria in the existing technology under short window and occlusion blur conditions.

[0020] (3) In this invention, when the residual value of external disturbance exceeds the residual threshold, the forward evidence band and the backward evidence band are constructed by freezing the time window before and after freezing. The residual value is recalculated after the median and quantile correction of background drift, sharpness and frame contact interference by the stable data in the evidence band. This realizes the secondary verification of external trigger candidates and effectively solves the problem of frequent alarms caused by instantaneous interference in the prior art.

[0021] (4) In this invention, a reliable suppression term is constructed by the audio-visual time difference offset and the inertial-visual coordination offset. Cross-modal synchronization is used as a penalty factor for the reliability value of abnormal triggering. By utilizing the characteristics that autonomous actions usually maintain audio-visual synchronization and inertial-visual coordination, and that external abnormalities often lead to cross-modal misalignment, the invention effectively screens out false abnormalities and effectively solves the technical problem of existing technologies that make it difficult to distinguish between true and false abnormalities due to the similarity of single-modal features.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0023] Figure 1 This is a flowchart of an abnormal audio and video event detection and early warning method for video glasses according to the present invention; Figure 2 This is a block diagram of an abnormal audio and video event detection and early warning device for video glasses according to the present invention; Figure 3 This is a surface plot showing the impact of cross-modal synchronization deviation on the abnormal reliability of the present invention. Figure 4 This is a diagram illustrating the effect of the autonomous external disturbance timing identification of abnormal events in the video glasses of this invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please see Figures 1-4This invention provides a technical solution: a method and apparatus for detecting and warning of abnormal audio and video events in video glasses, comprising: S1, collecting multi-source data from video glasses and performing preprocessing, performing feature analysis and extraction, storing the data, and constructing an abnormal event database; S2, constructing a baseline and performing external disturbance analysis using image motion features and action nearest neighbor center vectors, and performing disturbance segment identification and autonomous action baseline dynamic update operations based on the external disturbance analysis results; S3, performing audio-visual-inertial-visual collaborative correlation and abnormal credibility assessment based on multimodal temporal synchronization data, and performing pseudo-anomaly screening, abnormal segment marking, and event feature archiving operations based on the anomaly assessment results; S4, performing early warning decision calculation based on abnormal handling sequence features, and performing abnormal early warning, evidence solidification, and scheduling instruction generation based on the calculation results.

[0026] Specifically, the steps for collecting multi-source data from video glasses and performing preprocessing are as follows: When acquiring video data, the video frame sequence within the observation time window is obtained through the image acquisition unit. To ensure the accuracy of subsequent motion analysis, the optical flow method is used to extract the optical flow histogram as the motion feature vector of the image. This feature can effectively describe the pixel-level motion direction and amplitude distribution. The target region is extracted using a target detection algorithm, and the bounding box coordinate sequence is obtained as the target region feature vector to identify potential human bodies or objects in the image. The Laplacian operator is used to calculate the sharpness of each frame. Since sharpness is easily affected by instantaneous motion blur or occlusion, the p-th percentile within the window is taken as the sharpness retention value, where p is 80, that is, the sharpness value of the 80th percentile within the window is selected. This can eliminate the interference of a few extremely blurry frames and retain the sharpness of most frames as the overall characterization. The feature point matching method is used to extract static background feature points between adjacent frames, and the RANSAC algorithm is used to estimate the global homography matrix. Then, the average displacement of all static background feature points is calculated as the background drift value to quantify the overall field of view change caused by head movement.

[0027] When collecting sound data, a dual-microphone array is used to acquire ambient sound sequences and sound sequences of contact between the glasses frame and the frame. The dual microphones enable sound source localization and background noise suppression. Mel spectrograms are extracted and input into a lightweight convolutional neural network to output an impact spectrum feature vector, which can characterize the sudden impact components of the sound. Speech activity detection is used to separate human voices to avoid interference from human voices in anomaly detection. Then, short-time energy analysis in the mid-to-high frequency band of 2kHz-8kHz is used to extract the frame friction feature vector, because the frame friction sound is mainly concentrated in this frequency band, and short-time energy analysis can effectively capture its energy fluctuation pattern.

[0028] When acquiring inertial data, angular velocity, acceleration, and attitude angle sequences are obtained through a six-axis inertial measurement unit; the peak angular velocity within the window is calculated as the angular velocity mutation vector to characterize the severity of head rotation; the acceleration variance is calculated as the acceleration mutation vector to reflect the amplitude of acceleration fluctuations; and the attitude angle change rate is calculated as the attitude transition vector to describe rapid attitude changes.

[0029] After data acquisition, each observation record is synchronously written with an observation segment identifier, acquisition time, device identifier, and scene identifier. Time alignment and window aggregation are performed based on the acquisition time to ensure strict temporal correspondence between multi-source data. For video frame sequences, deblurring, occlusion segment marking, and brightness normalization are performed. Dimensionless processing is achieved by mapping brightness values ​​to the 0-1 range to improve image quality. For ambient sound sequences, wind noise suppression, broadband noise suppression, and frame contact sound separation are performed. The energy values ​​of the sound signals are mapped to the 0-1 range through linear transformation to achieve dimensionless processing and eliminate environmental noise. Anomaly peak correction is performed on angular velocity, acceleration, and attitude change sequences. Dimensionless processing is achieved by dividing by the historical maximum absolute value of each sequence, followed by normalization to eliminate sensor noise. Video features, sound features, contact sound features, and inertial features within the same observation time window are integrated into anomalous observation segments, and each anomalous observation segment is assigned a unique observation segment sequence number for subsequent module indexing and tracing.

[0030] In this implementation scheme, through systematic acquisition and refined preprocessing of multi-source data from the video glasses, this step achieves high-quality alignment and feature-level integration of three types of raw signals: video, audio, and inertial. At the video processing level, motion features are extracted using optical flow histograms and target region bounding box sequences. The Laplacian operator is employed, and the 80th percentile is selected as the sharpness-preserving value to enhance robustness against instantaneous motion blur and occlusion. The background drift value estimated by the RANSAC algorithm is combined to accurately quantify the field-of-view changes caused by head movements. At the audio processing level, a dual-microphone array is used to acquire ambient sound and frame contact sound. Impact spectrum feature vectors are extracted using Mel spectrograms and a lightweight convolutional neural network. Short-time energy analysis in the 2kHz-8kHz mid-to-high frequency band is used to separate the frame friction feature vectors, effectively separating the wearer's own friction sound from environmental interference. At the inertial processing level, abrupt change feature vectors are constructed using peak angular velocity, acceleration variance, and attitude angle change rate to fully characterize the dynamics of head movements. Finally, the multimodal features within the same time window were integrated into structured anomaly observation segments and assigned unique sequence numbers, providing a spatiotemporally aligned, clean, and reliable feature data foundation for subsequent autonomous action baseline construction and anomaly detection, significantly improving the usability and accuracy of the original data.

[0031] Specifically, the steps for performing feature parsing and extraction, storing the data, and constructing an anomaly event database are as follows: An individual autonomous action baseline is constructed by extracting image motion features, angular velocity change features, acceleration change features, and contact sound features from a historical autonomous action sample set. Specifically, the historical autonomous action sample set consists of multiple sets of feature data collected when the wearer performs typical actions such as walking, turning their head, and going up and down stairs in daily scenarios. Each set of samples includes synchronously aligned image motion features, angular velocity change features, acceleration change features, and contact sound features. K-means clustering analysis is performed on this sample set, with a cluster center number of 8, to obtain autonomous action nearest neighbor center vectors. Each center vector represents the cluster center of a typical autonomous action in the feature space and is used for subsequent deviation calculation.

[0032] The instantaneous collision response feature is defined as the peak amplitude of short-term energy exceeding the energy threshold in the sound sequence of eyeglass frame contact. The friction response feature is defined as the average short-term energy of the sound sequence of eyeglass frame contact in the mid-to-high frequency range, i.e., 2kHz to 8kHz. The short-term impulse response feature is defined as the synchronous response intensity where angular velocity and acceleration simultaneously peak in the inertial sequence. The three features are weighted and fused to obtain the eyeglass frame contact interference value, which is used to quantify the local interference intensity caused by the wearer touching, adjusting the glasses, or vibrating the eyeglass frame. The visual anomaly value is obtained by comprehensively scoring the displacement of the center point of the target area bounding box, the aspect ratio change rate, the confidence level of the human body key point posture, and the degree of abrupt change in optical flow amplitude. Its value range is set between 0 and 1. The larger the value, the higher the confidence level of the presence of abnormal targets such as falling or intrusion in the image.

[0033] The sound anomaly value is obtained by comprehensively scoring the matching degree between the impact spectrum feature vector and the dangerous sound template library, the amplitude of the short-term energy peak exceeding the energy threshold, and the angle of sudden change in the sound source direction. Its value ranges from 0 to 1, with a higher value indicating a higher confidence level in the presence of dangerous sounds such as breaking glass or screams in the environment. The motion feature vector and inertial change feature vector of the current segment are concatenated into a multimodal feature vector. The cosine similarity between this vector and all samples in the historical autonomous action reference feature set is calculated, and the maximum value is taken as the autonomous action similarity value. This value reflects the degree of matching between the current segment and the historical autonomous action features.

[0034] The preprocessed abnormal events are stored in multi-source data and an abnormal event database is constructed. The database uses abnormal observation segments as basic units. Each segment is associated with its segment identifier, acquisition time, device identifier, scene identifier, various feature vectors, frame contact interference value, visual anomaly value, sound anomaly value, autonomous action similarity value, and original data storage path. This provides a unified data access interface for subsequent autonomous action baseline construction, disturbance residual characterization, anomaly correlation assessment, and early warning decision-making modules.

[0035] In this implementation plan, a baseline for individual autonomous actions is constructed through cluster analysis of historical autonomous action sample sets, and the nearest neighbor center vector of autonomous actions is obtained, providing a personalized reference benchmark for the quantification of behavioral deviations. Response features are extracted from frame contact sound sequences and inertial sequences, and frame contact interference values ​​are output, enabling a quantitative representation of local disturbances such as wearer touching the frame. Visual anomaly features are extracted from video frame sequences, and visual anomaly values ​​are output; similarity features are extracted from environmental sound sequences, and sound anomaly values ​​are output, enabling quantitative assessment of abnormal targets in the scene and dangerous environmental sounds, respectively. Autonomous action similarity values ​​are calculated using scene motion features and inertial change features, enabling a quantitative measurement of the degree of matching between current behavior and historical autonomous actions. Finally, all feature data is stored and an anomaly event database is constructed, providing unified data support for subsequent disturbance residual calculation, anomaly credibility assessment, and early warning decision-making.

[0036] Specifically, the steps for baseline construction and external disturbance analysis using image motion features and action nearest neighbor center vectors are as follows: The image motion feature vector, action nearest neighbor center vector, clarity retention value, background drift value, and frame contact interference value of the i-th abnormal observation segment are obtained. The image motion feature vector is used to characterize the dynamic change state of the current abnormal observation segment. The action nearest neighbor center vector serves as the core reference feature of the individual's autonomous action baseline. The clarity retention value is used to reflect the stability of the image features to assist in calibrating the degree of offset. The background drift value is used to quantify the non-disturbance offset interference caused by changes in the environmental background. The frame contact interference value is used to characterize the degree of interference caused by the contact friction between the frame and the wearer.

[0037] The difference between the motion feature vector and the action nearest neighbor center vector of the i-th abnormal observation segment is calculated. An offset vector is obtained through vector subtraction, which directly reflects the direction and degree of deviation between the current motion features and the autonomous action baseline features. A L2 norm operation is performed on the offset vector to quantify the overall deviation intensity, yielding the offset amplitude. This amplitude provides the basis for subsequent residual calculations. The offset amplitude is multiplied by the sharpness retention value to obtain the residual amplification. This multiplication operation strengthens the influence of the offset amplitude under stable image features, highlighting the effects of real external disturbances. The deviation is identified by adding the background drift value and the frame contact interference value, and then adding one to obtain the residual suppression term. The addition operation can effectively avoid the calculation anomaly of zero denominator. At the same time, by superimposing the two interference values, the environmental background drift and frame contact interference are jointly suppressed, reducing the interference of non-external disturbance factors on the residual calculation. The offset amplitude is divided by the residual suppression term, and the influence of irrelevant interference is eliminated by the division operation. Finally, the external disturbance residual value of the i-th abnormal observation segment is obtained. This residual value can accurately quantify the degree of feature deviation caused by external disturbance in the current observation segment, and provide a core quantitative indicator for subsequent anomaly trigger confidence value calculation and anomaly identification.

[0038] The specific formula for the residual value of external disturbance is as follows: In the formula, Indicates the first The external perturbation residual value of each abnormal observation segment reflects the degree to which the current observation segment deviates from the wearer's autonomous action baseline; Indicates the first Motion feature vectors of a single abnormal observation segment; Indicates the first The action nearest neighbor center vector of each anomalous observation segment; Indicates the first The sharpness retention value of each abnormal observation segment is used to reflect the discernibility of the current image; Indicates the first Background drift values ​​of individual anomalous observation segments are used to characterize the overall field of view change caused by head movement; Indicates the first The frame contact interference value of each abnormal observation segment is used to characterize the local interference intensity caused by frame collision sound and wearing friction sound.

[0039] Table 1 shows the external disturbance residual evaluation data based on multi-feature fusion in this embodiment. The first observation segment has an offset amplitude of 0.320, a sharpness retention value of 0.920, a background drift value of 0.350, and a frame contact interference value of 0.280, resulting in a calculated external disturbance residual value of 0.197. The second observation segment has an offset amplitude of 0.450, a sharpness retention value of 0.880, a background drift value of 0.420, and a frame contact interference value of 0.310, resulting in a calculated external disturbance residual value of 0.232. The third observation segment has an offset amplitude of 0.680, a sharpness retention value of 0.890, and a background drift value of 0. The external disturbance residual value is calculated to be 0.474, with a frame contact interference value of 0.120 and an external disturbance residual value of 0.180. For the fourth observation segment, the offset amplitude is 0.850, the sharpness retention value is 0.860, the background drift value is 0.150, and the frame contact interference value is 0.090. The external disturbance residual value is calculated to be 0.602. For the fifth observation segment, the offset amplitude is 0.950, the sharpness retention value is 0.750, the background drift value is 0.080, and the frame contact interference value is 0.050. The external disturbance residual value is calculated to be 0.646.

[0040] Table 1. External perturbation residual evaluation data based on multi-feature fusion like Figure 3 The figure shows a surface plot illustrating the impact of cross-modal synchronization deviation on the anomaly reliability provided in this embodiment of the application. Combined with the data in Table 1... Figure 3 As can be seen from the comparison of the data in Table 1, although observation segments 1 and 2 show a certain offset amplitude, the background drift value and frame contact interference value are relatively large, indicating that the current movement is accompanied by significant head movement and frame friction. The external disturbance residual value remains at a low level, consistent with the characteristics of autonomous movement. In contrast, the offset amplitude of observation segments 3 to 5 increases significantly, while the background drift and frame interference are smaller, and the external disturbance residual value increases significantly, consistent with the characteristics of external triggering candidates. This distribution verifies the corrective effect of introducing the sharpness preservation value, background drift value, and frame contact interference value into the external disturbance residual formula: image sharpness can enhance the response to true deviation, while background drift and frame interference effectively suppress false anomalies caused by head movement and wearing friction. Further combining... Figure 3 It is evident that the reliability of anomaly detection decreases with increasing cross-modal synchronization deviation. The larger the time difference offset and coordination offset, the lower the reliability, indicating that audio-visual asynchrony and inertial-visual incoordination significantly weaken the weight of anomaly judgment. Therefore, cross-modal consistency is used as the core criterion for distinguishing autonomous actions from external anomalies. This dual verification mechanism jointly ensures the accuracy of anomaly detection, providing a reliable decision-making basis for the subsequent graded early warning module.

[0041] In this implementation scheme, by performing vector difference, norm operation and weighted scaling on the multi-dimensional features of the current abnormal observation segment, the feature offset corresponding to the real external disturbance is amplified while effectively suppressing the invalid interference caused by background drift and frame contact. Finally, the residual value of external disturbance that can accurately distinguish autonomous actions from external disturbances is obtained, providing a stable and reliable quantitative basis for subsequent anomaly credibility assessment and pseudo-anomaly filtering.

[0042] Specifically, the steps for performing disturbance segment identification and autonomous action baseline dynamic update operations based on the external disturbance analysis results are as follows: By comparing the residual value of external disturbance with the residual threshold in real time, when the residual value of external disturbance is less than the residual threshold, the corresponding abnormal observation segment is marked as the autonomous action-dominated observation segment. At this time, only the observation segment index, posture summary and sound summary are retained to reduce storage and transmission overhead. If the motion feature vector of the observation segment and the maximum cosine similarity in the autonomous action reference feature set are greater than the similarity threshold, the observation segment is written into the pending baseline buffer area and the individual autonomous action baseline is updated, so that the baseline can adapt to the slow changes in the wearer's individual habits and continuously improve the accuracy of autonomous action recognition.

[0043] When the residual value of external disturbance is greater than or equal to the residual threshold, it indicates that the current segment may contain external anomaly triggers, but misjudgments caused by transient interference need to be excluded. To this end, m consecutive time windows before and after the current observation segment are frozen to construct forward and backward evidence bands; where m is 15, corresponding to a time range of about 0.5 seconds. At a frame rate of 30fps, this length is sufficient to cover common transient jitter such as rapid head turning or brief occlusion, while avoiding the introduction of too much irrelevant data. Based on stable multi-frame data within the evidence band, robust statistics are used to correct key parameters: the median of the background reference point displacement sequence in the forward and backward evidence bands is used as the corrected background drift value. The median effectively suppresses extreme displacement values ​​caused by mismatches of locally moving objects or feature points, thus more realistically reflecting the overall field of view changes caused by head movement; the k% quantile of the video frame sharpness sequence within the evidence band is used as the corrected sharpness retention value, where k is set to 75, i.e., the 75th quantile. This value can eliminate a few extremely blurry frames caused by rapid movement or occlusion, retaining the typical sharpness of most frames as a stable representation of image discriminability; the mean of the short-time energy sequence of frame contact sound within the evidence band is used as the corrected contact interference value. The mean smooths out energy fluctuations caused by accidental collisions or friction, reflecting the average intensity of frame contact interference during that period. The external disturbance residual value is recalculated based on the corrected background drift value, sharpness retention value, and contact interference value, such as... Figure 4The diagram shown in this embodiment illustrates the effect of autonomous external disturbance timing identification for abnormal events in video glasses. If the recalculated external disturbance residual value is still greater than or equal to the residual threshold, it indicates that the deviation is still significant after the evidence band eliminates transient interference. In this case, the current observation segment is marked as an external trigger candidate segment, and the external disturbance residual value is output to the anomaly association and evaluation module for subsequent analysis. If the recalculated external disturbance residual value is less than the residual threshold, it indicates that the original trigger was caused by transient jitter. The corresponding abnormal observation segment should be remarked as an autonomous action-dominated observation segment to effectively avoid false alarms.

[0044] In this implementation scheme, by comparing the residual values ​​of external disturbances with the residual threshold in real time, accurate triage of abnormal observation segments is achieved: segments below the residual threshold are marked as autonomously driven, retaining only summary information and used for baseline dynamic updates, enabling the system to adapt to changes in individual wearer habits; segments above the residual threshold trigger a secondary confirmation mechanism using evidence bands. The residual values ​​are then recalculated after robustly correcting key parameters using the median of background drift values, the quantile of clarity retention values, and the mean of frame contact interference values ​​within the preceding and following time windows. This effectively eliminates false anomalies caused by instantaneous head movements, partial occlusion, or accidental friction, significantly improving end-side anti-interference capabilities and false alarm suppression. The final output of stable externally triggered candidate segments provides high-confidence input for subsequent multimodal correlation assessments, thereby ensuring that genuine anomalies are not missed at the source while significantly reducing the probability of false alarms, laying a solid foundation for the accuracy and reliability of the entire early warning system.

[0045] Specifically, the steps for conducting audio-visual-inertial-visual collaborative correlation and anomaly credibility assessment based on multimodal temporal synchronization data are as follows: Using environmental sound sequences, video frame sequences, angular velocity sequences, acceleration sequences, attitude change sequences, and time-synchronized records as basic inputs, an audio-visual time difference analysis model is constructed using an audio-visual temporal correlation method. This model first detects the impact initiation moment in the environmental sound sequence, i.e., the moment when the short-term energy exceeds the energy threshold, and simultaneously detects the moment when the target region in the video frame sequence undergoes a significant change, such as the starting frame of a sudden change in bounding box area or optical flow amplitude. The time difference between these two is the audio-visual time difference value, measured in milliseconds. To establish a personalized synchronization benchmark, a reference audio-visual time difference value is output by combining the statistical results of audio-visual time offsets from historical autonomous action observation segments: specifically, all segments marked as autonomous actions are extracted from the individual autonomous action sample library, the mean of their audio-visual time difference values ​​is calculated, and an exponentially weighted moving average is used for dynamic updates, ensuring that the reference value reflects both inherent individual characteristics and adapts to slow changes. An inertial vision co-analysis method is used to construct an inertial vision co-analysis model, outputting inertial vision co-change values ​​based on the synchronization response relationship between the optical flow change trend in the video frame sequence and the angular velocity, acceleration, and attitude change sequences. This value is obtained by calculating the cosine of the angle between the principal direction vector of optical flow and the principal direction vector of angular velocity within the current time window. The value ranges from -1 to +1, with a closer value to +1 indicating higher directional consistency. A reference inertial-visual coordination change value is output by combining the statistical results of inertial changes and visual changes in historical autonomous motion observation segments: the median inertial-visual coordination value within the same time window in historical autonomous motion samples is taken as the benchmark, and a sliding window is used to update it periodically to reflect the normal head movement and visual-motor coordination characteristics of an individual.

[0046] Obtain the external disturbance residual value, visual anomaly value, and sound anomaly value for the i-th anomaly observation segment. It should be noted that the above three values ​​have all been normalized and mapped to the interval between zero and one: the external disturbance residual value is explicit and its range is naturally within the interval between zero and one due to the constraint of the denominator; the visual anomaly value and the sound anomaly value are normalized to the interval between zero and one based on the scores of visual features and sound features, respectively, through minimum and maximum value normalization.

[0047] The visual and acoustic anomaly values ​​of the i-th anomalous observation segment are squared, and the sum of the results is taken as the square root to obtain the combined audiovisual anomaly intensity. This intensity is then added to the external disturbance residual value to obtain the combined anomaly amplitude, which also falls within the zero-to-two range but will subsequently match the denominator. The absolute difference between the audiovisual time difference and the reference time difference is calculated to obtain the time difference offset; the absolute difference between the inertial-visual co-variance value and the reference inertial-visual co-variance value is calculated to obtain the co-offset. To standardize the units, the time difference offset is normalized to zero to one based on the maximum possible offset. The co-offset is already within the zero-to-two range, but its absolute value can be used directly in practice. The external disturbance residual value is added to the combined anomaly amplitude to obtain the anomaly enhancement term; the time difference offset is added to the co-offset and then one is added to obtain the confidence suppression term. Finally, the anomaly enhancement term is divided by the confidence suppression term to obtain the anomaly trigger confidence value of the i-th anomalous observation segment. This design has a clear physical meaning in mathematics: the anomaly enhancement term reflects the overall anomaly intensity, while the credibility suppression term penalizes credibility based on cross-modal consistency. When the wearer performs voluntary actions, although the image motion or inertial changes may cause a large anomaly enhancement term, the audio-visual time difference and inertial-visual coordination are usually well maintained. The time difference offset and coordination offset are close to zero, the credibility suppression term is close to one, and the anomaly trigger credibility value is close to the anomaly enhancement term, without excessive suppression. However, when external anomalies occur, they are often accompanied by audio-visual misalignment or inertial-visual non-coordination, resulting in a significant increase in the time difference offset and coordination offset. The credibility suppression term increases accordingly, effectively reducing the anomaly trigger credibility value and suppressing false alarms. It can accurately distinguish between voluntary actions and real external anomalies, providing a reliable basis for subsequent early warning decisions.

[0048] The specific formula for the confidence value of anomaly triggering is as follows: In the formula, Indicates the first The anomaly trigger confidence value of each anomaly observation segment reflects the degree of confidence that the current observation segment was triggered by an external anomaly event; Indicates the first External disturbance residuals for each anomalous observation segment; Indicates the first The visual anomaly values ​​of each abnormal observation segment reflect the confidence level of the presence of abnormal targets such as falling or intruding in the image; Indicates the first The sound anomaly values ​​of each abnormal observation segment reflect the confidence level of the presence of dangerous sounds such as breaking glass or screams in the environment; Indicates the first The audio-visual time difference value of each abnormal observation segment reflects the time offset between the start time of the audio impact and the time of the change of the video target. Indicates the first The reference audio-visual time difference value of each abnormal observation segment reflects the statistical benchmark of the audio-visual correspondence in historical autonomous actions; Indicates the first The inertial-visual co-variance values ​​of each abnormal observation segment reflect the degree of directional consistency between changes in inertial angular velocity and changes in video optical flow; Indicates the first The reference inertial-visual coordination change value of each abnormal observation segment reflects the statistical benchmark of inertial-visual coordination relationship in historical autonomous actions.

[0049] This implementation plan fully utilizes the fundamental difference that autonomous actions usually maintain audio-visual synchronization and inertial-visual coordination, while external anomalies often lead to cross-modal misalignment. It achieves effective screening of false anomalies and accurate labeling of real anomalies, providing high-confidence event inputs for subsequent anomaly handling sequence construction and early warning decision-making.

[0050] Specifically, the steps for performing pseudo-anomaly screening, anomaly fragment labeling, and event feature archiving based on the anomaly assessment results are as follows: By comparing the anomaly trigger confidence value and the trigger threshold in real time, when the anomaly trigger confidence value is less than the trigger threshold, the current observation segment is determined to be a false anomaly, and the segment is moved to the pending baseline buffer area. If the anomaly trigger confidence value of the same type of observation segment is less than the trigger threshold for N consecutive time periods, it indicates that the movement pattern occurs frequently and is not triggered by a real anomaly. At this time, the individual autonomous movement baseline is updated again, and this type of feature is included in the autonomous movement reference feature set and archived in the anomaly event database, thereby achieving adaptive evolution of the baseline and continuously reducing the probability of misjudging similar false anomalies. Here, N is set to 3, that is, the baseline update is triggered when the condition is met for 3 consecutive time periods. This value can avoid false updates caused by single accidental fluctuations and respond promptly to the slow changes in the wearer's habits.

[0051] When the confidence value of an anomaly trigger is greater than or equal to the trigger threshold, the current observation segment is marked as an anomalous observation segment. An anomaly handling sequence is constructed based on the confidence value of the anomaly trigger, from high to low, ensuring that high-confidence anomalies are handled first. Subsequently, a continuous window correlation analysis method is used to perform temporal correlation and repetition matching on anomalous observation segments in adjacent observation time windows to determine whether they belong to the same anomalous event: if the temporal overlap rate of two adjacent segments is greater than d and the intersection-union ratio (IoU) of the spatial target region is greater than T, they are merged into the same event. The temporal overlap rate d is defined as the ratio of the intersection length to the union length of two segments on the time axis, where d is the overlap threshold, set to 0.5, requiring at least 50% temporal overlap to filter out isolated segments that are too temporally discrete. The spatial target region IoU is defined as the ratio of the intersection area to the union area of ​​the target region bounding boxes in two segments, where T is the IoU threshold, set to 0.3, to allow for a certain degree of deformation and displacement while maintaining the continuity of target movement, suitable for situations where the target region may change significantly in events such as falls or collisions. By using the aforementioned spatiotemporal correlation rules, multiple segments of the same abnormal event are merged and the number of consecutive confirmations and the duration of the abnormality are output and archived into the abnormal event database, providing accurate event-level feature inputs for subsequent early warning decisions.

[0052] In this implementation plan, by comparing the anomaly trigger confidence value with the trigger threshold in real time, effective separation of false anomalies and true anomalies is achieved: segments below the trigger threshold are identified as false anomalies and moved to the pending baseline buffer area. After cumulative verification over N consecutive time periods, the individual autonomous action baseline is updated again, enabling the baseline to continuously absorb frequently occurring non-abnormal movement patterns, thereby reducing the probability of misjudging similar false anomalies from the source. Segments above the trigger threshold are marked as true anomaly observation segments. An anomaly handling sequence is constructed in descending order based on the anomaly trigger confidence value to ensure that high-confidence events are handled first. A continuous window association analysis method is used to perform spatiotemporal matching of anomaly segments within adjacent time windows. When the temporal overlap rate exceeds d and the spatial target area intersection-union ratio exceeds T, they are merged into the same event. Finally, the number of consecutive confirmations and the duration of the anomaly are output and archived in the anomaly event database. This mechanism not only achieves dynamic filtering and baseline optimization of false anomalies but also completes event-level integration and feature extraction of true anomalies, providing accurate and reliable event input for subsequent early warning decisions.

[0053] Specifically, the steps for conducting early warning decision-making calculations based on the characteristics of abnormal handling sequences are as follows: The system acquires the anomaly trigger confidence value, consecutive confirmation count, anomaly duration, and autonomous action similarity value for anomaly handling sequences. To comprehensively assess the urgency of warnings for current anomalies, these multidimensional features need to be integrated into a unified warning decision value. Specifically, the consecutive confirmation count reflects the cumulative intensity of the same anomaly being repeatedly captured in adjacent observation time windows. However, the gain effect of this intensity gradually saturates as the number of consecutive confirmations increases. Therefore, the consecutive confirmation count of the i-th anomaly observation segment is incremented by one and then subjected to a natural logarithm operation to obtain a confirmation enhancement term. Using the natural logarithm allows a small number of confirmations to contribute significant gains, while the gain increase gradually flattens as the number of confirmations continues to increase, avoiding over-warning caused by the accidental outbreak of a single event. Similarly, the anomaly duration reflects the duration of the event from its first trigger to the current moment. Its contribution to the urgency of warnings also has a saturation characteristic. Therefore, the anomaly duration is incremented by one and then subjected to a natural logarithm operation to obtain a persistence enhancement term, ensuring that long-term continuous anomalies can accumulate sufficient weight, but without infinitely amplifying the decision value due to extremely long durations. The alarm enhancement item is obtained by adding the anomaly trigger credibility value, the confirmation enhancement item and the continuous enhancement item. This fusion method not only retains the main position of the anomaly trigger credibility value, but also dynamically enhances the credibility of the event through the number of confirmations and the duration, so that anomalies that are repeatedly confirmed and continue to exist will receive higher alarm priority.

[0054] To prevent false alarms caused by accidental factors due to frequent spontaneous actions such as head turning and running, a similarity value of spontaneous actions is introduced as a suppression factor. The similarity value reflects the degree of matching between the current observed segment and historical spontaneous action features. A higher value indicates that the current movement pattern is closer to daily spontaneous actions, and even if the confidence value of the abnormal trigger is high, the alarm weight should be appropriately reduced. Therefore, the similarity value of spontaneous actions is incremented by one to obtain a decision suppression term. The alarm enhancement term is then divided by the decision suppression term to obtain the alarm decision value for the i-th abnormal observation segment. Using division instead of subtraction ensures that when the similarity value of spontaneous actions is close to zero, the decision suppression term approaches one, and the alarm decision value is close to the alarm enhancement term, maintaining sensitivity to suspected real anomalies. Conversely, when the similarity value of spontaneous actions is large, the decision suppression term increases significantly, and the alarm decision value is effectively compressed, thereby suppressing false alarms caused by violent head movements and other spontaneous actions. This mechanism integrates the end-side false alarm suppression strategy into the calculation of the alarm decision value through mathematical modeling, ensuring timely response to real anomalies in complex wearing scenarios while minimizing interference with daily activities, significantly improving user experience and alarm confidence.

[0055] The specific formula for the early warning decision value is as follows: ; In the formula, Indicates the first The warning decision value of each observation segment is used to comprehensively determine the urgency of the warning for the current abnormal event; Indicates the first The anomaly trigger confidence value of each observation segment reflects the degree of confidence that the current segment was triggered by an external anomaly event; Indicates the first The number of times an observation segment is repeatedly confirmed within a continuous observation time window reflects the cumulative confirmation intensity of the same anomalous event being captured continuously. Indicates the first The duration of an anomaly in each observation segment reflects the length of time from the initial triggering of the anomaly to the current moment; Indicates the first The similarity value of autonomous actions for each observation segment reflects the degree of matching between the current segment and the wearer's historical autonomous action samples.

[0056] In this implementation, the anomaly trigger confidence value is used as the main strength term. The number of consecutive confirmations and the duration of the anomaly, after being transformed by the natural logarithm, are accumulated as enhancement terms to form the early warning enhancement term. Simultaneously, the similarity value of autonomous actions is incremented by one as a decision suppression term, and the early warning enhancement term is dynamically suppressed by division, ultimately outputting the early warning decision value. This design fully utilizes the gain saturation characteristic of the natural logarithm on the number of confirmations and duration, allowing multiple confirmed and persistent anomalies to accumulate sufficient weight, while avoiding excessive amplification caused by single random fluctuations or extremely long durations. By using the similarity value of autonomous actions as the denominator suppression term, segments with high matching autonomous action features can be effectively suppressed even if they have a high anomaly trigger confidence value, thereby significantly reducing the probability of false alarms in everyday head movements, running, and other scenarios. Through the above fusion mechanism, the early warning decision value can accurately reflect the urgency of real anomalies, providing a reliable basis for subsequent tiered early warning scheduling. While ensuring timely response to real anomalies, it minimizes interference with the wearer, significantly improving the robustness of the system and the user experience.

[0057] Specifically, the steps for performing anomaly warning, evidence consolidation, and scheduling instruction generation based on the calculation results are as follows: By comparing the warning decision value with the warning threshold in real time, when the warning decision value is less than the warning threshold, it indicates that the overall urgency of the current abnormal observation segment is low and insufficient to trigger an actual alarm. Therefore, it is marked as a false alarm abnormal observation segment, and the current sensor sampling rhythm remains unchanged. At the same time, the complete video, audio, and inertial raw records are converted into lightweight keyframe summaries, audio envelope summaries, and attitude change summaries for storage to reduce storage overhead and retain necessary information for subsequent baseline optimization.

[0058] When the warning decision value is greater than or equal to the warning threshold, the corresponding abnormal observation segment is marked as a warning observation segment, and a refined analysis and evidence consolidation process needs to be initiated. Specifically, taking the current observation time window as the center, f preceding observation segments are extracted, and b continuing observation segments are extracted, and backtracking and supplementary recording are initiated. The duration of each observation time window is 1 second, and both f and b are 3, that is, 3 seconds are extended forward and backward, for a total of 7 seconds of continuous data. This range is sufficient to cover the complete process before and after the occurrence of common abnormal events such as falls and collisions, while avoiding the introduction of too much irrelevant data. The corresponding original video, audio, and inertial records in the cache are solidified from the circular buffer into read-only evidence segments to prevent subsequent writes from overwriting them. Simultaneously, the target area tracking algorithm is switched from conventional detection mode to frame-by-frame trajectory reconstruction, restoring the target's motion trajectory with pixel-level precision. The impact sound detection algorithm is switched from energy threshold detection to short-window fine scanning, using higher temporal resolution short-time Fourier analysis to accurately locate the impact moment. The attitude change detection algorithm is switched from attitude angle difference to abrupt change point relocation, using Kalman smoothing backward difference to identify the precise starting point of attitude instability. Based on chronological order, the moments of image change, impact sound arrival, and attitude instability are chained into an event timeline, generating a trigger-based alert command. Event type markers, warning times, consecutive confirmation counts, and batch traceability information are archived in the abnormal event database, providing complete and traceable evidence support for subsequent manual review and model optimization.

[0059] In this implementation plan, a tiered response and refined processing of abnormal events are achieved by comparing the early warning decision value with the early warning threshold in real time. When the early warning decision value is lower than the early warning threshold, the current segment is marked as a false alarm abnormal observation segment, and only the keyframe summary, sound envelope summary, and attitude change summary are retained. This effectively reduces storage overhead while accumulating data for subsequent baseline optimization. When the early warning decision value reaches or exceeds the early warning threshold, it is marked as an early warning observation segment, and a backtracking and supplementary recording mechanism is initiated. A complete abnormal event context is constructed by expanding a preset number of observation segments forward and backward. Based on this, the original data in the cache is solidified into read-only evidence segments. At the same time, target area tracking, impact sound detection, and attitude change detection are switched to refined analysis modes such as frame-by-frame trajectory reconstruction, short-window fine scanning, and abrupt change point relocation, respectively. Multimodal events are chained into an event time sequence chain in chronological order, and finally, an end-side reminder instruction is generated and key traceability information is archived in the abnormal event database. This mechanism ensures that real abnormal events can obtain complete evidence solidification and accurate analysis, and effectively avoids the interference of false alarms to users through tiered processing, providing high-quality traceable data support for subsequent manual review and model optimization.

[0060] Specifically, this embodiment provides an abnormal audio and video event detection and early warning device for video glasses, applied to a method for abnormal audio and video event detection and early warning for video glasses, including: The data acquisition and preprocessing module is used to acquire and preprocess multi-source data from the video glasses. The multi-source data includes environmental sound sequences, video frame sequences, angular velocity sequences, acceleration sequences, attitude change sequences, and time synchronization records. The preprocessing process includes data time synchronization calibration, noise filtering, and feature normalization to ensure data temporal consistency and validity. Subsequently, feature analysis and extraction are performed on the preprocessed multi-source data to extract various core features such as image motion features, angular velocity change features, acceleration change features, contact sound features, visual anomaly features, and sound anomaly features. The extracted feature data and the original preprocessed data are stored together to build an abnormal event database, providing comprehensive and reliable data support for the analysis and calculation of subsequent modules. The perturbation residual characterization module is used to construct a baseline and analyze external perturbations using image motion features and action nearest neighbor center vectors. The baseline construction is based on features extracted from historical autonomous action samples and action nearest neighbor center vectors to form an individual autonomous action baseline. The external perturbation analysis quantifies the degree of feature deviation caused by external perturbations by performing a series of processing operations such as difference and norm operations on image motion features and action nearest neighbor center vectors. Based on the results of the external perturbation analysis, the module performs perturbation segment identification and dynamic update of autonomous action baseline to accurately distinguish autonomous behavior segments from external perturbation segments. When no effective external perturbation is detected for several consecutive time periods and new typical autonomous action features are present, the autonomous action baseline is dynamically updated to ensure real-time matching between the baseline and the wearer's autonomous behavior features. The anomaly correlation and evaluation module is used to conduct audio-visual-inertial-visual collaborative correlation and anomaly credibility evaluation based on multimodal temporal synchronization data. The multimodal temporal synchronization data integrates various types of data such as preprocessed audio, video, and inertial data. Audio-visual-inertial-visual collaborative correlation strengthens the correlation of multi-dimensional features by analyzing the temporal relationship between audio and video and the collaborative response relationship between inertial and visual data. The anomaly credibility evaluation is based on the quantitative calculation of anomaly trigger credibility value based on multi-dimensional features. According to the anomaly evaluation results, pseudo-anomaly screening, anomaly segment marking, and event feature archiving operations are performed. Segments with anomaly trigger credibility value lower than the trigger threshold are judged as pseudo-anomalies and screened out. Segments with anomaly trigger credibility value higher than or equal to the trigger threshold are marked as anomaly observation segments. At the same time, various feature parameters and evaluation results of anomaly segments are archived to the anomaly event database to achieve standardized management of anomaly data. The event confirmation and early warning module is used to perform early warning decision calculations based on the characteristics of the abnormal handling sequence. The abnormal handling sequence is constructed in descending order according to the confidence value of the abnormal trigger. The early warning decision calculation combines the temporal correlation of abnormal segments, the duration of the abnormality, and other characteristics to comprehensively judge the abnormality level and handling priority. Based on the calculation results, it executes abnormal early warning, evidence solidification, and scheduling instruction generation. According to the abnormality level, it outputs the corresponding level of early warning prompts, solidifies the audio-visual data, feature parameters, and other key evidence of the abnormal observation segment, and generates appropriate scheduling instructions to ensure that abnormal events are handled in a timely and accurate manner.

[0061] In this implementation plan, four modules work synergistically and progressively. The data acquisition and preprocessing module provides standardized and reliable multi-source data support for the entire anomaly detection and early warning scheme, ensuring the effectiveness of subsequent analysis and calculations. The disturbance residual characterization module enables the accurate construction and dynamic updating of autonomous action baselines, effectively distinguishing autonomous behavior from external disturbance segments and laying the foundation for anomaly assessment. The anomaly correlation and assessment module accurately assesses the credibility of anomalies through multimodal audio-visual-inertial collaborative correlation analysis, achieving pseudo-anomaly screening, anomaly segment marking, and feature archiving, ensuring the standardization and accuracy of anomaly data. The event confirmation and early warning module completes early warning decision calculations based on the anomaly handling sequence, outputs corresponding level anomaly warnings, solidifies key evidence of anomalies, and generates adaptive scheduling instructions. Ultimately, it achieves accurate detection, effective identification, and timely handling of abnormal audio and video events in video glasses, ensuring the reliability, relevance, and efficiency of anomaly detection, and providing comprehensive anomaly protection support for wearers.

[0062] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0063] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for detecting and warning of abnormal audio and video events in video glasses, characterized in that, Includes the following steps: S1: Collect multi-source data from video glasses and perform preprocessing, then perform feature parsing and extraction, store the data, and build an abnormal event database; S2 constructs a baseline and analyzes external disturbances by using image motion features and action nearest neighbor center vectors. Based on the results of the external disturbance analysis, it performs disturbance segment identification and autonomous action baseline dynamic update operations. S3 performs audio-visual-inertial-visual collaborative correlation and anomaly credibility assessment based on multimodal temporal synchronization data, and performs pseudo-anomaly screening, anomaly segment marking and event feature archiving operations based on the anomaly assessment results; S4 performs early warning decision calculations based on the characteristics of the abnormal handling sequence, and executes abnormal early warning, evidence solidification and scheduling instruction generation based on the calculation results.

2. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for collecting multi-source data from video glasses and performing preprocessing are as follows: The system collects multi-source data from video glasses, including video data, audio data, and inertial data. Video data acquisition involves obtaining a sequence of video frames within the observation time window using an image acquisition unit; extracting an optical flow histogram as a motion feature vector using optical flow; and extracting the target region using a target detection algorithm and obtaining the bounding box coordinate sequence as the target region feature vector. The sharpness of each frame is calculated using the Laplacian operator, and the p-th percentile within the window is taken as the sharpness retention value. The static background feature points between adjacent frames are extracted using the feature point matching method, and the average displacement of the global homography matrix is ​​estimated using the RANSAC algorithm as the background drift value. Acquiring sound data: Ambient sound sequences and sound sequences of contact between the glasses frame and the frame are acquired through a dual-microphone array; Mel spectrograms are extracted and input into a lightweight convolutional neural network, outputting an impact spectrum feature vector; human voice is separated using speech activity detection, and the frame friction feature vector is extracted based on short-time energy analysis in the mid-to-high frequency band; Acquire inertial data: Obtain angular velocity, acceleration, and attitude angle sequences through a six-axis inertial measurement unit; calculate the peak angular velocity within the window as the angular velocity mutation vector, calculate the acceleration variance as the acceleration mutation vector, and calculate the attitude angle change rate as the attitude transition vector; Each observation record is synchronously written with an observation segment identifier, acquisition time, device identifier, and scene identifier, and time alignment and window aggregation are performed based on the acquisition time. The video frame sequence is deblurred, occluded observation segments are marked, and brightness is normalized. The ambient sound sequence is subjected to wind noise suppression, broadband noise suppression, and frame contact sound separation. The angular velocity sequence, acceleration sequence, and attitude change sequence are subjected to abnormal peak correction and normalization. The video features, sound features, contact sound features, and inertial features within the same observation time window are integrated into abnormal observation segments, and a unique observation segment sequence number is assigned to each abnormal observation segment.

3. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for performing feature parsing and extraction, storing the data, and constructing the abnormal event database are as follows: By extracting motion features, angular velocity variation features, acceleration variation features, and contact sound features from a historical autonomous action sample set, an individual autonomous action baseline is constructed, and cluster analysis is performed to obtain the nearest neighbor center vector of the autonomous action. Instantaneous collision response features, friction response features, and short-time impulse response features are extracted from the frame contact sound sequence and inertial sequence, and intensity aggregation analysis is performed on the response features to output the frame contact interference value of the abnormal observation segment. Visual anomaly features are extracted based on the target area displacement change, contour change, posture change, and optical flow mutation in the video frame sequence, and visual anomaly values ​​are output. Sound anomaly features are extracted based on the impact spectrum change, short-time energy change, and sound source mutation in the environmental sound sequence, and sound anomaly values ​​are output. The autonomous action similarity value is calculated based on the distance relationship between the motion features, inertial change features of the current abnormal observation segment and the historical autonomous action reference feature set. The preprocessed multi-source data of the abnormal events is stored and an abnormal event database is constructed.

4. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for baseline construction and external disturbance analysis using image motion features and action nearest neighbor center vectors are as follows: Get the The motion feature vector, action nearest neighbor center vector, sharpness retention value, background drift value, and frame contact interference value of each abnormal observation segment; For the The difference between the motion feature vector and the action nearest neighbor center vector of each abnormal observation segment is calculated to obtain the offset vector; the offset vector is subjected to the L2 norm operation to obtain the offset amplitude; the offset amplitude is multiplied by the sharpness retention value to obtain the residual amplification. Add the background drift value and the frame contact interference value, then add one to obtain the residual suppression term; divide the offset amplitude by the residual suppression term to obtain the first term. External disturbance residuals of an anomalous observation segment.

5. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for performing disturbance segment identification and autonomous action baseline dynamic update operations based on the external disturbance analysis results are as follows: By comparing the residual value of external disturbance with the residual threshold in real time, when the residual value of external disturbance is less than the residual threshold, the corresponding abnormal observation segment is marked as the autonomous action-dominated observation segment, and only the observation segment index, posture summary and sound summary are retained; if the image motion feature vector of the observation segment has a maximum cosine similarity with the autonomous action reference feature set greater than the similarity threshold, the observation segment is written into the pending baseline buffer area and the individual autonomous action baseline is updated to adapt to the wearer's individual habit changes; When the external disturbance residual value is greater than or equal to the residual threshold, freeze m consecutive time windows before and after the current observation segment to construct forward and backward evidence bands. Use the median of the background reference point displacement sequence in the forward and backward evidence bands as the corrected background drift value, use the k% quantile of the video frame sharpness sequence in the evidence band as the corrected sharpness retention value, and use the mean of the short-time energy sequence of the frame contact sound in the evidence band as the corrected contact interference value. Recalculate the external disturbance residual value based on the corrected background drift value, sharpness retention value, and contact interference value. If the recalculated external disturbance residual value is still greater than or equal to the residual threshold, mark the current observation segment as an external trigger candidate segment and output the external disturbance residual value to the anomaly correlation and evaluation module. If the recalculated residual value of the external disturbance is less than the residual threshold, the corresponding abnormal observation segment will be marked as an autonomous action-dominated observation segment.

6. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for conducting audio-visual-inertial co-correlation and anomaly credibility assessment based on multimodal temporal synchronization data are as follows: Using environmental sound sequences, video frame sequences, angular velocity sequences, acceleration sequences, attitude change sequences, and time-synchronized records as basic inputs, an audio-visual time difference analysis model is constructed using the audio-visual time sequence correlation method. Based on the impact start time in the environmental sound sequence and the target area change time in the video frame sequence, the audio-visual time difference value is output, and a reference audio-visual time difference value is output by combining the audio-visual time offset statistics in historical autonomous action observation segments. An inertial vision co-analysis model is constructed using an inertial vision co-analysis method. Based on the synchronous response relationship between the optical flow change trend in the video frame sequence and the angular velocity sequence, acceleration sequence, and attitude change sequence, the inertial vision co-change value is output. The reference inertial vision co-change value is output by combining the statistical results of the inertial change and the image change in the historical autonomous action observation segments. Get the External disturbance residual values ​​of each anomalous observation segment Visual anomalies Abnormal sound values; for the first The visual and acoustic anomaly values ​​of each anomalous observation segment are squared, summed, and the square root is taken. This sum is then added to the external disturbance residual value to obtain the joint anomaly amplitude. The absolute difference between the audio-visual time difference and the reference time difference is calculated to obtain the time difference offset. The absolute difference between the inertial-visual co-variance value and the reference inertial-visual co-variance value is calculated to obtain the co-variance offset. The external disturbance residual value is added to the joint anomaly amplitude to obtain the anomaly enhancement term. The time difference offset and the co-variance offset are added together and then one is added to obtain the credible suppression term. The anomaly enhancement term is divided by the credible suppression term to obtain the first anomaly. The confidence value of the anomalous trigger for each anomalous observation segment.

7. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for performing pseudo-anomaly removal, anomaly fragment marking, and event feature archiving operations based on the anomaly assessment results are as follows: The system compares the anomaly trigger confidence value with the trigger threshold in real time. When the anomaly trigger confidence value is less than the trigger threshold, it is determined to be a false anomaly, and the observed segment is moved to the pending baseline buffer area. If the observed segment is in continuous... If the confidence value of an abnormal trigger is less than the trigger threshold within a certain time period, the individual autonomous action baseline will be updated again and archived to the abnormal event database. When the confidence value of an anomaly trigger is greater than or equal to the trigger threshold, the observation segment is marked as an anomaly observation segment. An anomaly handling sequence is constructed in descending order based on the confidence value of the anomaly trigger. The continuous window association analysis method is used to perform temporal association and repetition matching on the anomaly observation segments in adjacent observation time windows. If the time overlap rate is >d% and the spatial target region IoU is >T, they are merged into the same event, and the number of consecutive confirmations and the duration of the anomaly are output and archived to the anomaly event database.

8. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for conducting early warning decision-making calculations based on anomaly handling sequence features are as follows: Obtain the anomaly trigger confidence value, consecutive confirmation count, anomaly duration, and autonomous action similarity value of the anomaly handling sequence; for the first... The number of consecutive confirmations for each abnormal observation segment is incremented by one and then the natural logarithm is calculated to obtain the confirmation enhancement term; the duration of the abnormality is incremented by one and then the natural logarithm is calculated to obtain the persistence enhancement term; the anomaly trigger confidence value, the confirmation enhancement term, and the persistence enhancement term are added together to obtain the early warning enhancement term. Incrementing the similarity value of autonomous actions by one yields the decision inhibition term; Dividing the early warning enhancement term by the decision inhibition term yields the first... Early warning decision value for an abnormal observation segment.

9. The method for detecting and warning of abnormal audio and video events in video glasses according to claim 1, characterized in that: The specific steps for performing anomaly warning, evidence consolidation, and scheduling instruction generation based on the calculation results are as follows: By comparing the early warning decision value with the early warning threshold in real time, when the early warning decision value is less than the early warning threshold, the corresponding abnormal observation segment is marked as a false alarm abnormal observation segment, the current sampling rhythm is maintained, and the complete record is converted into keyframe summary, sound envelope summary, and attitude change summary. When the warning decision value is greater than or equal to the warning threshold, the corresponding abnormal observation segment is marked as a warning observation segment. The observation time window is expanded forward to extract f preceding observation segments and expanded backward to extract b continuing observation segments to initiate backtracking and supplementary recording. The original video, audio, and inertial records in the cache are solidified into read-only evidence segments. The target area tracking is switched to frame-by-frame trajectory reconstruction, the impact sound detection is switched to short-window fine scanning, and the attitude change detection is switched to abrupt point relocation. The image change, impact sound arrival, and attitude instability are linked into an event time sequence chain according to the chronological order. A triggering end-side reminder instruction is generated, and the event type mark, warning time, number of consecutive confirmations, and batch traceability information are archived to the abnormal event database.

10. An abnormal audio and video event detection and early warning device for video glasses, employing the abnormal audio and video event detection and early warning method for video glasses as described in any one of claims 1-9, characterized in that, include: The data acquisition and preprocessing module is used to acquire multi-source data from video glasses, perform preprocessing, analyze and extract features, store the data, and build an abnormal event database. The perturbation residual characterization module is used to construct the baseline and analyze external perturbations by using image motion features and action nearest neighbor center vectors. Based on the results of the external perturbation analysis, it performs perturbation segment identification and autonomous action baseline dynamic update operations. The anomaly correlation and evaluation module is used to conduct audio-visual inertial-visual collaborative correlation and anomaly credibility evaluation based on multimodal time-series synchronized data, and to perform pseudo-anomaly screening, anomaly segment marking and event feature archiving operations based on the anomaly evaluation results; The event confirmation and early warning module is used to perform early warning decision calculations based on the characteristics of abnormal handling sequences, and to execute abnormal early warnings, evidence solidification, and scheduling instruction generation based on the calculation results.

Images