Security internet of things intelligent early warning method and system based on data fusion

Abstract

The application relates to the security technology field and discloses a security Internet of Things intelligent early warning method and system based on data fusion. The method grids video frames, obtains a video confusion degree sequence according to grid motion vectors and image gradients, detects abnormal events from an array audio, obtains a sound source azimuth angle and an audio energy sequence, calculates bidirectional information transmission according to the audio energy sequence and the video confusion degree sequence, binds a unidirectional significant video segment and the abnormal event into a candidate compound event, and obtains a cause-effect confidence degree; obtains a visual confusion center azimuth angle according to a high confusion grid center of gravity of an event frame, matches the sound source azimuth angle to obtain a spatial confidence degree; fuses the cause-effect confidence degree and the spatial confidence degree to obtain a comprehensive confidence degree, and an alarm signal is sent when the comprehensive confidence degree exceeds a warning threshold. The application verifies abnormal events through time cause-effect relationships and spatial consistency, and reduces the false alarm rate of security early warning.

Description

Technical Field

[0001] This application relates to the field of intelligent security technology, and in particular to a security IoT intelligent early warning method and system based on data fusion. Background Technology

[0002] Security IoT systems achieve 24 / 7 monitoring of public places by deploying sensors such as video surveillance and audio acquisition. In densely populated areas such as subway platforms and shopping malls, abnormal events are often accompanied by visual confusion and unusual sounds. Existing technologies have three major drawbacks: pure video analysis fails when there is high-density crowd obstruction or drastic changes in lighting; pure audio detection has a high false alarm rate under complex background noise; and simple audio-visual fusion only uses time synchronization or threshold superposition, which cannot isolate common environmental disturbances or verify spatial consistency, resulting in insufficient accuracy and reliability of early warnings. Summary of the Invention

[0003] To address the technical problems of high false alarm rates in single-modal detection and the inability of simple multimodal fusion to effectively distinguish causal relationships and spatial consistency in security early warning systems, this application provides a security IoT intelligent early warning method and system based on data fusion.

[0004] Firstly, this application provides a data fusion-based intelligent early warning method for security IoT, employing the following technical solution: The data fusion-based intelligent early warning method for security IoT includes: acquiring audiovisual data at the current moment and extracting array audio from it; determining whether an abnormal event has occurred based on the array audio; if so, acquiring the azimuth angle of the sound source of the abnormal event, as well as the audio energy sequence and multiple video frames within a set time period before and after the current moment; dividing each video frame into multiple grids, calculating its local disorder degree based on the grid motion vector of each grid, and obtaining the video disorder degree sequence of each video frame based on the local disorder degree; calculating... The audio-visual data contains bidirectional information transmission between audio and video, and the audio-visual data is used to determine whether it is unidirectional significant data. If so, the audio-visual data is bound as a candidate composite event and the causal confidence of the candidate composite event is calculated. The video frame corresponding to the start time of the abnormal event is recorded as the event frame, the azimuth angle of the visual confusion center of the event frame is identified, and its azimuth difference with the azimuth angle of the sound source is calculated. If the azimuth difference is less than the spatial matching tolerance, the spatial confidence of the candidate composite event is generated. The causal confidence and the spatial confidence are weighted and fused to obtain the comprehensive confidence of the candidate composite event, and an alarm signal is issued when the comprehensive confidence exceeds the warning threshold.

[0005] Optionally, the signal of each channel of the array audio is processed by frame segmentation to obtain the short-time energy sequence of each channel and the comprehensive spectral centroid sequence formed by the spectral centroids of each channel. The abnormal event is then determined based on the energy background estimate corresponding to the short-time energy sequence and the comprehensive spectral centroid sequence.

[0006] Optionally, one channel of the array audio is used as a reference channel to obtain the measured arrival delay of the other channels relative to the reference channel, and the azimuth angle of the sound source of the abnormal event is determined based on the measured arrival delay and the geometric layout of the microphone array.

[0007] Optionally, a local orientation variance factor and a local kinetic entropy factor are obtained based on the mesh motion vector, and a texture weight is calculated based on the image gradient magnitude of the mesh; the local orientation variance factor and the local kinetic entropy factor are adaptively weighted and fused based on the texture weight to obtain the local disorder of the mesh.

[0008] Optionally, the local disorder of all grids within each video frame is statistically analyzed to obtain the video disorder of each video frame, and the video disorder is arranged in the temporal order of multiple video frames to form the video disorder sequence.

[0009] Optionally, the audio energy sequence and the video disorder sequence within the analysis window corresponding to the abnormal event are extracted, and the audio energy sequence and the video disorder sequence are converted into audio symbol sequences and video symbol sequences, respectively; based on the audio symbol sequences and the video symbol sequences, the video-to-audio information transfer amount and the audio-to-video information transfer amount are calculated, and the two are used as the bidirectional information transfer amount.

[0010] Optionally, randomized statistical tests are performed on the information transfer volume in the video-to-audio direction and the audio-to-video direction respectively to obtain the standardized significance score in the video-to-audio direction and the standardized significance score in the audio-to-video direction; when the standardized significance score in only one direction meets the significance condition, the audiovisual data is determined to be unidirectional significant data.

[0011] Optionally, if the audiovisual data is determined to be unidirectional significant data, the corresponding video segment is bound to the abnormal event as the candidate composite event, and the standardized significance score corresponding to the information transmission direction that satisfies the significance condition is used as the causal confidence score.

[0012] Optionally, in the event frame, high-disorder grids with local disorder levels higher than the background level are selected. The centroid of the high-disorder grid is obtained based on the center pixel coordinates of the high-disorder grid, and the centroid of the high-disorder grid is projected onto the global coordinate system to obtain the azimuth angle of the visual disorder center. The azimuth difference is calculated based on the azimuth angle of the visual disorder center and the azimuth angle of the sound source. If the azimuth difference is less than the spatial matching tolerance, the spatial confidence score is generated. The causal confidence score and the spatial confidence score are weighted and fused to obtain the comprehensive confidence score.

[0013] Secondly, this application provides a security IoT intelligent early warning system based on data fusion, which adopts the following technical solution: The security IoT intelligent early warning system based on data fusion includes a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the aforementioned security IoT intelligent early warning method based on data fusion is implemented.

[0014] This application has the following technical effects: 1. By meshing video frames and fusing local orientation variance factor and local kinetic entropy factor, combined with texture weights obtained from image gradient magnitude, the estimation bias of single visual features under high-density crowd occlusion and complex texture environments is effectively overcome, significantly improving the robustness and accuracy of video disorder assessment.

[0015] 2. By introducing the transfer entropy and random scrambling statistical test method, the common cause fluctuation interference of audiovisual modalities in security scenarios was successfully removed, the real one-way causal driving relationship between video and audio was accurately identified, the environmental synchronization disturbance and real abnormal events were effectively distinguished, and the false alarm rate was greatly reduced.

[0016] 3. By extracting the centroid of highly disordered grids and performing multi-frame median filtering projection, combined with a spatial matching tolerance mechanism, we successfully eliminated erroneous events that co-occur in time but are separated in physical space, thus achieving dual verification of temporal causal correlation and physical spatial consistency. Attached Figure Description

[0017] Figure 1 This is a flowchart of a security IoT intelligent early warning method based on data fusion. Detailed Implementation

[0018] This application discloses a security IoT intelligent early warning method based on data fusion, referring to... Figure 1 ,include: S1. Obtain the audio-visual data at the current moment and extract the array audio from it. Determine whether an abnormal event has occurred based on the array audio. If so, proceed to step S2.

[0019] In addition to video surveillance, a multi-channel microphone array inside the platform synchronously collects ambient sounds. Train hum, broadcasts, and conversations among people create a complex background noise field. Fixed beamforming technology can be used to enhance the sound source in the target area directly in front and suppress diffused noise. This is common knowledge and will not be elaborated on in this embodiment to avoid unnecessary details.

[0020] The method for determining whether an abnormal event has occurred based on the array audio at the current moment in this embodiment includes: To detect abnormal events, the array audio includes multiple channel signals from a microphone array. In this embodiment, each channel signal is divided into frames according to frame length sampling points and frame shift sampling points to obtain multiple audio frames corresponding to each channel signal; a short-time Fourier transform is applied to each audio frame to obtain the power spectrum corresponding to each audio frame, and the spectral centroid of the corresponding audio frame is calculated based on each power spectrum. The frame length sampling points can be the number of sampling points corresponding to 25 milliseconds, and the value range can be the number of sampling points corresponding to 20 milliseconds to 40 milliseconds; the frame shift sampling points can be the number of sampling points corresponding to 10 milliseconds, and the value range can be the number of sampling points corresponding to 5 milliseconds to 20 milliseconds, and is less than the frame length sampling points. The above settings achieve a balance between audio frequency resolution, abnormal event timing accuracy, and computational load. A larger frame length sampling number results in a more stable spectral centroid, but lower timing accuracy for abnormal events. A smaller frame length sampling number results in higher timing accuracy for abnormal events, but the spectral centroid is more susceptible to transient noise. A larger frame shift sampling number results in lower computational load, but weaker temporal continuity between adjacent audio frames. A smaller frame shift sampling number results in stronger temporal continuity, but higher computational load.

[0021] Specifically, the spectral centroid of the m-th channel in the t-th frame satisfies the following relationship: ; in, Indicates the first The first channel in the The centroid of the frame's spectrum; Indicates the channel index number of the microphone array; Indicates the audio frame number; Indicates the power spectrum frequency point index. ; This represents the total number of frequency points in the power spectrum. Indicates the first The frequency corresponding to each frequency point is expressed in Hertz. Indicates the first The first channel Frame in Power spectrum values ​​at each frequency point.

[0022] No. The first channel in the Frame spectral centroid element value The larger the value, the higher the proportion of high-frequency components in the audio signal acquired by that single channel in that frame, and the brighter the timbre; when When a sudden increase occurs, it indicates that the m-th channel picked up an abnormal sound rich in high-frequency components in the t-th frame.

[0023] The spectral centroids of each channel within the same frame of signal are obtained by iterating through the data. The mean of all the obtained spectral centroids is calculated, and this mean is used as the composite spectral centroid of the corresponding frame of signal. The centroid of the integrated spectrum represents the brightness of the integrated timbre.

[0024] Calculate the short-time root-mean-square energy for each channel per frame to obtain the short-time energy sequence for each channel; then average the short-time energy sequences for each channel to form the audio energy sequence. The frame number is used; the average of the spectral centroids of each channel is taken to form a comprehensive spectral centroid sequence; since relying solely on the energy threshold is easily triggered by noise, a large number of false alarms must be made in conjunction with frequency domain characteristics.

[0025] To adapt to the gradually changing background noise of the platform, a first-order recursive smoothing is performed on the short-time energy to maintain the energy background estimate. Specifically, the first-order recursive smoothing of short-time energy satisfies the following relationship: ; ; in, This represents the energy background estimate for the current frame; Indicates the smoothing coefficient; Indicates the audio energy of the current frame; This indicates the energy background estimate of the previous frame; short-time energy. Energy background estimation The smoothing coefficient is adaptively determined by the variance of the current frame audio energy and the variance of the energy background estimate according to the aforementioned relationship, and its value ranges from 0 to 1. The larger the smoothing coefficient, the faster the energy background estimate responds to the current frame audio energy, and the better it can adapt to rapidly changing background noise, but it is prone to including short-term abnormal energy in the energy background estimate; the smaller the smoothing coefficient, the more stable the energy background estimate, and the better it can suppress transient noise interference, but it is slower to adapt to continuously changing background noise.

[0026] Energy background estimation It integrates the current frame energy and historical energy background; when When a brief spike occurs, due to Smaller The changes are gradual, thus ensuring the stability of the background estimate and avoiding misjudging instantaneous noise as anomalies.

[0027] When the current frame audio energy is greater than Adaptive rate threshold for background estimation And the spectral centroid satisfies When an abnormal event occurs, the start time is recorded. The adaptive rate threshold is dynamically adjusted based on the energy background estimate, and satisfies the following relationship: ; in, Indicates the adaptive scaling threshold; This indicates an estimate of the energy background; This represents the mean of the long-term historical energy background estimate.

[0028] Adaptive scaling threshold With energy background estimation The noise level increases with the increase of background noise; When it rises, The increased size raises the threshold for anomaly detection, thereby effectively suppressing false alarms caused by environmental noise.

[0029] In this embodiment, based on the average energy background, when the audio energy of the current frame exceeds an adaptive multiplier threshold of the average energy background and the spectral centroid falls within a preset high-frequency range, an abnormal event is determined and the start time is recorded. The preset high-frequency range is set to 25 Hz to 8000 Hz. This range can be determined based on the spectral centroid distribution of normal background sound and abnormal sound samples in the deployment scenario. The value of the preset high-frequency range is between 0 Hz and half the audio sampling frequency. A higher lower limit of the preset high-frequency range provides stronger suppression of low-frequency train noise and conversations, but may miss abnormal sounds with low spectral centroids; a lower lower limit results in higher sensitivity for abnormal sound detection, but increases the risk of false alarms caused by background noise.

[0030] In this embodiment, the channel short-time energy sequence and the channel spectral centroid sequence are calculated for each channel signal of the array audio.

[0031] By introducing a first-order recursive smoothing mechanism that combines the spectral centroid with an adaptive dynamic threshold, the interference caused by complex background noise in subway stations (such as the rumble of trains entering the station) is effectively overcome, the false alarm rate of audio anomaly detection is greatly reduced, and the azimuth and start time of the sound source of the anomaly event are accurately located.

[0032] S2, obtain the azimuth angle of the sound source of the abnormal event, as well as the audio energy sequence and multiple video frames within a set time period before and after the current moment.

[0033] For the audio frame detecting the abnormal event, using one channel in the microphone array as a reference channel, the audio signals of the remaining channels are subjected to generalized cross-correlation-phase transformation with the audio signal of the reference channel to obtain the measured arrival delay of each channel relative to the reference channel. Each measured arrival delay characterizes the time difference between the arrival of the abnormal sound source signal at its corresponding channel and its arrival at the reference channel. Based on the known geometric layout of the microphone array, the spatial positional relationship between the microphone corresponding to the reference channel and the microphones corresponding to the other channels is determined. The measured arrival delays are then matched with the corresponding microphone spatial positional relationships. A spatial spectrum search is used to determine the sound source azimuth angle that minimizes the deviation between the theoretical and measured arrival delays, thereby obtaining the sound source azimuth angle of the abnormal event. The generalized cross-correlation-phase transformation and spatial spectrum search are techniques well-known to those skilled in the art and will not be elaborated upon here.

[0034] The update interval of the audio energy sequence is equal to the audio frame shift time, that is, the time difference between adjacent audio frames; when the abnormal event is detected, the start time and azimuth angle of the sound source of the abnormal event are recorded synchronously, and the start time, the azimuth angle of the sound source and the audio energy sequence are used as the output of this step.

[0035] After determining that the abnormal event has occurred, based on the start time of the abnormal event, the audio energy sequence within a set time period before and after the start time is extracted from the continuously updated audio energy sequence, and multiple video frames within the set time period before and after the start time are obtained from the video data synchronously acquired with the array audio. The start time, the azimuth angle of the sound source, the audio energy sequence, and the multiple video frames are used as the output of step S2. The set time period is half the total duration of the analysis window in step S4; the specific reasons for setting it, its value range, and adjustment rules are explained in step S4.

[0036] S3, each video frame is divided into multiple grids, the local disorder degree of each grid is calculated based on the grid motion vector, and the video disorder degree sequence of each video frame is obtained based on the local disorder degree.

[0037] In the high-density crowd environment of subway platforms, severe obstruction between passengers makes individual behavior analysis difficult. Therefore, single-frame images of the video are divided into... The pixels are non-overlapping square grids; motion estimation is performed on each grid using the Farneback dense optical flow algorithm, which is a well-known technique and will not be described in detail here, to obtain the grid motion vector of each grid in the current frame. , , For grid index coordinates, The frame number is used, and the magnitude of the mesh motion vector, expressed in pixels per frame, represents the speed of local motion. The mesh side length can be 16 pixels, ranging from 8 to 32 pixels. Setting the mesh side length to 16 pixels balances the preservation of local motion differences with the stability of optical flow calculation. A larger mesh side length means more pixels per mesh, resulting in more stable mesh motion vector estimation and lower computational cost, but lower spatial positioning accuracy in cluttered areas. Conversely, a smaller mesh side length results in higher spatial positioning accuracy in cluttered areas, but the mesh motion vector is more susceptible to insufficient texture and noise, and requires more computation.

[0038] When a crowd exhibits localized disordered movement, the movement directions within the neighborhood will show significant inconsistencies. Therefore, at each grid location, a pre-defined [mechanism / mechanism] is extracted. The motion direction angle of the grid motion vector of each grid in the neighborhood, in radians; treating each angle as a unit complex number, calculate the magnitude of the sum of all complex numbers and divide by the total number of neighborhood grids. Then subtract that ratio from 1 to obtain the local direction variance factor. The factor is zero when the directions are completely consistent and approaches 1 when the directions are completely random. The local direction variance factor satisfies the following relationship: ; in, Let represent the local orientation variance factor of the grid with coordinates (x, y) in the t-th video frame; , Indicates grid index coordinates; Indicates the frame number; express The motion direction angle (x, y) of each grid motion vector within the neighborhood, in radians; This indicates that each angle is considered as a unit complex number; This represents the total number of neighboring grids. The preset neighborhood can be a 3×3 neighborhood, with side lengths ranging from 3 to 7 (odd numbers). Using a 3×3 neighborhood reflects the differences in motion directions between adjacent grids while avoiding over-smoothing; a larger neighborhood results in a more stable local direction variance factor, but the boundaries of adjacent motion regions are easily smoothed; a smaller neighborhood is more sensitive to sudden local motions, but is more susceptible to errors in the motion vectors of individual grids.

[0039] Local direction variance factor The larger the value, the more dispersed and disordered the motion direction angles of the mesh motion vectors within the neighborhood; when... When the value approaches 0, it indicates that the directions are completely consistent and the local movement of the scene is orderly; when When the value approaches 1, it indicates that the direction is completely random and the local motion of the scene is extremely chaotic and disordered.

[0040] Taking one of the grids as an example, the instantaneous kinetic energy of this grid is defined to satisfy the following relationship: ; in, Represents the instantaneous kinetic energy of the grid, expressed in square pixels per square frame; Represents the grid motion vector of the current frame; Represents the grid motion vector The modulus is expressed in pixels per frame, representing the speed of local motion.

[0041] Instantaneous kinetic energy of the grid The larger the value, the faster the movement speed and the higher the kinetic energy of the people within that grid; when An increase indicates intense movement in a local area, which may lead to a rise in the overall level of chaos in the scene.

[0042] To avoid systematic deviations in kinetic energy amplitude caused by camera distance, all grids across the entire frame are... After sorting the values, they are divided into equal-frequency groups. There are n histogram intervals, where the total number of histogram intervals B satisfies the following relationship: ; Where B is the total number of histogram intervals. The total number of all grids in the image. This is the floor function.

[0043] The total number of grids generated by cameras with different resolutions They are different. The larger the data volume, the more intervals are automatically divided, ensuring that there are enough samples in each interval to support the calculation of Shannon entropy.

[0044] In the current grid Within the neighborhood, count the frequency at which the kinetic energy of each grid falls into each histogram interval. The Shannon entropy is calculated and used as the local kinetic energy entropy factor. Specifically, the local kinetic energy entropy factor satisfies the following relationship: ; in, Represents the local kinetic energy entropy factor; Indicates the current grid Within the neighborhood, the kinetic energy of each grid falls into the first... The frequency of each histogram interval; This represents the total number of histogram intervals.

[0045] Local kinetic energy entropy factor The higher the value, the more dispersed and uneven the distribution of kinetic energy within the neighborhood; when... When the value increases, it indicates a significant difference in kinetic energy in a local area, suggesting the possible presence of abnormally high-energy motion in a localized area within the scene.

[0046] In subway platform scenes, the camera's field of view often contains both sparse and rich texture regions. For the former, such as the ground obscured by dense crowds, the local orientation variance factor is more reflective of the disorder than the local kinetic entropy factor. For the latter, such as billboards, the local kinetic entropy factor is more discriminative of anomalies. Using fixed fusion weights will lead to bias in the estimation of local disorder. Therefore, texture weights based on image gradient magnitude are introduced.

[0047] Introducing texture weights, which satisfy the following relationship: ; in, express Texture weights of the position grid on frame t; express The image gradient magnitude of the position grid on frame t; Maximum difference magnitude of 8-bit image Round up.

[0048] Texture weights The smaller the value, the higher the gradient magnitude of the image in the current grid. The larger the size, the richer the texture; when When the value is close to 1, it indicates a small gradient magnitude and sparse texture; in this case, the local orientation variance factor contributes more to the disorder. When the value is close to 0, it indicates that the gradient magnitude is large and the texture is rich. At this time, the local kinetic entropy factor contributes more to the disorder.

[0049] Based on the texture weights, a linear combination of the local orientation variance factor and the local kinetic entropy factor yields the local disorder degree, which satisfies the following relationship: ; in, express Local clutter of the position grid on frame t; Indicates texture weight; Indicates the local directional variance factor; This represents the local kinetic energy entropy factor.

[0050] Local disorder The larger the value, the more disordered the motion direction and the more dispersed the energy distribution in the grid region, and the more chaotic the local scene; when... An increase in the value indicates a higher probability of abnormal crowding or disorderly movement in the area.

[0051] Local disorder of all grids in the current frame Calculate the arithmetic mean to obtain the video scrambling degree of the current video frame. Then, calculate the video scrambling level for each frame according to the video frame rate. This forms a time series with the sampling rate synchronized with the video frame rate, namely the video chaos sequence. The higher the value of this sequence, the more chaotic the overall movement of the crowd in the frame at the corresponding moment.

[0052] By dynamically fusing local orientation variance factor and local kinetic entropy factor as texture weights as adaptive weights, the single feature estimation bias caused by complex textures in subway platforms (such as dense crowds and billboards) is effectively overcome, significantly improving the robustness and accuracy of local disorder and video disorder assessment, and providing high-quality visual feature sequences for subsequent anomaly detection.

[0053] Thus, the video disorder sequence is obtained.

[0054] S4. Calculate the bidirectional information transfer between audio and video in the audiovisual data based on the audio energy sequence and the video disorder sequence, and determine whether the audiovisual data is unidirectional salient data based on the bidirectional information transfer. If so, proceed to step S5.

[0055] In subway platform conditions, disturbances in the public environment, such as trains entering the station, will simultaneously cause an increase in video disorder and a sharp rise in audio energy. Such common-cause fluctuations and real abnormal causal events can co-occur in time. Therefore, it is necessary to distinguish the direction and salience of information transmission between video disorder sequences and audio energy sequences.

[0056] When an abnormal event begins Then, this step is based on Expand forward and backward from the center Duration, the total duration is taken as The analysis window. For example, Get experience points 's' is a preset parameter, which can also be adjusted by the implementer according to the scenario. The total duration of the analysis window can be 2.0 seconds, ranging from 1.0 to 5.0 seconds. Using 2.0 seconds can cover short-term causal changes before and after the occurrence of the anomalous event and provide sufficient samples for probability estimation of bidirectional information transmission. A longer total analysis window duration can cover anomalous events with longer durations and increase the statistical sample, but background changes unrelated to the anomalous event are also more likely to enter the analysis window. A shorter total analysis window duration results in more concentrated time positioning, but the stability of bidirectional information transmission and standardized significance scores decreases when the sample size is insufficient.

[0057] From the video disorder sequence Extract all values ​​within the window to obtain From audio energy sequences Extract all values ​​within the same classmate to obtain The timestamps of the two sequences have been aligned uniformly within the system.

[0058] because and Since they are all continuous value sequences, directly using them for probability estimation would face the problem of an excessively large joint state space and susceptibility to amplitude noise interference. Therefore, they are discretized into symbol sequences.

[0059] The marginal cumulative distribution function upon which symbolization is based and It is obtained from long-term offline statistical monitoring data and has been pre-installed in the system.

[0060] In specific implementation, according to Will The disorder values ​​are divided into equal-frequency categories. There are several state intervals; each interval is sequentially assigned a discrete symbol, thereby... Convert to video symbol sequence The total number of state intervals satisfies the following relationship: ; in, Indicates the total number of state intervals; This represents the video disorder sequence within the analysis window. The maximum value, This represents the video disorder sequence within the analysis window. The minimum value is obtained by iterating through the data within the window; express The standard deviation represents the dispersion of the current scene disorder value within the window, and is obtained by calculating the standard deviation of the data within the window; This indicates the actual number of samples within the analysis window; This represents the floor function; This is the theoretical constant (Scotts rule) based on the optimal bin width according to the normal distribution.

[0061] Number of state intervals The video clutter level within the analysis window is extremely poor. It increases with the increase of standard deviation. The value decreases as the disorder increases; when the disorder value distribution within the scene is wide but locally clustered (large range, relatively small standard deviation), Automatically increments to retain more state details; when scene chaos values ​​fluctuate drastically and are extremely discrete (standard deviation) When (large), Automatic reduction and merging of adjacent states prevents probability estimates from becoming too sparse. This process is entirely driven by the actual statistical patterns of the scene data, requiring no human experience to set.

[0062] Similarly, according to right Perform the same operation to obtain the audio symbol sequence. ,in It is the temporal index of the sampling points within the window, and the two symbol sequences are of equal length and have the same value range.

[0063] Based on symbol sequences, a first-order Markov approximation is used to calculate the amount of video-to-audio information transferred. Its definition is: ; in, This represents the amount of video-to-audio information transferred, measured in bits. Indicates the audio symbol sequence at the index The sign value at the location; Indicates the audio symbol sequence at the index The sign value at the location; Indicates the video symbol sequence in the index The sign value at the location; This represents a joint probability estimate based on the frequency of occurrence of all triples within the window; and This represents the corresponding conditional probability.

[0064] Video to audio information transfer volume The larger the value, the greater the contribution of historical video symbols to predicting the next audio symbol, given the current audio symbol. When the value is significantly greater than 0, it indicates that video disorder precedes and drives audio anomalies in time, and there is a causal relationship between visual and auditory perception in the scene.

[0065] This transitivity measures the current symbol of the known audio. Under these conditions, video history symbols The reduction in uncertainty of predicting the audio symbol at the next moment, with a logarithmic base of . The unit is bits.

[0066] Using the first-order Markov approximation means considering only the current symbol state and ignoring longer histories, in order to reduce the amount of data required for probability estimation.

[0067] Similarly, in the formula and By reversing the roles, the amount of audio-to-video information transmitted can be obtained. .

[0068] The original transmission rate is affected by the statistical structure of the sequence itself, and its significance needs to be evaluated through statistical tests.

[0069] for ,Keep Unchanged, for Time index The number of random scrambles. For example, the number of random scrambles. Take the industry standard number of statistical tests (permutation tests). The number of random shuffling iterations is a preset parameter, which can be determined by the implementer according to their own needs. The number of random shuffling iterations can be 1000, with a range of 500 to 5000. Using 1000 iterations strikes a balance between the stability of the null hypothesis distribution estimation and computational latency. A higher number of random shuffling iterations results in higher statistical stability of the standardized significance score, but also increases computational load and warning latency. A lower number of random shuffling iterations results in faster computation, but also greater fluctuations in the mean and standard deviation of the null hypothesis distribution.

[0070] Each scrambling generates a sequence of proxy video symbols, and its relationship with the original is calculated. Between A total of Each scrambled propagation quantity constitutes the video-to-audio null hypothesis distribution.

[0071] Calculate the mean of the video-to-audio null hypothesis distribution. and standard deviation The standardized significance scores in the video-to-audio direction satisfy the following relationship: ; in, The standardized significance score represents the video-to-audio direction. This indicates the amount of original video-to-audio information transferred. express The mean of the null hypothesis distribution of the video to audio generated by the next random scrambling; It represents the standard deviation of the null hypothesis distribution from the video to the audio.

[0072] Standardized significance score The larger the value, the greater the deviation of the original transport quantity from a random distribution, and the more significant the causal relationship; when... When the threshold is exceeded, it indicates that the causal relationship between video-driven audio anomalies is statistically highly reliable and not a coincidence.

[0073] Similarly, for the audio-to-video direction, the video symbol sequence is preserved. Unchanged, for audio symbol sequences Time index Each random scrambling operation generates a proxy audio symbol sequence, and the amount of audio-to-video scrambling information transferred between this proxy audio symbol sequence and the original video symbol sequence is calculated, thus forming the null hypothesis distribution in the audio-to-video direction; the mean of the null hypothesis distribution is then calculated. and standard deviation Then the standardized significance scores in the audio-to-video direction satisfy the following relationship: ; in, Represents the standardized significance score in the audio-to-video direction, dimensionless; Represents a sequence of audio symbols; Represents a sequence of video symbols; This indicates that the direction of information transmission is from the audio symbol sequence to the video symbol sequence; This represents the amount of raw audio-to-video information transferred, measured in bits. Indicates the amount of information transmitted, subscript Indicates the audio-to-video direction; Indicates the path from audio to video. The mean of the null hypothesis distribution generated by the random scrambling, in bits, where This represents the mean. This represents the null hypothesis distribution generated by random scrambling; Indicates the path from audio to video. The standard deviation of the null hypothesis distribution generated by the random scrambling is given in bits. Indicates standard deviation; Indicates the number of times the object is randomly shuffled.

[0074] The causal direction is determined based on the obtained standardized significance score.

[0075] Set a significance threshold For example, the significance threshold is taken as a statistical standard parameter. Dimensionless, corresponding to a one-sided significance level of approximately The parameters are preset. The significance threshold can be 1.96, with a range of 1.64 to 2.58. Using 1.96 allows for stricter statistical exclusion of random information transmission; a higher significance threshold makes the criteria for determining unidirectional significant data more stringent, reducing the false alarm rate but increasing the risk of missed detections; a lower significance threshold increases the sensitivity of detecting abnormal events, but increases the risk of random fluctuations being misjudged as unidirectional significant data.

[0076] when and When the video is deemed to be malfunctioning and the audio is not driving properly, a direction indicator is set. ;when and When an audio anomaly is detected, causing video corruption, settings are adjusted accordingly. .

[0077] If both directions satisfy or do not satisfy the above conditions, it indicates that there is no significant unidirectional causality, and the current audiovisual data is not bound.

[0078] S5, bind the audiovisual data into candidate composite events and calculate the causal confidence of the candidate composite events.

[0079] When the audiovisual data is determined to be unidirectionally significant, the video segment in the current analysis window is bound to the abnormal event as a candidate composite event, and the standardized significance score in that significant direction is used as the causal confidence score. The causal confidence score satisfies the following relationship: ; in, Indicates the confidence level of causality, dimensionless; The standardized significance score represents the video-to-audio direction. This represents the standardized significance score in the audio-to-video direction.

[0080] Causal confidence The larger the value, the stronger the validation of the candidate composite event on the time causal sequence and the higher its statistical reliability; when... When the value increases, it indicates that the unidirectional causal driving relationship between the audiovisual modalities is more explicit, and the greater the likelihood that the event is a real anomaly.

[0081] The candidate composite events and their corresponding causal confidence scores are the output of this step.

[0082] By introducing transfer entropy and random scrambling statistical tests, the "common cause fluctuations" of audiovisual modalities in the subway platform environment (such as the simultaneous shaking of the screen and increase in volume caused by the train entering the station) were successfully removed, and the true one-way causal driving relationship was accurately identified, which greatly improved the logical rigor and anti-interference ability of candidate composite event binding.

[0083] Thus, we obtain the candidate composite events and their corresponding causal confidence scores.

[0084] S6, record the video frame corresponding to the start time of the abnormal event as the event frame, identify the azimuth angle of the visual confusion center of the event frame and calculate its azimuth difference with the azimuth angle of the sound source.

[0085] Since candidate composite events are only bound by statistical significance through information transmission, and there may be erroneous associations between video chaos areas and sound sources in physical space due to inconsistent directions, and it is impossible to accurately locate individuals under the high-density crowd conditions of subway platforms, the projection direction of the centroid of the high chaos grid is compared with the azimuth angle of the sound source, and spatial matching is used to determine the spatial consistency of events.

[0086] At the start time of the abnormal event In the corresponding video frame, extract the local disorder of each grid in that frame. Calculate the arithmetic mean of the grid disorder across the entire frame. And set an adaptive disorder threshold. ,in, This represents the adaptive disorder threshold, which is dimensionless. This represents the arithmetic mean of the overall grid clutter level. Only grids with significantly higher local clutter levels than the background are retained; these grids constitute visual clutter hotspots. The adaptive clutter threshold, relative to the arithmetic mean of the overall grid clutter level, can be a factor of 1.5, ranging from 1.2 to 2.0. A factor of 1.5 can filter out grids close to the background level while retaining local clutter hotspots; a larger factor results in fewer retained grids and less interference from background motion at the visual clutter center, but scattered clutter areas may be missed; a smaller factor retains more potentially cluttered grids, but the visual clutter center is more susceptible to the influence of normal pedestrian movement.

[0087] The coordinates of the center pixel of each of the highly disordered grids are denoted as The total number of highly chaotic grids is The centroid pixel coordinates are obtained by calculating their arithmetic mean: ; in, and Indicates the centroid pixel coordinates, in pixels; Indicates the total number of highly disordered grid cells; and Indicates the first The center pixel coordinates of the selected grid.

[0088] Centroid pixel coordinates The spatial center represents the focal point of visually chaotic hotspots; when When a shift occurs, it indicates that the spatial location of visually cluttered hotspots in the scene has changed.

[0089] Camera intrinsic parameter matrix calibrated offline and distortion coefficient Distortion correction is performed to obtain the corrected coordinates. Then through Will Projecting the image coordinates into normalized image coordinates, constructing the direction vector in the camera coordinate system. Based on the camera's extrinsic parameters in the platform's global coordinate system, i.e., the rotation matrix Translation vector ,Will Rotate the image to the global coordinate system and extract the horizontal azimuth angle to obtain the single-frame visual azimuth angle of the current frame. The unit is degrees.

[0090] Because the center of gravity of a highly chaotic grid in a single frame can deviate significantly due to the instantaneous movement of the crowd, it is necessary to extend the grid before and after the event frame. Frame by frame, repeat the above mesh selection, centroid calculation and projection process to obtain A sequence of visual azimuth angles from individual frames is used, and the median of this sequence is taken as the azimuth angle at the center of visual confusion. The unit is degrees. Among them, the number of extended frames... The following relationship must be satisfied: ; in, The number of extended frames before and after the event frame. The standard deviation of the azimuth angle difference between adjacent frames. This is the mean.

[0091] When the instantaneous movement of a crowd causes extremely drastic jitter in the azimuth sequence (variance) When (large), Automatically increases the size to enhance the smoothing filter effect; when the image is relatively stable. Automatically reduces noise while preserving more realistic motion details. It's based entirely on the sequence's own signal-to-noise ratio characteristics, requiring no fixed frame rate.

[0092] Read the azimuth angle of the sound source from the output of step S2. This angle is related to Within the same global coordinate system, in degrees; the azimuth difference is calculated according to the following relationship: ; in, This indicates the difference in azimuth, expressed in degrees. This indicates the azimuth angle of the sound source, in degrees. Indicates the azimuth angle of the center of visual confusion, in degrees.

[0093] Azimuth difference The smaller the value, the closer the visual confusion center matches the sound source in spatial direction; when When the value approaches 0, it indicates that the audiovisual anomalies highly overlap in physical space, greatly increasing the credibility of the event as a real anomaly.

[0094] S7. Determine whether the azimuth difference is less than the spatial matching tolerance; if so, proceed to step S8.

[0095] Set space matching tolerance This tolerance covers random errors in sound source localization and visual hotspot bias. Specifically, the spatial matching tolerance satisfies the following relationship: ,in, To match the space tolerance, The variance of the theoretical error in sound source localization is calculated based on the signal-to-noise ratio of the current audio frame. For visual calibration and projection error variance, For statistics The principle constant. Using 3 enables the spatial matching tolerance to cover most normal deviations caused by sound source localization errors and visual projection errors; the larger the statistical principle constant, the wider the spatial matching tolerance, reducing the risk of missed detections but increasing the risk of spatial misassociation; the smaller the statistical principle constant, the stricter the spatial direction matching, reducing the risk of spatial misassociation but potentially missing real abnormal events with large localization errors.

[0096] The signal-to-noise ratio (SNR) of the current audio frame is determined according to the following steps: subtract the background energy floor estimate from the total energy of the current audio frame to obtain the abnormal sound energy; when both the abnormal sound energy and the background energy floor estimate are greater than 0, divide the abnormal sound energy by the background energy floor estimate, take the logarithm to base 10 of the resulting ratio, and multiply by 10 to obtain the SNR expressed in decibels; when the abnormal sound energy is not greater than 0, determine the SNR of the current audio frame as the lowest SNR used in the pre-test; when the background energy floor estimate is not greater than 0, recalculate the background energy floor estimate for audio frames from which no abnormal event was detected, until the background energy floor estimate is greater than 0.

[0097] The variance of the sound source localization error is obtained through pre-testing. At least two calibrated signal-to-noise ratio (SNR) values ​​are pre-selected, including the lower and upper limits of the usable SNR range of the microphone array. Within the effective localization angle range of the microphone array, at least two known true azimuth angles are selected for each calibrated SNR value, and at least two generalized cross-correlation-phase transformations and spatial spectrum searches are performed at each true azimuth angle. For each test, the estimated and true sound source azimuth angles are first converted to a range of 0 to 360 degrees. Then, the estimated azimuth angle is subtracted from the true azimuth angle to obtain the initial error. If the initial error is greater than or equal to 180 degrees, 360 degrees are subtracted; if the initial error is less than -180 degrees, 360 degrees are added. Otherwise, the values ​​remain unchanged to obtain the sound source localization error. For the same calibrated signal-to-noise ratio (SNR), first calculate the average of all sound source localization errors. Then, calculate the square of the difference between each sound source localization error and this average. Sum all the squared values ​​and divide by the number of sound source localization errors for that calibrated SNR value minus one to obtain the variance of the sound source localization error corresponding to that calibrated SNR value. A table mapping calibrated SNR values ​​to sound source localization error variances is saved. During actual operation, the absolute value of the difference between the current audio frame's SNR and each calibrated SNR value in the table is calculated. The sound source localization error variance corresponding to the calibrated SNR value with the smallest absolute value is selected. If two identical smallest absolute values ​​exist, the sound source localization error variance corresponding to the smaller calibrated SNR value is selected.

[0098] The visual calibration and projection error variance are obtained through pre-testing of the camera. At least two calibration targets with known true global azimuth angles are set within the global azimuth angle range corresponding to the camera's effective field of view. These calibration targets include the lower and upper limits of the global azimuth angle range, and at least two sets of images are repeatedly acquired for each target. The visual azimuth angle is obtained by a process involving distortion correction using the same centroid pixel coordinates as the visual distortion center, inverse projection of camera intrinsic parameters, transformation of camera extrinsic parameters, and median value of the visual azimuth angles from multiple frames. For each test, the visual azimuth angle and the true global azimuth angle of the calibration target are first converted to the range of 0 to 360 degrees. Then, the initial error is obtained by subtracting the true global azimuth angle from the visual azimuth angle. If the initial error is greater than or equal to 180 degrees, 360 degrees are subtracted; if the initial error is less than -180 degrees, 360 degrees are added. Otherwise, the values ​​remain unchanged, resulting in the visual azimuth angle error. First, calculate the average of all visual azimuth errors. Then, calculate the square of the difference between each visual azimuth error and the average. Sum all the squared values ​​and divide by the number of visual azimuth errors minus one to obtain the visual calibration and projection error variance. If any of the camera position, orientation, focal length, intrinsic parameters, or extrinsic parameters changes, repeat the test and update the visual calibration and projection error variance.

[0099] Both sound source localization error and visual azimuth error are angular errors within the same global coordinate system. If we consider them independent, the variance of the difference between the sound source azimuth and the visual azimuth is equal to the sum of the two variances. When the platform environment is extremely noisy (low signal-to-noise ratio), the sound source localization error... As the size increases, the system automatically adjusts the space to match the tolerance. To prevent missed alarms, when the environment is quiet and the positioning is accurate, the system automatically tightens the spatial matching tolerance to prevent false alarms. This achieves adaptive spatial matching tolerance as the environment changes.

[0100] S8, generate the spatial confidence of the candidate composite event.

[0101] when At that time, spatial orientation matching is determined, and spatial confidence is generated. The following relationship must be satisfied: ; in, Represents spatial confidence level, dimensionless, with a range of values. ; This indicates the difference in azimuth, expressed in degrees. Indicates the space matching tolerance.

[0102] Spatial confidence The larger the value, the greater the azimuth difference. The smaller the size, the higher the consistency between the visually chaotic hotspots and the sound sources in terms of physical location; when When it is close to 1, it indicates a perfect match in spatial orientation; when hour, This indicates that the spatial orientation is inconsistent, ruling out the possibility of erroneous association.

[0103] By extracting the centroid of highly disordered grids and performing multi-frame median filtering projection, the centroid jitter deviation caused by the instantaneous movement of crowds in a single frame is effectively overcome. Combined with the spatial matching tolerance mechanism, erroneous association events that co-occur in time but are separated in space are successfully eliminated, significantly enhancing the verification reliability of candidate composite events in the physical space dimension.

[0104] Thus, the spatial confidence of the candidate composite events is obtained.

[0105] S9, the causal confidence and the spatial confidence are weighted and fused to obtain the comprehensive confidence of the candidate composite event, and an alarm signal is issued when the comprehensive confidence exceeds the warning threshold.

[0106] In obtaining the causal confidence of candidate composite events and spatial confidence Since the former only reflects the statistical significance of information transmission between the visual and auditory modalities, and the latter only characterizes the consistency between visual disturbance hotspots and sound sources in physical location, a single-dimensional confidence score is insufficient to provide sufficient evidence for the final warning. Therefore, the two confidence scores are weighted and fused to obtain a comprehensive confidence score that takes into account both the strength of causal relationship and the degree of spatial direction consistency. The calculation method is as follows: ; , ; in, Indicates the overall confidence level, dimensionless; Indicates the causal fusion weighting coefficient; Indicates the confidence level of causality; Indicates the spatial fusion weighting coefficient; Indicates spatial confidence; Indicates the causal confidence level of historical or current batches. The variance; Indicates the spatial confidence level of the historical or current batch. The variance.

[0107] Overall confidence level The higher the value, the stronger the degree to which the candidate composite event is verified both in terms of temporal causality and physical space; when When the value increases and exceeds the warning threshold, it indicates that the candidate composite event has a very high degree of authenticity, and the system should immediately trigger an alarm.

[0108] In this embodiment, the azimuth angle of the visual disorder center is extracted based on the centroid projection of the highly disordered grid in the event frame. It is then compared with the azimuth angle of the sound source. When the azimuth difference is less than the spatial matching tolerance, a spatial confidence score is generated.

[0109] In the process of setting up Higher than The reliability of determining the direction of causal information transmission usually contributes more to the identification of the abnormal event than the result of spatial matching.

[0110] Dimensionless, its range of values ​​is theoretically between and , Between the maximum values, the higher the value, the stronger the degree to which the candidate composite event is verified in both the temporal causal sequence and the physical space.

[0111] System setting warning threshold Take experience points The value is dimensionless and is a preset parameter that implementers can adjust based on their tolerance for false alarms and missed alarms. The warning threshold can be 0.7, with a range of 0.5 to 0.9. Using 0.7 strikes a balance between alarm sensitivity and false alarm suppression for candidate composite events; a higher warning threshold means only candidate composite events with high overall confidence trigger alarms, reducing the false alarm rate but increasing the risk of missed detections; a lower warning threshold increases alarm sensitivity but increases the risk of low-confidence candidate composite events triggering alarms.

[0112] when When the overall confidence level of the current candidate composite event is deemed to meet the alarm conditions, an alarm signal is immediately generated, and relevant information about the event is reported, including: the directional markers determined before this step. The obtained composite event types, the analysis time window range corresponding to the candidate composite events, and the calculated comprehensive confidence level. Directional signs The event type is labeled as "video chaos driving audio anomaly". The time stamp is marked as "audio-driven video disorder".

[0113] like If the event fails, no alarm will be triggered, and the system will discard the current candidate composite event binding relationship and will not perform any further processing.

[0114] This application also provides an automatic dye feeding system for dyeing vats, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the computer program is executed by the processor, it implements the steps of an automatic dye feeding method for dyeing vats according to any of the above embodiments.

[0115] The computer program used to perform the operations of this application may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this application, electronic circuits, including, for example, programmable logic circuits, Field-Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), may execute computer-readable program instructions using status information from computer-readable program instructions to personalize the electronic circuits.

[0116] Therefore, those skilled in the art should recognize that although many exemplary embodiments of this application have been shown and described in detail herein, many other variations or modifications conforming to the principles of this application can be directly determined or derived from the disclosure of this application without departing from the spirit and scope of this application. Thus, the scope of this application should be understood and construed as covering all such other variations or modifications.

Claims

1. A security IoT intelligent early warning method based on data fusion, characterized in that, include: Acquire the audio-visual data at the current moment and extract the array audio from it, and determine whether an abnormal event has occurred based on the array audio; If so, then obtain the azimuth angle of the sound source of the abnormal event, as well as the audio energy sequence and multiple video frames within a set time period before and after the current moment; Each video frame is divided into multiple grids, and the local disorder degree of each grid is calculated based on the grid motion vector. The video disorder degree sequence of each video frame is obtained based on the local disorder degree. The bidirectional information transfer between audio and video in the audiovisual data is calculated based on the audio energy sequence and the video disorder sequence, and the audiovisual data is determined to be unidirectional salient data based on the bidirectional information transfer. If so, the audiovisual data is bound as a candidate composite event and the causal confidence of the candidate composite event is calculated; The video frame corresponding to the start time of the abnormal event is recorded as the event frame. The azimuth angle of the visual confusion center of the event frame is identified and the azimuth difference between it and the azimuth angle of the sound source is calculated. The spatial confidence of the candidate composite event is generated when the azimuth difference is less than the spatial matching tolerance. The causal confidence score and the spatial confidence score are weighted and fused to obtain the comprehensive confidence score of the candidate composite event, and an alarm signal is issued when the comprehensive confidence score exceeds the warning threshold.

2. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of determining whether an abnormal event has occurred based on the array audio includes: The array audio signal is processed by frame segmentation to obtain the short-time energy sequence of each channel and the comprehensive spectral centroid sequence formed by the spectral centroids of each channel. The abnormal event is determined based on the energy background estimate corresponding to the short-time energy sequence and the comprehensive spectral centroid sequence.

3. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of obtaining the azimuth angle of the sound source of the abnormal event includes: Using one channel of the array audio as a reference channel, the measured arrival delay of the remaining channels relative to the reference channel is obtained, and the azimuth angle of the sound source of the abnormal event is determined based on the measured arrival delay and the geometric layout of the microphone array.

4. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The calculation of the local disorder degree based on the mesh motion vector of each mesh includes: The local orientation variance factor and local kinetic entropy factor are obtained based on the mesh motion vector, and the texture weight is calculated based on the image gradient magnitude of the mesh. The local disorder of the mesh is obtained by adaptively weighting and fusing the local orientation variance factor and the local kinetic entropy factor according to the texture weight.

5. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of obtaining the video disorder sequence for each video frame based on the local disorder includes: The local disorder of all grids within each video frame is statistically analyzed to obtain the video disorder of each video frame. Multiple video disorder values ​​are then arranged in chronological order according to the video frames to form the video disorder sequence.

6. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of calculating the bidirectional information transfer between audio and video in the audiovisual data based on the audio energy sequence and the video disorder sequence includes: Extract the audio energy sequence and the video disorder sequence from the analysis window corresponding to the abnormal event, and convert the audio energy sequence and the video disorder sequence into audio symbol sequences and video symbol sequences, respectively; The video-to-audio information transmission amount and the audio-to-video information transmission amount are calculated based on the audio symbol sequence and the video symbol sequence, respectively, and the video-to-audio information transmission amount and the audio-to-video information transmission amount are used as the bidirectional information transmission amount.

7. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of determining whether the audiovisual data is one-way salient data based on the bidirectional information transmission volume includes: Randomized statistical tests were performed on the information transfer amounts in the video-to-audio direction and the audio-to-video direction to obtain the standardized significance scores in the video-to-audio direction and the audio-to-video direction. When the standardized significance score in only one direction satisfies the significance condition, the audiovisual data is determined to be unidirectional significant data.

8. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of binding the audiovisual data into candidate composite events and calculating the causal confidence of the candidate composite events includes: When the audiovisual data is determined to be unidirectional significant data, the corresponding video segment is bound to the abnormal event as the candidate composite event, and the standardized significance score corresponding to the information transmission direction that satisfies the significance condition is used as the causal confidence score.

9. The intelligent early warning method for security IoT based on data fusion according to claim 1, characterized in that, The step of identifying the visual disturbance center azimuth angle of the event frame and calculating its azimuth difference with the sound source azimuth angle includes: In the event frame, select high-chaos meshes with local chaos levels higher than the background level, obtain the centroid of the high-chaos mesh based on the center pixel coordinates of the high-chaos mesh, and project the centroid of the high-chaos mesh onto the global coordinate system to obtain the azimuth angle of the visual chaos center. The azimuth difference is calculated based on the azimuth angle of the visual confusion center and the azimuth angle of the sound source. If the azimuth difference is less than the spatial matching tolerance, the spatial confidence score is generated. The causal confidence score and the spatial confidence score are then weighted and fused to obtain the comprehensive confidence score.

10. A security IoT intelligent early warning system based on data fusion, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the security IoT intelligent early warning method based on data fusion according to any one of claims 1-9.

Images