Anomaly event detection method and electronic device

Abstract

This invention provides an abnormal event detection method and electronic device, relating to the field of vehicle technology. The method includes: acquiring a video sequence, an audio sequence, and a condition vector, the condition vector including vehicle state information and vehicle operating environment information; generating a reconstructed video frame and a reconstructed audio spectrum at the current moment under normal driving conditions based on the video sequence, audio sequence, and condition vector; determining a reconstruction error based on the reconstructed video frame, reconstructed audio spectrum, and the actual video frame and actual audio spectrum at the current moment; and determining whether an abnormal event exists at the current moment based on the reconstruction error. This method can effectively detect various novel abnormal events in vehicle driving scenarios without relying on abnormal event samples, thereby improving detection reliability.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to an abnormal event detection method and electronic device. Background Technology

[0002] In related technologies, anomaly detection in vehicle driving scenarios typically employs supervised learning, relying on a large number of real anomaly event samples to train the recognition model. For example, detecting vehicle theft requires a large number of real theft event samples. However, anomaly event samples are scarce, and the ways in which anomalies occur are constantly changing, making it easy to miss anomalies not seen during the training phase, thus reducing detection reliability. Summary of the Invention

[0003] The purpose of this invention is to provide an abnormal event detection method and electronic device, which can solve the technical problems of high dependence on abnormal event samples and poor detection reliability in vehicle driving scenarios.

[0004] To address the aforementioned technical problems, this invention provides an abnormal event detection method, comprising: Acquire video sequences, audio sequences, and condition vectors, wherein the condition vectors include vehicle status information and vehicle operating environment information; Based on the video sequence, the audio sequence, and the conditional vector, generate the reconstructed video frame and the reconstructed audio spectrum under normal driving conditions at the current moment; The reconstruction error is determined based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and actual audio spectrum at the current moment; The existence of an abnormal event at the current moment is determined based on the reconstruction error.

[0005] Optionally, generating the reconstructed video frame and reconstructed audio spectrum at the current moment under normal driving conditions based on the video sequence, the audio sequence, and the conditional vector includes: The video sequence, the audio sequence, and the conditional vector are input into a trained audio-visual reconstruction model for processing to obtain the reconstructed video frame and the reconstructed audio spectrum.

[0006] Optionally, the step of inputting the video sequence, the audio sequence, and the conditional vector into a trained audio-visual reconstruction model for processing to obtain the reconstructed video frame and the reconstructed audio spectrum includes: The video sequence, the audio sequence, and the conditional vector are respectively embedded to obtain video embedding features, audio embedding features, and conditional embedding features; Cross-attention processing is performed on the video embedding features, the audio embedding features, and the conditional embedding features to obtain fused features; Based on the fusion features, latent distribution parameters are determined, and latent variables are obtained by sampling based on the latent distribution parameters; The reconstructed video frame and the reconstructed audio spectrum are obtained by decoding based on the latent variables and the conditional vector.

[0007] Optionally, the training process of the audio-visual reconstruction model includes: Acquire training data, which includes video sequences, audio sequences, and conditional vectors under normal driving conditions; The video sequence, audio sequence, and conditional vector in the training data are input into the initial audio-visual reconstruction model for processing to obtain the training reconstructed video frames, training reconstructed audio spectra, and potential distribution parameters. The reconstruction loss is determined based on the differences between the reconstructed video frames and the corresponding original video frames, and between the reconstructed audio spectra and the corresponding original audio spectra. Determine the KL divergence loss based on the potential distribution parameters; The trained and reconstructed video frames are input into a trained video semantic encoder to obtain video semantic features, and the trained and reconstructed audio spectra are input into a trained audio semantic encoder to obtain audio semantic features. The semantic consistency loss is determined based on the similarity between the video semantic features and the audio semantic features. The total loss is determined based on the reconstruction loss, the KL divergence loss, and the semantic consistency loss, and the parameters of the initial audio-visual reconstruction model are adjusted based on the total loss to obtain the trained audio-visual reconstruction model.

[0008] Optionally, determining the reconstruction error based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and actual audio spectrum at the current moment includes: Determine the image difference value between the reconstructed video frame and the actual video frame; Determine the audio difference value between the reconstructed audio spectrum and the actual audio spectrum; Determine the semantic consistency difference value between the reconstructed video frame and the reconstructed audio spectrum; The reconstruction error is determined based on the image difference value, the audio difference value, and the semantic consistency difference value.

[0009] Optionally, determining the image difference value between the reconstructed video frame and the actual video frame includes: Determine the pixel difference value and structural similarity difference value between each pixel in the reconstructed video frame and the corresponding pixel in the actual video frame; The reconstructed video frame and the actual video frame are respectively input into a pre-trained deep network for feature extraction to obtain the features of the reconstructed video frame and the features of the actual video frame. The difference between the features of the reconstructed video frame and the features of the actual video frame is determined to obtain the feature difference value. The image difference value is determined based on the pixel difference value, the structural similarity difference value, and the feature difference value.

[0010] Optionally, determining whether an abnormal event exists at the current moment based on the reconstruction error includes: Determine whether the reconstruction error is greater than a preset threshold; If the reconstruction error is less than or equal to the preset threshold, then it is determined that there is no abnormal event at the current moment; If the reconstruction error is greater than the preset threshold, it is determined that there is an abnormal event at the current moment, and an abnormal heatmap is generated based on the pixel difference value and the structural similarity difference value, so as to determine the regional location of the abnormal event in the actual video frame based on the abnormal heatmap.

[0011] Optionally, generating the abnormal heatmap based on the pixel difference value and the structural similarity difference value includes: The pixel difference value and structural similarity difference value of each pixel are weighted and fused to determine the difference map at the current time. Obtain the difference maps of multiple historical moments before the current moment, set the weight of each difference map according to the difference between each historical moment and the current moment, and perform weighted fusion of each difference map according to the weight to obtain the time-domain smoothed difference map. The time-domain smoothed difference map is filtered to obtain a spatially filtered difference map. The spatially filtered difference map is normalized to obtain the abnormal heatmap.

[0012] Optionally, it also includes: Obtain the most recent preset number of historical reconstruction errors, and determine the mean and standard deviation of the historical reconstruction errors; wherein, the historical reconstruction error is the reconstruction error when it is determined that there are no abnormal events; The preset threshold is determined based on the mean, the standard deviation, and the preset sensitivity coefficient.

[0013] The present invention also provides an electronic device, comprising: Memory, used to store computer programs; A processor for implementing the above-described steps of abnormal event detection when executing the computer program.

[0014] This invention provides an abnormal event detection method, comprising: acquiring a video sequence, an audio sequence, and a condition vector, wherein the condition vector includes vehicle status information and vehicle operating environment information; generating a reconstructed video frame and a reconstructed audio spectrum at the current moment under normal driving conditions based on the video sequence, the audio sequence, and the condition vector; determining a reconstruction error based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and the actual audio spectrum at the current moment; and determining whether an abnormal event exists at the current moment based on the reconstruction error.

[0015] The beneficial effects of this invention are as follows: By acquiring video sequences, audio sequences, and conditional vectors containing vehicle status information and vehicle operating environment information, this invention generates reconstructed video frames and reconstructed audio spectra under normal driving conditions at the current moment. It then calculates the difference between the reconstructed video frames, reconstructed audio spectra, and the actual video frames and actual audio spectra at the current moment to obtain the reconstruction error. The larger the reconstruction error, the more the vehicle's operating state deviates from normal driving conditions at the current moment, thus effectively identifying anomalies. In this way, this invention can effectively detect various abnormal events without relying on theft samples, and can effectively cover various novel abnormal events in open scenarios.

[0016] The present invention also provides an electronic device that has the above-mentioned beneficial effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of an abnormal event detection method provided in an embodiment of the present invention; Figure 2 A flowchart of another abnormal event detection method provided in an embodiment of the present invention; Figure 3 A training flowchart for an audio / video reconstruction model provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of an abnormal event detection device provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In related technologies, anomaly detection in vehicle driving scenarios typically employs supervised learning, relying on a large number of real anomaly event samples to train the recognition model. For example, detecting vehicle theft requires a large number of real theft event samples. However, anomaly event samples are scarce, and the ways in which anomalies occur are constantly changing, making it easy to miss anomalies not seen during the training phase, thus reducing detection reliability.

[0021] In view of this, to address the technical problem of high dependence on abnormal event samples and poor detection reliability in abnormal event detection during vehicle driving scenarios, this invention provides an abnormal event detection method. This method transforms the identification of abnormal events into predicting video frames and audio spectra that might be generated during normal vehicle driving based on currently collected video and audio sequences and vehicle operation information. The predicted video and audio frames and spectra are then compared with the actually collected video and audio frames and spectra. If there is a significant difference between the predicted and actual video and audio frames and spectra, it indicates a large deviation between the vehicle's actual driving state and the normal driving state, thus detecting an abnormal event. In this embodiment, during model training, it is only necessary to ensure that the model can predict video frames and audio spectra under normal driving conditions, avoiding dependence on abnormal event samples and improving detection reliability.

[0022] It should be noted that this embodiment does not limit the executing entity of this method. For example, the executing entity of this method can be an in-vehicle device. If the user allows its data to be uploaded to the server for testing, the executing entity of this method can also be the server.

[0023] The abnormal event detection method provided in this embodiment is described below. Please refer to... Figure 1 , Figure 1 A flowchart of an abnormal event detection method provided in an embodiment of the present invention, the method may include: S10. Obtain the video sequence, audio sequence, and condition vector. The condition vector includes vehicle status information and vehicle operating environment information.

[0024] A video sequence refers to a continuous set of video frames captured by an in-vehicle camera that are related to the vehicle's safety status. A video sequence can contain single-channel or multi-channel video. For example, a video sequence can include videos of the external road environment, the area surrounding the vehicle, and related areas. Videos of the external road environment can include videos of lanes, intersections, parking areas, surrounding vehicles, pedestrians, and non-motorized vehicles. Videos of the area surrounding the vehicle can include videos recording actions such as approaching, colliding with, prying, or damaging the vehicle, such as doors, windows, trunk, or people near the vehicle. Furthermore, a video sequence can also include in-vehicle view videos or surround-view videos. A video sequence can use continuous image frames within a preset time window. The sequence length can be a fixed value or dynamically adjusted, such as based on the camera frame rate, model computing power, and real-time requirements. For example, the video sequence length can be 16-32 consecutive frames; for example, under 10fps sampling conditions, this corresponds to a video time window of approximately 1.6-3.2 seconds.

[0025] An audio sequence refers to a collection of continuous audio segments related to vehicle operation and safety events, collected by an in-vehicle audio pickup device. An audio sequence can contain single-channel or multi-channel audio. For example, an audio sequence can include vehicle operating sounds, abnormal sounds related to the vehicle itself, and ambient sounds. Vehicle operating sounds include, for example, engine sounds, acceleration / deceleration sounds, braking sounds, tire friction sounds, horn sounds, and windshield wiper sounds. Abnormal sounds related to the vehicle itself include, for example, the sound of doors opening, abnormal lock sounds, glass breaking sounds, metallic knocking sounds, prying sounds, and collision sounds. Ambient sounds include, for example, people approaching, arguments / shouting, unusual gatherings, and road noise. An audio sequence is a continuous audio segment within a preset time window. The sequence length can be a fixed value or dynamically adjusted, for example, based on the sampling rate, model computing power, and real-time requirements. For example, the audio sequence length can be 1-3 seconds.

[0026] A condition vector is an information vector used to characterize the current driving conditions of a vehicle. The condition vector can include vehicle state information and vehicle operating environment information. Vehicle state information may include, for example, vehicle speed, acceleration, steering, braking, windshield wipers, doors, locks, windows, and lights. Vehicle operating environment information may include, for example, time, weather, lighting conditions, road type, and whether the vehicle is parked or in motion.

[0027] In this step, by simultaneously acquiring video sequences, audio sequences, and conditional vectors, comprehensive multimodal information can be provided for subsequent reconstruction, thereby improving the accuracy of anomaly detection.

[0028] S20. Based on the video sequence, audio sequence, and conditional vector, generate the reconstructed video frame and reconstructed audio spectrum at the current moment under normal driving conditions.

[0029] Reconstructed video frames and reconstructed audio spectra refer to the predicted video frames and audio spectra representing the normal driving state at the current moment, obtained from the aforementioned video sequence, audio sequence, and conditional vectors. The video sequence, audio sequence, and conditional vectors can be input into a trained audio-visual reconstruction model for processing to obtain the reconstructed video frames and reconstructed audio spectra. The audio-visual reconstruction model is a model specifically designed for audio-visual reconstruction and can be trained based on normal driving data.

[0030] It should be noted that this embodiment does not limit the type of audio and video reconstruction model. For example, it can be a Conditional Variational Autoencoder (CVAE) model, or other conditional generative reconstruction networks. For CVAE models, during the training phase, they can learn the conditional distribution of normal driving scenes in the latent space. During the inference phase, the current conditions are input, and predicted images and sounds that better conform to the normal distribution are generated and used as reconstruction references. By learning the deep semantic distribution of normal driving scenes through conditional generative networks, the model has a stronger reconstruction capability for normal data, while significantly increasing the reconstruction error for inputs containing structural anomalies or cross-modal semantic anomalies.

[0031] S30. Determine the reconstruction error based on the reconstructed video frames, the reconstructed audio spectrum, and the actual video frames and audio spectrum at the current moment.

[0032] Actual video frames and actual audio spectra refer to the video frames and audio spectra actually captured by the vehicle-mounted camera and vehicle-mounted audio pickup equipment at the current moment.

[0033] Reconstruction error can first include the difference between the reconstructed result and the actual acquired result, which can reflect the degree to which the vehicle's current operating state deviates from the normal driving state. For example, if the reconstructed video frame differs significantly from the actual video frame, or the reconstructed audio spectrum differs significantly from the actual audio spectrum, it indicates that the vehicle has encountered an abnormal event that significantly deviates from the normal driving state, thus enabling effective detection of abnormal events.

[0034] Furthermore, reconstruction errors can also include semantic differences between the reconstructed video frames and the reconstructed audio spectrum. Under normal circumstances, the reconstructed video frames should maintain semantic consistency with the reconstructed audio spectrum. For example, if the reconstructed video frame shows a vehicle parked in a parking space, empty inside, and the external environment is quiet, the reconstructed audio spectrum should correspond to extremely low ambient noise or no obvious vehicle noise; that is, the static state in the video and the quiet characteristics in the audio should be semantically consistent. However, if engine noise or unusual knocking sounds are present in the audio, it constitutes a semantic inconsistency, indicating the presence of an abnormal event.

[0035] As can be seen, by determining the difference between the reconstruction result and the actual acquisition result, or by simultaneously determining the difference between the reconstruction result and the actual acquisition result and the cross-modal semantic matching degree between the reconstruction results, this embodiment can more comprehensively capture abnormal features and reduce the risk of missed detection and false alarm caused by single-modal noise or cross-modal semantic mismatch.

[0036] S40. Determine whether there are any abnormal events at the current moment based on the reconstruction error.

[0037] Anomalies refer to traffic theft or other unknown abnormal behaviors that deviate from normal driving conditions. As mentioned above, when the reconstruction error is large, it can be determined that an anomaly exists at the current moment. Therefore, anomaly detection based on reconstruction error can effectively detect various types of anomalies, especially covering new types of anomalies in open scenarios.

[0038] Based on the above embodiments, this invention generates reconstructed video frames and reconstructed audio spectra under normal driving conditions at the current moment by acquiring video sequences, audio sequences, and conditional vectors containing vehicle status information and vehicle operating environment information. The reconstruction error is obtained by calculating the difference between the reconstructed video frames, reconstructed audio spectra, and the actual video frames and actual audio spectra at the current moment. A larger reconstruction error indicates that the vehicle's operating state deviates more from normal driving conditions at the current moment, thus effectively identifying anomalies. In this way, this invention can effectively detect various abnormal events without relying on theft samples, and can effectively cover various novel abnormal events in open scenarios.

[0039] Based on the above embodiments, the method for determining reconstruction error is described below. In one embodiment, determining the reconstruction error based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and the actual audio spectrum at the current moment may include: S31. Determine the image difference value between the reconstructed video frame and the actual video frame.

[0040] In this embodiment, the image difference value characterizes the difference in image content between the reconstructed video frame and the actual video frame. The image difference value can consist of multiple parts, such as pixel difference values, structural similarity difference values, and feature difference values. Pixel difference values ​​are determined based on the difference between corresponding pixel values ​​in the two images, while structural difference values ​​are determined based on the structural similarity index of the two images. Specifically, to further compare the high-dimensional differences between the reconstructed video frame and the actual video frame, such as differences in contours, textures, target shapes, scene layouts, and semantics, this embodiment can also extract image features from both the reconstructed and actual video frames separately and compare the differences between them to obtain feature difference values. By determining the differences between the two types of video frames from multiple perspectives, abnormal changes in the video frames can be more comprehensively reflected, avoiding missed detections and ensuring detection effectiveness.

[0041] The following describes the method for determining image difference values. In one implementation, determining the image difference value between the reconstructed video frame and the actual video frame may include: S311. Determine the pixel difference value and structural similarity difference value between each pixel in the reconstructed video frame and the corresponding pixel in the actual video frame.

[0042] The pixel difference value is determined based on the difference between the pixel values ​​of each pixel in the reconstructed video frame and the corresponding pixel in the actual video frame.

[0043] The structural similarity difference value can be determined based on the Structural Similarity Index Measure (SSIM).

[0044] It is worth noting that pixel value differences reflect local grayscale changes, are sensitive to illumination changes, and ignore structural information; while SSIM reflects changes in structural information and is less affected by illumination changes. Therefore, by simultaneously introducing pixel value differences and structural similarity differences, the reliability of image difference detection can be improved.

[0045] S312. Input the reconstructed video frame and the actual video frame into the pre-trained deep network for feature extraction to obtain the features of the reconstructed video frame and the features of the actual video frame. Determine the difference between the features of the reconstructed video frame and the features of the actual video frame to obtain the feature difference value.

[0046] A pre-trained deep network refers to a neural network used to extract features from video frames, and its network structure can be configured as needed. This embodiment uses a pre-trained deep network to extract features from reconstructed video frames and actual video frames, obtaining features for both the reconstructed and actual video frames, and determining the differences between them to obtain feature difference values. These feature difference values ​​reflect differences between the two image frames in high-level information such as contours, textures, target shapes, scene layouts, and action semantics, thus effectively supplementing the image difference values ​​and further improving the reliability of image difference detection.

[0047] S313. Determine the image difference value based on the pixel difference value, structural similarity difference value, and feature difference value.

[0048] Image difference values ​​can be determined by combining pixel difference values, structural similarity difference values, and feature difference values, for example, by weighted fusion of these values. By integrating pixel, structural, and high-level semantic differences, image difference values ​​focus more on structural changes and improve sensitivity to edge and texture anomalies.

[0049] S32. Determine the audio difference value between the reconstructed audio spectrum and the actual audio spectrum.

[0050] The audio difference value characterizes the difference in audio content between the reconstructed audio spectrum and the actual audio spectrum, and is used to measure the degree of deviation between the reconstructed audio spectrum and the actual audio spectrum at the spectral level. This embodiment does not limit the method of determining the audio difference value, and can be set according to actual application requirements.

[0051] For example, the energy difference between the reconstructed audio spectrum and the actual audio spectrum in each frequency band can be determined to obtain the audio difference value. Alternatively, the reconstructed audio spectrum and the actual audio spectrum can be input into a pre-trained deep network for feature extraction, yielding reconstructed audio spectrum features and actual audio spectrum features. The difference between these features can then be determined to obtain the audio difference value. This audio difference value can reflect higher-level differences such as acoustic event type, rhythmic features, frequency band energy structure, and semantic voiceprint.

[0052] Of course, multiple audio differences can also be fused to obtain the final audio difference value.

[0053] S33. Determine the semantic consistency difference value between the reconstructed video frame and the reconstructed audio spectrum.

[0054] In this embodiment, the semantic consistency difference value represents the semantic difference between the reconstructed video frame and the reconstructed audio spectrum. By introducing the semantic consistency difference value, this embodiment can detect the semantic shift between the image dimension and the audio dimension when an abnormal event occurs, such as a still image but high-speed wind noise in the sound, thereby further improving the detection effect of abnormal events.

[0055] Specifically, the semantic consistency difference value can be determined as follows: 1. Input the reconstructed video frames into the trained video semantic encoder to obtain video semantic features; 2. Input the reconstructed audio spectrum into the trained audio semantic encoder to obtain audio semantic features; 3. Determine the semantic consistency difference value based on the similarity between video semantic features and audio semantic features.

[0056] In this embodiment, both the video semantic encoder and the audio semantic encoder are pre-trained models for cross-modal semantic alignment. The video semantic encoder maps video frames to a semantic feature space, and the audio semantic encoder maps audio spectra to a semantic feature space. By mapping the reconstructed video frames and the reconstructed audio spectra to the same semantic feature space, the semantic differences between the two can be detected.

[0057] Specifically, the semantic consistency difference value can be represented as: ; in, For video semantic encoders, For audio semantic encoders, To reconstruct video frames To reconstruct the audio spectrum, The cosine similarity function has the following range of values: .

[0058] S34. Determine the reconstruction error based on the image difference value, audio difference value, and semantic consistency difference value.

[0059] In this embodiment, the reconstruction error can be determined based on image difference values, audio difference values, and semantic consistency difference values, for example, by weighted fusion of these three factors. By combining multimodal and multidimensional differences, this embodiment can effectively improve the accuracy of anomaly detection and reduce the risk of false positives and false negatives in anomaly detection.

[0060] Based on the above embodiments, the detection method for abnormal events is described below. In one implementation, determining whether an abnormal event exists at the current moment based on the reconstruction error may include: S41. Determine whether the reconstruction error is greater than the preset threshold.

[0061] In this embodiment, if the reconstruction error is greater than a preset threshold, it can be determined that the current moment is abnormal and an abnormal event exists; implying that if the reconstruction error is less than or equal to the preset threshold, it can be determined that the current moment is normal and no abnormal event exists.

[0062] It should be noted that the preset threshold can be a fixed threshold or a dynamic threshold. The reason for setting a dynamic threshold is that the baseline of reconstruction error differs under different road conditions, and a fixed threshold is not easy to operate stably. Therefore, this embodiment can dynamically determine the preset threshold according to the actual normal driving state of the vehicle.

[0063] The dynamic setting method for the preset threshold is described below. In one embodiment, this method may further include: S441. Obtain the most recent preset number of historical reconstruction errors, and determine the mean and standard deviation of the historical reconstruction errors; wherein, the historical reconstruction error is the reconstruction error when it is determined that there are no abnormal events.

[0064] In this step, a sliding window can be set to collect a preset number of historical reconstruction errors. These historical reconstruction errors are those where no abnormal events are determined to exist. In other words, only reconstruction errors determined to be normal are included in the sliding window, while those determined to be abnormal are excluded, to prevent the dynamic threshold from being contaminated by outliers. Subsequently, for the sliding window, this embodiment can determine the mean and standard deviation of the historical reconstruction errors.

[0065] It should be noted that if the reconstruction error at the current moment is determined to be abnormal, the reconstruction error at the current moment will not be added to the sliding window, and therefore the corresponding mean and standard deviation will not be updated.

[0066] S442. Determine the preset threshold based on the mean, standard deviation and preset sensitivity coefficient.

[0067] In this embodiment, the preset threshold can be determined by the following formula: ; in, This is the current dynamic alarm threshold. This represents the mean reconstruction error within the sliding window at the previous time step. The standard deviation of the reconstruction error within the sliding window at the previous time step. This is the preset sensitivity coefficient, and its value can be set arbitrarily, for example, to 3.

[0068] By setting dynamic thresholds, this embodiment can ensure that anomaly detection can adjust the anomaly judgment criteria according to the actual situation, making the thresholds in different scenarios more in line with the actual situation and ensuring the anomaly detection effect.

[0069] S42. If the reconstruction error is less than or equal to the preset threshold, then it is determined that there is no abnormal event at the current moment.

[0070] When the reconstruction error is less than or equal to the preset threshold, it indicates that the vehicle's current operating state is not significantly different from the normal driving state and there are no abnormal events.

[0071] S43. If the reconstruction error is greater than the preset threshold, it is determined that there is an abnormal event at the current moment, and an abnormal heat map is generated based on the pixel difference value and the structural similarity difference value, so as to determine the regional location of the abnormal event in the actual video frame based on the abnormal heat map.

[0072] When the reconstruction error exceeds a preset threshold, it indicates a significant deviation between the vehicle's current operating state and normal driving status, thus indicating an abnormal event. To facilitate user identification of the abnormal event's location, this embodiment can also construct an anomaly heatmap based on video frames. The anomaly heatmap is a heat map used to identify the location and intensity of abnormal areas within a video frame. The highlighted areas of the anomaly heatmap correspond to the actual abnormal locations in the video frame, allowing operators to visually see the location of the abnormal area in the image, facilitating rapid location and verification of suspicious targets.

[0073] The following describes the method for generating abnormal heatmaps. In one implementation, an abnormal heatmap is generated based on pixel difference values ​​and structural similarity difference values, including: S431. Perform weighted fusion of pixel difference values ​​and structural similarity difference values ​​for each pixel to determine the difference map at the current time.

[0074] A difference map is a two-dimensional matrix recording the degree of anomaly at each pixel location. It is obtained by weighted fusion of pixel differences and structural similarity differences between the reconstructed video frame and the actual video frame at the current moment. A difference map can be represented as: ; in, This represents the actual video frame at the current time t. , representing the reconstructed video frame at the current time t. This represents the difference value at the current position (x, y). The preset fusion weights can be set arbitrarily.

[0075] By fusing pixel and structural information, the difference map focuses more on changes in structural information such as edges and textures.

[0076] S432. Obtain the difference maps of multiple historical moments before the current moment, set the weight of each difference map according to the difference between each historical moment and the current moment, and perform weighted fusion of each difference map according to the weight to obtain the time-domain smoothed difference map.

[0077] In this embodiment, considering that abnormal events occur dynamically, it is difficult to effectively label abnormal events based solely on the difference map at the current moment. Therefore, this embodiment can also obtain the difference maps of multiple historical moments prior to the current moment, and combine the difference map at the current moment and the difference maps of the historical moments to form a time series, thereby generating the final difference map. The time series can be represented as: ; Where t represents the current time, This indicates the length of the time window used for temporal smoothing, such as 5 frames. This time window length can be adjusted according to the video frame rate, real-time requirements, and noise level.

[0078] After obtaining the time series, this embodiment can set the weights of each difference map based on the difference between each historical moment and the current moment, and then perform weighted fusion of each difference map according to the weights to obtain a time-smoothed difference map. The weight allocation method is: the closer the frame is to the current moment, the greater the weight is assigned, that is, the weight of the current frame is higher than the weight of the historical frames. Time-domain smoothing can also be achieved using an equal-weighted moving average method. The time-smoothed difference map can be represented as: ; in, This represents the difference plot after time-domain smoothing. Time weighting.

[0079] Temporal smoothing can suppress single-frame error spikes caused by sensor jitter, compression noise, and transient illumination, reduce sporadic false alarms, and enhance the stable response of persistent abnormal regions in adjacent frames.

[0080] S433. Filter the time-domain smoothed difference map to obtain the spatially filtered difference map.

[0081] Filtering can be performed using a spatial filter kernel for neighborhood filtering. The spatial filter kernel can be any of Gaussian filtering, median filtering, or mean filtering. Gaussian filtering is used to smooth local noise while maintaining the overall continuity of abnormal regions. Median filtering is used to remove discrete noise or salt-and-pepper interference. Mean filtering is used to average local differences.

[0082] The filter kernel size can be 3×3 or 5×5, which can be set according to the image resolution and the size of the anomalous target. Spatial filtering can remove spatially isolated scattered noise, improve the connectivity between adjacent anomalous pixels, and make the boundaries of anomalous regions in the final heatmap smoother and more stable.

[0083] The difference map after spatial filtering can be represented as: ; in, This represents the difference map after spatial filtering, where K represents the spatial filtering kernel. This indicates a convolution or neighborhood filtering operation.

[0084] S434. Normalize the spatially filtered difference map to obtain an anomaly heatmap.

[0085] Normalization maps the spatially filtered difference map to a uniform numerical range, such as 0 to 1. This processing yields the final anomaly heatmap, where highlighted areas correspond to suspected anomaly locations. The anomaly heatmap can be overlaid with actual video frames to visually identify the location, extent, and intensity of anomaly areas. Higher response values ​​in the heatmap indicate a greater deviation from the normal scene distribution, and are more likely to correspond to theft, vandalism, abnormal proximity, or other unknown abnormal behavior.

[0086] Of course, after the normalization operation is completed, the following processing can be performed: 1. Threshold the heatmap to retain high-response areas above the preset threshold; 2. Perform morphological opening and closing operations on the thresholding results to remove small-area pseudo-anomaly areas and fill local holes; 3. Filter the connected regions by area to remove noise areas with areas below the set threshold.

[0087] To better understand the online anomaly detection and heatmap generation process, please refer to [link / reference]. Figure 2 , Figure 2This is a flowchart illustrating another abnormal event detection method provided in an embodiment of the present invention. In the diagram, the real-time input is used to acquire a video sequence, an audio sequence, and a conditional vector. The video sequence is a set of continuous image frames captured by an in-vehicle camera, the audio sequence is a continuous audio segment captured by an in-vehicle audio pickup device, and the conditional vector is an auxiliary information vector containing vehicle status information and vehicle operating environment information. The trained conditional variational autoencoder model, i.e., the audio-video reconstruction model, is used to generate reconstructed video frames and reconstructed audio spectra under normal driving conditions at the current moment based on the video sequence, audio sequence, and conditional vector. The reconstructed video frames are predicted video frames output by the model trained based on normal driving data, and the reconstructed audio spectra are predicted audio spectra output by the model trained based on normal driving data. The difference calculation module is used to determine the reconstruction error based on the reconstructed video frames, reconstructed audio spectra, and the actual video frames and actual audio spectra at the current moment. The reconstruction error is a measure of the difference between the reconstruction result and the actual acquisition result.

[0088] In the difference analysis, pixel difference refers to the pixel difference value between the reconstructed video frame and the actual video frame, which is the absolute value of the difference between the pixel values ​​at corresponding positions. Structural difference refers to the structural similarity difference value, calculated based on the structural similarity index. Feature perception difference refers to the feature difference value, which is the difference between the features extracted by the pre-trained deep network from the reconstructed video frame and the features of the actual video frame. The weighted aggregation module fuses the pixel difference value, structural similarity difference value, and feature difference value into an image difference value, and further aggregates them into a total anomaly score, i.e., the reconstruction error. The anomaly heatmap module generates an anomaly heatmap based on the pixel difference value and the structural similarity difference value; the anomaly heatmap is a two-dimensional heatmap that identifies the location and intensity of abnormal regions in the video frame.

[0089] In dynamic judgment, the historical statistics module is used to obtain the most recent preset number of historical reconstruction errors and determine their mean and standard deviation. The historical reconstruction error is the reconstruction error when no abnormal event is determined. The dynamic threshold module is used to determine a preset threshold based on the mean, standard deviation, and preset sensitivity coefficient. The preset threshold is used to determine anomalies. The total anomaly score is compared with the dynamic threshold. If the total anomaly score is greater than the preset threshold, an alarm and heatmap localization are triggered to determine that an abnormal event exists at the current time and to determine the location of the abnormal event in the actual video frame based on the anomaly heatmap.

[0090] Based on the above embodiments, the generation process of reconstructed video frames and reconstructed audio spectra is described below. In one implementation, the video sequence, audio sequence, and conditional vector are input into a trained audio-visual reconstruction model for processing to obtain reconstructed video frames and reconstructed audio spectra, including: S21. Embedding processes are performed on the video sequence, audio sequence, and conditional vector respectively to obtain video embedding features, audio embedding features, and conditional embedding features.

[0091] In this embodiment, to capture long temporal dependencies, a Transformer can be used as the backbone network, and the encoder and decoder can be implemented separately. The encoder executes steps S21-S23, and the decoder executes step S24.

[0092] In this step, embedding refers to the linear projection transformation process that maps the original data to a high-dimensional feature space. Video embedding features refer to the high-dimensional feature representation obtained after linear projection embedding of a video sequence. Audio embedding features refer to the high-dimensional feature representation obtained after linear projection embedding of an audio sequence. Conditional embedding features refer to the high-dimensional feature representation obtained after embedding of conditional vectors.

[0093] The above features can be generated as follows: flatten each frame of the video sequence and map it to the target dimension through a linear projection matrix to obtain video embedding features; map the spectral features of the audio sequence to the target dimension through a linear projection matrix to obtain audio embedding features; and map the conditional vector to the target dimension through an embedding layer to obtain conditional embedding features.

[0094] Video embedding features can be represented as: ; in, Represents video embedding features, Represents a video sequence. This represents a linear function.

[0095] Audio embedding features can be represented as: ; in, Represents audio embedding features. This represents an audio sequence.

[0096] Conditional embedding features can be represented as: ; in, Represents a condition vector. This indicates the embedding layer.

[0097] S22. Perform cross-attention processing on video embedding features, audio embedding features, and conditional embedding features to obtain fused features.

[0098] In this step, cross-attention processing refers to the multi-head self-attention mechanism in the Transformer module used to achieve cross-modal feature fusion. By performing cross-attention processing on video embedding features, audio embedding features, and conditional embedding features, fused features are obtained. This embodiment can achieve multimodal fusion of video, audio, and conditional information, ensuring that the model can better capture long-term temporal dependencies and cross-modal associations.

[0099] The fusion feature can be represented as: ; Where H represents the fusion feature, () indicates the Transformer module.

[0100] S23. Determine the latent distribution parameters based on the fusion characteristics, and obtain latent variables by sampling based on the latent distribution parameters.

[0101] In this step, the latent distributed parameters refer to the parameters output by the encoder. ,in, The mean is represented, which defines the central location of the latent space (a multidimensional Gaussian distribution); The standard deviation represents the uncertainty or dispersion of the latent space in each dimension, and together they define the multidimensional Gaussian distribution in the latent space. The latent distribution parameters can be determined as follows: ; in, This refers to a multilayer perceptron. This indicates global pooling.

[0102] Latent variables capture essential feature representations of normal driving scenarios, enabling the model to learn the normal scenario distribution within a compact latent space. The latent variables can be represented as latent representation vectors, denoted as z, sampled from the latent distribution through reparameterization. The latent variables can be expressed as: ; in, This indicates a normal distribution.

[0103] S24. Decode the video frame and the audio spectrum based on the latent variables and condition vectors.

[0104] The decoding process can be summarized as follows: A query vector is generated based on latent variables and conditional vectors; the query vector is then input into the Transformer decoder for processing, outputting reconstructed video frames and reconstructed audio spectra. Specifically, this can be represented as: ; ; in, Let Q represent the condition, and let Q represent the query vector. These represent the reconstructed video frames and the reconstructed audio spectra, respectively. This indicates the Transformer decoder.

[0105] Based on the above embodiments, the training process of the audio-visual reconstruction model is described below. In one implementation, the training process of the audio-visual reconstruction model may include: S51. Obtain training data, which includes video sequences, audio sequences, and conditional vectors under normal driving conditions.

[0106] In this embodiment, the training data includes video sequences, audio sequences, and conditional vectors under normal driving conditions, without including anomalous samples. By training with normal driving data, the model becomes better at reconstructing normal driving scenarios, thus transforming theft identification into a problem of measuring differences in normal scenario reconstruction, and supporting open-set anomaly detection.

[0107] S52. Input the video sequence, audio sequence and conditional vector in the training data into the initial audio and video reconstruction model for processing to obtain the training reconstructed video frames, training reconstructed audio spectra and potential distribution parameters.

[0108] The description of step S52 is similar to that of steps S21 to S24, and will not be repeated here.

[0109] S53. Determine the reconstruction loss based on the differences between the reconstructed video frames and the corresponding original video frames, and between the reconstructed audio spectra and the corresponding original audio spectra.

[0110] In this embodiment, reconstruction loss refers to the differences in image content and audio content between the training reconstructed video frame and the original video frame, and between the training reconstructed audio spectrum and the original audio spectrum. The specific determination method is similar to the determination method of the image difference value and audio difference value mentioned above, and can be referred to the above embodiment, which will not be repeated here.

[0111] S54. Determine the KL divergence loss based on the potential distribution parameters.

[0112] The Kullback-Leibler (KL) divergence loss is used to constrain the latent spatial distribution to approximate a standard normal prior. This loss is based on the latent distribution parameters. Calculating the KL divergence between the model and the standard normal prior can regularize the latent spatial distribution, thereby improving the diversity of generated samples and the model's generalization ability.

[0113] S55. Determine the semantic consistency loss based on the semantic differences between the reconstructed video frames and the reconstructed audio spectra.

[0114] In this embodiment, to avoid semantic separation between video and audio reconstruction, a semantic consistency loss can be introduced and the reconstructed audiovisual semantic alignment can be constrained, thereby ensuring that the reconstructed video frames and reconstructed audio frames in the inference stage are semantically comparable.

[0115] The following describes how semantic consistency loss is determined. Based on the semantic differences between the reconstructed video frames and the reconstructed audio spectra, semantic consistency loss is determined, and this can include: S551. Input the reconstructed video frames into the trained video semantic encoder to obtain video semantic features.

[0116] A video semantic encoder is a pre-trained model based on a multimodal alignment framework used to map video frames to a semantic feature space. This encoder keeps its parameters fixed during training, i.e., freezes the parameter settings, to prevent the semantic space from drifting during training and maintain its stability.

[0117] S552. Input the reconstructed audio spectrum into the trained audio semantic encoder to obtain audio semantic features.

[0118] An audio semantic encoder is a pre-trained model based on a multimodal alignment framework used to map audio spectra to a semantic feature space. This encoder also employs frozen parameter settings, keeping the parameters fixed and not updated to maintain the stability of the semantic space.

[0119] S553. Determine the semantic consistency loss based on the similarity between video semantic features and audio semantic features.

[0120] The semantic consistency loss can be expressed as: ; in, For video semantic encoders, For audio semantic encoders, To reconstruct video frames To reconstruct the audio spectrum, The cosine similarity function has the following range of values: .

[0121] This loss makes the reconstructed video and audio more aligned in semantic space, and makes the on-screen actions and sound events more closely matched.

[0122] S56. Determine the total loss based on the reconstruction loss, KL divergence loss, and semantic consistency loss, and adjust the parameters of the initial audio and video reconstruction model based on the total loss to obtain the trained audio and video reconstruction model.

[0123] The total loss function can be expressed as: ; in: Reconstruction loss is used to measure the difference between the input and the reconstructed output.

[0124] The KL divergence loss is used to constrain the potential spatial distribution.

[0125] This represents cross-modal consistency loss.

[0126] The weight for the KL divergence loss (dimensionless, typical value: 0.1, range: ) ).

[0127] The consistency loss weight (dimensionless, typical value: 0.5, range: ) ).

[0128] By using multi-objective joint optimization, the model can be ensured to simultaneously meet the requirements of reconstruction accuracy, distribution constraints and semantic alignment, thereby improving the reliability of the model in reconstructing normal scenes.

[0129] To better understand the model training process, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the training process of an audio / video reconstruction model provided in an embodiment of the present invention.

[0130] The beneficial effects of this invention are as follows: 1. The system can provide more sensitive difference responses to abnormal scenarios even when trained using only normal driving data, thus making it easier to cover unknown theft methods that do not appear in the training set, and is suitable for open set recognition scenarios.

[0131] 2. By constraining cross-modal semantic consistency, the reconstruction results of video and audio are more likely to maintain semantic matching. The system is less likely to be misled when sporadic noise exists in a single modality, thereby reducing the risk of false alarms.

[0132] 3. Based on the structural differences and the output format of the anomaly heatmap, operators can more intuitively see the location of abnormal areas in the image, which facilitates quick location of suspicious targets and verification.

[0133] 4. The dynamic threshold mechanism enables the system to adjust the judgment scale according to changes in road conditions and noise baseline, making the alarm threshold more controllable and improving stability in different scenarios.

[0134] The following describes the abnormal event detection device, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present invention. The abnormal event detection device, electronic device, computer-readable storage medium, and computer program product described below can be referred to in correspondence with the abnormal event detection method described above.

[0135] Please refer to Figure 4 , Figure 4 This is a structural block diagram of an abnormal event detection device provided in an embodiment of the present invention. The device may include: The acquisition module 401 is used to acquire video sequences, audio sequences, and condition vectors. The condition vectors include vehicle status information and vehicle operating environment information. The reconstruction module 402 is used to generate a reconstructed video frame and a reconstructed audio spectrum at the current moment under normal driving conditions, based on the video sequence, audio sequence and condition vector. The error determination module 403 is used to determine the reconstruction error based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and the actual audio spectrum at the current moment. The anomaly detection module 404 is used to determine whether there is an abnormal event at the current moment based on the reconstruction error.

[0136] Optionally, the reconstruction module 402 can be used for: The video sequence, audio sequence, and conditional vector are input into the trained audio-visual reconstruction model for processing to obtain reconstructed video frames and reconstructed audio spectra.

[0137] Optionally, the reconstruction module 402 may include: The embedding submodule is used to perform embedding processing on video sequences, audio sequences, and conditional vectors respectively to obtain video embedding features, audio embedding features, and conditional embedding features; The fusion submodule is used to perform cross-attention processing on video embedding features, audio embedding features, and conditional embedding features to obtain fused features. The parameter sampling submodule is used to determine the latent distribution parameters based on the fusion features, and to sample the latent variables based on the latent distribution parameters. The decoding submodule is used to perform decoding processing based on latent variables and conditional vectors to obtain reconstructed video frames and reconstructed audio spectra.

[0138] Optionally, it may also include: The training data acquisition module is used to acquire training data, which includes video sequences, audio sequences, and conditional vectors under normal driving conditions. The forward propagation submodule is used to input the video sequence, audio sequence and conditional vector in the training data into the initial audio-video reconstruction model for processing, and to obtain the training reconstructed video frames, training reconstructed audio spectra and potential distribution parameters. The reconstruction loss determination submodule is used to determine the reconstruction loss based on the differences between the training reconstructed video frames and the corresponding original video frames, as well as the training reconstructed audio spectra and the corresponding original audio spectra. The KL divergence loss determination submodule is used to determine the KL divergence loss based on the potential distribution parameters; The semantic consistency loss determination submodule is used to input the trained reconstructed video frames into the trained video semantic encoder to obtain video semantic features, input the trained reconstructed audio spectrum into the trained audio semantic encoder to obtain audio semantic features, and determine the semantic consistency loss based on the similarity between the video semantic features and the audio semantic features. The parameter tuning submodule is used to determine the total loss based on the reconstruction loss, KL divergence loss, and semantic consistency loss, and to tune the parameters of the initial audio and video reconstruction model based on the total loss to obtain the trained audio and video reconstruction model.

[0139] Optionally, the error determination module 403 may include: The image difference value determination submodule is used to determine the image difference value between the reconstructed video frame and the actual video frame; The audio difference value determination submodule is used to determine the audio difference value between the reconstructed audio spectrum and the actual audio spectrum; The semantic consistency difference value determination submodule is used to determine the semantic consistency difference value between the reconstructed video frame and the reconstructed audio spectrum; The error fusion submodule is used to determine the reconstruction error based on image difference values, audio difference values, and semantic consistency difference values.

[0140] Optionally, the image difference value determination submodule may include: The first difference value determination unit is used to determine the pixel difference value and structural similarity difference value between each pixel in the reconstructed video frame and the corresponding pixel in the actual video frame; The second difference value determination unit is used to input the reconstructed video frame and the actual video frame into the pre-trained deep network for feature extraction, obtain the features of the reconstructed video frame and the features of the actual video frame, and determine the difference between the features of the reconstructed video frame and the features of the actual video frame to obtain the feature difference value. The image difference fusion unit is used to determine the image difference value based on the pixel difference value, structural similarity difference value, and feature difference value.

[0141] Optionally, the anomaly detection module 404 may include: The threshold judgment submodule is used to determine whether the reconstruction error is greater than a preset threshold. The normal determination submodule is used to determine that there are no abnormal events at the current time if the reconstruction error is less than or equal to a preset threshold. The anomaly determination submodule is used to determine that an abnormal event exists at the current moment if the reconstruction error is greater than a preset threshold, and to generate an anomaly heatmap based on the pixel difference value and the structural similarity difference value, so as to determine the regional location of the abnormal event in the actual video frame based on the anomaly heatmap.

[0142] Optionally, the exception determination submodule may include: The difference map determination unit is used to perform weighted fusion of pixel difference values ​​and structural similarity difference values ​​for each pixel to determine the difference map at the current time. The temporal smoothing unit is used to obtain the difference map of multiple historical moments before the current moment. The weight of each difference map is set according to the difference between each historical moment and the current moment, and the difference maps are weighted and fused according to the weight to obtain the temporally smoothed difference map. The spatial filtering unit is used to filter the time-domain smoothed difference map to obtain a spatially filtered difference map. The normalization unit is used to normalize the spatially filtered difference map to obtain an anomaly heatmap.

[0143] Optionally, it also includes: The threshold update module is used to obtain the most recent preset number of historical reconstruction errors and determine the mean and standard deviation of the historical reconstruction errors; wherein, the historical reconstruction error is the reconstruction error when it is determined that there are no abnormal events; the preset threshold is determined based on the mean, standard deviation and preset sensitivity coefficient.

[0144] Please refer to Figure 5 , Figure 5 This invention provides a structural block diagram of an electronic device 10, which includes a processor 11 and a memory 12. The memory 12 is used to store a computer program, and the processor 11 is used to execute the abnormal event detection method provided in the foregoing embodiment when executing the computer program.

[0145] The specific process of the above-mentioned abnormal event detection method can be found in the corresponding content provided in the foregoing embodiments, and will not be repeated here.

[0146] Furthermore, the memory 12, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0147] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15, and a communication bus 16; wherein, the power supply 13 is used to provide operating voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface 15 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0148] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the abnormal event detection method described in the above embodiments.

[0149] Since the embodiments of the computer program product section correspond to the embodiments of the abnormal event detection method section, please refer to the description of the embodiments of the abnormal event detection method section for the embodiments of the computer program product section, and will not be repeated here.

[0150] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the abnormal event detection method described in the above embodiments.

[0151] Since the embodiments of the computer-readable storage medium portion correspond to the embodiments of the abnormal event detection method portion, the embodiments of the storage medium portion are described in the description of the embodiments of the abnormal event detection method portion, and will not be repeated here.

[0152] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0154] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0155] The above provides a detailed description of the abnormal event detection method and electronic device provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. An abnormal event detection method characterized by, include: Acquire video sequences, audio sequences, and condition vectors, wherein the condition vectors include vehicle status information and vehicle operating environment information; Based on the video sequence, the audio sequence, and the conditional vector, generate the reconstructed video frame and the reconstructed audio spectrum under normal driving conditions at the current moment; The reconstruction error is determined based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and actual audio spectrum at the current moment; The existence of an abnormal event at the current moment is determined based on the reconstruction error.

2. The abnormal event detection method according to claim 1, characterized in that, The step of generating the reconstructed video frame and reconstructed audio spectrum under normal driving conditions at the current moment based on the video sequence, the audio sequence, and the conditional vector includes: The video sequence, the audio sequence, and the conditional vector are input into a trained audio-visual reconstruction model for processing to obtain the reconstructed video frame and the reconstructed audio spectrum.

3. The abnormal event detection method according to claim 2, characterized in that, The step of inputting the video sequence, the audio sequence, and the conditional vector into a trained audio-visual reconstruction model for processing to obtain the reconstructed video frames and the reconstructed audio spectrum includes: The video sequence, the audio sequence, and the conditional vector are respectively embedded to obtain video embedding features, audio embedding features, and conditional embedding features; Cross-attention processing is performed on the video embedding features, the audio embedding features, and the conditional embedding features to obtain fused features; Based on the fusion features, latent distribution parameters are determined, and latent variables are obtained by sampling based on the latent distribution parameters; The reconstructed video frame and the reconstructed audio spectrum are obtained by decoding based on the latent variables and the conditional vector.

4. The abnormal event detection method according to claim 2, characterized in that, The training process of the audio and video reconstruction model includes: Acquire training data, which includes video sequences, audio sequences, and conditional vectors under normal driving conditions; The video sequence, audio sequence, and conditional vector in the training data are input into the initial audio-visual reconstruction model for processing to obtain the training reconstructed video frames, training reconstructed audio spectra, and potential distribution parameters. The reconstruction loss is determined based on the differences between the reconstructed video frames and the corresponding original video frames, and between the reconstructed audio spectra and the corresponding original audio spectra. Determine the KL divergence loss based on the potential distribution parameters; The trained and reconstructed video frames are input into a trained video semantic encoder to obtain video semantic features, and the trained and reconstructed audio spectra are input into a trained audio semantic encoder to obtain audio semantic features. The semantic consistency loss is determined based on the similarity between the video semantic features and the audio semantic features. The total loss is determined based on the reconstruction loss, the KL divergence loss, and the semantic consistency loss, and the parameters of the initial audio-visual reconstruction model are adjusted based on the total loss to obtain the trained audio-visual reconstruction model.

5. The abnormal event detection method according to any one of claims 1 to 4, characterized in that, The step of determining the reconstruction error based on the reconstructed video frame, the reconstructed audio spectrum, and the actual video frame and actual audio spectrum at the current moment includes: Determine the image difference value between the reconstructed video frame and the actual video frame; Determine the audio difference value between the reconstructed audio spectrum and the actual audio spectrum; Determine the semantic consistency difference value between the reconstructed video frame and the reconstructed audio spectrum; The reconstruction error is determined based on the image difference value, the audio difference value, and the semantic consistency difference value.

6. The abnormal event detection method according to claim 5, characterized in that, Determining the image difference value between the reconstructed video frame and the actual video frame includes: Determine the pixel difference value and structural similarity difference value between each pixel in the reconstructed video frame and the corresponding pixel in the actual video frame; The reconstructed video frame and the actual video frame are respectively input into a pre-trained deep network for feature extraction to obtain the features of the reconstructed video frame and the features of the actual video frame. The difference between the features of the reconstructed video frame and the features of the actual video frame is determined to obtain the feature difference value. The image difference value is determined based on the pixel difference value, the structural similarity difference value, and the feature difference value.

7. The abnormal event detection method according to claim 6, characterized in that, The step of determining whether there is an abnormal event at the current moment based on the reconstruction error includes: Determine whether the reconstruction error is greater than a preset threshold; If the reconstruction error is less than or equal to the preset threshold, then it is determined that there is no abnormal event at the current moment; If the reconstruction error is greater than the preset threshold, it is determined that there is an abnormal event at the current moment, and an abnormal heatmap is generated based on the pixel difference value and the structural similarity difference value, so as to determine the regional location of the abnormal event in the actual video frame based on the abnormal heatmap.

8. The abnormal event detection method according to claim 7, characterized in that, The step of generating an abnormal heatmap based on the pixel difference value and the structural similarity difference value includes: The pixel difference value and structural similarity difference value of each pixel are weighted and fused to determine the difference map at the current time. Obtain the difference maps of multiple historical moments before the current moment, set the weight of each difference map according to the difference between each historical moment and the current moment, and perform weighted fusion of each difference map according to the weight to obtain the time-domain smoothed difference map. The time-domain smoothed difference map is filtered to obtain a spatially filtered difference map. The spatially filtered difference map is normalized to obtain the abnormal heatmap.

9. The abnormal event detection method according to claim 7, characterized in that, Also includes: Obtain the most recent preset number of historical reconstruction errors, and determine the mean and standard deviation of the historical reconstruction errors; wherein, the historical reconstruction error is the reconstruction error when it is determined that there are no abnormal events; The preset threshold is determined based on the mean, the standard deviation, and the preset sensitivity coefficient.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the abnormal event detection method as described in any one of claims 1 to 9 when executing the computer program.

Images