A subway emergency project event perception system and method based on machine vision

Abstract

The present invention relates to the field of machine vision technology, and invents a subway emergency project event perception system and method based on machine vision, including performing video splitting and enhanced noise removal operations on real-time subway monitoring videos to obtain subway monitoring image sequences and subway audio sequences; performing background object annotation on subway monitoring image sequences to obtain annotated image sequences; performing motion tracking and weighted feature extraction on the annotated image sequences to obtain image event feature sequences, and extracting audio event feature sequences corresponding to subway audio sequences; performing alignment fusion and time series convolution on the image event feature sequences and audio event feature sequences to obtain event time series features, and performing variational reconstruction analysis on the event time series features to obtain primary abnormal events; performing event classification on the event time series features to obtain secondary abnormal events, and generating emergency project events according to the primary abnormal events and secondary abnormal events. The present invention can reduce the limitations of event perception.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a subway emergency project event perception system and method based on machine vision. Background Art

[0002] Subway emergencies refer to events that occur suddenly in the subway system, are usually potentially dangerous, or cause disruption to the normal operation of the subway. These events may threaten the safety of passengers, cause equipment failures, or affect the normal operation of the transportation system. In order to ensure passenger safety and improve operational efficiency, it is necessary to perceive and monitor subway emergencies.

[0003] Existing perception methods for subway emergency events are mostly based on sensor monitoring, that is, they rely on various physical sensor systems to monitor specific types of events in the subway, such as fire monitoring through disgust sensors. However, in actual applications, perception methods based on sensor monitoring can only detect specific physical phenomena and cannot handle complex behaviors or social events, which leads to greater limitations in event perception. Summary of the invention

[0004] The present invention provides a subway emergency project event perception system and method based on machine vision, the main purpose of which is to solve the problem of large limitations in event perception.

[0005] To achieve the above purpose, the present invention provides a subway emergency event perception system based on machine vision, which includes a video segmentation module, a background annotation module, a feature extraction module, a feature fusion module and an event analysis module, wherein:

[0006] The video splitting module is used to perform video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence. When the video splitting module performs video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence, the module includes: performing audio decomposition on the pre-acquired real-time subway monitoring video to obtain a monitoring audio sequence; performing video frame decomposition on the real-time subway monitoring video to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum denoising algorithm to obtain a denoised amplitude spectrum signal:

[0007] ;

[0008] in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k; generating a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal; reconstructing the de-noised frequency domain signal in the time domain to obtain a subway audio sequence; performing neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence;

[0009] The background annotation module is used to annotate the background objects of the subway monitoring image sequence to obtain an annotated image sequence;

[0010] The feature extraction module is used to perform motion tracking and weighted feature extraction on the annotated image sequence to obtain an image event feature sequence, and to extract an audio event feature sequence corresponding to the subway audio sequence;

[0011] The feature fusion module is used to perform alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and perform variational reconstruction analysis on the event time series features to obtain primary abnormal events;

[0012] The event analysis module is used to classify the event time series features to obtain secondary abnormal events, and generate sudden project events according to the primary abnormal events and the secondary abnormal events.

[0013] Optionally, when the video splitting module performs neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence, the video splitting module includes:

[0014] The real-time monitoring image sequence is subjected to image denoising using the following gradient neighborhood denoising algorithm to obtain a denoised monitoring image sequence: ;

[0015] in, is the pixel value of the pixel point with pixel coordinate x in the denoised monitoring image sequence, is the filtering neighborhood of the pixel with pixel coordinate x, is the filtering neighborhood The pixel coordinates of the pixel point in is the exponential function symbol, is the Euclidean distance symbol, N is the filtering neighborhood The total number of pixels in is the pixel gradient value of the pixel point with pixel coordinate x in the real-time monitoring image sequence, The pixel coordinates in the real-time monitoring image sequence are The pixel value of the pixel point, is the pixel value of the pixel point with pixel coordinate x in the real-time monitoring image sequence;

[0016] Performing global illumination estimation on the denoised monitoring image sequence to obtain an illumination histogram sequence;

[0017] Performing a brightness equalization operation on the denoised monitoring image sequence according to the illumination histogram sequence to obtain a balanced monitoring image sequence;

[0018] Color correction is performed on the balanced monitoring image sequence to obtain a subway monitoring image sequence.

[0019] Optionally, when the background annotation module annotates the background objects of the subway monitoring image sequence to obtain the annotated image sequence, the background annotation module includes:

[0020] Performing multi-scale image convolution on the subway monitoring map sequence to obtain a subway map feature group sequence;

[0021] Performing batch normalization and pooling operations on the subway map feature group sequence to obtain a pooled map feature group sequence;

[0022] Performing residual connection on the pooling graph feature group sequence according to the subway monitoring graph sequence to obtain a residual graph feature group sequence;

[0023] Performing grid annotation analysis on the residual image feature group sequence to obtain an object annotation frame group sequence;

[0024] Performing a non-maximum suppression operation on the object annotation frame group sequence to obtain a standard annotation frame group sequence;

[0025] The subway monitoring map sequence is annotated with objects using the standard annotation frame group sequence to obtain an annotated map sequence.

[0026] Optionally, when the feature extraction module performs motion tracking and weighted feature extraction on the labeled image sequence to obtain the image event feature sequence, the feature extraction module includes:

[0027] Performing frame difference motion detection on the labeled image sequence to obtain a motion image sequence;

[0028] Performing event analysis on the motion graph sequence to obtain an event feature sequence;

[0029] Extracting a standard annotation frame group sequence from the annotation image sequence;

[0030] Performing object weighting on the standard annotation frame group sequence to obtain a weighted annotation frame group sequence;

[0031] The weighted annotation frame group sequence is used to perform feature weighting operation on the event feature sequence to obtain a graph event feature sequence.

[0032] Optionally, when the feature extraction module performs frame difference motion detection on the labeled image sequence to obtain a motion image sequence, the feature extraction module includes:

[0033] The following frame difference mask algorithm is used to perform frame difference detection on the labeled image sequence to obtain a frame difference image sequence: ;

[0034] in, is the frame difference value of the yth pixel of the tth frame difference image in the frame difference image sequence, t is the sequence index, y is the pixel index, is the modulo symbol, is the pixel value of the yth pixel of the tth labeled image in the labeled image sequence, is the first The pixel value of the yth pixel in the labeled image, is the first The pixel value of the yth pixel of the labeled image;

[0035] Performing a frame difference binarization operation on the frame difference image sequence to obtain a frame difference mask sequence;

[0036] Performing an opening operation on the frame difference mask sequence to obtain a noise removal mask sequence;

[0037] The denoising mask sequence is used to perform a mask operation on the labeled image sequence to obtain a motion image sequence.

[0038] Optionally, when the feature extraction module extracts the audio event feature sequence corresponding to the subway audio sequence, it includes:

[0039] Performing audio windowing on the subway audio sequence to obtain a windowed audio sequence;

[0040] Performing frequency domain conversion on the windowed audio sequence to obtain a windowed audio frequency domain signal;

[0041] Extracting a windowed power spectrum signal from the windowed audio frequency domain signal;

[0042] Performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain an audio event spectrum signal;

[0043] The audio event spectrum signal is reconstructed in the time domain to obtain an audio event feature sequence.

[0044] Optionally, when the feature fusion module performs alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain the event time series feature, the feature fusion module includes:

[0045] Synchronously aligning the image event feature sequence and the audio event feature sequence to obtain an aligned image feature sequence and an aligned audio feature sequence;

[0046] Performing weighted feature fusion on the alignment graph feature sequence and the alignment audio feature sequence to obtain an alignment event feature sequence;

[0047] Performing long-term time series convolution on the aligned event feature sequence to obtain long-term event time series features;

[0048] Performing short-term time series convolution on the aligned event feature sequence to obtain short-term event time series features;

[0049] The long-term event time series features and the short-term event time series features are fully connected and fused to obtain event time series features.

[0050] Optionally, when the feature fusion module performs variational reconstruction analysis on the event time series features to obtain a primary abnormal event, the feature fusion module includes:

[0051] Performing latent space encoding on the event temporal features to obtain a latent feature distribution;

[0052] Reparameterizing the latent feature distribution to obtain latent variable features;

[0053] Performing latent space decoding on the latent variable features to obtain reconstructed time series features;

[0054] Calculating a variational error between the event time series feature and the reconstructed time series feature;

[0055] Anomaly detection is performed on the variational error to obtain a primary abnormal event.

[0056] Optionally, when the event analysis module performs event classification on the event time series features to obtain a secondary abnormal event, the event analysis module includes:

[0057] Performing kernel function mapping on the event time series features to obtain mapped event features;

[0058] Performing a nonlinear activation operation on the mapped event features to obtain abnormal event features;

[0059] Performing weight regularization and batch normalization operations on the abnormal event features to obtain abnormal event codes;

[0060] The abnormal event codes are classified and outputted to obtain secondary abnormal events.

[0061] In order to solve the above problems, the present invention also provides a method for sensing subway emergency events based on machine vision, the method comprising:

[0062] The pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, wherein the pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, including: performing audio decomposition on the pre-acquired real-time monitoring video of the subway to obtain a monitoring audio sequence; performing video frame decomposition on the real-time monitoring video of the subway to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum denoising algorithm to obtain a denoised amplitude spectrum signal: ;

[0063] in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k; generating a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal; reconstructing the de-noised frequency domain signal in the time domain to obtain a subway audio sequence; performing neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence;

[0064] Annotating the subway monitoring image sequence with background objects to obtain an annotated image sequence;

[0065] Performing motion tracking and weighted feature extraction on the labeled image sequence to obtain an image event feature sequence, and extracting an audio event feature sequence corresponding to the subway audio sequence;

[0066] Performing alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and performing variational reconstruction analysis on the event time series features to obtain primary abnormal events;

[0067] Event classification is performed on the event time series features to obtain secondary abnormal events, and sudden project events are generated according to the primary abnormal events and the secondary abnormal events.

[0068] The embodiment of the present invention can extract subway audio data and image data from the surveillance video by performing video splitting, thereby achieving more comprehensive perception of sudden events; by performing enhanced noise removal, the details of the image and audio data can be improved, thereby improving the accuracy of sudden event perception; by performing background object marking, subway items that need attention can be marked in the frame image of the surveillance video, thereby facilitating subsequent perception of sudden events; by performing motion tracking and weighted feature extraction, event features of image areas where movement occurs in subway monitoring can be extracted by the frame difference method, and events can be weighted in combination with whether or not there is contact with important background objects; by extracting audio event feature sequences, audio event features in subway surveillance videos can be extracted, thereby facilitating subsequent multi-dimensional event perception, improving the comprehensiveness of event perception, and reducing the limitations of event perception.

[0069] By performing alignment fusion and time convolution, we can perform time series analysis on the event features of images and audio to obtain more complete event features. By performing variational reconstruction analysis, we can combine unsupervised training methods to perceive abnormal events and reduce the limitations of event perception. By performing event classification and generating sudden project events, we can combine supervised learning models and unsupervised models to perceive sudden project events, thereby improving the dimension of event perception and reducing the limitations of subway sudden event perception, thereby reducing the limitations of event perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A functional module diagram of a subway emergency project event perception system based on machine vision provided by an embodiment of the present invention;

[0071] Figure 2 A flowchart for background object labeling provided by an embodiment of the present invention;

[0072] Figure 3 A flowchart of extracting a feature sequence of graph events provided by an embodiment of the present invention;

[0073] Figure 4 A schematic diagram of a process flow of a method for sensing sudden subway project events based on machine vision provided by an embodiment of the present invention;

[0074] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0075] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0076] The embodiment of the present application provides a subway emergency project event perception system based on machine vision. The execution subject of the subway emergency project event perception system based on machine vision includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the system provided by the embodiment of the present application. In other words, the subway emergency project event perception system based on machine vision can be executed by software or hardware installed on a terminal device or a server device. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0077] like Figure 1 , which is a functional module diagram of a subway emergency project event perception system based on machine vision provided by an embodiment of the present invention.

[0078] The subway emergency project event perception system 100 based on machine vision of the present invention can be installed in an electronic device. According to the functions implemented, the subway emergency project event perception system 100 based on machine vision can include a video segmentation module 101, a background annotation module 102, a feature extraction module 103, a feature fusion module 104 and an event analysis module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0079] In this embodiment, the functions of each module / unit are as follows:

[0080] The video splitting module 101 is used to perform video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence.

[0081] In detail, the subway real-time monitoring video refers to the real-time video obtained by monitoring the subway stations and carriages using a monitoring system, each subway monitoring image in the subway monitoring image sequence corresponds to a video frame image of each frame in the subway real-time monitoring video, and the subway audio sequence corresponds to the audio data of the subway real-time monitoring video.

[0082] In the embodiment of the present invention, when the video splitting module 101 performs video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence, the video splitting module 101 includes:

[0083] Perform audio decomposition on the pre-acquired real-time subway surveillance video to obtain a surveillance audio sequence;

[0084] Decomposing the subway real-time monitoring video into video frames to obtain a real-time monitoring image sequence;

[0085] Performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal;

[0086] Extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal;

[0087] Calculating the signal-to-noise ratio of the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio;

[0088] Performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio to obtain a denoised amplitude spectrum signal;

[0089] Generate a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal;

[0090] Reconstructing the noise-removed frequency domain signal in the time domain to obtain a subway audio sequence;

[0091] The real-time monitoring image sequence is subjected to neighborhood denoising and brightness equalization to obtain a subway monitoring image sequence.

[0092] In detail, the audio decomposition refers to extracting audio data of the video from the real-time monitoring video of the subway, and each monitoring audio in the monitoring audio sequence corresponds to the audio signal of each frame in the real-time monitoring video of the subway. Among them, FFmpeg, Python's MoviePy or Pydub and other components can be used for audio decomposition.

[0093] Specifically, the video frame decomposition refers to extracting the image of each frame of the subway real-time monitoring video, and each real-time monitoring image in the real-time monitoring image sequence corresponds to the video frame image of each frame of the subway real-time monitoring video. Among them, video frame decomposition can be performed using components such as FFmpeg, Python's OpenCV, and scikit video.

[0094] In detail, the frequency domain conversion can be performed using the fast Fourier transform or wavelet transform method, the frequency domain amplitude spectrum signal is the intensity of the monitoring audio frequency domain signal in the frequency component, and the frequency domain phase spectrum signal is the phase of the monitoring audio frequency domain signal in the frequency.

[0095] Specifically, the signal-to-noise ratio calculation includes calculating the a posteriori signal-to-noise ratio using noise estimation, minimum pursuit method or spectral subtraction method, and calculating the a priori signal-to-noise ratio using a direct method, decision-oriented method or minimum mean square error estimation method.

[0096] Specifically, the denoised amplitude spectrum signal is calculated using the following amplitude spectrum denoising algorithm: ;

[0097] in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, It's the multiplication sign. is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k.

[0098] Specifically, the amplitude spectrum denoising algorithm can model the statistical characteristics of signals and noise, and calculate conditional expectations through Bayesian reasoning, so as to more accurately retain the signal details related to speech, and is particularly suitable for processing emergencies in speech signals.

[0099] In detail, the method of generating a noise-removed frequency domain signal based on the noise-removed amplitude spectrum signal and the frequency domain phase spectrum signal is the inverse method of the method of extracting the frequency domain amplitude spectrum signal and the frequency domain phase spectrum signal from the monitoring audio frequency domain signal, and the method of performing time domain reconstruction on the noise-removed frequency domain signal to obtain a subway audio sequence is the inverse method of the method of performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal, which will not be repeated here.

[0100] Specifically, when the video splitting module 101 performs neighborhood noise removal and brightness equalization on the real-time monitoring image sequence to obtain the subway monitoring image sequence, it includes:

[0101] Performing image denoising on the real-time monitoring image sequence to obtain a denoised monitoring image sequence;

[0102] Performing global illumination estimation on the denoised monitoring image sequence to obtain an illumination histogram sequence;

[0103] Performing a brightness equalization operation on the denoised monitoring image sequence according to the illumination histogram sequence to obtain a balanced monitoring image sequence;

[0104] Color correction is performed on the balanced monitoring image sequence to obtain a subway monitoring image sequence.

[0105] In detail, the following gradient neighborhood denoising algorithm is used to calculate the denoised monitoring image sequence: ;

[0106] in, is the pixel value of the pixel point with pixel coordinate x in the denoised monitoring image sequence, is the filtering neighborhood of the pixel with pixel coordinate x, is the filtering neighborhood The pixel coordinates of the pixel point in is the exponential function symbol, is the Euclidean distance symbol, N is the filtering neighborhood The total number of pixels in is the pixel gradient value of the pixel point with pixel coordinate x in the real-time monitoring image sequence, The pixel coordinates in the real-time monitoring image sequence are The pixel value of the pixel point, It is the pixel value of the pixel point with pixel coordinate x in the real-time monitoring image sequence.

[0107] Specifically, the gradient neighborhood denoising algorithm can combine the gradient change of the image and the pixel value change trend of the neighborhood to perform denoising of different intensities, thereby effectively protecting the image details of the real-time monitoring image sequence and improving the efficiency of denoising.

[0108] In detail, global illumination estimation can be performed using the cv2.calcHist function, the numpy.histogram function, or the skimage.exposure.histogram function, brightness equalization can be performed using the cumulative distribution function, gamma correction, or an offset adjustment method based on a Gaussian distribution curve, and color correction can be performed using white balance correction, gamma correction, or a color constancy algorithm.

[0109] In the embodiment of the present invention, by performing video splitting, the audio data and image data of the subway can be extracted from the surveillance video, thereby achieving a more comprehensive perception of sudden project events. By performing enhanced noise removal, the details of the image and audio data can be improved, thereby improving the accuracy of sudden project event perception.

[0110] The background annotation module 102 is used to annotate the background objects of the subway monitoring image sequence to obtain an annotated image sequence.

[0111] In detail, the labeled image sequence refers to the subway monitoring image sequence that annotates the priority of each background object or area in the image. When perceiving sudden events in the subway, sudden events are often related to damage to items on the subway, such as structural failure of subway car doors, passengers pressing emergency buttons, etc. In order to achieve more accurate perception of sudden events in the subway, it is necessary to annotate the background objects in the subway monitoring image sequence.

[0112] In the embodiment of the present invention, refer to Figure 2 As shown, when the background annotation module 102 annotates the background objects of the subway monitoring image sequence to obtain the annotated image sequence, it includes:

[0113] S21, performing multi-scale image convolution on the subway monitoring map sequence to obtain a subway map feature group sequence;

[0114] S22, performing batch normalization and pooling operations on the subway map feature group sequence to obtain a pooled map feature group sequence;

[0115] S23, performing residual connection on the pooled graph feature group sequence according to the subway monitoring graph sequence to obtain a residual graph feature group sequence;

[0116] S24, performing grid annotation analysis on the residual image feature group sequence to obtain an object annotation frame group sequence;

[0117] S25, performing a non-maximum suppression operation on the object annotation frame group sequence to obtain a standard annotation frame group sequence;

[0118] S26. Use the standard annotation frame group sequence to annotate objects in the subway monitoring map sequence to obtain an annotated map sequence.

[0119] In detail, the multi-scale image convolution refers to using a feature pyramid network (FPN) to perform image feature convolution of three scales, large, medium and small, on the subway monitoring map sequence, thereby improving the detection capability of objects of different sizes. Each subway map feature group in the subway map feature group sequence contains image features of subway monitoring maps of three different sizes, large, medium and small.

[0120] Specifically, the batch normalization (BN) is to perform a normalization operation on the subway map feature group sequence of each layer, thereby accelerating the convergence and training effect of the network, and the residual connection refers to scaling the subway monitoring map sequence to the corresponding size of the pooled map feature group sequence, thereby performing a residual operation to obtain a residual feature group sequence.

[0121] In detail, a pre-trained YOLOv2 model or YOLOv3 model can be used to perform grid annotation analysis. Each object annotation frame in the object annotation frame group sequence includes the analyzed position coordinates of the object, the height and width of the annotation frame, the confidence of the object, and the category probability of the object classification. The non-maximum suppression is to retain the object annotation frame with the highest confidence within a given neighborhood, and suppress other non-maximum object annotation frames.

[0122] In the embodiment of the present invention, by marking background objects, subway objects that need attention can be marked in the frame image of the surveillance video, thereby facilitating the subsequent perception of emergencies.

[0123] The feature extraction module 103 is used to perform motion tracking and weighted feature extraction on the annotated image sequence to obtain an image event feature sequence, and to extract an audio event feature sequence corresponding to the subway audio sequence.

[0124] In detail, the image event feature sequence refers to the event features generated by moving objects in the standard image sequence, and the audio event features refer to the event-related features in the subway audio sequence, such as sudden noise features and people's cries for help features.

[0125] In the embodiment of the present invention, refer to Figure 3 As shown, when the feature extraction module 103 performs motion tracking and weighted feature extraction on the labeled image sequence to obtain the image event feature sequence, it includes:

[0126] S31, performing frame difference motion detection on the labeled image sequence to obtain a motion image sequence;

[0127] S32, performing event analysis on the motion graph sequence to obtain an event feature sequence;

[0128] S33, extracting a standard annotation frame group sequence from the annotation image sequence;

[0129] S34, weighting the objects on the standard annotation frame group sequence to obtain a weighted annotation frame group sequence;

[0130] S35. Perform a feature weighting operation on the event feature sequence using the weighted annotation frame group sequence to obtain a graph event feature sequence.

[0131] In detail, when the feature extraction module 103 performs frame difference motion detection on the labeled image sequence to obtain a motion image sequence, it includes:

[0132] Performing frame difference detection on the labeled image sequence to obtain a frame difference image sequence;

[0133] Performing a frame difference binarization operation on the frame difference image sequence to obtain a frame difference mask sequence;

[0134] Performing an opening operation on the frame difference mask sequence to obtain a noise removal mask sequence;

[0135] The denoising mask sequence is used to perform a mask operation on the labeled image sequence to obtain a motion image sequence.

[0136] In detail, the frame difference image sequence is calculated using the following frame difference mask algorithm: ;

[0137] in, is the frame difference value of the yth pixel of the tth frame difference image in the frame difference image sequence, t is the sequence index, y is the pixel index, is the modulo symbol, is the pixel value of the yth pixel of the tth labeled image in the labeled image sequence, is the first The pixel value of the yth pixel in the labeled image, is the first The pixel value of the y-th pixel in the labeled image.

[0138] Specifically, the frame difference mask algorithm can determine the frame difference by combining the pixel change values ​​of the previous frame and the next frame, and can capture the motion area more accurately. Simply using the difference between the current frame and the next frame may be affected by noise or short-term changes, while combining the previous and next frames can better balance the errors and reduce detection errors and missed detections.

[0139] In detail, the frame difference binarization operation refers to taking the pixels in the frame difference image sequence whose frame difference values ​​are greater than a preset frame difference threshold as mask pixels, and aggregating all the mask pixels into a frame difference mask sequence; the opening operation refers to first performing an erosion operation to remove isolated noise points, and then performing an expansion operation to enhance the continuity of the mask area.

[0140] Specifically, the event analysis includes detecting the position features of the motion center of mass, bounding box, etc. of the motion image sequence; extracting geometric features such as the motion area, perimeter, aspect ratio and shape of the motion region of the motion image sequence; extracting dynamic features such as the speed, acceleration and direction of the moving object; and extracting image features such as the color and texture of the motion image.

[0141] In detail, the object weighting refers to assigning different importance weights to different types of background objects. For example, the weight of the emergency brake button is 3, and the weight of the car window is 2. The feature weighting refers to assigning corresponding feature weights to the event features in the event feature sequence that are located in the weighted annotation frame group sequence.

[0142] Specifically, when the feature extraction module 103 extracts the audio event feature sequence corresponding to the subway audio sequence, it includes:

[0143] Performing audio windowing on the subway audio sequence to obtain a windowed audio sequence;

[0144] Performing frequency domain conversion on the windowed audio sequence to obtain a windowed audio frequency domain signal;

[0145] Extracting a windowed power spectrum signal from the windowed audio frequency domain signal;

[0146] Performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain an audio event spectrum signal;

[0147] The audio event spectrum signal is reconstructed in the time domain to obtain an audio event feature sequence.

[0148] In detail, audio windowing can be performed using methods such as Hamming Window, Hanning Window or Blackman Window. The frequency domain conversion and the time domain reconstruction methods are consistent with the frequency domain conversion and the time domain reconstruction methods in the above-mentioned video splitting module 101, and will not be repeated here.

[0149] Specifically, the windowed power spectrum signal represents the energy of each frequency component of the windowed audio frequency domain signal in the frequency domain, and is usually used to analyze the energy distribution of the signal. The windowed power spectrum signal can be obtained by calculating the square of the amplitude spectrum of the windowed audio frequency domain signal. The windowed power spectrum signal emphasizes the energy characteristics of the signal rather than simple amplitude changes.

[0150] In the embodiment of the present invention, by performing motion tracking and weighted feature extraction, the event features of the image area where movement occurs in subway monitoring can be extracted by the frame difference method, and the events can be weighted in combination with the presence or absence of contact with important background objects. By extracting the audio event feature sequence, the audio event features in the subway monitoring video can be extracted, thereby facilitating subsequent multi-dimensional event perception, improving the comprehensiveness of event perception, and reducing the limitations of event perception.

[0151] The feature fusion module 104 is used to perform alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and perform variational reconstruction analysis on the event time series features to obtain primary abnormal events.

[0152] In detail, the event timing feature refers to the timing feature of the event that is a fusion of the graphic event feature sequence and the audio event feature sequence. The primary abnormal event refers to the abnormal event corresponding to the event timing feature obtained by the unsupervised learning model of variational reconstruction. The abnormal event may be a passenger calling for help, quarreling, or being injured.

[0153] In the embodiment of the present invention, when the feature fusion module 104 performs alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain the event time series feature, it includes:

[0154] Synchronously aligning the image event feature sequence and the audio event feature sequence to obtain an aligned image feature sequence and an aligned audio feature sequence;

[0155] Performing weighted feature fusion on the alignment graph feature sequence and the alignment audio feature sequence to obtain an alignment event feature sequence;

[0156] Performing long-term time series convolution on the aligned event feature sequence to obtain long-term event time series features;

[0157] Performing short-term time series convolution on the aligned event feature sequence to obtain short-term event time series features;

[0158] The long-term event time series features and the short-term event time series features are fully connected and fused to obtain event time series features.

[0159] In detail, the synchronization alignment refers to an operation of upsampling or downsampling the image event feature sequence and the audio event feature sequence so that the frame rate, resolution and length of the image event feature sequence and the audio event feature sequence are consistent.

[0160] Specifically, weighted features can be performed according to preset graph feature weights and audio feature weights, or different weights can be assigned to the aligned graph feature sequence and the aligned audio feature sequence according to a self-attention mechanism, thereby achieving weighted feature fusion.

[0161] In detail, the long-term temporal convolution and the short-term temporal convolution refer to using the gate structure and memory unit of the Long Short Term Memory (LSTM) model for memory and output.

[0162] Specifically, when the feature fusion module 104 performs variational reconstruction analysis on the event time series features to obtain a primary abnormal event, it includes:

[0163] Performing latent space encoding on the event temporal features to obtain a latent feature distribution;

[0164] Reparameterizing the latent feature distribution to obtain latent variable features;

[0165] Performing latent space decoding on the latent variable features to obtain reconstructed time series features;

[0166] Calculating a variational error between the event time series feature and the reconstructed time series feature;

[0167] Anomaly detection is performed on the variational error to obtain a primary abnormal event.

[0168] In detail, the latent space encoding refers to compressing the event timing features into a low-dimensional latent space using an encoder composed of multiple convolutional layers, a recurrent neural network, or a long-short time series neural network. The latent feature distribution can be the normal distribution of the event timing features in the latent space. The reparameterization represents a random variable as a function of another random variable so that it can be processed by a standard optimization algorithm.

[0169] Specifically, the latent space decoding is the inverse operation of the latent space encoding. The mean square error algorithm or the cosine distance algorithm can be used to calculate the variational error. The anomaly detection refers to taking the event timing characteristics of the variational error greater than a preset error threshold as the abnormal feature, performing event mapping on the corresponding coding error, and obtaining the primary abnormal event.

[0170] In the embodiment of the present invention, by performing alignment fusion and time convolution, the event features of the picture and audio can be analyzed in time to obtain more complete event features. By performing variational reconstruction analysis, abnormal events can be perceived in combination with unsupervised training methods to reduce the limitations of event perception.

[0171] The event analysis module 105 is used to classify the event time series features to obtain secondary abnormal events, and generate sudden project events according to the primary abnormal events and the secondary abnormal events.

[0172] In the embodiment of the present invention, when the event analysis module 105 performs event classification on the event time series features to obtain a secondary abnormal event, it includes:

[0173] Performing kernel function mapping on the event time series features to obtain mapped event features;

[0174] Performing a nonlinear activation operation on the mapped event features to obtain abnormal event features;

[0175] Performing weight regularization and batch normalization operations on the abnormal event features to obtain abnormal event codes;

[0176] The abnormal event codes are classified and outputted to obtain secondary abnormal events.

[0177] In detail, the kernel function can be a pre-trained Gaussian kernel function or polynomial kernel function, nonlinear activation operations can be performed using activation functions such as ReLU or Sigmoid, weight regularization can be performed using L2 regularization, dropout and other techniques, and the classification output refers to the activation output using the output layer of a pre-trained support vector machine model or neural network model.

[0178] Specifically, generating a sudden project event according to the primary abnormal event and the secondary abnormal event refers to using the secondary abnormal event to update and supplement the corresponding abnormal event in the primary abnormal event to obtain a sudden project event.

[0179] In the embodiment of the present invention, by classifying events and generating sudden project events, the sudden project events can be perceived by combining the supervised learning model and the unsupervised model, thereby improving the dimension of event perception and reducing the limitations of subway sudden event perception.

[0180] Reference Figure 4 FIG. 1 is a flow chart of a method for sensing sudden subway project events based on machine vision according to an embodiment of the present invention. In this embodiment, the method for sensing sudden subway project events based on machine vision includes:

[0181] S1. Performing video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence, wherein the performing video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence includes: performing audio decomposition on the pre-acquired real-time subway monitoring video to obtain a monitoring audio sequence; performing video frame decomposition on the real-time subway monitoring video to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio noise removal on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum noise removal algorithm to obtain a noise-removed amplitude spectrum signal:

[0182] ;

[0183] in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is; a denoised frequency domain signal is generated according to the denoised amplitude spectrum signal and the frequency domain phase spectrum signal; the denoised frequency domain signal is reconstructed in the time domain to obtain a subway audio sequence; the real-time monitoring image sequence is subjected to neighborhood denoising and brightness equalization to obtain a subway monitoring image sequence.

[0184] S2. Annotate the background objects of the subway monitoring image sequence to obtain an annotated image sequence.

[0185] S3. Perform motion tracking and weighted feature extraction on the labeled image sequence to obtain an image event feature sequence, and extract an audio event feature sequence corresponding to the subway audio sequence.

[0186] S4. Align and fuse the image event feature sequence and the audio event feature sequence and perform time convolution to obtain event time series features, and perform variational reconstruction analysis on the event time series features to obtain primary abnormal events.

[0187] S5. Classify the event time series features to obtain secondary abnormal events, and generate sudden project events according to the primary abnormal events and the secondary abnormal events.

[0188] The embodiment of the present invention can extract subway audio data and image data from the surveillance video by performing video splitting, thereby achieving more comprehensive perception of sudden events; by performing enhanced noise removal, the details of the image and audio data can be improved, thereby improving the accuracy of sudden event perception; by performing background object marking, subway items that need attention can be marked in the frame image of the surveillance video, thereby facilitating subsequent perception of sudden events; by performing motion tracking and weighted feature extraction, event features of image areas where movement occurs in subway monitoring can be extracted by the frame difference method, and events can be weighted in combination with whether or not there is contact with important background objects; by extracting audio event feature sequences, audio event features in subway surveillance videos can be extracted, thereby facilitating subsequent multi-dimensional event perception, improving the comprehensiveness of event perception, and reducing the limitations of event perception.

[0189] By performing alignment fusion and time convolution, we can perform time series analysis on the event features of images and audio to obtain more complete event features. By performing variational reconstruction analysis, we can combine unsupervised training methods to perceive abnormal events and reduce the limitations of event perception. By performing event classification and generating sudden project events, we can combine supervised learning models and unsupervised models to perceive sudden project events, thereby improving the dimension of event perception and reducing the limitations of subway sudden event perception, thereby reducing the limitations of event perception.

[0190] In detail, the subway emergency project event perception method based on machine vision in the embodiment of the present invention is used in the same manner as above. Figure 1 The same technical means are used as the subway emergency project event perception system based on machine vision described in the text, and can produce the same technical effects, so I will not go into details here.

[0191] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0192] The pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, wherein the pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, including: performing audio decomposition on the pre-acquired real-time monitoring video of the subway to obtain a monitoring audio sequence; performing video frame decomposition on the real-time monitoring video of the subway to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum denoising algorithm to obtain a denoised amplitude spectrum signal: ;

[0193] in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k; generating a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal; reconstructing the de-noised frequency domain signal in the time domain to obtain a subway audio sequence; performing neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence;

[0194] Annotating the subway monitoring image sequence with background objects to obtain an annotated image sequence;

[0195] Performing motion tracking and weighted feature extraction on the labeled image sequence to obtain an image event feature sequence, and extracting an audio event feature sequence corresponding to the subway audio sequence;

[0196] Performing alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and performing variational reconstruction analysis on the event time series features to obtain primary abnormal events;

[0197] Event classification is performed on the event time series features to obtain secondary abnormal events, and sudden project events are generated according to the primary abnormal events and the secondary abnormal events.

[0198] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and systems can be implemented in other ways. For example, the system embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0199] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0201] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0202] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0203] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is a theory, system, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0204] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or systems stated in the system embodiment can also be implemented by one unit or system through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0205] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A subway emergency event perception system based on machine vision, characterized in that: The system includes a video segmentation module, a background annotation module, a feature extraction module, a feature fusion module and an event analysis module, wherein: The video splitting module is used to perform video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence. When the video splitting module performs video splitting and enhanced noise removal operations on the pre-acquired real-time subway monitoring video to obtain a subway monitoring image sequence and a subway audio sequence, the module includes: performing audio decomposition on the pre-acquired real-time subway monitoring video to obtain a monitoring audio sequence; performing video frame decomposition on the real-time subway monitoring video to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum denoising algorithm to obtain a denoised amplitude spectrum signal: ; in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k; generating a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal; reconstructing the de-noised frequency domain signal in the time domain to obtain a subway audio sequence; performing neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence; The background annotation module is used to annotate the background objects of the subway monitoring image sequence to obtain an annotated image sequence; The feature extraction module is used to perform motion tracking and weighted feature extraction on the annotated image sequence to obtain an image event feature sequence, and to extract an audio event feature sequence corresponding to the subway audio sequence; The feature fusion module is used to perform alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and perform variational reconstruction analysis on the event time series features to obtain primary abnormal events; The event analysis module is used to classify the event time series features to obtain secondary abnormal events, and generate sudden project events according to the primary abnormal events and the secondary abnormal events.

2. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the video splitting module performs neighborhood noise removal and brightness equalization on the real-time monitoring image sequence to obtain the subway monitoring image sequence, it includes: The real-time monitoring image sequence is subjected to image denoising using the following gradient neighborhood denoising algorithm to obtain a denoised monitoring image sequence: ； in, is the pixel value of the pixel point with pixel coordinate x in the denoised monitoring image sequence, is the filtering neighborhood of the pixel with pixel coordinate x, is the filtering neighborhood The pixel coordinates of the pixel point in is the exponential function symbol, is the Euclidean distance symbol, N is the filtering neighborhood The total number of pixels in is the pixel gradient value of the pixel point with pixel coordinate x in the real-time monitoring image sequence, The pixel coordinates in the real-time monitoring image sequence are The pixel value of the pixel point, is the pixel value of the pixel point with pixel coordinate x in the real-time monitoring image sequence; Performing global illumination estimation on the denoised monitoring image sequence to obtain an illumination histogram sequence; Performing a brightness equalization operation on the denoised monitoring image sequence according to the illumination histogram sequence to obtain a balanced monitoring image sequence; Color correction is performed on the balanced monitoring image sequence to obtain a subway monitoring image sequence.

3. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the background annotation module annotates the background objects of the subway monitoring image sequence to obtain the annotated image sequence, the background annotation module includes: Performing multi-scale image convolution on the subway monitoring map sequence to obtain a subway map feature group sequence; Performing batch normalization and pooling operations on the subway map feature group sequence to obtain a pooled map feature group sequence; Performing residual connection on the pooling graph feature group sequence according to the subway monitoring graph sequence to obtain a residual graph feature group sequence; Performing grid annotation analysis on the residual image feature group sequence to obtain an object annotation frame group sequence; Performing a non-maximum suppression operation on the object annotation frame group sequence to obtain a standard annotation frame group sequence; The subway monitoring map sequence is annotated with objects using the standard annotation frame group sequence to obtain an annotated map sequence.

4. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the feature extraction module performs motion tracking and weighted feature extraction on the labeled image sequence to obtain the image event feature sequence, it includes: Performing frame difference motion detection on the labeled image sequence to obtain a motion image sequence; Performing event analysis on the motion graph sequence to obtain an event feature sequence; Extracting a standard annotation frame group sequence from the annotation image sequence; Performing object weighting on the standard annotation frame group sequence to obtain a weighted annotation frame group sequence; The weighted annotation frame group sequence is used to perform feature weighting operation on the event feature sequence to obtain a graph event feature sequence.

5. The machine vision-based subway emergency event perception system according to claim 4, characterized in that: When the feature extraction module performs frame difference motion detection on the labeled image sequence to obtain a motion image sequence, the feature extraction module includes: The following frame difference mask algorithm is used to perform frame difference detection on the labeled image sequence to obtain a frame difference image sequence: ; in, is the frame difference value of the yth pixel of the tth frame difference image in the frame difference image sequence, t is the sequence index, y is the pixel index, is the modulo symbol, is the pixel value of the yth pixel of the tth labeled image in the labeled image sequence, is the first The pixel value of the yth pixel in the labeled image, is the first The pixel value of the yth pixel of the labeled image; Performing a frame difference binarization operation on the frame difference image sequence to obtain a frame difference mask sequence; Performing an opening operation on the frame difference mask sequence to obtain a noise removal mask sequence; The denoising mask sequence is used to perform a mask operation on the labeled image sequence to obtain a motion image sequence.

6. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the feature extraction module extracts the audio event feature sequence corresponding to the subway audio sequence, it includes: Performing audio windowing on the subway audio sequence to obtain a windowed audio sequence; Performing frequency domain conversion on the windowed audio sequence to obtain a windowed audio frequency domain signal; Extracting a windowed power spectrum signal from the windowed audio frequency domain signal; Performing Mel-cepstrum filtering on the windowed power spectrum signal to obtain an audio event spectrum signal; The audio event spectrum signal is reconstructed in the time domain to obtain an audio event feature sequence.

7. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the feature fusion module performs alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain the event time series feature, the feature fusion module includes: Synchronously aligning the image event feature sequence and the audio event feature sequence to obtain an aligned image feature sequence and an aligned audio feature sequence; Performing weighted feature fusion on the alignment graph feature sequence and the alignment audio feature sequence to obtain an alignment event feature sequence; Performing long-term time series convolution on the aligned event feature sequence to obtain long-term event time series features; Performing short-term time series convolution on the aligned event feature sequence to obtain short-term event time series features; The long-term event time series features and the short-term event time series features are fully connected and fused to obtain event time series features.

8. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the feature fusion module performs variational reconstruction analysis on the event time series features to obtain a primary abnormal event, it includes: Performing latent space encoding on the event temporal features to obtain a latent feature distribution; Reparameterizing the latent feature distribution to obtain latent variable features; Performing latent space decoding on the latent variable features to obtain reconstructed time series features; Calculating a variational error between the event time series feature and the reconstructed time series feature; Anomaly detection is performed on the variational error to obtain a primary abnormal event.

9. The machine vision-based subway emergency event perception system according to claim 1, characterized in that: When the event analysis module performs event classification on the event time series features to obtain a secondary abnormal event, it includes: Performing kernel function mapping on the event time series features to obtain mapped event features; Performing a nonlinear activation operation on the mapped event features to obtain abnormal event features; Performing weight regularization and batch normalization operations on the abnormal event features to obtain abnormal event codes; The abnormal event codes are classified and outputted to obtain secondary abnormal events.

10. A method for sensing sudden subway project events based on machine vision, characterized in that: The method comprises: The pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, wherein the pre-acquired real-time monitoring video of the subway is subjected to video splitting and enhanced noise removal operations to obtain a subway monitoring image sequence and a subway audio sequence, including: performing audio decomposition on the pre-acquired real-time monitoring video of the subway to obtain a monitoring audio sequence; performing video frame decomposition on the real-time monitoring video of the subway to obtain a real-time monitoring image sequence; performing frequency domain conversion on the monitoring audio sequence to obtain a monitoring audio frequency domain signal; extracting a frequency domain amplitude spectrum signal and a frequency domain phase spectrum signal from the monitoring audio frequency domain signal; performing signal-to-noise ratio calculation on the frequency domain amplitude spectrum signal to obtain a priori signal-to-noise ratio and a posteriori signal-to-noise ratio; performing audio denoising on the frequency domain amplitude spectrum signal according to the priori signal-to-noise ratio and the a posteriori signal-to-noise ratio using the following amplitude spectrum denoising algorithm to obtain a denoised amplitude spectrum signal: ; in, It refers to the amplitude spectrum signal of the denoised amplitude spectrum signal when the frequency is k, is the modulo symbol, It refers to the amplitude value corresponding to the denoised amplitude spectrum signal when the frequency is k, is the a priori signal-to-noise ratio at frequency k, is the a posteriori signal-to-noise ratio at frequency k, is the exponential function symbol, It refers to the amplitude spectrum signal of the frequency domain amplitude spectrum signal when the frequency is k, It refers to the amplitude value corresponding to the frequency domain amplitude spectrum signal when the frequency is k; generating a de-noised frequency domain signal according to the de-noised amplitude spectrum signal and the frequency domain phase spectrum signal; reconstructing the de-noised frequency domain signal in the time domain to obtain a subway audio sequence; performing neighborhood denoising and brightness equalization on the real-time monitoring image sequence to obtain a subway monitoring image sequence; Annotating the subway monitoring image sequence with background objects to obtain an annotated image sequence; Performing motion tracking and weighted feature extraction on the labeled image sequence to obtain an image event feature sequence, and extracting an audio event feature sequence corresponding to the subway audio sequence; Performing alignment fusion and time series convolution on the image event feature sequence and the audio event feature sequence to obtain event time series features, and performing variational reconstruction analysis on the event time series features to obtain primary abnormal events; Event classification is performed on the event time series features to obtain secondary abnormal events, and sudden project events are generated according to the primary abnormal events and the secondary abnormal events.

Images