Audio / video and image fusion monitoring analysis system and method based on deep learning in track scene

Abstract

The invention relates to the technical field of rail traffic safety management, in particular to an audio and video and image fusion monitoring analysis system and method based on deep learning in a rail scene, and the system comprises a data collection module which is used for collecting audio data and video data in the rail scene; the audio processing module is used for receiving the collected audio data and carrying out noise removal and segmentation on the audio data to obtain a plurality of audio segments; the video processing module is used for receiving the collected video data and performing signal enhancement and splicing on the video data; and the data analysis module is used for keeping the plurality of audio segments and the spliced video consistent in time and completing a classification function based on a neural network. According to the invention, noise removal and audio segmentation are carried out on the audio data, image enhancement and splicing are carried out on the video data, and the audio data and the video data are fused, so that the classification accuracy of various events in the track scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit safety management, and in particular to a deep learning-based audio, video and image fusion monitoring and analysis system and method in a rail scenario. Background Art

[0002] In today's security monitoring field, traditional monitoring systems are mainly based on a single type of data for analysis, such as relying solely on image data or audio data. Image-based monitoring systems can capture visual information, but in some cases there are obvious deficiencies; for example, in scenes with low light, severe occlusion, or low image resolution, image information alone may not be able to accurately identify target objects or behaviors; in addition, simple image analysis is difficult to effectively detect some hidden actions or situations without obvious visual features.

[0003] Although monitoring systems based only on audio can capture sound information, problems such as background noise interference and inaccurate audio source positioning frequently occur. Audio signals are easily drowned out by environmental noise, leading to misjudgment or omission of abnormal audio events. Moreover, audio data alone cannot provide visual scene information, making it difficult to determine the specific location of the event and the visual characteristics of the participants.

[0004] The utility model patent with announcement number CN219812199U discloses a station video intelligent monitoring system, which includes machine room equipment, video monitoring terminals and field equipment; by deploying and controlling the field equipment, the line is monitored, and the sound and light alarms are actively turned on and off to remind the trains entering the station and the operations are completed; the information such as area intrusion, personnel entering the area, and personnel leaving the area is received and sent to the video monitoring terminal through the machine room equipment. The video display terminal can locate the alarm position, actively call the real-time video of the target object, and provide the function of processing the alarm; at the same time, a panoramic camera is added to provide a 360-degree panoramic picture of the station, and it supports adding any object annotations in the panoramic picture to realize three-dimensional monitoring and high and low linkage.

[0005] However, this patent can assist managers in making decisions, reduce the workload of personnel patrolling and monitoring equipment, form a camera linkage system, and improve the human-computer interaction experience. However, relying solely on video to achieve the purpose of monitoring, the single mode leads to insufficient accuracy and reliability. Therefore, a monitoring and analysis system that combines audio and video is needed. Summary of the invention

[0006] The purpose of the present invention is to provide an audio, video and image fusion monitoring and analysis system and method based on deep learning in a track scenario to solve the problems raised in the above background technology.

[0007] In order to solve the above technical problems, the present invention provides an audio, video and image fusion monitoring and analysis system based on deep learning in a track scene, including:

[0008] Data acquisition module: used to collect audio and video data in track scenes;

[0009] Audio processing module: used to receive the collected audio data, remove noise and segment the audio data to obtain multiple audio segments;

[0010] Video processing module: used to receive the collected video data and perform signal enhancement and splicing on the video data;

[0011] Data analysis module: used to make multiple audio segments and spliced ​​videos consistent in time, and complete the classification function based on neural network.

[0012] As a further improvement of the technical solution, the data acquisition module includes a plurality of cameras, which are installed at different positions of the station and are equipped with microphones.

[0013] As a further improvement of the technical solution, the specific steps of removing noise from audio data in the audio processing module are:

[0014] A1. Assuming that the audio signal is a linear dynamic system, its state equation is:

[0015] x k =Ax k-1 +w k-1

[0016] Among them, x k is the system state at time k, A is the state transfer matrix, x k-1 is the system state at time k-1, w k-1 is the process noise at time k-1, which is a Gaussian white noise vector with zero mean and covariance matrix Q;

[0017] Then its observation equation is:

[0018] y k =Hx k +v k

[0019] Among them, y k is the observed value at time k, H is the observation matrix, v k is the observation noise at time k, which is a Gaussian white noise with zero mean and covariance matrix R;

[0020] A2. Calculate the state prediction value and prediction error covariance. The corresponding formula is:

[0021]

[0022]

[0023] in, is the prior state estimate at time k, that is, the predicted value, is the posterior state estimate at time k-1, which is the optimal estimate after update. is the prior estimation error covariance matrix, is the posterior estimation error covariance matrix at time k-1, A T is the transposed matrix of the state transfer matrix A;

[0024] A3. Optimize the parameters in the Kalman filter algorithm based on the improved grey wolf algorithm;

[0025] A4. Calculate the Kalman gain. The corresponding formula is:

[0026]

[0027] Among them, K k is the Kalman gain matrix at time k, H T is the transposed matrix of the observation matrix H;

[0028] A5. Update state estimation and error covariance. The corresponding formula is:

[0029]

[0030] in, is the posterior state estimate at time k, I is the unit matrix, is the updated posterior estimation error covariance matrix for the next iteration.

[0031] As a further improvement of the technical solution, the specific steps of optimizing the parameters in the Kalman filter based on the improved grey wolf algorithm in A3 are:

[0032] A31, population initialization, setting the population size, spatial dimension and maximum number of iterations, where the position vector of each gray wolf represents the parameters to be optimized in the process noise and observation noise respectively;

[0033] A32. Calculate the fitness of each gray wolf. Define the top three best gray wolves as α, β, and δ, and the rest of the gray wolves as ω.

[0034] A33. Surround the prey:

[0035]

[0036]

[0037] Where D represents the distance between the gray wolf and its prey. and are all coefficient vectors, t represents the current number of iterations, represents the position vector of the prey at the tth iteration, represents the position vector of the gray wolf at the t+1th iteration, is the convergence factor, T is the maximum number of iterations, and The modulus of gets a random number between [0,1];

[0038] A34. Hunting prey:

[0039]

[0040] Among them, D α , D β and D δ Respectively represent the distances between gray wolf α, gray wolf β, gray wolf δ and the current gray wolf ω, and Represent the current positions of gray wolf α, gray wolf β and gray wolf δ respectively, and is a random vector, is the current position vector of the gray wolf ω;

[0041]

[0042] in, and They represent the step length and direction of the gray wolf ω moving toward the gray wolf α, gray wolf β, and gray wolf δ, respectively. and are coefficient vectors;

[0043] A35, Gaussian mutation operation:

[0044] Set high and low mutation probability p m , for each gray wolf individual Y i For each dimension j, generate a random number rand between [0,1] j , when rand j ≥p m When rand j <p m When , Gaussian mutation is performed on the dimension, and the mutation formula is:

[0045]

[0046] in, Y is the gray wolf iThe value of the j dimension after Gaussian mutation, Y i,j Y is the gray wolf i The original j-dimensional value of The mean is 0 and the standard deviation is Normally distributed random number, ub j is the upper bound of the search space in the jth dimension, lb j is the lower bound of the search space for the jth dimension;

[0047] A36. Termination condition judgment: terminate when the maximum number of iterations is reached.

[0048] As a further improvement of the technical solution, the root mean square of the estimation error is used as the fitness function in A32, and the corresponding formula is:

[0049]

[0050] Among them, RMSE is the mean square error, m is the time index, which represents the value at different time points, representing the different sampling moments of the audio signal, M is the total number of sampling points, and x m is the real signal value at time m, is the estimated value of the signal at time m.

[0051] As a further improvement of the technical solution, the audio processing module performs segmentation according to energy. p (q)(q=0,1,…,L-1), its short-time energy E p The calculation formula is:

[0052]

[0053] Where q represents the sample point number in the frame, and L represents the frame length;

[0054] Set the energy threshold E 0 , when E p >E 0 When E p ≤E 0 , the effective part is considered to be ended.

[0055] As a further improvement of the technical solution, the video processing module performs image enhancement based on an improved single-scale Retinex algorithm, and the specific steps are as follows:

[0056] B1. Divide each frame of the video into blocks, and divide the input image I(x, y) into M×N non-overlapping windows C ij(i=1,2,…,M;j=1,2,…,N), I(x,y) is the pixel value of the input image at the coordinate (x,y), let the width of each window be a, the height be b, and the number of pixels in the window be L mn , m and n represent the coordinates of the pixels within the window;

[0057] B2. Calculate each window C ij Variance V ij , the corresponding formula is:

[0058]

[0059] Among them, G ij For window C ij The number of pixels inside, is the average value of the inner pixels;

[0060] B3. Determine the adaptive scale parameter, then:

[0061] σ ij =s-dV ij

[0062] Among them, σ ij is the scale parameter, s and d are set parameters;

[0063] B4. Logarithmic transformation: perform logarithmic transformation on the input image I(x,y). The transformation formula is: z(x,y)=logI(x,y), where z(x,y) is the pixel value of the image at the coordinate (x,y) after logarithmic transformation. The multiplication operation is converted into an addition operation to facilitate subsequent processing.

[0064] B5. Estimate the illumination component. Use Gaussian function to estimate the illumination component. For each window C ij ,have:

[0065] L ij (x,y)=F(x,y)*G(x,y;σ ij )

[0066]

[0067] Among them, L ij (x, y) is the illumination component, G(x, y; σ ij ) is the Gaussian function, F(x,y) is the image z(x,y) after logarithmic transformation, * is the convolution operation;

[0068] B6. Calculate the reflection component and restore the image. For each window C ij ,have:

[0069] r ij (x,y)=z(x,y)-Lij (x,y)

[0070] R ij (x,y)=exp(r ij (x,y)

[0071] Among them, r ij (x, y) is the reflection component, R ij (x,y) is the restored image.

[0072] As a further improvement of the present technical solution, the video processing module extracts features based on the SIFT algorithm, matches corresponding feature points between different images, and estimates the geometric transformation relationship between the images based on the matching points to complete the stitching.

[0073] As a further improvement of the present technical solution, the data analysis module adopts a dual-stream neural network architecture, and the two branches are used to process audio data and video data respectively. The video data branch is based on the architecture of a convolutional neural network to extract spatial features and temporal features in video frames, and the audio data branch is based on the architecture of a recurrent neural network to extract temporal features and spectral features of audio signals. The output layer of the network architecture weightedly fuses the fully connected layers of the two branches, and the output layer outputs the classification results, using the cross entropy loss function as the loss function.

[0074] The present invention also provides a method for monitoring and analyzing audio, video and image fusion based on deep learning in a track scene. According to the above-mentioned system for monitoring and analyzing audio, video and image fusion based on deep learning in a track scene, the method comprises the following steps:

[0075] S1, obtain audio data and video data;

[0076] S2, performing noise removal processing on the acquired audio data to obtain optimized audio data;

[0077] S3, dividing the optimized audio data according to energy to obtain multiple audio segments;

[0078] S4, performing signal enhancement processing on the acquired video data to obtain optimized video data;

[0079] S5, splicing the optimized video data;

[0080] S6. Make the spliced ​​video data correspond to each audio segment in time, and use the audio segment and the spliced ​​video data as input data to train a two-stream neural network, and output the classification result.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows: in the audio, video and image fusion monitoring and analysis system and method based on deep learning in the track scene, noise is removed from the audio data, which improves the ability to extract real information from the audio signal, and then the audio is segmented according to energy to accurately identify the effective part of the audio signal for further processing, and the video data is enhanced to improve possible problems such as uneven lighting and low contrast, and a spliced ​​panoramic view is obtained for comprehensive monitoring, and finally the audio data and video data are fused to improve the accuracy of classifying various events in the track scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 is a system structure diagram of the present invention;

[0083] Figure 2 is a flow chart of the method of the present invention;

[0084] The meanings of the marks in the figure are as follows: 100, data acquisition module; 200, audio processing module; 300, video processing module; 400, data analysis module. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] like Figure 1 As shown, this embodiment provides an audio, video and image fusion monitoring and analysis system based on deep learning in a track scenario, including:

[0087] Data collection module 100: used to collect audio data and video data in track scenes;

[0088] The audio processing module 200 is used to receive the collected audio data, remove noise and segment the audio data to obtain multiple audio segments;

[0089] Video processing module 300: used to receive the collected video data, and perform signal enhancement and splicing on the video data;

[0090] Data analysis module 400: used to make multiple audio segments and the spliced ​​video consistent in time, and complete the classification function based on the neural network.

[0091] The data acquisition module 100 includes several cameras, which are installed in different locations of the station, including entrances and exits, platforms, waiting areas, etc. The cameras are equipped with microphones to comprehensively collect audio and video data in track scenes to ensure that there are no blind spots in monitoring and provide sufficient raw data sources for subsequent analysis and processing.

[0092] The specific steps of removing noise from audio data in the audio processing module 200 are:

[0093] A1. Assuming that the audio signal is a linear dynamic system, its state equation is:

[0094] x k =Ax k-1 +w k-1

[0095] Among them, x k is the system state at time k, where is the amplitude of the audio signal, A is the state transfer matrix, which describes the state change law of the system from time k-1 to time k, x k-1 is the system state at time k-1, w k-1 is the process noise at time k-1, which is a Gaussian white noise vector with a mean of zero and a covariance matrix of Q. It is used to represent the uncertainty in the system dynamic model, such as the slight changes in the audio signal due to various unknown factors;

[0096] Then its observation equation is:

[0097] y k =Hx k +v k

[0098] Among them, y k is the observed value at time k, H is the observation matrix, which maps the system state to the observation space, v k is the observation noise at time k, which is a Gaussian white noise with a mean of zero and a covariance matrix of R, representing the uncertainty in the observation process, such as the noise introduced by the microphone;

[0099] A2. Calculate the state prediction value and prediction error covariance. The corresponding formula is:

[0100]

[0101] in, is the prior state estimate at time k, that is, the predicted value, is the posterior state estimate at time k-1, which is the optimal estimate after update. is the prior estimation error covariance matrix, which measures the uncertainty of the predicted state, is the posterior estimation error covariance matrix at time k-1, A T is the transposed matrix of the state transfer matrix A;

[0102] A3. Optimize the parameters in the Kalman filter algorithm based on the improved grey wolf algorithm;

[0103] A31, population initialization, set the population size, spatial dimension and maximum number of iterations, where the position vector of each gray wolf represents the parameters to be optimized in the process noise and observation noise in turn. Taking the amplitude of dual-channel audio data as an example, at this time, the process noise covariance matrix Observation noise covariance matrix is the process noise variance of the left channel amplitude, is the amplitude process noise variance of the right channel, is the left channel observation noise variance, is the variance of the right channel observation noise, ρ is the correlation coefficient of the left and right channel observation noise, and its value range is [-1,1]. When ρ = 0, the left and right channel observation noises are independent of each other. When ρ = 1, the left and right channel observation noises are completely positively correlated. When ρ = -1, the left and right channel observation noises are completely negatively correlated. The corresponding position vector of the gray wolf is

[0104] A32. Calculate the fitness. Calculate the fitness of each gray wolf separately. Define the top three best gray wolves as α, β and δ respectively. The rest of the gray wolves are defined as ω. The root mean square of the estimated error is used as the fitness function. The corresponding formula is:

[0105]

[0106] Among them, RMSE is the mean square error, m is the time index, which represents the value at different time points, representing the different sampling moments of the audio signal, M is the total number of sampling points, and x m is the real signal value at time m, is the estimated value of the signal at time m;

[0107] A33. Surround the prey:

[0108]

[0109]

[0110] Where D represents the distance between the gray wolf and its prey. and are all coefficient vectors, t represents the current number of iterations, represents the position vector of the prey at the tth iteration, represents the position vector of the gray wolf at the t+1th iteration, is the convergence factor, T is the maximum number of iterations, and The modulus of takes a random number between [0,1], and introduces an adaptive convergence factor. In the early stage, the value of the convergence factor is large, and the gray wolf individual can explore possible optimal solutions in a larger parameter space to avoid falling into the local optimum. As the iteration proceeds, the value of the convergence factor gradually decreases, and the search range of the gray wolf individual narrows. It is more inclined to conduct local searches near the currently found better solution, which helps to fine-tune the parameters in the determined potential area and improve the accuracy of the solution.

[0111] A34. Hunting prey:

[0112]

[0113] Among them, D α , D β and D δ Respectively represent the distances between gray wolf α, gray wolf β, gray wolf δ and the current gray wolf ω, and Represent the current positions of gray wolf α, gray wolf β and gray wolf δ respectively, and is a random vector, is the current position vector of the gray wolf ω;

[0114]

[0115] in, and They represent the step length and direction of the gray wolf ω moving toward the gray wolf α, gray wolf β, and gray wolf δ, respectively. and are coefficient vectors;

[0116] A35, Gaussian mutation operation:

[0117] Set high and low mutation probability p m , for each gray wolf individual Y i For each dimension j, generate a random number rand between [0,1] j , when rand j ≥p m When rand j <p m When , Gaussian mutation is performed on the dimension, and the mutation formula is:

[0118]

[0119] in, Y is the gray wolf iThe value of the j dimension after Gaussian mutation, Y i,j Y is the gray wolf i The original j-dimensional value of The mean is 0 and the standard deviation is Normally distributed random number, ub j is the upper bound of the search space in the jth dimension, lb j is the lower bound of the j-th dimension search space. It changes the position of individual gray wolves to a certain extent, making the individuals in the population have more diverse characteristics, allowing the algorithm to jump out of possible local optimal solutions and explore new parameter combinations.

[0120] A36. Termination condition judgment: terminate when the maximum number of iterations is reached.

[0121] A4. Calculate the Kalman gain. The corresponding formula is:

[0122]

[0123] Among them, K k is the Kalman gain matrix at time k, which determines the degree of trust in the observed value when updating the estimated value, H T is the transposed matrix of the observation matrix H;

[0124] A5. Update state estimation and error covariance. The corresponding formula is:

[0125]

[0126] in, is the posterior state estimate at time k, which is the optimal estimate after combining the predicted value and the observed value, I is the unit matrix, is the updated posterior estimation error covariance matrix for the next iteration;

[0127] The Kalman filter algorithm based on the improved Grey Wolf algorithm is used to remove audio noise, which effectively copes with the complex noise interference in the track environment. By accurately optimizing the parameters in the Kalman filter algorithm, the ability to extract real information from the audio signal is improved, making the processed audio clearer and reducing the risk of misjudgment due to noise. In the noisy station environment, useful voice commands or abnormal sound signals can be better separated.

[0128] The audio processing module 200 performs segmentation according to energy. p (q)(q=0,1,…,L-1), its short-time energy E p The calculation formula is:

[0129]

[0130] Where q represents the sample point number in the frame, and L represents the frame length;

[0131] Set the energy threshold E 0 , when E p >E 0 When E p ≤E 0 When the effective part is considered to end, audio segmentation is performed based on energy, which can accurately identify the effective part of the audio signal and divide the audio into multiple meaningful audio segments, which is helpful for the subsequent classification and analysis of the audio content. Multiple thresholds can be set to effectively separate different types of audio such as train running sound, broadcast sound, and passenger conversation sound for further processing.

[0132] The video processing module 300 performs image enhancement based on the improved single-scale Retinex algorithm, and the specific steps are as follows:

[0133] B1. Divide each frame of the video into blocks, and divide the input image I(x, y) into M×N non-overlapping windows C ij (i=1,2,…,M;j=1,2,…,N), I(x,y) is the pixel value of the input image at the coordinate (x,y), let the width of each window be a, the height be b, and the number of pixels in the window be L mn , m and n represent the coordinates of the pixels within the window;

[0134] B2. Calculate each window C ij Variance V ij , the corresponding formula is:

[0135]

[0136] Among them, G ij For window C ij The number of pixels inside, is the average value of the inner pixels;

[0137] B3. Determine the adaptive scale parameter, then:

[0138] σ ij =s-dV ij

[0139] Among them, σ ij is the scale parameter, s and d are both set parameters, s determines the upper limit of the scale parameter, d controls the rate at which the scale parameter changes with the local variance, and can adaptively adjust the brightness and contrast according to the local features of the image;

[0140] B4. Logarithmic transformation: perform logarithmic transformation on the input image I(x,y). The transformation formula is: z(x,y)=logI(x,y), where z(x,y) is the pixel value of the image at the coordinate (x,y) after logarithmic transformation. The multiplication operation is converted into an addition operation to facilitate subsequent processing.

[0141] B5. Estimate the illumination component. Use Gaussian function to estimate the illumination component. For each window C ij ,have:

[0142] L ij (x,y)=F(x,y)*G(x,y;σ ij )

[0143]

[0144] Among them, L ij (x, y) is the illumination component, G(x, y; σ ij ) is the Gaussian function, F(x,y) is the image z(x,y) after logarithmic transformation, * is the convolution operation;

[0145] B6. Calculate the reflection component and restore the image. For each window C ij ,have:

[0146] r ij (x,y)=z(x,y)-L ij (x,y)

[0147] R ij (x,y)=exp(r ij (x,y)

[0148] Among them, r ij (x, y) is the reflection component, R ij (x,y) is the restored image.

[0149] The video processing module 300 extracts features based on the SIFT algorithm, matches corresponding feature points between different images, and estimates the geometric transformation relationship between images based on the matching points to complete the stitching. It can find accurate matching points between video images at different perspectives or positions, thereby achieving precise stitching.

[0150] The data analysis module 400 adopts a dual-stream neural network architecture, with two branches used to process audio data and video data respectively. The video data branch is based on the architecture of a convolutional neural network, which extracts spatial features and temporal features in video frames, and can well capture information such as the shape and motion trajectory of objects in the video. The audio data branch is based on the architecture of a recurrent neural network, which extracts temporal features and spectral features of audio signals, and effectively analyzes the changing patterns of audio signals. The output layer of the network architecture weightedly fuses the fully connected layers of the two branches, and the output layer outputs classification results, such as distinguishing normal passenger flow from abnormal congestion, passengers falling, items falling on the track, identifying the running status of trains, etc. The cross entropy loss function is used as the loss function, which improves the accuracy of classifying various events in track scenes.

[0151] like Figure 2 As shown, this embodiment also provides an audio, video and image fusion monitoring and analysis method based on deep learning in a track scenario. According to the above-mentioned audio, video and image fusion monitoring and analysis system based on deep learning in a track scenario, the following steps are included:

[0152] S1, obtain audio data and video data;

[0153] S2, performing noise removal processing on the acquired audio data to obtain optimized audio data;

[0154] S3, dividing the optimized audio data according to energy to obtain multiple audio segments;

[0155] S4, performing signal enhancement processing on the acquired video data to obtain optimized video data;

[0156] S5, splicing the optimized video data;

[0157] S6. Make the spliced ​​video data correspond to each audio segment in time, and use the audio segment and the spliced ​​video data as input data to train a two-stream neural network, and output the classification result.

[0158] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene is characterized by: include: Data collection module (100): used to collect audio data and video data in track scenes; An audio processing module (200): used for receiving the collected audio data, removing noise and segmenting the audio data to obtain a plurality of audio segments; Video processing module (300): used to receive collected video data, and perform signal enhancement and splicing on the video data; Data analysis module (400): used to make multiple audio segments and the spliced ​​video consistent in time, and complete the classification function based on the neural network.

2. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1 is characterized by: The data collection module (100) comprises a plurality of cameras, which are installed at different positions of the station and are equipped with microphones.

3. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1 is characterized in that: The specific steps of removing noise from audio data in the audio processing module (200) are: A1. Assuming that the audio signal is a linear dynamic system, its state equation is: x k =A x k -1 +w k-1 Among them, x k is the system state at time k, A is the state transfer matrix, x k-1 is the system state at time k-1, w k-1 is the process noise at time k-1, which is a Gaussian white noise vector with zero mean and covariance matrix Q; Then its observation equation is: y k =Hx k +v k Among them, y k is the observed value at time k, H is the observation matrix, v k is the observation noise at time k, which is a Gaussian white noise with zero mean and covariance matrix R; A2. Calculate the state prediction value and prediction error covariance. The corresponding formula is: in, is the prior state estimate at time k, that is, the predicted value, is the posterior state estimate at time k-1, which is the optimal estimate after update. is the prior estimation error covariance matrix, is the posterior estimation error covariance matrix at time k-1, A T is the transposed matrix of the state transfer matrix A; A3. Optimize the parameters in the Kalman filter algorithm based on the improved grey wolf algorithm; A4. Calculate the Kalman gain. The corresponding formula is: Among them, K k is the Kalman gain matrix at time k, H T is the transposed matrix of the observation matrix H; A5. Update state estimation and error covariance. The corresponding formula is: in, is the posterior state estimate at time k, I is the unit matrix, is the updated posterior estimation error covariance matrix for the next iteration.

4. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 3 is characterized in that: The specific steps of optimizing the parameters in the Kalman filter based on the improved grey wolf algorithm in A3 are: A31, population initialization, setting the population size, spatial dimension and maximum number of iterations, where the position vector of each gray wolf represents the parameters to be optimized in the process noise and observation noise respectively; A32. Calculate the fitness of each gray wolf. Define the top three best gray wolves as α, β, and δ, and the rest of the gray wolves as ω. A33. Surround the prey: Where D represents the distance between the gray wolf and its prey. and are all coefficient vectors, t represents the current number of iterations, represents the position vector of the prey at the tth iteration, represents the position vector of the gray wolf at the t+1th iteration, is the convergence factor, T is the maximum number of iterations, and The modulus of gets a random number between [0,1]; A34. Hunting prey: Among them, D α , D β and D δ Respectively represent the distances between gray wolf α, gray wolf β, gray wolf δ and the current gray wolf ω, and Represent the current positions of gray wolf α, gray wolf β and gray wolf δ respectively, and is a random vector, is the current position vector of the gray wolf ω; in, and They represent the step length and direction of the gray wolf ω moving toward the gray wolf α, gray wolf β, and gray wolf δ, respectively. and are coefficient vectors; A35, Gaussian mutation operation: Set high and low mutation probability p m , for each gray wolf individual Y i For each dimension j, generate a random number rand between [0,1] j , when rand j ≥p m When rand j <p m When , Gaussian mutation is performed on the dimension, and the mutation formula is: in, Y is the gray wolf i The value of the j dimension after Gaussian mutation, Y i,j Y is the gray wolf i The original j-dimensional value of The mean is 0 and the standard deviation is Normally distributed random number, ub j is the upper bound of the search space in the jth dimension, lb j is the lower bound of the search space for the jth dimension; A36. Termination condition judgment: terminate when the maximum number of iterations is reached.

5. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 4 is characterized in that: In A32, the root mean square of the estimation error is used as the fitness function, and the corresponding formula is: Among them, RMSE is the mean square error, m is the time index, which represents the value at different time points, representing the different sampling moments of the audio signal, M is the total number of sampling points, and x m is the real signal value at time m, is the estimated value of the signal at time m.

6. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1 is characterized in that: The audio processing module (200) performs segmentation according to energy. For the p-th frame signal x p (q)(q=0,1,…,L-1), its short-time energy E p The calculation formula is: Where q represents the sample point number in the frame, and L represents the frame length; Set the energy threshold E0, when E p >E0, the effective part of the audio signal is considered to begin. p When ≤E0, the effective part is considered to be ended.

7. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1 is characterized in that: The video processing module (300) performs image enhancement based on an improved single-scale Retinex algorithm, and the specific steps are as follows: B1. Divide each frame of the video into blocks, and divide the input image I(x, y) into M×N non-overlapping windows C ij (i=1,2,…,M;j=1,2,…,N), I(x,y) is the pixel value of the input image at the coordinate (x,y), let the width of each window be a, the height be b, and the number of pixels in the window be L mn , m and n represent the coordinates of the pixels within the window; B2. Calculate each window C ij Variance V ij , the corresponding formula is: Among them, G ij For window C ij The number of pixels inside, is the average value of the inner pixels; B3. Determine the adaptive scale parameter, then: s ij =s-dV ij Among them, σ ij is the scale parameter, s and d are set parameters; B4. Logarithmic transformation: perform logarithmic transformation on the input image I(x,y). The transformation formula is: z(x,y)=logI(x,y), where z(x,y) is the pixel value of the image at the coordinate (x,y) after logarithmic transformation. The multiplication operation is converted into an addition operation to facilitate subsequent processing. B5. Estimate the illumination component. Use Gaussian function to estimate the illumination component. For each window C ij ,have: L ij (x,y)=F(x,y)*G(x,y;σ ij ) Among them, L ij (x, y) is the illumination component, G(x, y; σ ij ) is the Gaussian function, F(x,y) is the image z(x,y) after logarithmic transformation, * is the convolution operation; B6. Calculate the reflection component and restore the image. For each window C ij ,have: r ij (x,y)=z(x,y)-L ij (x,y) R ij (x,y)=exp(r ij (x,y)) Among them, r ij (x, y) is the reflection component, R ij (x,y) is the restored image.

8. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1, characterized in that: The video processing module (300) extracts features based on the SIFT algorithm, matches corresponding feature points between different images, and estimates the geometric transformation relationship between the images based on the matching points to complete the splicing.

9. The audio, video and image fusion monitoring and analysis system based on deep learning in the track scene according to claim 1, characterized in that: The data analysis module (400) adopts a dual-stream neural network architecture, wherein the two branches are used to process audio data and video data respectively. The video data branch is based on a convolutional neural network architecture to extract spatial features and temporal features in video frames, and the audio data branch is based on a recurrent neural network architecture to extract temporal features and spectral features of audio signals. The output layer of the network architecture weightedly fuses the fully connected layers of the two branches, and the output layer outputs the classification result, using a cross entropy loss function as a loss function.

10. A method for monitoring and analyzing audio, video and image fusion based on deep learning in a track scene, according to a system for monitoring and analyzing audio, video and image fusion based on deep learning in a track scene according to any one of claims 1 to 9, characterized in that: The steps include: S1, obtain audio data and video data; S2, performing noise removal processing on the acquired audio data to obtain optimized audio data; S3, dividing the optimized audio data according to energy to obtain multiple audio segments; S4, performing signal enhancement processing on the acquired video data to obtain optimized video data; S5, splicing the optimized video data; S6. Make the spliced ​​video data correspond to each audio segment in time, and use the audio segment and the spliced ​​video data as input data to train a two-stream neural network, and output the classification result.

Images