Mobile audio-visual evidence collection management system

Abstract

The present application belongs to the technical field of digital forensics, and discloses a mobile audio-video evidence collection and management system, which comprises an evidence capturing module, a data processing module, an audio-video evidence matching module, a content matching degree analysis module and an abnormality evaluation module.The evidence capturing module is used for collecting audio-video comprehensive data and environmental monitoring data.The data processing module is used for preliminarily processing the comprehensive data to obtain time-synchronized audio-video feature data sets.The audio-video evidence matching module is used for fusing the time-synchronized audio-video feature data sets and the environmental monitoring data to obtain comprehensive feature data sets.The content matching degree analysis module is used for training an audio-video evidence matching model according to the comprehensive feature data sets and predicting the comprehensive content matching degree based on the audio-video evidence matching model.The abnormality evaluation module is used for judging whether the audio-video comprehensive data is abnormal based on the comprehensive content matching degree, and sending an alarm information to a management terminal if the audio-video comprehensive data is abnormal.The present application improves the convenience and efficiency of evidence use and provides strong support for subsequent investigation and processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital forensics, and more particularly, to a mobile audio-video evidence collection and management system. BACKGROUND

[0002] The patent with the patent publication number CN105206059A discloses a traffic accident scene evidence collection guidance system and method. The system includes: a user uses a mobile terminal to complete the missing information in the accident report. The accident report includes accident ID, location (GPS information), time, accident type, number of participants, and license plate number. After generating the accident report, the system starts the accident evidence collection module, calls the corresponding evidence collection sub-module according to the accident type and the number of participants in the accident report, and guides the user to take evidence photos. After the photos are taken, the user can record additional audio according to the on-site situation. Finally, the accident report, images, and audio are packaged as an evidence compression package, which is sent to a server and stored by the server for handling.

[0003] The existing mobile audio-video evidence collection and management system mainly has the following problems:

[0004] The existing method does not fully utilize the similarity measurement of non-local mean filtering, which can easily lead to important texture and edge information being destroyed in the denoising process, weakening the authenticity of the video picture. The weight decay parameter is not dynamically adjusted, which can show an excessively strong or weak denoising effect in different frames of the video, and cannot be flexibly optimized according to the noise intensity difference. The existing technology does not process the video jitter problem, which can cause unstable pictures and affect the viewing experience, especially in scenes that require high precision such as law enforcement or justice. The stability and clarity of the video picture cannot be enhanced, which can produce trailing or blur in dynamic scenes, damaging the visual experience.

[0005] The video image frame and the audio signal frame cannot be completely aligned in the time dimension, resulting in a loss of synchronization of the audio-video data and affecting the accuracy of subsequent processing and analysis. Without dynamic window adjustment design, the algorithm cannot adapt to the differences in video frame and audio frame characteristics in complex scenes, such as rapidly changing background noise or scene switching. The synchronization algorithm without dynamic adjustment of the window size can lose the accurate capture of similarity in scenes with large feature changes, resulting in an increase in time alignment error.

[0006] The algorithm cannot rely on a single initial hyperparameter combination, and the search range of the algorithm is limited, and the solution space is not fully covered; a linear decreasing factor is not introduced to dynamically adjust the search step, and the initial step may not be sufficient to effectively explore the vast solution space; the position and step of the grey wolf individual are not dynamically adjusted, and it is difficult to adapt to complex hyperparameter space and complex model optimization requirements; the algorithm with fixed or lack of dynamic adjustment strategy cannot adjust the strategy according to the real-time state of the search process, and the adaptability is significantly reduced; due to insufficient optimization of the search step and solution space exploration, the final obtained hyperparameter combination may deviate from the global optimum, affecting the model performance.

[0007] In view of this, the present application provides a mobile audio-video evidence collection management system to solve the above problems. SUMMARY

[0008] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: a mobile audio-video evidence collection management system, comprising:

[0009] An evidence capture module is configured to collect audio-video integrated data and environmental monitoring data.

[0010] A data processing module is configured to preliminarily process the audio-video integrated data to obtain a time-synchronized audio-video feature data set.

[0011] A multi-modal data fusion module is configured to fuse the time-synchronized audio-video feature data set and the environmental monitoring data to obtain an integrated feature data set.

[0012] A content matching degree analysis module is configured to train an audio-video evidence matching model based on the integrated feature data set, and predict an integrated content matching degree based on the audio-video evidence matching model.

[0013] An abnormality evaluation module is configured to determine whether the audio-video integrated data is abnormal based on the integrated content matching degree, and send an alarm information to a management terminal if the audio-video integrated data is abnormal. The modules are connected through wired and / or wireless means.

[0014] Further, the audio-video integrated data includes video evidence data and audio evidence data; the video evidence data includes a video image frame sequence, a video frame timestamp, a video resolution, a video resolution, and a video compression encoding; the audio evidence data includes an audio signal frame sequence, an audio signal waveform, an audio sampling rate, an audio signal timestamp, and a sound frequency feature; and the environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay, and network signal strength.

[0015] Further, the method for preliminarily processing the audio-video integrated data to obtain a time-synchronized audio-video feature data set comprises:

[0016] The collected video evidence data and audio evidence data are preliminarily processed to obtain a video evidence feature data set and an audio evidence feature data set; a flow method is used to estimate the motion between each adjacent frame in the video image frame sequence in the video evidence data; according to the motion between each adjacent frame, a translation motion model is used to fit the overall motion of the camera, and a geometric transformation is used to adjust the video image frame sequence according to the translation motion model, and each frame of image is de-jittered; a non-local mean filter is used to denoise the de-jittered video evidence data to obtain the video evidence feature data set;

[0017] The DBSCAN algorithm is used to identify the outliers existing in the audio evidence data, and the identified outliers are removed to obtain the audio evidence feature data set; the video evidence feature data set and the audio evidence feature data set are subjected to standard deviation normalization processing to convert to standard normal distribution, and the normalized video evidence feature data set and the normalized audio evidence feature data set are obtained;

[0018] The normalized video evidence feature data set and the normalized audio evidence feature data set are synchronized by using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature data set.

[0019] Further, the method for denoising the de-jittered video evidence data by using the non-local mean filter comprises:

[0020] Each frame of image in the video image frame sequence in the video evidence data is extracted, for each frame of image in the video image frame sequence, a frame of image is randomly selected as a current frame, and the non-local mean filter is executed, for each pixel in the current frame , the similarity between the pixel and all remaining pixels is calculated, the original pixel value of the pixel is taken as the pixel value of the pixel , the original pixel value of the remaining pixel is taken as the pixel value of the remaining pixel , and the parameter controlling the weight decay is taken as

[0021] The parameter controlling the weight decay is limited by a weight decay parameter limiting formula, and the weight decay parameter limiting formula is: ; wherein, is the limited parameter controlling the weight decay; is the total number of pixels of the image; is a scale factor adjusting the range of the parameter controlling the weight decay;

[0022] The weighted average is used to calculate the pixel value after denoising, and the specific mathematical formula is: ; wherein, is the search window of the current frame image, including pixels surrounding pixels;

[0023] Use the denoised pixel values To update all pixels in the current frame image until all pixels are processed; repeat until all frame images are denoised.

[0024] Furthermore, the method of synchronizing the normalized video evidence feature dataset and the audio evidence feature dataset using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset includes:

[0025] S51. Express the video image frame sequence in the normalized video evidence feature dataset as: ;in, For video images Feature vector of the frame; is the total number of frames in the video image frame sequence;

[0026] The normalized audio signal frame sequence in the audio evidence feature dataset is represented as ;in, The first frame of the audio signal feature vectors; is the total number of frames in the audio signal frame sequence;

[0027] S52. Construct a cross-correlation function to measure the similarity between the video image frame sequence and the audio signal frame sequence at different time delays through the cross-correlation function; find the time alignment between the video image frame sequence and the audio signal frame sequence by maximizing the cross-correlation;

[0028] The cross-correlation function is: ;in, The video image frame sequence and the audio signal frame sequence are delayed at different times Weighted similarity under ; is the first frame in the audio signal sequence Frame, indicating that the audio signal has been delayed by time The next frame; is a time delay parameter, which represents the time difference between the video image frame sequence and the audio signal frame sequence; It is the minimum length that can be processed after the video image frame sequence and the audio signal frame sequence are time synchronized; For the Frames are delayed The window size below; is the index of the video image frame sequence and the audio signal frame;

[0029] S53, for Frames are delayed The window size is dynamically adjusted by comparing the video image frame sequence and the audio signal frame sequence at different time delays. The weighted similarity under the condition of , select the optimal time delay corresponding to the maximum weighted similarity , according to the best time delay , adjust the audio signal frame to synchronize it with the video image frame; if , the audio signal moves forward frame; if , the audio signal moves backward frame; if , no time adjustment is made; after adjusting the video image frame sequence and the audio signal frame sequence, a time-synchronized video and audio feature dataset is obtained.

[0030] Furthermore, the Frames are delayed The following methods are used to dynamically adjust the window size:

[0031] By adjusting the formula of dynamic window Frames are delayed The window size under the dynamic adjustment design; the dynamic window adjustment formula is: ;in, After dynamic adjustment Frames are delayed The window size below; is the rate of change of video image frames; is the rate of change of audio signal frames; is the video impact factor; is the audio impact factor; is the scaling factor.

[0032] Furthermore, the method of fusing the time-synchronized audio and video feature dataset and the environmental monitoring data to obtain a comprehensive feature dataset includes:

[0033] The outliers in the environmental monitoring data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the environmental monitoring feature data set; the environmental monitoring feature data set is normalized by standard deviation and converted into a standard normal distribution to obtain the normalized environmental monitoring feature data set;

[0034] The time-aligned audio and video feature dataset and the normalized environmental monitoring feature dataset are fused through a weighted formula to obtain a comprehensive feature dataset; the time-aligned audio and video feature dataset is recorded as , the normalized environmental monitoring feature dataset is recorded as ;

[0035] The weighting formula is: ; wherein, is a comprehensive feature dataset; is a weight coefficient of the time-aligned audio-visual feature dataset; is a weight coefficient of the normalized environmental monitoring feature dataset.

[0036] Further, the training method of the audio-visual evidence matching model comprises:

[0037] Divide the dataset into a training set, a validation set, and a test set, train the model, and evaluate the model performance; build an audio-visual evidence matching model, and each sample set includes a historical comprehensive feature dataset and a corresponding comprehensive content matching degree;

[0038] The audio-visual evidence matching model includes an input layer, a hidden layer, and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used to input the historical comprehensive feature dataset; the output layer of the model is used to output the comprehensive content matching degree; and the audio-visual evidence matching model is a multi-modal neural network model;

[0039] The training set is used to train the audio-visual evidence matching model, the model parameters are updated through a back propagation algorithm and a gradient descent method to minimize a loss function; the validation set is used to evaluate the performance of the audio-visual evidence matching model by calculating a determination coefficient, and the hyperparameters of the model are optimized;

[0040] The Adam optimization algorithm is selected as an optimizer, the hyperparameters of the model are adjusted until the performance is no longer improved or a preset number of iterations is reached; the test set is used to evaluate the performance of the model in a prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature dataset to obtain a comprehensive content matching degree.

[0041] Further, the method of optimizing the hyperparameters of the model comprises:

[0042] S91, a set of initial hyperparameter combinations is randomly generated, each hyperparameter combination corresponds to a grey wolf individual; the position of each grey wolf individual is a dimensional vector; wherein, is the number of hyperparameters that need to be optimized; there are grey wolf individuals, each grey wolf individual contains hyperparameters, and the position of the i-th grey wolf individual is denoted as: ; wherein, is the position of the i-th grey wolf individual; is the position of the i-th grey wolf individual; is the position of the i-th grey wolf individual; is the position of the i-th grey wolf individual. the first hyperparameter value of the gray wolf individual; is an index of the gray wolf individual;

[0043] S92, using a mean square error loss function to measure the fitness value of each hyperparameter combination, each gray wolf individual calculating the fitness value according to the mean square error loss function, the smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual; according to the fitness value of the gray wolf individual, selecting the gray wolf individual with the highest fitness value ranking as the alpha wolf, selecting the gray wolf individual with the second highest fitness value ranking as the beta wolf, and selecting the gray wolf individual with the third highest fitness value ranking as the delta wolf;

[0044] S93, each gray wolf individual adjusting its own position by approaching the alpha wolf, the beta wolf, and the delta wolf, exploring and updating the hyperparameter combination, the adjustment of the position of the gray wolf individual complying with a gray wolf position update formula; the gray wolf position update formula is: ; wherein, the first hyperparameter position of the th generation of the gray wolf individual, that is, the hyperparameter combination; is the first hyperparameter position of the th generation of the gray wolf individual; , and represent random factors related to position updating, controlling the step size and search direction of the gray wolf individual; is the distance between the th gray wolf individual and the alpha wolf; is the current iteration number; is the next iteration number;

[0045] The calculation formula of the random factor is: ; wherein, is a linear decreasing factor; is a random number controlling the randomness of the search direction, ranging from ;

[0046] The calculation formula of the random factor is: ; wherein, is a random number controlling the search step size of the gray wolf individual, ranging from ;

[0047] The calculation formula of the random factor The calculation formula of is: ; wherein, is a random number for controlling the search direction of the grey wolf individual, and the range is ;

[0048] The linear decreasing factor is dynamically adjusted by a linear decreasing factor adjustment formula; the linear decreasing factor adjustment formula is: ; wherein, is the dynamically adjusted linear decreasing factor; is the maximum value of the linear decreasing factor; is the maximum iteration number; is a parameter for controlling the decreasing rate;

[0049] S94, the position of each grey wolf individual is updated one by one by a grey wolf position updating formula, so as to adjust the hyperparameter combination; in each iteration, the fitness value of each grey wolf individual is recalculated, if the fitness value of a certain grey wolf individual is higher, the position of the grey wolf individual is updated. wolf, wolf and wolf, a preset maximum iteration number is stopped when the iteration number reaches the maximum iteration number; in the final iteration, the position of the wolf is selected as the optimal hyperparameter combination, that is, the performance of the model is optimal under the hyperparameter combination.

[0050] Further, the method for judging whether the audio-video comprehensive data is abnormal based on the comprehensive content matching degree comprises:

[0051] If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is judged that the audio-video comprehensive data is abnormal;

[0052] If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is judged that the audio-video comprehensive data is not abnormal.

[0053] The technical effects and advantages of the mobile video and audio evidence collection and management system are as follows:

[0054] The present application can effectively remove the noise in the video while preserving the edge and texture details of the original image by calculating the similarity between pixels through non-local mean filtering, and improving the image quality. It is suitable for video processing in complex scenes, such as multiple noise interference in evidence; removing video jitter and noise, enhancing the clarity and stability of the video picture, making it more authoritative and credible in law enforcement, judicial or research applications; using the weight decay parameter limiting formula to dynamically adjust and control the weight decay parameter, so that the algorithm can adapt flexibly according to the characteristics of different frames of images in the video, further optimizing the denoising effect;

[0055] ​The time delay of the video image frame and the audio signal frame can be accurately calculated through the cross-correlation function, so that high-precision time synchronization is realized, and the consistency of the video and audio data in the time dimension is ensured; the dynamic window size adjustment design is adopted, the synchronization strategy is flexibly adjusted according to the real-time changing characteristic similarity, and the adaptability to complex scenes is improved; the window size can be dynamically adjusted according to the change rate of the video frame and the audio frame, so that the synchronization algorithm is more suitable for the dynamic change of the data characteristics, and the precision of time alignment is improved; through the dynamic adjustment of the window size, the real similarity between the video and audio frames can be more accurately captured; the high-precision data after time synchronization reduces the false alarm and missed alarm in the abnormal detection, and improves the reliability of the system;

[0056] A plurality of gray wolf individuals (i.e., a plurality of hyperparameter combinations) are preset to perform parallel search, so that the solution space can be quickly covered, and the search efficiency is improved; the gray wolf individual dynamically adjusts its position according to the positions of the wolf (optimal solution), the wolf (suboptimal solution) and the wolf (third optimal solution), which helps to balance exploration and utilization in the search process and avoid premature convergence; with the increase of the number of iterations, the gray wolf individual gradually approaches the positions of the wolf, the wolf and the wolf through the gray wolf position updating formula, so as to gradually optimize the hyperparameter combination; the search step is dynamically adjusted through a linear decreasing factor, so that the search process has a larger exploration range in the early stage, and pays more attention to local fine search in the later stage, which helps to find more accurate optimal solution. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 It is a mobile video and audio evidence collection management system structure schematic view of the present application;

[0058] Figure 2 It is a mobile video and audio evidence collection management method flowchart of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0060] Embodiment one

[0061] Please refer to Figure 1 The mobile video and audio evidence collection management system described in the embodiment includes:

[0062] The evidence capture module is used for collecting video and audio integrated data and environment monitoring data.

[0063] The data processing module is configured to preliminarily process the audio-visual comprehensive data to obtain a time-synchronized audio-visual feature data set.

[0064] The multi-modal data fusion module is configured to fuse the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set.

[0065] The content matching degree analysis module is configured to train an audio-visual evidence matching model according to the comprehensive feature data set, and predict a comprehensive content matching degree based on the audio-visual evidence matching model.

[0066] The abnormality evaluation module is configured to determine whether the audio-visual comprehensive data is abnormal based on the comprehensive content matching degree, and send an alarm information to the management terminal if the audio-visual comprehensive data is abnormal.

[0067] The audio-visual comprehensive data includes video evidence data and audio evidence data. The video evidence data includes a video image frame sequence, a video frame timestamp, a video resolution, a video resolution, and a video compression encoding. The audio evidence data includes an audio signal frame sequence, an audio signal waveform, an audio sampling rate, an audio signal timestamp, and a sound frequency feature. The environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay, and network signal strength.

[0068] The method for preliminarily processing the audio-visual comprehensive data to obtain a time-synchronized audio-visual feature data set includes:

[0069] The method includes preliminarily processing the collected video evidence data and audio evidence data to obtain a video evidence feature data set and an audio evidence feature data set. The method estimates the motion between each adjacent frame in the video image frame sequence in the video evidence data using an optical flow method. The method uses a translation motion model to fit the overall motion of the camera according to the motion between each adjacent frame, and uses geometric transformation to adjust the video image frame sequence for each frame of image to perform de-jittering processing.

[0070] For example, assume that there is a 10-second monitoring video data set, with a frame rate of 30 frames per second. The video resolution is 1920x1080, and there is image blur and noise in the video due to slight camera jitter.

[0071] 1. The optical flow method estimates the motion between adjacent frames:

[0072] The optical flow method estimates the motion between adjacent frames by estimating the motion of each pixel in the image. Assume that two frames in the video, frame A and frame B, are to be processed. Through optical flow estimation, the motion vector of each pixel in the image is obtained, i.e., how much distance each pixel moves from frame A to frame B, and the direction and speed of the motion.

[0073] For example, the optical flow method calculates that:

[0074] The motion vector of the image center is (dx=2, dy=-1), that is, the image center is offset by 2 pixels and 1 pixel to the right and down. The upper right corner of the image has more motion, with a motion vector of (dx=5, dy=3), indicating that the upper right corner is offset by a large amount.

[0075] 2. Construct a motion model:

[0076] Based on the motion vector of each pixel estimated by the optical flow, a translation transformation model is used to fit the motion of the entire camera. For example, in this case, it is assumed that the camera in the video has a slight translation (moving forward or backward or left or right).

[0077] Assuming that the motion model of the camera is a translation model: by statistical analysis of the optical flow estimation results, it is found that the overall translation displacement of the camera is: x-axis displacement: dx=2.5 pixels; y-axis displacement: dy=-1.8 pixels;

[0078] 3. Geometric transformation for de-shaking processing:

[0079] According to the constructed translation motion model, geometric transformation is performed on each frame of image; the purpose of this step is to correct the offset in the image caused by camera motion, thereby removing the shaking in the image;

[0080] For example:

[0081] For frame A, it is assumed that according to the estimated translation displacement, inverse translation transformation is performed to restore frame A to its normal position.

[0082] For frame B, according to the transformation information of the translation motion model, the content of frame B is translated to compensate for the image displacement caused by camera motion, ensuring that the positions of objects in the image are consistent with frame A.

[0083] The de-shaking processed video evidence data is denoised using non-local mean filtering to obtain a video evidence feature data set; DBSCAN algorithm is used to identify outliers in the audio evidence data, and the identified outliers are removed to obtain an audio evidence feature data set; the video evidence feature data set and the audio evidence feature data set are subjected to standard deviation normalization processing to convert them to standard normal distribution, thereby obtaining normalized video evidence feature data set and audio evidence feature data set;

[0084] The normalized video evidence feature data set and the audio evidence feature data set are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized audio-visual feature data set.

[0085] The method for de-noising the video evidence data after de-jittering processing by using non-local mean filtering comprises:

[0086] Extracting each frame image in the video image frame sequence in the video evidence data, for each frame image in the video image frame sequence, randomly selecting a frame image as a current frame, performing non-local mean filtering, for each pixel in the current frame , calculating the similarity of all remaining pixels with the pixel ; is the original pixel value of the remaining pixel , is the control weight decay parameter;

[0087] Limiting the control weight decay parameter by a weight decay parameter limiting formula, the weight decay parameter limiting formula is: ; wherein, is the limited control weight decay parameter; is the total number of pixels of the image; is a scale factor for adjusting the range of the control weight decay parameter;

[0088] For example, the total number of pixels of the image is 16, and the scale factor for adjusting the range of the control weight decay parameter is 10, then the limited control weight decay parameter .

[0089] Using weighted average to calculate the de-noised pixel value , the specific mathematical formula is: ; wherein, is the search window of the current frame image, containing the pixels around the pixel ;

[0090] Using the de-noised pixel value to update all pixels in the current frame image until all pixels are processed; repeat until all frame images are de-noised.

[0091] The method for synchronizing the normalized video evidence feature data set and the audio evidence feature data set by using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature data set comprises:

[0092] S51, the video image frame sequence in the normalized video evidence feature data set is represented as: ; wherein, is the feature vector of the th frame of the video image; is the total number of frames of the video image frame sequence;

[0093] The audio signal frame sequence in the normalized audio evidence feature dataset is represented as ; wherein, is the i-th feature vector of the audio signal frame; is the total number of frames of the audio signal frame sequence;

[0094] S52, a cross-correlation function is constructed to measure the similarity between the video image frame sequence and the audio signal frame sequence at different time delays; the time alignment between the video image frame sequence and the audio signal frame sequence is found by maximizing the cross-correlation;

[0095] The cross-correlation function is: ; wherein, is the weighted similarity between the video image frame sequence and the audio signal frame sequence at different time delays ; wherein, is the i-th frame of the audio signal frame sequence, and represents the frame of the audio signal after a time delay ; wherein, is the time delay parameter, representing the time difference between the video image frame sequence and the audio signal frame sequence; is the minimum length that can be processed after the time synchronization of the video image frame sequence and the audio signal frame sequence; is the window size of the i-th frame at the delay ; wherein, is the index of the video image frame sequence and the audio signal frame; S53, the window size of the i-th frame at the delay is selected by comparing the weighted similarity between the video image frame sequence and the audio signal frame sequence at different time delays

[0096] , the best time delay corresponding to the maximum weighted similarity is selected, and the audio signal frame is adjusted according to the obtained best time delay ; if , the audio signal needs to be moved forward by frames; if , the audio signal needs to be moved backward by frames; if , no time adjustment is needed; and the time-synchronized audio-visual feature dataset is obtained after adjusting the video image frame sequence and the audio signal frame sequence. The window size of the i-th frame at the delay ; wherein,

[0097] The window size of the i-th frame at the delay ; wherein, ​​The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps:

[0098] The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps: The method for dynamically adjusting the window size of the video image frame at the delay time T includes the following steps:

[0099] If the video content or the audio signal changes dramatically (i.e. or is large), the window size will be increased so as to pay more attention to the part with large changes; if the video and audio changes are both gentle (i.e. or is small), the window size will be reduced to avoid over-attention to the part with small changes.

[0100] For example, assuming that a video image frame sequence and an audio signal frame sequence are being synchronized and the above dynamic window adjustment formula is used to dynamically adjust the window size; at time T=5, assuming that the change rate of the video image frame is 0.8, the change rate of the audio signal frame is 0.2, the influence factor of the change rate of the video image frame on the window size is 1, the influence factor of the change rate of the audio signal frame on the window size is 0.5, and the scaling coefficient is 2, then the window size of the video image frame at the delay time T after dynamic adjustment is 0.3. The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps: The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps: The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps:

[0101] The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps: The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps:

[0102] The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps: The method for fusing the time-synchronized audio-visual feature data set and the environmental monitoring data to obtain a comprehensive feature data set includes the following steps:

[0103] The time-aligned audio-visual feature dataset and the normalized environmental monitoring feature dataset are fused by a weighting formula to obtain a comprehensive feature dataset; the time-aligned audio-visual feature dataset is denoted as , and the normalized environmental monitoring feature dataset is denoted as .

[0104] The weighting formula is: ; wherein, is the comprehensive feature dataset; is a weight coefficient of the time-aligned audio-visual feature dataset; is a weight coefficient of the normalized environmental monitoring feature dataset.

[0105] The training method of the audio-visual evidence matching model comprises:

[0106] The dataset is divided into a training set, a validation set and a test set, the model is trained and the model performance is evaluated; the audio-visual evidence matching model is constructed, the sample set is a subset in the dataset, and each sample set comprises a historical comprehensive feature dataset and a corresponding comprehensive content matching degree;

[0107] The audio-visual evidence matching model comprises an input layer, a hidden layer and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used for inputting the historical comprehensive feature dataset; the output layer of the model is used for outputting the comprehensive content matching degree; and the audio-visual evidence matching model is a multi-modal neural network model.

[0108] The mean square error is used as a loss function to measure the error between the predicted value and the actual value of the model; the mean square error loss function is: ; wherein, is the number of datasets; is the actual value of a data point in the dataset; is the predicted value of a data point in the dataset;

[0109] The training set is used to train the audio-visual evidence matching model, the model parameters are updated by a back propagation algorithm and a gradient descent method to minimize the loss function; the validation set is used to evaluate the performance of the audio-visual evidence matching model by calculating the determination coefficient, and the hyperparameters of the model are optimized;

[0110] The Adam optimization algorithm is selected as the optimizer to adjust the hyperparameters of the model until the performance is no longer improved or the preset number of iterations is reached; the test set is used to evaluate the performance of the model in the prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature dataset to obtain the comprehensive content matching degree.

[0111] Methods for tuning model hyperparameters include:

[0112] S91. Randomly generate a set of initial hyperparameter combinations, each hyperparameter combination corresponds to a gray wolf individual; the position of each gray wolf individual is a dimensional vector; where is the number of hyperparameters that need to be tuned; Gray wolf individuals, each gray wolf individual contains hyperparameters, The location of the gray wolf individuals is: ;in, For the The location of individual gray wolves; For the Gray wolf individuals Hyperparameter values; is the index of the individual gray wolf;

[0113] S92. Use the mean square error loss function to measure the fitness value of each hyperparameter combination. Each gray wolf individual calculates the fitness value according to the mean square error loss function. The smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual. According to the fitness value of the gray wolf individual, the gray wolf individual with the highest fitness value is selected as Wolf, select the gray wolf individual with the second highest fitness value as Wolf and select the gray wolf individual with the third highest fitness value as Wolf;

[0114] S93, each gray wolf individual Wolf, Wolf and The wolf approaches to adjust its position, explore and update the hyperparameter combination, and the adjustment of the individual position of the gray wolf follows the gray wolf position update formula; the gray wolf position update formula is: ;in, No. Gray wolf individuals in The hyperparameter position of the generation, that is, the hyperparameter combination; For the Gray wolf individuals in The hyperparameter position of the generation, that is, the hyperparameter combination; 、 and Represents a random factor related to position update, controlling the step size and search direction of the individual gray wolf; For the Gray wolf individuals and The distance between wolves, simulating the movement of gray wolves Wolf, Wolf and the process of the wolf approaching; is the current iteration number; is the next iteration number;

[0115] random factor is calculated by the formula: ; wherein, is a linearly decreasing factor; is a random number for controlling the randomness of the search direction, ranging from ;

[0116] random factor is calculated by the formula: ; wherein, is a random number for controlling the search step size of the grey wolf individual, ranging from ;

[0117] random factor is calculated by the formula: ; wherein, is a random number for controlling the search direction of the grey wolf individual, ranging from ;

[0118] The linearly decreasing factor is dynamically adjusted by the linearly decreasing factor adjustment formula; the linearly decreasing factor adjustment formula is: ; wherein, is the dynamically adjusted linearly decreasing factor; is the maximum value of the linearly decreasing factor; is the maximum iteration number; is a parameter for controlling the decreasing rate;

[0119] For example, the maximum value of the linearly decreasing factor is 2, the maximum iteration number is 1000, and the parameter for controlling the decreasing rate is 2;

[0120] When = 0, the dynamically adjusted linearly decreasing factor ;

[0121] When = 500, the dynamically adjusted linearly decreasing factor ;

[0122] When = 1000, the dynamically adjusted linearly decreasing factor ;

[0123] As the iteration number increases, the factor starts to decrease and gradually approaches a smaller stable value as the iteration proceeds; by adjusting the parameter for controlling the decreasing rate The rate of decrease can be controlled, and the larger The decrease will be faster.

[0124] S94, update the position of each gray wolf individual by a gray wolf position update formula, thereby adjusting the hyperparameter combination, recalculate the fitness value of each gray wolf individual in each iteration, and if the fitness value of a certain gray wolf individual is higher, update The position of the wolf, The position of the wolf and The position of the wolf, preset a maximum number of iterations, and stop when the number of iterations reaches the maximum number of iterations; in the final iteration, the position of the wolf is selected as the optimal hyperparameter combination, that is, the performance of the model is optimal under the hyperparameter combination. The position of the wolf,

[0125] The method for judging whether the audio-video comprehensive data is abnormal based on the comprehensive content matching degree comprises the following steps:

[0126] If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is judged that the audio-video comprehensive data is abnormal.

[0127] If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is judged that the audio-video comprehensive data is not abnormal.

[0128] The preset comprehensive content matching degree is set by a staff member, different comprehensive content matching degrees are collected through a management terminal, and the average value of the multiple comprehensive content matching degrees is taken as the preset comprehensive content matching degree.

[0129] In this embodiment, the similarity between pixels is calculated by non-local mean filtering, which can effectively remove noise in the video while preserving the edges and texture details of the original image, and improve the image quality. It is suitable for video processing in complex scenes, such as multiple noise interference in evidence; remove video jitter and noise, enhance the clarity and stability of the video picture, make it more authoritative and credible in law enforcement, judicial or research applications; use the weight decay parameter limit formula to dynamically adjust and control the weight decay parameter, so that the algorithm can adapt flexibly according to the characteristics of different frame images in the video, further optimize the denoising effect;

[0130] The time delay of the video image frame and the audio signal frame can be accurately calculated through the cross-correlation function, so that high-precision time synchronization is realized, and the consistency of the audio and video data in the time dimension is ensured; the dynamic window size adjustment design is adopted, the synchronization strategy is flexibly adjusted according to the real-time changing characteristic similarity, and the adaptability to complex scenes is improved; the window size can be dynamically adjusted according to the change rate of the video frame and the audio frame, so that the synchronization algorithm is more suitable for the dynamic change of the data characteristics, and the precision of time alignment is improved; through the dynamic adjustment of the window size, the real similarity between the video and audio frames can be more accurately captured; the high-precision data after time synchronization reduces the false alarm and missed alarm in the abnormal detection, and improves the reliability of the system;

[0131] Through presetting multiple gray wolf individuals (i.e., multiple groups of hyperparameter combinations) for parallel search, the solution space can be quickly covered, and the search efficiency is improved; the gray wolf individual dynamically adjusts its position according to the positions of the wolf (optimal solution), the wolf (suboptimal solution) and the wolf (third optimal solution), which helps to balance exploration and utilization in the search process and avoid premature convergence; with the increase of the number of iterations, the gray wolf individual gradually approaches the positions of the wolf, the wolf and the wolf through the gray wolf position updating formula, thereby gradually optimizing the hyperparameter combination; through the linear decreasing factor, the search step is dynamically adjusted, so that the search process has a larger exploration range in the early stage, and pays more attention to local fine search in the later stage, which helps to find more accurate optimal solution.

[0132] Embodiment two

[0133] Please refer to Figure 2 The embodiment not described in detail is described in embodiment 1, and a mobile audio and video evidence collection and management method is provided, which comprises:

[0134] S1, collecting audio and video comprehensive data and environmental monitoring data;

[0135] S2, preliminarily processing the audio and video comprehensive data to obtain time-synchronized audio and video feature data set;

[0136] S3, fusing the time-synchronized audio and video feature data set and the environmental monitoring data to obtain comprehensive feature data set;

[0137] S4, training an audio and video evidence matching model according to the comprehensive feature data set, and predicting a comprehensive content matching degree based on the audio and video evidence matching model;

[0138] S5, judging whether the audio and video comprehensive data is abnormal based on the comprehensive content matching degree, and if there is an abnormality, sending an alarm information to a management terminal.

[0139] Since the electronic device introduced in the embodiment is the electronic device used in the mobile video and audio evidence collection and management system in the embodiment, the specific implementation of the electronic device and various changes thereof can be understood by those skilled in the art based on the mobile video and audio evidence collection and management system introduced in the embodiment, so the implementation of the electronic device in the method in the embodiment will not be introduced in detail. As long as the electronic device used in the mobile video and audio evidence collection and management system in the embodiment is implemented by those skilled in the art, it belongs to the scope of protection of the present application.

[0140] The above formulas are dimensionless values, and the formulas are obtained by collecting a large amount of data to simulate a formula of the nearest real situation. The preset parameters and threshold values in the formula are set by those skilled in the art according to the actual situation.

[0141] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application is within the protection scope of the present application. It should be noted that, for ordinary technical users in the technical field, some improvements and refinements without departing from the principles of the present application are also considered to be within the protection scope of the present application.

Claims

1. A mobile video and audio evidence collection and management system, characterized in that: include: Evidence capture module, used to collect comprehensive audio and video data and environmental monitoring data; The data processing module is used to perform preliminary processing on the audio and video comprehensive data to obtain a time-synchronized audio and video feature data set; The normalized video evidence feature dataset and audio evidence feature dataset are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset, including: S52. Construct a cross-correlation function to measure the similarity between the video image frame sequence and the audio signal frame sequence at different time delays through the cross-correlation function; find the time alignment between the video image frame sequence and the audio signal frame sequence by maximizing the cross-correlation; The cross-correlation function is: ;in, The video image frame sequence and the audio signal frame sequence are delayed at different times Weighted similarity under ; The first frame in the video image sequence frame, is the first frame in the audio signal sequence Frame, indicating that the audio signal has been delayed by time The next frame; is a time delay parameter, which represents the time difference between the video image frame sequence and the audio signal frame sequence; It is the minimum length that can be processed after the video image frame sequence and the audio signal frame sequence are time synchronized; For the Frames are delayed The window size below; S53, adjust the formula through the dynamic window Frames are delayed The window size under the dynamic adjustment design; the dynamic window adjustment formula is: ;in, After dynamic adjustment Frames are delayed The window size below; is the rate of change of video image frames; is the rate of change of audio signal frames; is the video impact factor; is the audio impact factor; is the scaling factor; by comparing the video image frame sequence and the audio signal frame sequence at different time delays The weighted similarity under the condition of , select the optimal time delay corresponding to the maximum weighted similarity , according to the best time delay obtained , adjust the audio signal frame to synchronize it with the video image frame; if , the audio signal moves forward frame; if , the audio signal moves backward Frame; if , then no time adjustment is made; after adjusting the video image frame sequence and the audio signal frame sequence, a time-synchronized video and audio feature dataset is obtained; Multimodal data fusion module, used to fuse the time-synchronized audio and video feature datasets and environmental monitoring data to obtain a comprehensive feature dataset; The content matching analysis module is used to train an audio-visual evidence matching model based on a comprehensive feature dataset, and predict the comprehensive content matching degree based on the audio-visual evidence matching model; The anomaly assessment module determines whether there is an anomaly in the audio and video comprehensive data based on the comprehensive content matching degree. If there is an anomaly, an alarm message is sent to the management terminal; each module is connected by wired and / or wireless means.

2. A mobile video and audio evidence collection and management system according to claim 1, characterized in that: The comprehensive audio-visual data includes video evidence data and audio evidence data; the video evidence data includes video image frame sequence, video frame timestamp, video resolution and video compression coding; the audio evidence data includes audio signal frame sequence, audio signal waveform, audio sampling rate, audio signal timestamp and sound frequency characteristics; the environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay and network signal strength.

3. The mobile video and audio evidence collection and management system according to claim 2, characterized in that: The method of performing preliminary processing on the audio-visual integrated data to obtain a time-synchronized audio-visual feature data set includes: The collected video evidence data and audio evidence data are preliminarily processed to obtain video evidence feature datasets and audio evidence feature datasets; the optical flow method is used to estimate the motion between each adjacent frame in the video image frame sequence in the video evidence data; based on the motion between each adjacent frame, a translation motion model is used to fit the overall motion of the camera, and geometric transformation is used based on the translation motion model to adjust the video image frame sequence and de-jitter each frame; the de-jittered video evidence data is denoised using non-local mean filtering to obtain a video evidence feature dataset; The outliers in the audio evidence data are identified using the DBSCAN algorithm, and the identified outliers are removed to obtain the audio evidence feature dataset; the video evidence feature dataset and the audio evidence feature dataset are normalized by standard deviation and converted into a standard normal distribution to obtain the normalized video evidence feature dataset and the audio evidence feature dataset; The normalized video evidence feature dataset and audio evidence feature dataset are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset.

4. The mobile video and audio evidence collection and management system according to claim 3, characterized in that: The method for performing denoising on the video evidence data after de-jittering using non-local means filtering includes: Extract each frame of the video image frame sequence in the video evidence data, randomly select a frame as the current frame for each frame in the video image frame sequence, perform non-local mean filtering, and for each pixel in the current frame , calculate and all remaining pixels Similarity ;in, Pixels The original pixel value of The remaining pixels The original pixel value, Parameters that control weight decay; The weight attenuation parameter restriction formula is used to restrict the parameters that control weight attenuation. The weight attenuation parameter restriction formula is: ;in, is the parameter that controls weight decay after restriction; is the total number of pixels in the image; A scaling factor for adjusting the range of parameters that control weight decay; Use weighted average to calculate the denoised pixel value , the specific mathematical formula is: ;in, is the search window of the current frame image, including pixels surrounding pixels; Use the denoised pixel values To update all pixels in the current frame image until all pixels are processed; repeat until all frame images are denoised.

5. The mobile video and audio evidence collection and management system according to claim 4, characterized in that: The method of synchronizing the normalized video evidence feature dataset and the audio evidence feature dataset using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset further includes: S51. Express the video image frame sequence in the normalized video evidence feature dataset as: ;in, For video images Feature vector of the frame; is the total number of frames in the video image frame sequence; The normalized audio signal frame sequence in the audio evidence feature dataset is represented as ;in, The first frame of the audio signal feature vectors; is the total number of frames in the audio signal frame sequence.

6. The mobile video and audio evidence collection and management system according to claim 5, characterized in that: The method of fusing the time-synchronized audio and video feature dataset and the environmental monitoring data to obtain a comprehensive feature dataset includes: The outliers in the environmental monitoring data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the environmental monitoring feature data set; the environmental monitoring feature data set is normalized by standard deviation and converted into a standard normal distribution to obtain the normalized environmental monitoring feature data set; The time-aligned audio and video feature dataset and the normalized environmental monitoring feature dataset are fused through a weighted formula to obtain a comprehensive feature dataset; the time-aligned audio and video feature dataset is recorded as , the normalized environmental monitoring feature dataset is recorded as ; The weighted formula is: ;in, It is a comprehensive feature dataset; is the weight coefficient of the time-aligned audio and video feature dataset; is the weight coefficient of the normalized environmental monitoring feature dataset.

7. The mobile video and audio evidence collection and management system according to claim 6, characterized in that: The training method of the video and audio evidence matching model includes: The dataset is divided into training, validation, and test sets to train the model and evaluate its performance. A video and audio evidence matching model is constructed, where the sample set is a subset of the dataset. Each sample set includes a historical comprehensive feature dataset and the corresponding comprehensive content matching degree. The audio-visual evidence matching model includes an input layer, a hidden layer, and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used to input a historical comprehensive feature dataset; the output layer of the model is used to output a comprehensive content matching degree; the audio-visual evidence matching model is a multimodal neural network model; The mean squared error (MSE) is used as the loss function to measure the error between the model's predicted value and the actual value. The audio-visual evidence matching model is trained using the training set, and the model parameters are updated using the backpropagation algorithm and gradient descent to minimize the loss function. The performance of the audio-visual evidence matching model is evaluated using the validation set by calculating the coefficient of determination and tuning the model's hyperparameters. The Adam optimization algorithm is selected as the optimizer, and the model's hyperparameters are adjusted until the performance no longer improves or the preset number of iterations is reached. The test set is used to evaluate the model's performance in the prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature dataset to obtain the comprehensive content matching degree.

8. The mobile video and audio evidence collection and management system according to claim 7, characterized in that: The method for tuning the hyperparameters of the model includes: S91. Randomly generate a set of initial hyperparameter combinations, each hyperparameter combination corresponds to a gray wolf individual; the position of each gray wolf individual is a dimensional vector; where is the number of hyperparameters that need to be tuned; Gray wolf individuals, each gray wolf individual contains hyperparameters, The location of the gray wolf individuals is: ;in, For the The location of individual gray wolves; For the Gray wolf individuals Hyperparameter values; is the index of the individual gray wolf; S92. Use the mean square error loss function to measure the fitness value of each hyperparameter combination. Each gray wolf individual calculates the fitness value according to the mean square error loss function. The smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual. According to the fitness value of the gray wolf individual, the gray wolf individual with the highest fitness value is selected as Wolf, select the gray wolf individual with the second highest fitness value as Wolf and select the gray wolf individual with the third highest fitness value as Wolf; S93, each gray wolf individual Wolf, Wolf and The wolf approaches to adjust its position, explore and update the hyperparameter combination, and the adjustment of the individual position of the gray wolf follows the gray wolf position update formula; the gray wolf position update formula is: ;in, For the Gray wolf individuals in The hyperparameter position of the generation, that is, the hyperparameter combination; For the Gray wolf individuals in The hyperparameter positions of the generations; 、 and Represents a random factor related to position update, controlling the step size and search direction of the individual gray wolf; For the Gray wolf individuals and the distance between wolves; is the current iteration number; is the next iteration number; Random Factor The calculation formula is: ;in, is a linear decreasing factor; A random number that controls the randomness of the search direction, ranging from ; Random Factor The calculation formula is: ;in, A random number that controls the size of the gray wolf's individual search step, ranging from ; Random Factor The calculation formula is: ;in, A random number that controls the search direction of individual gray wolves, ranging from ; The linear decline factor is dynamically adjusted through the linear decline factor adjustment formula; the linear decline factor adjustment formula is: ;in, is the linear decreasing factor after dynamic adjustment; is the maximum value of the linear decreasing factor; is the maximum number of iterations; is the parameter to control the deceleration rate; S94, update the position of each individual gray wolf one by one through the gray wolf position update formula, thereby adjusting the hyperparameter combination. In each iteration, recalculate the fitness value of each individual gray wolf. If a gray wolf individual has a higher fitness value, update Wolf, Wolf and The position of the wolf, preset the maximum number of iterations, and stop when the number of iterations reaches the maximum number of iterations; in the final iteration, select The position of the wolf is taken as the optimal hyperparameter combination, that is, under this optimal hyperparameter combination, the performance of the model is optimal.

9. The mobile video and audio evidence collection and management system according to claim 8, characterized in that: The method for determining whether the audio and video comprehensive data has an abnormality based on the comprehensive content matching degree includes: If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is determined that there is an abnormality in the audio and video comprehensive data; If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is determined that there is no abnormality in the audio and video comprehensive data.

Images