Mobile video and audio evidence collection and management system

Abstract

The invention belongs to the technical field of digital forensics, and discloses a mobile video and audio evidence collection management system, which comprises an evidence capture module used for collecting video and audio comprehensive data and environment monitoring data; the data processing module is used for performing primary processing on the comprehensive data to obtain a video and audio feature data set after time synchronization; the video and audio evidence matching module is used for fusing the video and audio feature data set after time synchronization and the environment monitoring data to obtain a comprehensive feature data set; the content matching degree analysis module is used for training according to the comprehensive feature data set to obtain a video and audio evidence matching model, and predicting to obtain a comprehensive content matching degree based on the video and audio evidence matching model; the abnormity evaluation module is used for judging whether the audio and video comprehensive data is abnormal or not based on the comprehensive content matching degree, and sending alarm information to the management terminal if the audio and video comprehensive data is abnormal; the convenience and efficiency of evidence use are improved, and powerful support is provided for subsequent investigation and processing.

Description

Technical Field

[0001] The present invention relates to the technical field of digital evidence collection, and more specifically, to a mobile video and audio evidence collection and management system. Background Art

[0002] The patent with the patent announcement number CN105206059A discloses a traffic accident scene evidence collection guidance system and method. The system includes: the user uses a mobile terminal to complete the blank information in the accident form, and the accident form includes the accident ID, the location of the accident (GPS information), time, accident type, number of participants, and license plate number; after the accident form is generated, the system starts the accident evidence collection module, calls the corresponding evidence collection submodule according to the accident type and number of participants in the accident form, and guides the user to take evidence photos; after the photo is taken, the user can record additional recordings according to the scene; finally, the accident form, image, and audio are packaged into an evidence compression package, sent to the server, and the server stores the evidence and waits for acceptance. The present invention uses the user's mobile phone as an image or video acquisition device, and the owner of the accident car takes photos and collects evidence independently, which simplifies the evidence collection procedure for minor traffic accidents, realizes a fast and efficient self-collection process, and facilitates insurance companies or relevant departments to quickly settle claims.

[0003] The existing mobile video and audio evidence collection and management systems mainly have the following problems:

[0004] The existing methods do not make full use of the similarity measurement of non-local mean filtering, which may easily lead to the destruction of important texture and edge information in the denoising process, weakening the authenticity of the video image; the weight decay parameters are not adjusted dynamically, which may show too strong or too weak denoising effects in different frames of the video, and cannot be flexibly optimized for differences in noise intensity; the existing technology does not deal with the problem of video jitter, which may cause unstable images and affect the viewing experience, especially in scenes that require high precision such as law enforcement or justice; it fails to enhance the stability and clarity of the video image, which may produce ghosting or blur in dynamic scenes, damaging the visual experience;

[0005] The video image frames and audio signal frames are not fully aligned in the time dimension, resulting in the loss of synchronization of audio and video data, affecting the accuracy of subsequent processing and analysis; when the dynamic window adjustment design is not adopted, the algorithm is difficult to adapt to the differences in the changes in the characteristics of video frames and audio frames in complex scenes, such as rapidly changing background noise or scene switching; the synchronization algorithm that does not dynamically adjust the window size may lose the accurate capture of similarities in scenes with large feature changes, resulting in an increase in the time alignment error;

[0006] Failure to use multiple gray wolf individuals for parallel search may mean that the algorithm can only rely on a single initial hyperparameter combination, resulting in limited exploration scope of the algorithm and insufficient coverage of the solution space. Failure to introduce a linear decreasing factor to dynamically adjust the search step size may result in the algorithm being unable to effectively explore the vast solution space due to insufficient initial step size. Failure to dynamically adjust the position and step size of gray wolf individuals may make it difficult to adapt to complex hyperparameter spaces and complex model tuning requirements. Algorithms that are fixed or lack dynamic adjustment strategies cannot adjust strategies according to the real-time status of the search process, and their adaptability is significantly reduced. Due to insufficient optimization of the search step size and solution space exploration, the final hyperparameter combination may deviate from the global optimum, affecting model performance.

[0007] In view of this, the present invention proposes a mobile video and audio evidence collection and management system to solve the above problems. Summary of the invention

[0008] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a mobile video and audio evidence collection and management system, comprising:

[0009] Evidence capture module, used to collect comprehensive audio and video data and environmental monitoring data;

[0010] A data processing module is used to perform preliminary processing on the audio and video comprehensive data to obtain a time-synchronized audio and video feature data set;

[0011] The multimodal data fusion module is used to fuse the time-synchronized audio and video feature data set and the environmental monitoring data to obtain a comprehensive feature data set;

[0012] A content matching degree analysis module is used to obtain a video and audio evidence matching model based on a comprehensive feature data set training, and to predict a comprehensive content matching degree based on the video and audio evidence matching model;

[0013] The abnormality assessment module determines whether there is an abnormality in the comprehensive audio and video data based on the comprehensive content matching degree. If there is an abnormality, an alarm message is sent to the management terminal; each module is connected by wire and / or wireless means.

[0014] Furthermore, the comprehensive audio-visual data includes video evidence data and audio evidence data; the video evidence data includes video image frame sequence, video frame timestamp, video resolution, video resolution and video compression coding; the audio evidence data includes audio signal frame sequence, audio signal waveform, audio sampling rate, audio signal timestamp and sound frequency characteristics; the environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay and network signal strength.

[0015] Furthermore, the method of preliminarily processing the audio-visual comprehensive data to obtain the time-synchronized audio-visual feature data set includes:

[0016] The collected video evidence data and audio evidence data are preliminarily processed to obtain video evidence feature data sets and audio evidence feature data sets; the optical flow method is used to estimate the motion between each adjacent frame in the video image frame sequence in the video evidence data; according to the motion between each adjacent frame, a translation motion model is used to fit the overall motion of the camera, and the video image frame sequence is adjusted by geometric transformation according to the translation motion model, and each frame image is de-jittered; the video evidence data after de-jittering is denoised by non-local mean filtering to obtain a video evidence feature data set;

[0017] The outliers in the audio evidence data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the audio evidence feature data set; the video evidence feature data set and the audio evidence feature data set are normalized by standard deviation and converted into a standard normal distribution to obtain the normalized video evidence feature data set and the audio evidence feature data set;

[0018] The normalized video evidence feature dataset and audio evidence feature dataset are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset.

[0019] Furthermore, the method of performing denoising on the video evidence data after de-jittering using non-local mean filtering includes:

[0020] Extract each frame of the video image frame sequence in the video evidence data. For each frame of the video image frame sequence, randomly select a frame as the current frame and perform non-local mean filtering. For each pixel in the current frame, , calculate and all remaining pixels Similarity Pixel The original pixel value of The remaining pixels The original pixel value, Parameters for controlling weight decay;

[0021] The weight attenuation parameter restriction formula is used to restrict the parameters that control weight attenuation. The weight attenuation parameter restriction formula is: ;in, is the parameter that controls weight decay after restriction; is the total number of pixels in the image; A scaling factor for adjusting the range of parameters that control weight decay;

[0022] Use weighted average to calculate the denoised pixel value , the specific mathematical formula is: ;in, is the search window of the current frame image, including pixels The surrounding pixels;

[0023] Use the denoised pixel values To update all pixels in the current frame image until all pixels are processed; repeat until all frame images are denoised.

[0024] Furthermore, the method of synchronizing the normalized video evidence feature dataset and the audio evidence feature dataset using an optimized time synchronization algorithm to obtain the time-synchronized video and audio feature dataset includes:

[0025] S51, the video image frame sequence in the normalized video evidence feature dataset is expressed as: ;in, For video images Feature vector of the frame; is the total number of frames in the video image frame sequence;

[0026] The normalized audio signal frame sequence in the audio evidence feature dataset is represented as ;in, is the first frame of the audio signal feature vectors; is the total number of frames in the audio signal frame sequence;

[0027] S52, constructing a cross-correlation function, measuring the similarity between the video image frame sequence and the audio signal frame sequence under different time delays through the cross-correlation function; finding the time alignment between the video image frame sequence and the audio signal frame sequence by maximizing the cross-correlation;

[0028] The cross-correlation function is: ;in, The video image frame sequence and the audio signal frame sequence are delayed at different times Weighted similarity under ; is the first frame in the audio signal sequence Frame, indicating that the audio signal has been delayed by time The following frame; is a time delay parameter, which represents the time difference between the video image frame sequence and the audio signal frame sequence; It is the minimum length that can be processed after the video image frame sequence and the audio signal frame sequence are time synchronized; For the Frames are delayed The window size below; is the index of the video image frame sequence and the audio signal frame;

[0029] S53, for Frames are delayed The window size is dynamically adjusted by comparing the video image frame sequence and the audio signal frame sequence at different time delays. The weighted similarity under the condition of , select the optimal time delay corresponding to the maximum weighted similarity , according to the best time delay , adjust the audio signal frame to synchronize it with the video image frame; if , the audio signal moves forward Frame; if , the audio signal moves backward Frame; if , no time adjustment is made; after adjusting the video image frame sequence and the audio signal frame sequence, a time-synchronized video and audio feature data set is obtained.

[0030] Furthermore, the Frames are delayed The following methods are used to dynamically adjust the window size:

[0031] Through the dynamic window adjustment formula Frames are delayed The window size is dynamically adjusted and designed; the dynamic window adjustment formula is: ;in, After dynamic adjustment Frames are delayed The window size below; is the rate of change of video image frames; is the rate of change of audio signal frames; is the video impact factor; is the audio impact factor; is the scaling factor.

[0032] Furthermore, the method of fusing the time-synchronized audio and video feature data set with the environmental monitoring data to obtain a comprehensive feature data set includes:

[0033] The outliers in the environmental monitoring data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the environmental monitoring feature data set; the environmental monitoring feature data set is normalized by standard deviation and converted into a standard normal distribution to obtain the normalized environmental monitoring feature data set;

[0034] The time-aligned audio and video feature dataset and the normalized environmental monitoring feature dataset are fused through a weighted formula to obtain a comprehensive feature dataset. The time-aligned audio and video feature dataset is recorded as , the normalized environmental monitoring feature data set is recorded as ;

[0035] The weighted formula is: ;in, It is a comprehensive feature dataset; is the weight coefficient of the time-aligned audio and video feature dataset; is the weight coefficient of the normalized environmental monitoring feature dataset.

[0036] Furthermore, the training method of the video and audio evidence matching model includes:

[0037] Divide the dataset into training set, validation set and test set, train the model and evaluate the model performance; build an audio-visual evidence matching model, where the sample set is a subset of the dataset, and each sample set includes a historical comprehensive feature dataset and the corresponding comprehensive content matching degree;

[0038] The audio-visual evidence matching model includes an input layer, a hidden layer and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used to input a historical comprehensive feature data set; the output layer of the model is used to output a comprehensive content matching degree; the audio-visual evidence matching model is a multimodal neural network model;

[0039] Use the training set to train the audio-visual evidence matching model, and update the model parameters through the back-propagation algorithm and gradient descent method to minimize the loss function; use the validation set to evaluate the performance of the audio-visual evidence matching model by calculating the coefficient of determination and tune the model's hyperparameters;

[0040] The Adam optimization algorithm is selected as the optimizer, and the hyperparameters of the model are adjusted until the performance no longer improves or the preset number of iterations is reached. The test set is used to evaluate the performance of the model in the prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature data set to obtain the comprehensive content matching degree.

[0041] Furthermore, the method for tuning the hyperparameters of the model includes:

[0042] S91. Randomly generate a set of initial hyperparameter combinations, each of which corresponds to a gray wolf individual; the position of each gray wolf individual is a dimensional vector; where is the number of hyperparameters that need to be tuned; Gray wolf individuals, each gray wolf individual contains hyperparameters, The location of the individual gray wolf is: ;in, For the The location of individual gray wolves; For the The first Hyperparameter values; is the index of the individual gray wolf;

[0043] S92. Use the mean square error loss function to measure the fitness value of each hyperparameter combination. Each gray wolf individual calculates the fitness value according to the mean square error loss function. The smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual. According to the fitness value of the gray wolf individual, select the gray wolf individual with the highest fitness value as Wolf, select the gray wolf individual with the second highest fitness value as The wolf and the gray wolf with the third highest fitness value are selected as Wolf;

[0044] S93, each gray wolf individual Wolf, Wolf and The wolf approaches to adjust its position, explore and update the hyperparameter combination, and the adjustment of the individual position of the gray wolf follows the gray wolf position update formula; the gray wolf position update formula is: ;in, No. The gray wolf individual The hyperparameter positions of the generations, i.e., the hyperparameter combinations; For the The gray wolf individual The hyperparameter positions of the generations; , and Represents the random factor related to position update, controlling the step size and search direction of the individual gray wolf; For the Gray wolf individuals and the distance between wolves; is the current iteration number; is the next iteration number;

[0045] Random Factor The calculation formula is: ;in, is a linear decreasing factor; A random number to control the randomness of the search direction, ranging from ;

[0046] Random Factor The calculation formula is: ;in, A random number that controls the size of the gray wolf's individual search step, ranging from ;

[0047] Random Factor The calculation formula is: ; Among them, is a random number that controls the search direction of the gray wolf individual, ranging from ;

[0048] The linear decline factor is dynamically adjusted through the linear decline factor adjustment formula; the linear decline factor adjustment formula is: ;in, is the linear decreasing factor after dynamic adjustment; is the maximum value of the linear decreasing factor; is the maximum number of iterations; Parameters for controlling the rate of deceleration;

[0049] S94. Update the position of each individual gray wolf one by one through the gray wolf position update formula, so as to adjust the hyperparameter combination. In each iteration, recalculate the fitness value of each individual gray wolf. If a gray wolf individual has a higher fitness value, update Wolf, Wolf and The position of the wolf, preset the maximum number of iterations, and stop when the number of iterations reaches the maximum number of iterations; in the final iteration, select The position of the wolf is taken as the optimal hyperparameter combination, that is, under this hyperparameter combination, the performance of the model is optimal.

[0050] Furthermore, the method for judging whether there is an abnormality in the audio and video comprehensive data based on the comprehensive content matching degree includes:

[0051] If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is determined that the audio and video comprehensive data is abnormal;

[0052] If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is determined that there is no abnormality in the audio and video comprehensive data.

[0053] The technical effects and advantages of a mobile video and audio evidence collection and management system of the present invention are as follows:

[0054] The present invention calculates the similarity between pixels through non-local mean filtering, which can effectively remove noise in the video while retaining the edge and texture details of the original image and improving the image quality. It is suitable for video processing in complex scenes, such as multi-noise interference in evidence; removes video jitter and noise, enhances the clarity and stability of the video image, and makes it more authoritative and credible in applications such as law enforcement, justice or research; uses the weight decay parameter restriction formula to dynamically adjust the parameters that control weight decay, so that the algorithm can flexibly adapt according to the characteristics of different frame images in the video, and further optimize the denoising effect;

[0055] Through the cross-correlation function, the time delay of video image frames and audio signal frames can be accurately calculated, thereby achieving high-precision time synchronization and ensuring the consistency of audio and video data in the time dimension; the dynamic window size adjustment design is adopted to flexibly adjust the synchronization strategy according to the real-time changing feature similarity, thereby improving the adaptability to complex scenes; the window size can be dynamically adjusted according to the change rate of video frames and audio frames, so that the synchronization algorithm can better adapt to the dynamic changes of data features and improve the accuracy of time alignment; by dynamically adjusting the window size, the true similarity between video and audio frames can be captured more accurately; the high-precision data after time synchronization reduces false positives and false negatives in anomaly detection and improves the reliability of the system;

[0056] By presetting multiple gray wolf individuals (i.e., multiple sets of hyperparameter combinations) for parallel search, the solution space can be quickly covered and the search efficiency can be improved; the gray wolf individuals dynamically adjust their own positions according to the positions of the wolf (optimal solution), wolf (suboptimal solution), and wolf (third optimal solution). This strategy helps to balance exploration and utilization during the search process and avoid premature convergence; as the number of iterations increases, the gray wolf individuals gradually approach the positions of wolves, wolves, and wolves through the gray wolf position update formula, thereby gradually optimizing the hyperparameter combination; the search step size is dynamically adjusted through a linear decreasing factor, so that the search process has a larger exploration range in the early stage, and more emphasis is placed on local fine search in the later stage, which helps to find a more accurate optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of the structure of a mobile video and audio evidence collection and management system of the present invention;

[0058] Figure 2 The present invention is a flowchart of a mobile video and audio evidence collection and management method. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1

[0061] See also Figure 1 As shown, the mobile video and audio evidence collection and management system described in this embodiment includes:

[0062] Evidence capture module, used to collect comprehensive audio and video data and environmental monitoring data;

[0063] A data processing module is used to perform preliminary processing on the audio and video comprehensive data to obtain a time-synchronized audio and video feature data set;

[0064] The multimodal data fusion module is used to fuse the time-synchronized audio and video feature data set and the environmental monitoring data to obtain a comprehensive feature data set;

[0065] A content matching degree analysis module is used to obtain a video and audio evidence matching model based on a comprehensive feature data set training, and to predict a comprehensive content matching degree based on the video and audio evidence matching model;

[0066] The abnormality assessment module determines whether there is an abnormality in the comprehensive audio and video data based on the comprehensive content matching degree. If there is an abnormality, an alarm message is sent to the management terminal; each module is connected by wire and / or wireless means.

[0067] Comprehensive audio-visual data includes video evidence data and audio evidence data; video evidence data includes video image frame sequence, video frame timestamp, video resolution, video resolution and video compression coding; audio evidence data includes audio signal frame sequence, audio signal waveform, audio sampling rate, audio signal timestamp and sound frequency characteristics; environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay and network signal strength.

[0068] The method of preliminarily processing the audio-visual comprehensive data to obtain the time-synchronized audio-visual feature data set includes:

[0069] The collected video evidence data and audio evidence data are preliminarily processed to obtain video evidence feature data sets and audio evidence feature data sets; the optical flow method is used to estimate the motion between each adjacent frame in the video image frame sequence in the video evidence data; according to the motion between each adjacent frame, a translation motion model is used to fit the overall motion of the camera, and geometric transformation is used to adjust the video image frame sequence according to the translation motion model, and each frame image is de-jittered;

[0070] For example, suppose there is a surveillance video dataset containing 10 seconds, with a frame rate of 30 frames per second; the video resolution is 1920x1080, and there is image blur and noise in the video due to slight camera shaking;

[0071] 1. Optical flow method estimates the motion between adjacent frames:

[0072] The optical flow method calculates the motion between adjacent frames by estimating the motion of pixels in an image. Suppose we want to process two frames in a video, frame A and frame B. Through the optical flow method, we can estimate the motion vector of each pixel in the image, that is, how far each pixel moves from frame A to frame B, and what the direction and speed of the motion are.

[0073] For example, the optical flow method calculates:

[0074] The motion vector of the center of the image is (dx=2,dy=-1), which means that the center of the image is shifted 2 pixels and 1 pixel to the lower right. There is more motion in the upper right corner of the image, with a motion vector of (dx=5,dy=3), indicating that the upper right corner is shifted more.

[0075] 2. Build a motion model:

[0076] Based on the motion vector of each pixel estimated by optical flow, a translation transformation model is used to fit the motion of the entire camera. For example, in this case, it is assumed that the camera in the video has a slight translation (front and back or left and right).

[0077] Assuming that the camera's motion model is a translation model: through statistics of the optical flow estimation results, the overall translation displacement of the camera is obtained as follows: x-axis displacement: dx = 2.5 pixels; y-axis displacement: dy = -1.8 pixels;

[0078] 3. Geometric transformation for de-jittering:

[0079] According to the constructed translational motion model, each frame of the image is geometrically transformed; the purpose of this step is to correct the offset caused by camera motion in the image, thereby removing the jitter in the image;

[0080] For example:

[0081] For frame A, it is assumed that an inverse translation transformation is performed on it according to the estimated translation displacement so that frame A is restored to its normal position.

[0082] For frame B, the content of frame B is translated according to the transformation information of the translation motion model to compensate for the image displacement caused by the camera motion and ensure that the position of the object in the image is consistent with that in frame A.

[0083] The video evidence data after de-jittering is denoised using non-local mean filtering to obtain a video evidence feature data set; the outliers in the audio evidence data are identified using the DBSCAN algorithm, and the identified outliers are removed to obtain an audio evidence feature data set; the video evidence feature data set and the audio evidence feature data set are normalized by standard deviation and converted into a standard normal distribution to obtain a normalized video evidence feature data set and an audio evidence feature data set;

[0084] The normalized video evidence feature dataset and audio evidence feature dataset are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset.

[0085] The method of using non-local mean filtering to denoise the video evidence data after de-jittering includes:

[0086] Extract each frame of the video image frame sequence in the video evidence data. For each frame of the video image frame sequence, randomly select a frame as the current frame and perform non-local mean filtering. For each pixel in the current frame, , calculate and all remaining pixels Similarity Pixel The original pixel value of The remaining pixels The original pixel value, Parameters for controlling weight decay;

[0087] The weight attenuation parameter restriction formula is used to restrict the parameters that control weight attenuation. The weight attenuation parameter restriction formula is: ;in, is the parameter that controls weight decay after restriction; is the total number of pixels in the image; A scaling factor for adjusting the range of parameters that control weight decay;

[0088] For example, if the total number of pixels in the image is 16, adjust the scale factor of the parameter range that controls the weight decay is 10, then the parameter controlling weight decay after restriction .

[0089] Use weighted average to calculate the denoised pixel value , the specific mathematical formula is: ;in, is the search window of the current frame image, including pixels The surrounding pixels;

[0090] Use the denoised pixel values To update all pixels in the current frame image until all pixels are processed; repeat until all frame images are denoised.

[0091] The method of synchronizing the normalized video evidence feature dataset and the audio evidence feature dataset by using an optimized time synchronization algorithm to obtain the time-synchronized video and audio feature dataset includes:

[0092] S51, the video image frame sequence in the normalized video evidence feature dataset is expressed as: ;in, For video images Feature vector of the frame; is the total number of frames in the video image frame sequence;

[0093] The normalized audio signal frame sequence in the audio evidence feature dataset is represented as ;in, is the first frame of the audio signal feature vectors; is the total number of frames in the audio signal frame sequence;

[0094] S52, constructing a cross-correlation function, measuring the similarity between the video image frame sequence and the audio signal frame sequence under different time delays through the cross-correlation function; finding the time alignment between the video image frame sequence and the audio signal frame sequence by maximizing the cross-correlation;

[0095] The cross-correlation function is: ;in, The video image frame sequence and the audio signal frame sequence are delayed at different times Weighted similarity under ; is the first frame in the audio signal sequence Frame, indicating that the audio signal has been delayed by time The following frame; is a time delay parameter, which represents the time difference between the video image frame sequence and the audio signal frame sequence; It is the minimum length that can be processed after the video image frame sequence and the audio signal frame sequence are time synchronized; For the Frames are delayed The window size below; is the index of the video image frame sequence and the audio signal frame;

[0096] S53, for Frames are delayed The window size is dynamically adjusted by comparing the video image frame sequence and the audio signal frame sequence at different time delays. The weighted similarity under the condition of , select the optimal time delay corresponding to the maximum weighted similarity , according to the best time delay , adjust the audio signal frame to synchronize it with the video image frame; if , the audio signal needs to move forward Frame; if , the audio signal needs to be moved backward Frame; if , no time adjustment is required; the video image frame sequence and the audio signal frame sequence are adjusted to obtain a time-synchronized audio and video feature dataset.

[0097] For Frames are delayed The following methods are used to dynamically adjust the window size:

[0098] Through the dynamic window adjustment formula Frames are delayed The window size is dynamically adjusted and designed; the dynamic window adjustment formula is: ;in, After dynamic adjustment Frames are delayed The window size below; is the rate of change of video image frames; is the rate of change of audio signal frames; is the video impact factor, which is used to control the impact of the change rate of the video image frame on the window size; is the audio impact factor, used to control the change rate of the audio signal frame to the window size; It is the scaling factor, which controls the combined effect of the change rate of video image frames and audio signal frames.

[0099] If the video content or audio signal changes drastically (i.e. or If the video and audio are changing slowly (i.e. or If the size of the window is smaller, the window size will be reduced to avoid focusing too much on the parts that do not change much;

[0100] For example, suppose that a video image frame sequence and an audio signal frame sequence are being synchronized, and the above dynamic window adjustment formula is used to dynamically adjust the window size; at time T=5, suppose that the rate of change of the video image frame is 0.8, the rate of change of the audio signal frame is 0.2, the influence factor of the rate of change of the video image frame on the window size is 1, the influence factor of the rate of change of the audio signal frame on the window size is 0.5, and the scaling factor is 2, then after dynamic adjustment, Frames are delayed The window size below is 0.3.

[0101] The method of fusing the time-synchronized audio and video feature dataset with the environmental monitoring data to obtain a comprehensive feature dataset includes:

[0102] The outliers in the environmental monitoring data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the environmental monitoring feature data set; the environmental monitoring feature data set is normalized by standard deviation and converted into a standard normal distribution to obtain the normalized environmental monitoring feature data set;

[0103] The time-aligned audio and video feature dataset and the normalized environmental monitoring feature dataset are fused through a weighted formula to obtain a comprehensive feature dataset. The time-aligned audio and video feature dataset is recorded as , the normalized environmental monitoring feature data set is recorded as ;

[0104] The weighted formula is: ;in, It is a comprehensive feature dataset; is the weight coefficient of the time-aligned audio and video feature dataset; is the weight coefficient of the normalized environmental monitoring feature dataset.

[0105] The training method of the audio-visual evidence matching model includes:

[0106] Divide the dataset into training set, validation set and test set, train the model and evaluate the model performance; build an audio-visual evidence matching model, where the sample set is a subset of the dataset, and each sample set includes a historical comprehensive feature dataset and the corresponding comprehensive content matching degree;

[0107] The audio-visual evidence matching model includes an input layer, a hidden layer and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used to input a historical comprehensive feature data set; the output layer of the model is used to output a comprehensive content matching degree; the audio-visual evidence matching model is a multimodal neural network model;

[0108] The mean square error is used as the loss function to measure the error between the model's predicted value and the actual value; the mean square error loss function is: ;in, is the number of data sets; Data points in the dataset The actual value of Data points in the dataset The predicted value of

[0109] Use the training set to train the audio-visual evidence matching model, and update the model parameters through the back-propagation algorithm and gradient descent method to minimize the loss function; use the validation set to evaluate the performance of the audio-visual evidence matching model by calculating the coefficient of determination and tune the model's hyperparameters;

[0110] The Adam optimization algorithm is selected as the optimizer, and the hyperparameters of the model are adjusted until the performance no longer improves or the preset number of iterations is reached. The test set is used to evaluate the performance of the model in the prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature data set to obtain the comprehensive content matching degree.

[0111] Methods for tuning model hyperparameters include:

[0112] S91. Randomly generate a set of initial hyperparameter combinations, each of which corresponds to a gray wolf individual; the position of each gray wolf individual is a dimensional vector; where is the number of hyperparameters that need to be tuned; Gray wolf individuals, each gray wolf individual contains hyperparameters, The location of the individual gray wolf is: ;in, For the The location of individual gray wolves; For the The first Hyperparameter values; is the index of the individual gray wolf;

[0113] S92. Use the mean square error loss function to measure the fitness value of each hyperparameter combination. Each gray wolf individual calculates the fitness value according to the mean square error loss function. The smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual. According to the fitness value of the gray wolf individual, select the gray wolf individual with the highest fitness value as Wolf, select the gray wolf individual with the second highest fitness value as The wolf and the gray wolf with the third highest fitness value are selected as Wolf;

[0114] S93, each gray wolf individual Wolf, Wolf and The wolf approaches to adjust its position, explore and update the hyperparameter combination, and the adjustment of the individual position of the gray wolf follows the gray wolf position update formula; the gray wolf position update formula is: ;in, No. The gray wolf individual The hyperparameter positions of the generations, i.e., the hyperparameter combinations; For the The gray wolf individual The hyperparameter positions of the generations, i.e., the hyperparameter combinations; , and Represents the random factor related to position update, controlling the step size and search direction of the individual gray wolf; For the Gray wolf individuals and The distance between wolves simulates the movement of gray wolves Wolf, Wolf and The wolf approaches; is the current iteration number; is the next iteration number;

[0115] Random Factor The calculation formula is: ;in, is a linear decreasing factor; A random number to control the randomness of the search direction, ranging from ;

[0116] Random Factor The calculation formula is: ;in, A random number that controls the size of the gray wolf's individual search step, ranging from ;

[0117] Random Factor The calculation formula is: ; Among them, is a random number that controls the search direction of the gray wolf individual, ranging from ;

[0118] The linear decline factor is dynamically adjusted through the linear decline factor adjustment formula; the linear decline factor adjustment formula is: ;in, is the linear decreasing factor after dynamic adjustment; is the maximum value of the linear decreasing factor; is the maximum number of iterations; Parameters for controlling the rate of deceleration;

[0119] For example, the maximum linear reduction factor is 2, and the maximum number of iterations is 1000, the parameter that controls the deceleration rate is 2;

[0120] when =0, the dynamically adjusted linear decreasing factor ;

[0121] when =500, the dynamically adjusted linear decreasing factor ;

[0122] when =1000, the dynamically adjusted linear decreasing factor ;

[0123] As the number of iterations As the factor increases, it starts to decrease and gradually approaches a smaller stable value as the iteration proceeds. By adjusting the parameter that controls the decreasing rate , can control the decreasing speed, the larger It will decrease faster.

[0124] S94. Update the position of each individual gray wolf one by one through the gray wolf position update formula, so as to adjust the hyperparameter combination. In each iteration, recalculate the fitness value of each individual gray wolf. If a gray wolf individual has a higher fitness value, update Wolf, Wolf and The position of the wolf, preset the maximum number of iterations, and stop when the number of iterations reaches the maximum number of iterations; in the final iteration, select The position of the wolf is taken as the optimal hyperparameter combination, that is, under this hyperparameter combination, the performance of the model is optimal.

[0125] Based on the comprehensive content matching degree, the method for judging whether the audio and video comprehensive data has anomalies includes:

[0126] If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is determined that the audio and video comprehensive data is abnormal;

[0127] If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is determined that there is no abnormality in the audio and video comprehensive data.

[0128] The preset comprehensive content matching degree is set by the staff, and different comprehensive content matching degrees are collected through the management terminal, and the average value of multiple comprehensive content matching degrees is taken as the preset comprehensive content matching degree.

[0129] This embodiment calculates the similarity between pixels through non-local mean filtering, which can effectively remove noise in the video while retaining the edge and texture details of the original image and improving the image quality. It is suitable for video processing in complex scenarios, such as multi-noise interference in evidence; removes video jitter and noise, enhances the clarity and stability of the video image, and makes it more authoritative and credible in applications such as law enforcement, justice or research; uses the weight decay parameter restriction formula to dynamically adjust the parameters that control weight decay, so that the algorithm can flexibly adapt according to the characteristics of different frame images in the video, and further optimize the denoising effect;

[0130] Through the cross-correlation function, the time delay of video image frames and audio signal frames can be accurately calculated, thereby achieving high-precision time synchronization and ensuring the consistency of audio and video data in the time dimension; the dynamic window size adjustment design is adopted to flexibly adjust the synchronization strategy according to the real-time changing feature similarity, thereby improving the adaptability to complex scenes; the window size can be dynamically adjusted according to the change rate of video frames and audio frames, so that the synchronization algorithm can better adapt to the dynamic changes of data features and improve the accuracy of time alignment; by dynamically adjusting the window size, the true similarity between video and audio frames can be captured more accurately; the high-precision data after time synchronization reduces false positives and false negatives in anomaly detection and improves the reliability of the system;

[0131] By presetting multiple gray wolf individuals (i.e., multiple sets of hyperparameter combinations) for parallel search, the solution space can be quickly covered and the search efficiency can be improved; the gray wolf individuals dynamically adjust their own positions according to the positions of the wolf (optimal solution), wolf (suboptimal solution), and wolf (third optimal solution). This strategy helps to balance exploration and utilization during the search process and avoid premature convergence; as the number of iterations increases, the gray wolf individuals gradually approach the positions of wolves, wolves, and wolves through the gray wolf position update formula, thereby gradually optimizing the hyperparameter combination; the search step size is dynamically adjusted through a linear decreasing factor, so that the search process has a larger exploration range in the early stage, and more emphasis is placed on local fine search in the later stage, which helps to find a more accurate optimal solution.

[0132] Embodiment 2

[0133] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a mobile video and audio evidence collection and management method, including:

[0134] S1. Collect comprehensive audio and video data and environmental monitoring data;

[0135] S2, preliminarily processing the audio and video comprehensive data to obtain a time-synchronized audio and video feature data set;

[0136] S3, fusing the time-synchronized audio and video feature data set with the environmental monitoring data to obtain a comprehensive feature data set;

[0137] S4. A video and audio evidence matching model is obtained by training the comprehensive feature data set, and a comprehensive content matching degree is predicted based on the video and audio evidence matching model;

[0138] S5. Based on the comprehensive content matching degree, determine whether there is any abnormality in the audio and video comprehensive data. If there is any abnormality, send an alarm message to the management terminal.

[0139] Since the electronic device introduced in this embodiment is an electronic device used to implement a mobile audio-visual evidence collection and management system in the embodiment of this application, based on the mobile audio-visual evidence collection and management system introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as the technical personnel of this field implement the electronic device used in a mobile audio-visual evidence collection and management system in the embodiment of this application, it belongs to the scope of protection of this application.

[0140] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0141] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A mobile video and audio evidence collection and management system, characterized in that: include: Evidence capture module, used to collect comprehensive audio and video data and environmental monitoring data; A data processing module is used to perform preliminary processing on the audio and video comprehensive data to obtain a time-synchronized audio and video feature data set; The multimodal data fusion module is used to fuse the time-synchronized audio and video feature data set and the environmental monitoring data to obtain a comprehensive feature data set; A content matching degree analysis module is used to obtain a video and audio evidence matching model based on a comprehensive feature data set training, and to predict a comprehensive content matching degree based on the video and audio evidence matching model; The abnormality assessment module determines whether there is an abnormality in the comprehensive audio and video data based on the comprehensive content matching degree. If there is an abnormality, an alarm message is sent to the management terminal; each module is connected by wire and / or wireless means.

2. A mobile video and audio evidence collection and management system according to claim 1, characterized in that: The comprehensive audio-visual data includes video evidence data and audio evidence data; the video evidence data includes video image frame sequence, video frame timestamp, video resolution, video resolution and video compression coding; the audio evidence data includes audio signal frame sequence, audio signal waveform, audio sampling rate, audio signal timestamp and sound frequency characteristics; the environmental monitoring data includes temperature, humidity, noise intensity, network transmission delay and network signal strength.

3. A mobile video and audio evidence collection and management system according to claim 2, characterized in that: The method of performing preliminary processing on the audio-visual comprehensive data to obtain the time-synchronized audio-visual feature data set includes: The collected video evidence data and audio evidence data are preliminarily processed to obtain video evidence feature data sets and audio evidence feature data sets; the optical flow method is used to estimate the motion between each adjacent frame in the video image frame sequence in the video evidence data; according to the motion between each adjacent frame, a translation motion model is used to fit the overall motion of the camera, and the video image frame sequence is adjusted by geometric transformation according to the translation motion model, and each frame image is de-jittered; the video evidence data after de-jittering is denoised by non-local mean filtering to obtain a video evidence feature data set; The outliers in the audio evidence data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the audio evidence feature data set; the video evidence feature data set and the audio evidence feature data set are normalized by standard deviation and converted into a standard normal distribution to obtain the normalized video evidence feature data set and the audio evidence feature data set; The normalized video evidence feature dataset and audio evidence feature dataset are synchronized using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature dataset.

4. A mobile video and audio evidence collection and management system according to claim 3, characterized in that: The method for performing denoising on the video evidence data after de-jittering using non-local mean filtering comprises: Extract each frame of the video image frame sequence in the video evidence data. For each frame of the video image frame sequence, randomly select a frame as the current frame and perform non-local mean filtering. For each pixel in the current frame, , calculate and all remaining pixels Similarity ;in, Pixel The original pixel value of The remaining pixels The original pixel value, Parameters for controlling weight decay; The weight attenuation parameter restriction formula is used to restrict the parameters that control weight attenuation. The weight attenuation parameter restriction formula is: ;in, is the parameter that controls weight decay after restriction; is the total number of pixels in the image; A scaling factor for adjusting the range of parameters that control weight decay; Use weighted average to calculate the denoised pixel value , the specific mathematical formula is: ;in, is the search window of the current frame image, including pixels The surrounding pixels; Use the denoised pixel values To update all pixels in the current frame image until all pixels are processed; repeat until all frame images are denoised.

5. A mobile video and audio evidence collection and management system according to claim 4, characterized in that: The method of synchronizing the normalized video evidence feature data set and the audio evidence feature data set by using an optimized time synchronization algorithm to obtain a time-synchronized video and audio feature data set includes: S51, the video image frame sequence in the normalized video evidence feature dataset is expressed as: ;in, For video images Feature vector of the frame; is the total number of frames in the video image frame sequence; The normalized audio signal frame sequence in the audio evidence feature dataset is represented as ;in, is the first frame of the audio signal feature vectors; is the total number of frames in the audio signal frame sequence; S52, constructing a cross-correlation function, measuring the similarity between the video image frame sequence and the audio signal frame sequence under different time delays through the cross-correlation function; finding the time alignment between the video image frame sequence and the audio signal frame sequence by maximizing the cross-correlation; The cross-correlation function is: ;in, The video image frame sequence and the audio signal frame sequence are delayed at different times Weighted similarity under ; is the first frame in the audio signal sequence Frame, indicating that the audio signal has been delayed by time The following frame; is a time delay parameter, which represents the time difference between the video image frame sequence and the audio signal frame sequence; It is the minimum length that can be processed after the video image frame sequence and the audio signal frame sequence are time synchronized; For the Frames are delayed The window size below; is the index of the video image frame sequence and the audio signal frame; S53, for Frames are delayed The window size is dynamically adjusted by comparing the video image frame sequence and the audio signal frame sequence at different time delays. The weighted similarity under the condition of , select the optimal time delay corresponding to the maximum weighted similarity , according to the best time delay , adjust the audio signal frame to synchronize it with the video image frame; if , the audio signal moves forward frame; if , the audio signal moves backward frame; if , no time adjustment is made; after adjusting the video image frame sequence and the audio signal frame sequence, a time-synchronized video and audio feature data set is obtained.

6. The mobile video and audio evidence collection and management system according to claim 5, characterized in that: The said Frames are delayed The following methods are used to dynamically adjust the window size: Through the dynamic window adjustment formula Frames are delayed The window size is dynamically adjusted and designed; the dynamic window adjustment formula is: ;in, After dynamic adjustment Frames are delayed The window size below; is the rate of change of video image frames; is the rate of change of audio signal frames; is the video impact factor; is the audio impact factor; is the scaling factor.

7. The mobile video and audio evidence collection and management system according to claim 6, characterized in that: The method of fusing the time-synchronized audio and video feature data set with the environmental monitoring data to obtain a comprehensive feature data set includes: The outliers in the environmental monitoring data are identified by the DBSCAN algorithm, and the identified outliers are removed to obtain the environmental monitoring feature data set; the environmental monitoring feature data set is normalized by standard deviation and converted into a standard normal distribution to obtain the normalized environmental monitoring feature data set; The time-aligned audio and video feature dataset and the normalized environmental monitoring feature dataset are fused through a weighted formula to obtain a comprehensive feature dataset. The time-aligned audio and video feature dataset is recorded as , the normalized environmental monitoring feature data set is recorded as ; The weighted formula is: ;in, It is a comprehensive feature dataset; is the weight coefficient of the time-aligned audio and video feature dataset; is the weight coefficient of the normalized environmental monitoring feature dataset.

8. The mobile video and audio evidence collection and management system according to claim 7, characterized in that: The training method of the video and audio evidence matching model includes: Divide the dataset into training set, validation set and test set, train the model and evaluate the model performance; build an audio-visual evidence matching model, where the sample set is a subset of the dataset, and each sample set includes a historical comprehensive feature dataset and the corresponding comprehensive content matching degree; The audio-visual evidence matching model includes an input layer, a hidden layer and an output layer; the hidden layer uses a ReLU activation function; the input layer of the model is used to input a historical comprehensive feature data set; the output layer of the model is used to output a comprehensive content matching degree; the audio-visual evidence matching model is a multimodal neural network model; The mean square error is used as the loss function to measure the error between the model's predicted value and the actual value; the audio-visual evidence matching model is trained using the training set, and the model parameters are updated through the back-propagation algorithm and the gradient descent method to minimize the loss function; the performance of the audio-visual evidence matching model is evaluated using the validation set by calculating the coefficient of determination, and the hyperparameters of the model are tuned; The Adam optimization algorithm is selected as the optimizer, and the hyperparameters of the model are adjusted until the performance no longer improves or the preset number of iterations is reached. The test set is used to evaluate the performance of the model in the prediction task, and the trained audio-visual evidence matching model is used to predict the current comprehensive feature data set to obtain the comprehensive content matching degree.

9. The mobile video and audio evidence collection and management system according to claim 8, characterized in that: The method for tuning the hyperparameters of the model includes: S91. Randomly generate a set of initial hyperparameter combinations, each of which corresponds to a gray wolf individual; the position of each gray wolf individual is a dimensional vector; where is the number of hyperparameters that need to be tuned; Gray wolf individuals, each gray wolf individual contains hyperparameters, The location of the gray wolf individuals is: ;in, For the The location of individual gray wolves; For the The first Hyperparameter values; is the index of the individual gray wolf; S92. Use the mean square error loss function to measure the fitness value of each hyperparameter combination. Each gray wolf individual calculates the fitness value according to the mean square error loss function. The smaller the loss value of the loss function, the higher the fitness value of the gray wolf individual. According to the fitness value of the gray wolf individual, select the gray wolf individual with the highest fitness value as Wolf, select the gray wolf individual with the second highest fitness value as The wolf and the gray wolf with the third highest fitness value are selected as Wolf; S93, each gray wolf individual Wolf, Wolf and The wolf approaches to adjust its position, explore and update the hyperparameter combination, and the adjustment of the individual position of the gray wolf follows the gray wolf position update formula; the gray wolf position update formula is: ;in, No. The gray wolf individual The hyperparameter positions of the generations, i.e., the hyperparameter combinations; For the The gray wolf individual The hyperparameter positions of the generations; , and Represents the random factor related to position update, controlling the step size and search direction of the individual gray wolf; For the Gray wolf individuals and the distance between wolves; is the current iteration number; is the next iteration number; Random Factor The calculation formula is: ;in, is a linear decreasing factor; A random number to control the randomness of the search direction, ranging from ; Random Factor The calculation formula is: ;in, A random number that controls the size of the gray wolf's individual search step, ranging from ; Random Factor The calculation formula is: ; Among them, is a random number that controls the search direction of the gray wolf individual, ranging from ; The linear decline factor is dynamically adjusted through the linear decline factor adjustment formula; the linear decline factor adjustment formula is: ;in, is the linear decreasing factor after dynamic adjustment; is the maximum value of the linear decreasing factor; is the maximum number of iterations; Parameters for controlling the rate of deceleration; S94. Update the position of each individual gray wolf one by one through the gray wolf position update formula, so as to adjust the hyperparameter combination. In each iteration, recalculate the fitness value of each individual gray wolf. If a gray wolf individual has a higher fitness value, update Wolf, Wolf and The position of the wolf, preset the maximum number of iterations, and stop when the number of iterations reaches the maximum number of iterations; in the final iteration, select The position of the wolf is taken as the optimal hyperparameter combination, that is, under this hyperparameter combination, the performance of the model is optimal.

10. The mobile video and audio evidence collection and management system according to claim 9, characterized in that: The method for judging whether the audio and video comprehensive data is abnormal based on the comprehensive content matching degree includes: If the predicted comprehensive content matching degree is less than the preset comprehensive content matching degree, it is determined that the audio and video comprehensive data is abnormal; If the predicted comprehensive content matching degree is greater than or equal to the preset comprehensive content matching degree, it is determined that there is no abnormality in the audio and video comprehensive data.

Images