Monitoring audio and video joint optimization method based on cross-modal attention mechanism

Abstract

The invention discloses a monitoring audio and video joint optimization method based on a cross-modal attention mechanism. The monitoring audio and video joint optimization method comprises the steps of performing noise reduction on an audio in an audio file obtained by monitoring; constructing a feature mapping relationship between the user features in the video file obtained by monitoring and the voice features in the audio file after noise reduction; performing feature identification on the audio after noise reduction through the feature mapping relation to obtain an identification result; performing audio and video synchronous matching based on the identification result and the deviation verification time interval to obtain a synchronous matching result; and determining the number of feature frames of the video according to the correction demand information, generating correction feature frames according to the number of the feature frames and the synchronous matching result, and performing synchronous adjustment of the audio and the video according to the correction feature frames. The problems that in a traditional monitoring system, audios and videos are not synchronous in an outdoor scene, and the on-site sound quality is not clear are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video synchronization processing, and in particular to a monitoring audio and video joint optimization method based on a cross-modal attention mechanism. Background Art

[0002] With the development and popularization of mobile Internet technology, data transmission technology and smart terminals, the monitoring technology used in the security field has become increasingly mature; at the same time, personal safety and social stability have also become a consensus among more people, and more and more monitoring equipment is being deployed in all aspects of various fields.

[0003] In the current traditional security field, people tend to prefer high-quality video recording but ignore the collection and processing of audio data, which results in only images but noisy and unclear sound when retrieved, or the retrieved images and sound are out of sync, etc. This will undoubtedly bring great difficulties to the rapid and accurate detection of abnormal audio and video events in key areas and scenes, leading to potential major risks to public safety.

[0004] The actual application in monitoring scenarios requires not only high-definition video monitoring, but also a monitoring system with audio and video linkage. Audio and video joint optimization monitoring has gradually become a technical development trend in the current security industry. Therefore, the intelligent upgrade of audio and video linkage in the security industry is imminent. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies in the current monitoring field in audio and video linkage, and to provide a monitoring audio and video joint optimization method based on a cross-modal attention mechanism, aiming to specifically solve the problems of audio and video asynchrony and unclear on-site sound quality in outdoor scenes in traditional monitoring systems.

[0006] A surveillance audio and video joint optimization method based on a cross-modal attention mechanism, comprising:

[0007] Perform noise reduction on the audio in the audio file obtained through monitoring;

[0008] Construct a feature mapping relationship between user features in the video file obtained by monitoring and speech features in the denoised audio file;

[0009] Perform feature identification of the denoised audio using the feature mapping relationship to obtain an identification result;

[0010] Perform synchronization matching of audio and video based on the identification result and the deviation verification time interval to obtain a synchronization matching result;

[0011] The number of feature frames of the video is determined according to the correction requirement information, a correction feature frame is generated according to the number of feature frames and the synchronization matching result, and the synchronization adjustment of the audio and video is performed according to the correction feature frame.

[0012] The user feature includes at least one of the user's age, occupation, and gender in the video file; and the voice feature includes at least one of the loudness, pitch, and timbre of the user's voice in the audio file.

[0013] The identification result includes multiple time node identifications, and the time node mark includes a start identification result obtained by marking the time node of the node where the sound starts in the audio.

[0014] Among them, the deviation verification time interval is a preset deviation time interval that does not affect the audio and video synchronization. When the time difference between the audio and video is within the deviation verification time interval, it indicates that the audio and video are synchronized, otherwise they are not synchronized and need to be synchronized.

[0015] The correction requirement information includes an allowable playback time difference between audio and video preset according to an audio and video synchronization accuracy target, and the allowable playback time difference is in milliseconds.

[0016] Wherein, the generating of the correction feature frame according to the feature frame quantity and the synchronization matching result includes:

[0017] The video is divided into frame intervals according to the total duration of the video file and the number of the characteristic frames to obtain interpolation constraint intervals, and the correction characteristic frames are obtained according to the interpolation constraint intervals.

[0018] Wherein, the step of obtaining the corrected feature frame according to the interpolation frame constraint interval includes:

[0019] It is determined whether there is at least one synchronization matching result in each of the interpolation frame constraint intervals, and the correction feature frame is obtained according to the interpolation frame constraint intervals in which the synchronization matching result exists.

[0020] The video and audio in the original monitoring scene are respectively acquired through an audio and video acquisition device, and the audio and video acquisition device includes an on-site sound pickup device and / or a camera.

[0021] The step of reducing the noise of the audio in the audio file obtained through monitoring includes:

[0022] Preprocess the audio to be processed to form target data that can be processed by the recurrent neural network;

[0023] The processed target data is input into a trained recurrent neural network implemented based on a cross-modal attention mechanism network layer for processing to obtain noise-reduced audio.

[0024] The step of preprocessing the audio to be processed to form target data that can be processed by a recurrent neural network includes:

[0025] The audio to be processed is subjected to reverberation enhancement through convolution and filtering through digital filters;

[0026] The filtered audio is subjected to feature extraction through a convolutional neural network to obtain feature values;

[0027] Transforming the characteristic values ​​into target data that can be processed by a recurrent neural network;

[0028] The step of performing shape conversion on the eigenvalues ​​comprises:

[0029] The eigenvalues ​​are Fourier transformed and converted into frequency domain signals. According to the size of the frame, the imaginary and real parts of the frequency domain signals are respectively transformed and concatenated to generate target data processed by a recurrent neural network.

[0030] The present invention enables the audio and video collected by the video collection device and the audio collection device to be synchronized and optimized, thereby solving the problems of audio and video asynchronism and unclear on-site sound quality in outdoor scenes in traditional monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the monitoring audio and video joint optimization method based on the cross-modal attention mechanism of the present invention.

[0032] Figure 2 The flowchart of performing noise reduction processing on audio according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] See also Figure 1 As shown, a monitoring audio and video joint optimization method based on a cross-modal attention mechanism of the present invention includes:

[0035] S1. Denoise the audio in the audio file obtained by monitoring;

[0036] S2. Constructing a feature mapping relationship between user features in the video file obtained by monitoring and speech features in the denoised audio file;

[0037] S3. The feature mapping relationship is used to identify the characteristics of the audio after noise reduction, and obtain the identification result;

[0038] S4. Synchronize the audio and video based on the identification result and the deviation verification time interval to obtain a synchronization matching result;

[0039] S5. Determine the number of feature frames of the video according to the correction requirement information, generate correction feature frames according to the number of feature frames and the synchronization matching result, and perform synchronization adjustment of audio and video according to the correction feature frames.

[0040] In step S1, the audio in the audio file obtained through monitoring can be denoised by a recurrent neural network based on a cross-modal attention mechanism, so that the audio denoising process meets the required requirements.

[0041] In step S2, the feature mapping relationship reflects the correspondence between the audio sound data and the user in the video. By establishing a mapping or correspondence, it is convenient to synchronously process the audio and video of different users, and the corresponding voice features can be found according to the user characteristics.

[0042] Among them, the user characteristics include at least one of the user's age, occupation, gender in the video file, or other physiological or biological characteristics that can distinguish one user from another, and may be but not limited to the user's age, occupation, and gender; the voice characteristics include at least one of the loudness, pitch, and timbre of the user's voice in the audio file, or other audio data characteristics that can distinguish the voices of different users, and may be but not limited to loudness, pitch, timbre, etc.

[0043] In some implementation schemes, the identification result includes multiple time node identifications, and the time node marking includes a start identification result obtained by marking the time node of the node where the sound starts in the audio, that is, a start speech marking result.

[0044] In some implementation schemes, the identification result also includes a starting text recognition result, and the sound in the audio file is converted into text by voice to obtain the content of the initial text of each sentence in the audio, thereby obtaining the starting text recognition result. The starting voice identification result can be corrected through the starting text identification result. If the starting voice identification result is inconsistent with the starting text mark, the starting text identification result shall prevail as the user's identification result.

[0045] Through the processing of step S3, the start node of the audio file is identified, which lays a foundation for improving the accuracy of audio and video synchronization processing.

[0046] Among them, in step S4, the deviation verification time interval is a preset deviation time interval that does not affect the synchronization of audio and video. Since the propagation speeds of sound signals during audio acquisition and light signals during video acquisition in the air are different, as well as the audio acquisition position and the video acquisition position are different, there is an inevitable deviation between the collected audio files and video files, that is, the deviation time interval that does not affect the synchronization of audio and video. For this purpose, a corresponding deviation verification time interval is set to solve this problem. When the time difference between the audio and video is within the deviation verification time interval, it indicates that the audio and video are synchronized. Otherwise, they are out of sync and need to be synchronized. It can be determined specifically according to the requirements for audio and video synchronization, in units of time, such as 2 / 3 seconds.

[0047] Among them, in step S5, the correction requirement information includes the allowed playback time difference between audio and video preset according to the audio and video synchronization accuracy target, which is used to generate a correction feature frame for subsequent fine-tuning, and the allowed playback time difference is in milliseconds.

[0048] The number of feature frames is the number of feature frames that are mainly processed when the original video file is synchronized according to the correction requirement information. The more feature frames there are, the higher the adjustment accuracy and the higher the audio and video synchronization quality. By analyzing the requirements of different tasks for synchronization accuracy and setting the number of feature frames according to the requirements, the maximum balance between quality and cost can be achieved, and the maximum synchronization effect can be achieved.

[0049] In some implementation schemes, generating a correction feature frame according to the feature frame quantity and the synchronization matching result includes:

[0050] The video is divided into frame intervals according to the total duration of the video file and the number of the characteristic frames to obtain interpolation constraint intervals, and the correction characteristic frames are obtained according to the interpolation constraint intervals.

[0051] Wherein, the step of obtaining the corrected feature frame according to the interpolation frame constraint interval includes:

[0052] It is determined whether there is at least one synchronization matching result in each of the interpolation frame constraint intervals, and the correction feature frame is obtained according to the interpolation frame constraint intervals in which the synchronization matching result exists.

[0053] The interpolation constraint interval in which the matching recognition result does not exist is determined as an abnormal interpolation constraint interval; and it is determined whether the abnormal interpolation interval is a human voice interval. When the abnormal interpolation interval is a human voice interval, the sound correction is not performed on the abnormal interpolation constraint interval.

[0054] The video and audio in the original monitoring scene are respectively acquired through the audio and video acquisition device, and the audio and video acquisition device includes a field sound pickup device and / or a camera. The audio is acquired by acquiring the sound of the object to be acquired through the field sound pickup device, etc., including the original sound data, and the video is acquired by acquiring the video image of the object through an acquisition device such as a camera.

[0055] In some embodiments, the step of reducing the noise of the audio in the audio file obtained through monitoring includes:

[0056] The audio to be processed is preprocessed to form target data that can be processed by a recurrent neural network; the processed target data is input into a trained recurrent neural network implemented based on a cross-modal attention mechanism network layer for processing to obtain denoised audio.

[0057] After being processed in the trained recurrent neural network implemented based on the cross-modal attention mechanism network layer, the output data is masked, inverse Fourier transformed, and reconstructed to obtain the final noise-reduced speech. The overall processing flow is as follows: Figure 2 shown.

[0058] In the present application, the recurrent neural network for noise reduction processing of the present application is obtained by pre-collecting audio data and training the constructed recurrent neural network. The construction process of the recurrent neural network includes: real-time acquisition and pre-processing of original audio data through a live sound pickup device, etc.; time-domain processing of the pre-processed original audio data, converting it into a frequency domain signal corresponding to the time domain, and performing shape transformation and splicing processing on the real part and the imaginary part of the frequency domain signal according to the size of the frame, respectively, to generate input data of the convolutional neural network;

[0059] Extracting corresponding feature data from the generated convolutional neural network input data using the convolutional neural network;

[0060] The extracted corresponding feature data is input into the recurrent neural network built based on the cross-modal attention mechanism network layer, and the network is iteratively trained to generate an optimized recurrent neural network as a speech denoising network model.

[0061] Among them, when performing network training, the feature data is combined with the cross-modal attention mechanism network layer to generate recurrent neural network data, the corresponding loss function of the cross-modal attention mechanism network is calculated and generated, and the corresponding weight matrix of the cross-modal attention mechanism network is updated and processed in combination with gradient descent to generate an optimized recurrent neural network. Specifically, according to the loss function value and the initialization of the cross-modal attention mechanism layer, the weight matrix gradient data corresponding to the self-attention mechanism is generated; combined with multiple iterative training, it is determined whether the recurrent neural network data meets the preset condition value. After determining that the preset condition value is met, the iteration is terminated to form an optimized recurrent neural network as a speech denoising network model.

[0062] During the training iteration of the cross-modal attention mechanism network layer, the input data is converted into the input of the cross-modal attention mechanism encoding structure after passing through the convolution layer, and the size model is reduced. In the encoding and decoding parts, a multi-head self-attention mechanism and forward propagation are used; the cross-modal attention mechanism network loss function takes into account the input data characteristics and the role of the cross-modal attention mechanism network, and adopts the logarithmic mean square error loss. During the training iteration of the cross-modal attention mechanism network layer, the weight matrix should be continuously updated, and q, k and v are initialized according to the loss value and the self-attention mechanism layer. The weight matrix gradient of q, k and v in each self-attention mechanism is calculated, and then reversely derived by stochastic gradient descent, and the weight matrix update value is calculated. After calculating with the initial weight value, the weight is updated, and the recurrent neural network parameters are obtained after multiple iterative training or when the conditions are met; wherein, the weight matrix is ​​updated using the following formula:

[0063]

[0064] Among them, q represents query, k represents key, and v represents value. is the latitude of the input, Attention(q,k,v) represents the output of each attention head; the three vectors are obtained by multiplying the input by the corresponding weight matrix. First, the input sequence is multiplied by three different weight matrices to obtain q, k, and v vectors. Secondly, for each q vector, a dot product operation is performed with all k vectors, and then a scaling operation and a Softmax function are performed to obtain the attention distribution of the q vector for the k vector. Finally, the distribution is applied to the v vector to obtain the final output result for the current input.

[0065] Among them, the cross-modal attention mechanism is a neural network technology that assigns attention weights by calculating the similarity or correlation between different modalities. These weights are used to weight the features of different modalities to generate a fused representation that contains the most relevant information from all modalities. Convolutional Neural Network (CNN) is a classic feedforward neural network that has made significant progress in other fields such as image recognition and target detection, and has become an important part of computer vision and deep learning research. The present invention addresses the challenge of speech denoising in complex environments in the field of monitoring, makes full use of the powerful ability of the cross-modal attention mechanism network layer in processing sequence data, combines the advantages of convolutional neural networks in feature extraction, performs speech denoising, removes noise data in audio, obtains pure speech data, achieves the expected denoising effect, and facilitates accurate or precise synchronization of audio and video in the later stage.

[0066] In some embodiments, the audio to be processed is preprocessed to form target data that can be processed by a recurrent neural network, see the process as follows Figure 2 As shown, the method includes: performing reverberation enhancement on the audio (original speech) to be processed by convolution, and filtering by a digital filter; extracting features of the filtered audio (speech data) by a convolutional neural network to obtain eigenvalues, and forming target data that can be processed by a recurrent neural network; performing shape transformation on the eigenvalues, transferring them to the Transformer network layer, and finally processing to obtain denoised speech data.

[0067] Among them, the shape transformation of the eigenvalues ​​includes Fourier transforming the eigenvalues ​​to convert them into frequency domain signals, and according to the size of the frame, the imaginary and real parts of the frequency domain signals are respectively transformed and concatenated to generate target data processed by a recurrent neural network, which is input into the Transformer network layer and finally processed to obtain noise-reduced speech data.

[0068] In the present invention, by performing reverberation convolution on the speech and using digital filtering for preliminary calculation, the pre-processed speech signal can effectively filter some special noise or useless information, effectively avoiding its interference with the subsequent noise reduction processing; on the other hand, after enhancing the speech by means of reverberation convolution, the pure speech in the noise-reduced speech can have a certain gain, which is conducive to obtaining a clearer and more accurate noise reduction effect. The speech feature value after pre-processing is subjected to short-time fast Fourier transform, and the speech signal is converted into a frequency domain signal in this way. By converting the time series signal into a spectrum, and then using a convolutional neural network to extract the features of the speech signal, the target data processed by the recurrent neural network is generated.

[0069] The data extracted by the convolutional neural network layer is converted and input into the recurrent neural network built based on the cross-modal attention mechanism network layer. The significant advantages of the cross-modal attention mechanism structure for sequence data processing are utilized. After its decoding structure, the forward calculation result of the entire network, that is, the mask of the input data, is obtained. Finally, the final gain value is obtained by performing point multiplication operations on the result obtained by the cross-modal attention mechanism network layer and the result obtained by the initial fast Fourier transform. After inverse fast Fourier transform, windowing and signal reconstruction, the final result is obtained to complete speech denoising.

[0070] The present invention can solve the technical problems of inaccurate audio information of the sound pickup device in existing monitoring scenarios and low synchronization rate when linked with the camera, can effectively improve the monitoring sound quality, realize efficient audio and video linkage, and help to quickly and accurately detect abnormal audio and video events in monitoring scenarios.

[0071] The present invention enables the audio and video collected by the video collection device and the audio collection device to be synchronized and optimized, thereby solving the problems of audio and video asynchronism and unclear on-site sound quality in outdoor scenes in traditional monitoring systems.

[0072] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention.

[0073] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the claims are included in the present invention.

[0074] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A surveillance audio and video joint optimization method based on a cross-modal attention mechanism, characterized in that: include: Perform noise reduction on the audio in the audio file obtained through monitoring; Construct a feature mapping relationship between user features in the video file obtained by monitoring and speech features in the denoised audio file; Perform feature identification of the denoised audio using the feature mapping relationship to obtain an identification result; Perform synchronization matching of audio and video based on the identification result and the deviation verification time interval to obtain a synchronization matching result; The number of feature frames of the video is determined according to the correction requirement information, a correction feature frame is generated according to the number of feature frames and the synchronization matching result, and the synchronization adjustment of the audio and video is performed according to the correction feature frame.

2. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 1 is characterized in that: The user feature includes at least one of the user's age, occupation, and gender in the video file; and the voice feature includes at least one of the loudness, pitch, and timbre of the user's voice in the audio file.

3. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 1 is characterized in that: The identification result includes multiple time node identifications, and the time node mark includes a start identification result obtained by marking the time node of the node where the sound starts in the audio.

4. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 1 is characterized in that: The deviation verification time interval is a preset deviation time interval that does not affect the audio and video synchronization. When the time difference between the audio and the video is within the deviation verification time interval, it indicates that the audio and the video are synchronized, otherwise they are not synchronized and need to be synchronized.

5. According to claim 1, the monitoring audio and video joint optimization method based on the cross-modal attention mechanism is characterized in that: The correction requirement information includes an allowable playback time difference between audio and video preset according to an audio and video synchronization accuracy target, and the allowable playback time difference is in milliseconds.

6. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 4 is characterized in that: The step of generating a correction feature frame according to the feature frame quantity and the synchronization matching result includes: The video is divided into frame intervals according to the total duration of the video file and the number of the characteristic frames to obtain interpolation constraint intervals, and the correction characteristic frames are obtained according to the interpolation constraint intervals.

7. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 6 is characterized in that: The step of obtaining the corrected feature frame according to the interpolation frame constraint interval includes: It is determined whether there is at least one synchronization matching result in each of the interpolation frame constraint intervals, and the correction feature frame is obtained according to the interpolation frame constraint intervals in which the synchronization matching result exists.

8. According to claim 1, the monitoring audio and video joint optimization method based on the cross-modal attention mechanism is characterized in that: The video and audio in the original monitoring scene are respectively acquired through an audio and video acquisition device, wherein the audio and video acquisition device includes an on-site sound pickup device and / or a camera.

9. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 1 is characterized in that: It is characterized in that The step of reducing the noise of the audio in the audio file obtained through monitoring includes: Preprocess the audio to be processed to form target data that can be processed by the recurrent neural network; The processed target data is input into a trained recurrent neural network implemented based on a cross-modal attention mechanism network layer for processing to obtain noise-reduced audio.

10. The monitoring audio and video joint optimization method based on the cross-modal attention mechanism according to claim 9 is characterized in that: The method of preprocessing the audio to be processed to form target data that can be processed by a recurrent neural network includes: The audio to be processed is subjected to reverberation enhancement through convolution and filtering through digital filters; The filtered audio is subjected to feature extraction through a convolutional neural network to obtain feature values; Transforming the characteristic values ​​into target data that can be processed by a recurrent neural network; The step of performing shape conversion on the eigenvalues ​​comprises: The eigenvalues ​​are Fourier transformed and converted into frequency domain signals. According to the size of the frame, the imaginary and real parts of the frequency domain signals are respectively transformed and concatenated to generate target data processed by a recurrent neural network.

Images