An audio and video signal processing system and a video conference terminal device using the same

Abstract

The application provides an audio and video signal processing system and a video conference terminal device using the same, and relates to the technical field of Internet.The system comprises an audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a master control module, the audio and video processing module is used for separating audio signals and video signals in input audio and video data signals, the signal preprocessing module is used for preprocessing the audio signals, the signal delay measurement module is used for measuring relative delay amounts of the audio signals and the video signals, the signal synchronization processing module is used for performing delay compensation according to the relative delay amounts, realizing the synchronization of audio and video display, and the video conference terminal device further comprises a microphone, a camera, a display screen and a sound box.The system and the video conference terminal device can test the relative delay time of audio and video signals, perform delay compensation, ensure the synchronization of output audio and video pictures, and improve the video conference quality and user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, in particular to an audio and video signal processing system and a video conference terminal device using the same. BACKGROUND

[0002] With the development of electronic information technology and Internet technology, more and more online conferences replace traditional offline conferences. Online conferences save the entity cost and labor cost such as site and transportation compared with offline conferences, and can save a lot of time, so they are favored by more and more enterprises.

[0003] Online conferences are divided into remote voice conferences and remote video conferences. The voice communication quality of the commonly used remote conference software and remote conference terminal is good at present, but in the case of video communication, due to network communication delay and other problems, the audio and picture are easy to be out of synchronization, that is, there is a delay between the audio and the picture, which affects the conference quality and the user's participation experience.

[0004] In view of the above reasons, the present application provides an audio and video signal processing system and a video conference terminal using the same to solve the above technical problems. SUMMARY

[0005] The present application aims to provide an audio and video signal processing system and a video conference terminal device using the same, which can test the relative delay time of audio and video signals, and perform delay compensation to ensure the synchronization of output audio and video pictures, and improve the video conference quality and user experience.

[0006] In one aspect, the present application provides an audio and video signal processing system, comprising: an audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a main control module, the main control module is used for outputting audio and video data signals to the audio and video processing module, the audio and video processing module is used for separating the audio signal and the video signal in the input audio and video data signal, the signal preprocessing module is used for preprocessing the audio signal, the signal delay measurement module is used for measuring the relative delay amount of the audio signal and the video signal, and the signal synchronization processing module is used for performing delay compensation according to the relative delay amount measured by the signal delay measurement module to realize the synchronization of audio and video display.

[0007] Preferably, the preprocessing operation of the signal preprocessing module on the audio signal includes noise reduction, equalization, dynamic range control and reverb suppression.

[0008] Preferably, the signal delay measurement module comprises a synchronization signal generator and an oscilloscope connected thereto, and the audio signal and the video signal to be measured are input into the synchronization signal generator, and then measured and displayed by the oscilloscope connected thereto.

[0009] Preferably, the audio signal and the video signal are de-embedded by the LipSync function of the synchronization signal generator, then pass through the audio and video links, and finally are embedded by the AV Delay function of the oscilloscope to measure the relative delay amount of the audio signal and the video signal.

[0010] Preferably, the audio and video processing module comprises a demultiplexing unit, a digital-to-analog conversion unit and a signal separation unit,

[0011] The demultiplexing unit is configured to demultiplex the audio and video data signal to obtain an audio decoding signal and a video decoding signal.

[0012] The digital-to-analog conversion unit is configured to convert the audio decoding signal and the video decoding signal into the audio signal and the video signal.

[0013] The signal separation unit is configured to separate the audio signal and the video signal.

[0014] Preferably, the signal preprocessing module comprises a noise reduction unit, an audio equalizer, a dynamic range control unit and a reverb suppression unit, the noise reduction unit is configured to reduce noise of the audio signal, the audio equalizer is configured to adjust equalization of the audio signal, the dynamic range control unit is configured to control dynamic range of the audio signal, and the reverb suppression unit is configured to suppress reverb of the audio signal.

[0015] Preferably, the signal synchronization processing module can generate delay compensation time and time stamp according to the relative delay amount measured by the signal delay measurement module, so as to realize synchronization of audio playing and video displaying.

[0016] In another aspect, the present application provides a video conference terminal device using the above-mentioned audio and video signal processing system, which further comprises a microphone, a camera, a display screen and a sound box, wherein the microphone is connected to the main control module, the main control module outputs the audio and video data signal collected by the microphone to the audio and video processing module, and the signal synchronization processing module is connected to the display screen and the sound box, and the audio signal and the video signal after delay compensation are respectively transmitted to the sound box and the display screen.

[0017] Preferably, the microphone is a cardioid directional microphone.

[0018] Preferably, the signal synchronization processing module is connected with the sound box through a power amplifier module.

[0019] Compared with the prior art, the application has the following beneficial effects:

[0020] The system can control the audio and video data signal to be decoded and output to the audio and video processing module through the main control module, and the input audio and video data signal can be decoded, digital-to-analog converted and separated through the audio and video processing module, so that the audio signal and the video signal are separated, the signal preprocessing module is used for preprocessing the audio signal to improve the quality of the audio signal, and the signal delay measurement module is used for measuring the relative delay amount of the audio signal and the video signal, so that the relative delay amount can be materialized, and the signal synchronization processing module is used for delay compensation according to the measured size of the relative delay amount, so that the synchronization of audio and video display is realized, and the quality of the remote video conference is ensured, and the use experience of the user is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art of the present application, the drawings needed in the specific embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 The composition block diagram of the audio and video signal processing system of the present application is shown in the figure.

[0023] Figure 2 The specific composition block diagram of the audio and video processing module in the present application is shown in the figure.

[0024] Figure 3 The specific composition block diagram of the audio preprocessing module in the present application is shown in the figure.

[0025] Figure 4 The composition block diagram of the video conference terminal device of the present application is shown in the figure. DETAILED DESCRIPTION

[0026] The technical solutions of the present application will be described in detail below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0027] In the description of the present application, it is to be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", and the like indicate the orientation or positional relationship shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0028] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features referred to. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, "a plurality of" means two or more, unless otherwise specifically limited. In addition, the terms "mounting", "connecting", "connecting" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0029] As Figure 1 shown, the present application provides an audio and video signal processing system, which comprises an audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a main control module, wherein the main control module is used to control the work of other modules, the main control module can control the output of the collected audio and video data signal to the audio and video processing module, the audio and video processing module is used to separate the audio signal and the video signal in the input audio and video data signal, and then the audio signal is preprocessed through the signal preprocessing module, the signal delay measurement module is used to measure the relative delay amount of the processed audio signal and the video signal, the signal synchronization processing module is used to compensate for the delay according to the relative delay amount measured by the signal delay measurement module, and the synchronization of audio and video display is realized.

[0030] As Figure 2As shown, specifically, the audio and video processing module includes a demultiplexing unit, a digital-to-analog conversion unit, and a signal separation unit. The demultiplexing unit is used to demultiplex the audio and video data signals acquired by audio and video acquisition devices such as microphones and cameras, thereby obtaining audio decoded signals and video decoded signals for output. The digital-to-analog conversion unit is used to convert the audio decoded signals and video decoded signals obtained by the demultiplexing unit into digital signals and video signals. The signal separation unit is used to separate the audio signals and video signals to facilitate subsequent processing of the audio signals.

[0031] like Figure 3 As shown, specifically, the preprocessing operations performed on the audio signal by the signal preprocessing module include noise reduction, equalization, dynamic range control, and reverberation suppression. The signal preprocessing module includes a noise reduction unit, an audio equalizer, a dynamic range control unit, and a reverberation suppression unit. The noise reduction unit is used to reduce noise in the audio signal. In situations such as video conferencing, both steady-state noise signals (such as continuous background noise like air conditioning sounds) and non-steady-state noise signals (such as footsteps and applause) are easily generated. The noise reduction unit mainly uses Kalman filtering and Wiener filtering methods to analyze and process the audio signal to suppress noise. Kalman filtering is a state estimation-based noise reduction method that estimates the state of the speech signal and filters the estimation error to obtain a relatively clean speech signal, suitable for filtering steady-state noise signals in audio signals. Wiener filtering, based on the minimum mean square error criterion, estimates the power spectrum of the noisy speech and noise, calculates an optimal filtering coefficient, and then applies this filtering coefficient to the noisy speech to obtain a relatively clean speech signal, suitable for filtering non-steady-state noise signals in audio signals.

[0032] An audio equalizer is used to adjust the equalization of audio signals, adjusting the frequency band, gain, and quality of the audio, thereby improving the final playback quality on the playback device. A dynamic range control unit is used to control the dynamic range of the audio signal. The specific control method is as follows: an initial value for the effective value detection time is set; the effective value of the input signal is detected based on the initial value of the effective value detection time to obtain the effective value of the signal; the effective value detection time is adjusted based on the effective value of the signal and a preset parameter threshold; the effective value of the input signal is detected again based on the adjusted effective value detection time to obtain the signal effective value after the effective value detection time adjustment; dynamic gain adjustment is performed based on the signal effective value after the effective value detection time adjustment and the preset parameter threshold; the adjusted gain is smoothed based on preset start and release times; and the smoothed gain is then applied to the input signal to obtain the output signal.

[0033] The reverberation suppression unit is used for reverberation suppression on the audio signal, and a speech enhancement unit can be configured to enhance the speech in the audio signal. Through the reverberation suppression unit and the speech enhancement unit, the important part of the speech signal can be enhanced by algorithm, so that it is easier to identify, and the influence of reverberation on speech is reduced. Among them, the reverberation suppression unit uses a deep learning algorithm, such as a recurrent neural network (RNN), to detect and suppress the reflection of sound in space, thereby reducing the influence of reverberation on speech; the speech enhancement unit adopts an artificial intelligence (AI) system with speech interaction function, which can use a neural network to perform speech optimization and semantic recognition on the input speech audio.

[0034] Because the analog signal in the audio and video data is directly transmitted, while the video signal such as HDMI signal needs to be coded and decoded, digital-to-analog converted and processed, the delay of the analog direct transmission signal is tens of milliseconds, and the different transmission modes also cause different degrees of time delay of the video signal compared with the analog signal. Therefore, the relative delay time needs to be measured to facilitate accurate compensation of the delay.

[0035] The signal delay measurement module includes a synchronous signal generator and an oscilloscope connected thereto, and the audio signal and the video signal to be measured are input into the synchronous signal generator, and then measured and displayed by the oscilloscope connected thereto. When measuring, first turn on the LipSync (Leader) or AV Timing (Telestream) function of the synchronous signal generator, input the test signal from an external channel, and after demultiplexing, pass through the audio and video links, and finally embedded by the AV Delay (Telestream) or AV phase (Leader) function of the oscilloscope to measure the relative delay amount of the audio and video.

[0036] The signal synchronization processing module can generate delay compensation time and time stamp according to the relative delay amount measured by the signal delay measurement module, and synchronize the playback of the audio signal in the audio playback device and the playback of the video signal in the display terminal such as a display screen when the system time reaches the time stamp, so as to realize the synchronization of audio playback and video display. When the audio transmission mode is different, the delay compensation time is also different. After the delay compensation of the audio signal, the audio signal is amplified by the power amplifier unit and played in the audio playback device such as a sound box.

[0037] As Figure 4As shown, the application further provides a video conference terminal device using the above-mentioned audio and video signal processing system, which further comprises a microphone, a camera, a display screen and a sound box, wherein the microphone is connected with the master control module, the master control module can output the audio and video data signals collected by the microphone after decoding to the audio and video processing module, separate the audio signals and video signals, the signal synchronization processing module is connected with the display screen and the sound box, and the audio signals and video signals after delay compensation are respectively transmitted to the sound box and the display screen for playing and displaying, wherein the sound box is connected with the signal synchronization processing module through the power amplifier module.

[0038] As a preferred solution, the microphone is a heart-shaped directional microphone array, and other spatially arranged multi-microphone arrays can also be used, which can capture sounds such as voices in all directions and suppress environmental noise.

[0039] The camera is configured with a gimbal, a device module and a picture quality module, wherein the device module is used to connect the camera with a computer or a video conference terminal device; the picture module is used to adjust the camera angle of view and focal length, determine the shooting range, picture quality and direction.

[0040] The video conference terminal device is further configured with a conference platform and a conference management system, which can invite relevant personnel to participate in the conference, test the network, debug the conference environment, record and store the conference agenda and other contents in the cloud.

[0041] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. An audio video signal processing system, characterized by, It includes: Audio and video processing module, signal preprocessing module, signal delay measurement module, signal synchronization processing module and main control module, the main control module is used for outputting audio and video data signal to the audio and video processing module, the audio and video processing module is used for separating the audio signal and the video signal in the input audio and video data signal, the signal preprocessing module is used for pre-processing the audio signal, the pre-processing operation of the signal preprocessing module to the audio signal includes noise reduction, equalization, dynamic range control and reverberation suppression, the signal preprocessing module includes noise reduction unit, audio equalizer, dynamic range control unit and reverberation suppression unit, the noise reduction unit is used for noise reduction of the audio signal, the noise reduction unit uses Kalman filtering and Wiener filtering method to analyze and process the audio signal to suppress noise, the audio equalizer is used for adjusting the equalization of the audio signal, the dynamic range control unit is used for dynamic range control of the audio signal, the reverberation suppression unit is used for reverberation suppression of the audio signal, the reverberation suppression unit uses deep learning algorithm to detect and suppress the reflection of sound in space, thereby reducing the influence of reverberation on voice, the signal delay measurement module is used for measuring the relative delay amount of the audio signal and the video signal, the signal delay measurement module includes a sync signal generator and an oscilloscope connected thereto, the audio signal and the video signal to be measured are input into the sync signal generator, and then measured and displayed by the oscilloscope connected thereto, the signal synchronization processing module is used for delay compensation according to the relative delay amount measured by the signal delay measurement module, and delay compensation time and time stamp can be generated according to the relative delay amount measured by the signal delay measurement module, so as to realize the synchronization of audio playing and video display.

2. The audio-video signal processing system of claim 1, wherein, The audio signal and the video signal are disembedded by the LipSync function of the sync signal generator, pass through the audio and video link, and then are embedded at the end stage, and the relative delay amount of the audio signal and the video signal is measured by the AV Delay function of the oscilloscope.

3. The audio-video signal processing system of claim 1, wherein, The audio and video processing module includes a demultiplexing unit, a digital-to-analog conversion unit and a signal separation unit, The demultiplexing unit is used for demultiplexing the audio and video data signal to obtain an audio decoding signal and a video decoding signal; The digital-to-analog conversion unit is used for digital-to-analog conversion of the audio decoding signal and the video decoding signal to obtain the audio signal and the video signal; The signal separation unit is used for separating the audio signal and the video signal.

4. A video conference terminal apparatus using the audio video signal processing system according to any one of claims 1 to 3, characterized by It also includes a microphone, a camera, a display screen and a sound box, the microphone is connected with the main control module, the main control module outputs the audio and video data signal collected by the microphone to the audio and video processing module, the signal synchronization processing module is connected with the display screen and the sound box, and the audio signal and the video signal after delay compensation are respectively transmitted to the sound box and the display screen.

5. The video conference terminal device according to claim 4, characterized by The microphone is a heart-shaped directional microphone.

6. The video conference terminal device according to claim 4, characterized by The signal synchronization processing module is connected with the sound box through a power amplifier module.

Images