Audio and video signal processing system and video conference terminal equipment using same

Abstract

The invention provides an audio and video signal processing system and video conference terminal equipment using the same, and relates to the technical field of Internet. The system comprises an audio and video processing module, a signal preprocessing module, a signal delay measuring module, a signal synchronous processing module and a main control module, the audio and video processing module is used for separating audio signals and video signals in input audio and video data signals, and the signal preprocessing module is used for preprocessing the audio signals and processing the video signals; the signal delay measurement module is used for measuring the relative delay amount of an audio signal and a video signal, the signal synchronization processing module is used for carrying out delay compensation according to the relative delay amount to realize synchronization of audio and video display, and the video conference terminal equipment further comprises a microphone, a camera, a display screen and a sound box. The system and the video conference terminal equipment can test the relative delay time of the audio and video signals, perform delay compensation, ensure the synchronization of the output audio and video pictures, and improve the video conference quality and the user experience.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to an audio and video signal processing system and a video conferencing terminal device using the same. Background Art

[0002] With the development of electronic information technology and Internet technology, more and more online meetings have replaced traditional offline meetings. Compared with offline meetings, online meetings save physical costs such as venues and transportation, as well as labor costs, and can save a lot of time, and are favored by more and more companies.

[0003] Online meetings are divided into remote voice meetings and remote video meetings. For commonly used remote meeting software and remote meeting terminals, the current voice communication quality is relatively good. However, in scenarios where video communication is required, due to problems such as network communication delays, it is easy for the audio and picture to be out of sync, that is, there is a delay between the audio and the picture, which affects the quality of the meeting and the user's meeting experience.

[0004] In view of the above reasons, the present invention provides an audio and video signal processing system and a video conferencing terminal using the same to solve the above technical problems. Summary of the invention

[0005] The purpose of the present invention is to provide an audio and video signal processing system and a video conferencing terminal device using the same. The system and the video conferencing terminal device can test the relative delay time of audio and video signals and perform delay compensation to ensure the synchronization of the output audio and video images, thereby improving the quality of video conferencing and user experience.

[0006] On the one hand, the present invention provides an audio and video signal processing system, comprising: an audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a main control module, wherein the main control module is used to output audio and video data signals to the audio and video processing module, the audio and video processing module is used to separate audio signals and video signals in the input audio and video data signals, the signal preprocessing module is used to preprocess the audio signal, the signal delay measurement module is used to measure the relative delay amount of the audio signal and the video signal, and the signal synchronization processing module is used to perform delay compensation according to the relative delay amount measured by the signal delay measurement module to achieve synchronization of audio and video display.

[0007] Preferably, the preprocessing operations performed by the signal preprocessing module on the audio signal include noise reduction, equalization, dynamic range control and reverberation suppression.

[0008] Preferably, the signal delay measurement module includes a synchronization signal generator and an oscilloscope connected thereto, and after the audio signal and the video signal to be measured are input into the synchronization signal generator, they are measured and displayed by the oscilloscope connected thereto.

[0009] Preferably, the audio signal and the video signal are de-embedded by the LipSync function of the synchronization signal generator and pass through the audio and video link, and are embedded at the final stage, and the relative delay of the audio signal and the video signal is measured by the AV Delay function of the oscilloscope.

[0010] Preferably, the audio and video processing module includes a demultiplexing unit, a digital-to-analog conversion unit and a signal separation unit.

[0011] The demultiplexing unit is used to demultiplex the audio and video data signals to obtain an audio decoding signal and a video decoding signal;

[0012] The digital-to-analog conversion unit is used to perform digital-to-analog conversion on the audio decoded signal and the video decoded signal to obtain the audio signal and the video signal;

[0013] The signal separation unit is used to separate the audio signal and the video signal.

[0014] Preferably, the signal preprocessing module includes a noise reduction unit, an audio equalizer, a dynamic range control unit and a reverberation suppression unit, the noise reduction unit is used to reduce noise on the audio signal, the audio equalizer is used to adjust the equalization of the audio signal, the dynamic range control unit is used to perform dynamic range control on the audio signal, and the reverberation suppression unit is used to suppress reverberation on the audio signal.

[0015] Preferably, the signal synchronization processing module can generate a delay compensation time and a timestamp according to the relative delay measured by the signal delay measurement module, so as to achieve synchronization of audio playback and video display.

[0016] On the other hand, the present invention provides a video conferencing terminal device using the above-mentioned audio and video signal processing system, the video conferencing terminal device also includes a microphone, a camera, a display screen and a speaker, wherein the microphone is connected to a main control module, the main control module outputs the audio and video data signals collected by the microphone to the audio and video processing module, the signal synchronization processing module is connected to both the display screen and the speaker, and transmits the audio signal and the video signal after delay compensation to the speaker and the display screen respectively.

[0017] Preferably, the microphone is a cardioid directional microphone.

[0018] Preferably, the signal synchronization processing module is connected to the sound box through a power amplifier module.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The system can control the audio and video data signals to be decoded and output to the audio and video processing module through the main control module. The audio and video processing module can decode, convert digital to analog and separate the input audio and video data signals, so as to obtain separated audio signals and video signals. The signal preprocessing module is used to preprocess the audio signal to improve the quality of the audio signal. The signal delay measurement module is used to measure the relative delay of the audio signal and the video signal so that the relative delay can be visualized. According to the measured relative delay, the signal synchronization processing module performs delay compensation to achieve synchronization of audio and video display, thereby ensuring the quality of remote video conferencing and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a block diagram of the composition of the audio and video signal processing system of the present invention;

[0023] Figure 2 The specific block diagram of the audio and video processing module in the present invention;

[0024] Figure 3 The block diagram of the specific composition of the audio preprocessing module in the present invention;

[0025] Figure 4 The figure is a block diagram of the video conferencing terminal device of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0028] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0029] like Figure 1 As shown, the present invention provides an audio and video signal processing system, which includes: an audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a main control module, wherein the main control module is used to control the operation of other modules, the main control module can control the output of the collected audio and video data signal to the audio and video processing module, the audio and video processing module is used to separate the audio signal and the video signal in the input audio and video data signal, and then preprocess the audio signal through the signal preprocessing module, the signal delay measurement module is used to measure the relative delay of the processed audio signal and video signal, and the signal synchronization processing module is used to perform delay compensation according to the relative delay measured by the signal delay measurement module, so as to realize the synchronization of audio and video display.

[0030] like Figure 2As shown, specifically, the audio and video processing module includes a demultiplexing unit, a digital-to-analog conversion unit and a signal separation unit, wherein the demultiplexing unit is used to demultiplex the audio and video data signals collected by audio and video collection devices such as microphones and cameras, thereby obtaining audio decoding signals and video decoding signals for output; the digital-to-analog conversion unit is used to perform digital-to-analog conversion on the audio decoding signals and video decoding signals decoded by the demultiplexing unit to obtain audio signals and video signals; the signal separation unit is used to separate the audio signals and the video signals to facilitate subsequent processing of the audio signals.

[0031] like Figure 3 As shown, specifically, the preprocessing operations performed by the signal preprocessing module on the audio signal include noise reduction, equalization, dynamic range control and reverberation suppression. The signal preprocessing module includes a noise reduction unit, an audio equalizer, a dynamic range control unit and a reverberation suppression unit, wherein the noise reduction unit is used to reduce the noise of the audio signal. In video conferences and other occasions, steady-state noise signals (such as continuous background noise such as air conditioning sound) and non-steady-state noise signals (such as footsteps, applause, etc.) are easily generated on the scene. The noise reduction unit mainly uses Kalman filtering and Wiener filtering methods to analyze and process the audio signal to suppress noise. Among them, Kalman filtering is a noise reduction method based on state estimation. By estimating the state of the speech signal and filtering the estimation error, a relatively pure speech signal is obtained, which is suitable for filtering steady-state noise signals in audio signals; Wiener filtering, this method is based on the minimum mean square error criterion, by estimating the power spectrum of the noisy speech and noise, and calculating an optimal filter coefficient, and then applying the filter coefficient to the noisy speech, a relatively pure speech signal is obtained, which is suitable for filtering non-steady-state noise signals in audio signals.

[0032] The audio equalizer is used to adjust the equalization of the audio signal, and can adjust the frequency band, gain and quality of the audio, thereby improving the final playback quality of the audio on the playback device; the dynamic range control unit is used to control the dynamic range of the audio signal, and its specific control method is: setting the initial value of the effective value detection time; performing effective value detection on the input signal according to the initial value of the effective value detection time to obtain the signal effective value; adjusting the effective value detection time according to the signal effective value and the preset parameter threshold; continuing to perform effective value detection on the input signal according to the adjusted effective value detection time to obtain the signal effective value after the effective value detection time is adjusted; performing dynamic gain adjustment according to the signal effective value after the effective value detection time is adjusted and the preset parameter threshold, smoothing the adjusted gain according to the preset start time and release time, and then applying the smoothed gain to the input signal to obtain the output signal.

[0033] The reverberation suppression unit is used to suppress reverberation of the audio signal. In addition, a speech enhancement unit can be set to enhance the speech in the audio signal. Through the reverberation suppression unit and the speech enhancement unit, the important parts of the speech signal can be enhanced through algorithms to make it easier to distinguish and reduce the impact of reverberation on the speech. Among them, the reverberation suppression unit uses deep learning algorithms, such as recursive neural networks (RNN), to detect and suppress the reflection of sound in space, thereby reducing the impact of reverberation on speech; the speech enhancement unit uses an artificial intelligence (AI) system with speech interaction function, which can use neural networks to optimize speech and semantic recognition of input speech audio.

[0034] Since the analog signal in audio and video data is transmitted directly, while the video signal such as HDMI signal needs to be processed by encoding and decoding, digital-to-analog conversion, etc., it is delayed by tens of milliseconds compared with the analog direct transmission signal. Different transmission methods will also cause different degrees of time delay of the video signal compared with the analog signal. Therefore, it is necessary to measure the relative delay time to facilitate accurate compensation of the delay.

[0035] The signal delay measurement module includes a synchronization signal generator and an oscilloscope connected thereto. After the audio signal and video signal to be measured are input into the synchronization signal generator, the oscilloscope connected thereto performs measurement and display. During measurement, the LipSync (Leader) or AV Timing (Telestream) function of the synchronization signal generator is first turned on, and the test signal is input from an external channel. After de-embedding, it passes through the audio and video links respectively. After the final stage of embedding, the relative delay of audio and video is measured by the AV Delay (Telestream) or AV phase (Leader) function of the oscilloscope.

[0036] The signal synchronization processing module can generate the delay compensation time and timestamp according to the relative delay measured by the signal delay measurement module. When the system time reaches the timestamp, the audio signal is played synchronously in the audio playback device, and the video signal is played synchronously on the display terminal such as the display screen, so as to achieve the synchronization of audio playback and video display. When the audio transmission mode is different, the delay compensation time is also different. After the audio signal is delayed, it is amplified by the power amplifier unit and played on the audio playback device such as the speaker.

[0037] like Figure 4As shown, the present invention also provides a video conferencing terminal device using the above-mentioned audio and video signal processing system, the video conferencing terminal device also includes a microphone, a camera, a display screen and a speaker, wherein the microphone is connected to a main control module, the main control module can decode the audio and video data signals collected by the microphone and output them to the audio and video processing module, separate the audio signal and the video signal, the signal synchronization processing module is connected to both the display screen and the speaker, and transmits the audio signal and the video signal after delay compensation to the speaker and the display screen respectively for playback and display, wherein the speaker is connected to the signal synchronization processing module through the power amplifier module.

[0038] As a preferred solution, the microphone is a cardioid directional microphone array, and other multi-microphone arrays arranged in space may also be used, which can capture sounds such as voices from all directions and suppress environmental noise.

[0039] The camera is equipped with a pan / tilt head, a device module and a picture quality module. The device module is used to connect the camera to a computer or a video conferencing terminal device; the picture module is used to adjust the camera's viewing angle and focal length, and determine the shooting range, picture quality and direction.

[0040] The video conferencing terminal equipment is also equipped with a conference platform and conference management system, which can invite relevant personnel to attend the meeting, test the network, debug the conference environment, and record and store meeting agendas and other content in the cloud.

[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio and video signal processing system, characterized in that: include: An audio and video processing module, a signal preprocessing module, a signal delay measurement module, a signal synchronization processing module and a main control module, wherein the main control module is used to output the audio and video data signal to the audio and video processing module, the audio and video processing module is used to separate the audio signal and the video signal in the input audio and video data signal, the signal preprocessing module is used to preprocess the audio signal, the signal delay measurement module is used to measure the relative delay amount of the audio signal and the video signal, and the signal synchronization processing module is used to perform delay compensation according to the relative delay amount measured by the signal delay measurement module to achieve synchronization of audio and video display.

2. The audio and video signal processing system according to claim 1, characterized in that: The preprocessing operations performed by the signal preprocessing module on the audio signal include noise reduction, equalization, dynamic range control and reverberation suppression.

3. The audio and video signal processing system according to claim 1, characterized in that: The signal delay measurement module includes a synchronous signal generator and an oscilloscope connected thereto. After the audio signal and the video signal to be measured are input into the synchronous signal generator, they are measured and displayed by the oscilloscope connected thereto.

4. The audio and video signal processing system according to claim 3, characterized in that: The audio signal and the video signal are de-embedded by the LipSync function of the synchronization signal generator and pass through the audio and video link. After being embedded at the final stage, the relative delay of the audio signal and the video signal is measured by the AV Delay function of the oscilloscope.

5. The audio and video signal processing system according to claim 1, characterized in that: The audio and video processing module includes a demultiplexing unit, a digital-to-analog conversion unit and a signal separation unit. The demultiplexing unit is used to demultiplex the audio and video data signals to obtain an audio decoding signal and a video decoding signal; The digital-to-analog conversion unit is used to perform digital-to-analog conversion on the audio decoded signal and the video decoded signal to obtain the audio signal and the video signal; The signal separation unit is used to separate the audio signal and the video signal.

6. The audio and video signal processing system according to claim 2, characterized in that: The signal preprocessing module includes a noise reduction unit, an audio equalizer, a dynamic range control unit and a reverberation suppression unit. The noise reduction unit is used to reduce noise on the audio signal, the audio equalizer is used to adjust the equalization of the audio signal, the dynamic range control unit is used to control the dynamic range of the audio signal, and the reverberation suppression unit is used to suppress reverberation on the audio signal.

7. The audio and video signal processing system according to claim 1, characterized in that: The signal synchronization processing module can generate a delay compensation time and a timestamp according to the relative delay measured by the signal delay measurement module, so as to achieve synchronization of audio playback and video display.

8. A video conference terminal device using the audio and video signal processing system according to any one of claims 1 to 7, characterized in that: It also includes a microphone, a camera, a display screen and a speaker. The microphone is connected to the main control module. The main control module outputs the audio and video data signals collected by the microphone to the audio and video processing module. The signal synchronization processing module is connected to both the display screen and the speaker, and transmits the audio signal and the video signal after delay compensation to the speaker and the display screen respectively.

9. The video conferencing terminal device according to claim 8, characterized in that: The microphone is a cardioid directional microphone.

10. The video conference terminal device according to claim 8, characterized in that: The signal synchronization processing module is connected to the sound box through the power amplifier module.

Images