Audio and video synchronization method and system for video conference

Abstract

The invention relates to the technical field of audio and video synchronization, in particular to an audio and video synchronization method and system for a video conference. The method comprises the following steps: acquiring audio and video data segments to be transmitted in a video conference process in real time; dividing the audio data segment and the video data segment into a plurality of audio data packets and video data packets according to frame data distribution characteristics of the audio data segment and the video data segment; the method comprises the following steps: firstly, determining a video feature coding matrix corresponding to a video frame at each moment in each video data packet, and then determining a video feature coding matrix corresponding to an audio frame at each moment according to a time sequence corresponding relationship between the audio frame and the video frame before transmission; and transmitting all the new audio data packets and video data packets, so that the video frame and the audio frame at each moment correspond to each other, and obtaining synchronously played audio and video data segments. According to the method, the audio and video synchronization effect for the video conference is effectively improved by adaptively dividing the audio and the video and determining the video feature coding matrix.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video synchronization, and in particular to an audio and video synchronization method and system for video conferencing. Background Art

[0002] Video conferencing refers to a meeting in which people in two or more locations talk face to face through communication equipment and the Internet. In a video conference, when there is a lack of synchronization between audio and video, that is, lip misalignment, distorted emotional expression, and even picture loss, it will distract the participants. Good audio and video synchronization can effectively enhance the authenticity and coherence of communication, reduce cognitive burden, and improve meeting efficiency. With the increase in the demand for remote collaboration, the requirements for audio and video synchronization are also gradually increasing.

[0003] When conducting a video conference, in order to deal with network delays, codec delays and other issues, the audio and video data is first divided into packets of fixed time or frame number, and then the audio and video after transmission are aligned using timestamp synchronization, adaptive buffering technology and adaptive coding technology. However, during a video conference, affected by the speaker and the playback of the demonstration images, some key images and voices may be lost. Based on this, after the data transmission is completed, the audio and video synchronization effect used for the video conference is low due to the loss of some information. Summary of the invention

[0004] In order to solve the technical problem of poor audio and video synchronization effect of the above-mentioned existing methods, the purpose of the present invention is to provide an audio and video synchronization method and system for video conferencing, and the technical solution adopted is as follows:

[0005] An embodiment of the present invention provides an audio and video synchronization method for a video conference, the method comprising the following steps:

[0006] Real-time collection of audio and video data segments to be transmitted during a video conference; wherein the audio and video data segments include audio data segments and video data segments;

[0007] Dividing the audio data segment and the video data segment into a plurality of audio data packets and video data packets according to the frame data distribution characteristics of the audio data segment and the video data segment;

[0008] First, determine the video feature coding matrix corresponding to the video frame at each moment in each video data packet, and then determine the video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission;

[0009] Transmit all new audio and video data packets so that the video frame at each moment corresponds to the audio frame, and obtain the audio and video data segments played synchronously;

[0010] The new audio data packet is composed of an audio frame and a video feature coding matrix corresponding to the audio frame.

[0011] Furthermore, the audio data segment and the video data segment are divided into a plurality of audio data packets and video data packets according to the frame data distribution characteristics of the audio data segment and the video data segment, including:

[0012] Analyze the voice pauses and breathing conditions of the conference speaker according to the audio data segment to determine the voice pause baseline value;

[0013] Using the voice pause reference value, a number of conference voice pause points are screened out, and then an initial division index of each conference voice pause point is determined; wherein the conference voice pause point is a moment when the sound amplitude is greater than the pause threshold;

[0014] Determine the visual change index of the video frame at each moment according to the gray value of each pixel corresponding to the video frame at each moment in the video data segment; wherein the visual change index represents the degree of change of the video frame at any moment relative to the video frame at the previous moment;

[0015] Determining a final division index for each conference voice pause point according to the initial division index for each conference voice pause point and the visual image replacement index;

[0016] Set the division threshold and determine the data packet division point from all conference voice pause points based on the final division index;

[0017] The audio data segment and the video data segment are divided and processed by using the data packet division points to obtain a plurality of audio data packets and video data packets.

[0018] Further, the step of analyzing the voice pause and breathing conditions of the conference speaker according to the audio data segment to determine the voice pause reference value includes:

[0019] Fitting the audio data segment to obtain a fitted audio sequence;

[0020] Extract all minimum points from the fitted audio sequence, and determine the mean, mode and median corresponding to the sound amplitudes of all minimum points;

[0021] Performing weighted sum processing on the mean, mode and median with different preset weights to obtain a weighted sum result;

[0022] The weighted sum result is adjusted upward, and the adjusted result is used as the speech pause reference value.

[0023] Furthermore, the determining of the initial division index of each conference speech pause point includes:

[0024] For any conference voice pause point, calculate the average value of the sound amplitudes of the audio frame corresponding to the conference voice pause point and a first preset number of adjacent audio frames;

[0025] A negatively correlated normalization process is performed on the difference between the speech pause reference value and the average value of the sound amplitude, and the normalized value is used as an initial division index of the conference speech pause point.

[0026] Furthermore, the final division index of each conference voice pause point is determined according to the initial division index of each conference voice pause point and the visual change index, including:

[0027] For any conference voice pause point, obtain the number of changed pixels in the video frame corresponding to the conference voice pause point and a second preset number of adjacent video frames; wherein the changed pixels are pixels whose grayscale values ​​change;

[0028] The final division index F of the dth conference speech pause point d The calculation formula is:

[0029] In the formula, exp represents an exponential function with a natural constant as the base. represents the average number of changed pixels of the video frame corresponding to the dth conference voice pause point and the second preset number of adjacent video frames, V d represents the initial division index of the dth conference speech pause point, G d Indicates the visual change index of the video frame corresponding to the dth conference voice pause point.

[0030] Furthermore, the step of determining the video feature coding matrix corresponding to the video frame at each moment in each video data packet includes:

[0031] For any video data packet, when processing the video frame at each moment in the video data packet using the frame difference method, obtaining each difference feature point in the video frame at each moment; wherein the difference feature point is obtained by comparing the pixel difference between the current frame and the two frames above and below it;

[0032] Performing a fourth preset number of division processes on the video frame at each moment in the video data packet to determine a division position sequence where each of the difference feature points in the video frame at each moment is located;

[0033] The division position sequences of all difference feature points in the same video frame are used to form a matrix, and the matrix is ​​used as a video feature coding matrix corresponding to the corresponding video frame.

[0034] Furthermore, performing a fourth preset number of division processes on the video frame at each moment in the video data packet to determine a division position sequence at which each of the difference feature points in the video frame at each moment is located includes:

[0035] For any video frame, firstly divide the video frame into a third preset number of first local areas, and mark each first local area with a corresponding position serial number;

[0036] Then, the first local area is divided into a third preset number of second local areas, and each second local area is marked with a corresponding position number; the local areas are divided iteratively until the number of divisions meets a fourth preset number, thereby obtaining a divided video frame;

[0037] Based on the divided video frames, a division position sequence of each difference feature point in the video frame is determined; wherein the elements in the division position sequence are position serial numbers of each local area where the difference feature point is located.

[0038] Furthermore, the determining of the video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission includes:

[0039] For audio frames and video frames at the same time, the timestamp technology is combined to assign the audio frame with the video feature coding matrix corresponding to the video frame at the same time.

[0040] Furthermore, after determining the video feature coding matrix corresponding to the audio frame at each moment, the method further includes:

[0041] The local area corresponding to the last division where each difference feature point in the video frame is located is used as a feature area, and the local area other than the feature area in the video frame is used as a non-feature area;

[0042] All the non-feature areas in the video frame are compressed to obtain a compressed video frame.

[0043] Another embodiment of the present invention provides an audio and video synchronization system for video conferencing, including a processor and a memory, wherein the processor is used to process instructions stored in the memory to implement an audio and video synchronization method for video conferencing.

[0044] The present invention has the following beneficial effects:

[0045] The present invention provides an audio and video synchronization method and system for video conferencing. Firstly, the audio data segments and the video data segments are divided into a plurality of audio data packets and video data packets in combination with the frame data distribution characteristics of the audio data segments and the video data segments. Compared with the existing data packet division method based on a fixed time or a fixed number of frames, the present invention makes the information before and after the division point of each data packet less critical, that is, avoids the loss of some critical information, which can effectively improve the synchronization effect of subsequent audio and video playback; secondly, the corresponding video feature coding matrix is ​​determined for the audio frame at each moment, which helps to directly decode and obtain the corresponding coding information according to the video frame information after transmission, and then obtain more accurate audio frames corresponding to the video frames, avoiding audio and video asynchronism due to transmission misalignment and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A flowchart of a method for synchronizing audio and video for video conferencing according to an embodiment of the present invention;

[0048] Figure 2 This is a flowchart of step S2 in an embodiment of the present invention;

[0049] Figure 3 is a flowchart of step S31 in an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of each first local area marked in an embodiment of the present invention;

[0051] Figure 5 A schematic diagram of each second local area located in any marked completed area in a video frame according to an embodiment of the present invention;

[0052] Figure 6 Schematic diagram of the formation of a video feature coding matrix in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0054] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0055] The application scenarios targeted by the present invention may be:

[0056] During an actual video conference, participants will point out the corresponding key points or make marks on the presentation picture while explaining their ideas. That is, each voice segment in the video conference may be accompanied by a corresponding specific presentation picture, and the audio and video data segments are split into data packets. After the audio and video are split into frames, the feature points of each frame of the video have their corresponding audio frames. Based on the timing correspondence between the audio frame and the video frame, the synchronous output of the audio and video can be restored after the transmission is completed.

[0057] Specifically, an embodiment of the present invention provides an audio and video synchronization method and system for video conferencing, such as Figure 1 As shown, the following steps are included:

[0058] S1, real-time collection of audio and video data segments to be transmitted during the video conference.

[0059] Specifically, a high-definition camera and a microphone are set at the participants of the video conference, which can be used to collect the audio and video data segments to be transmitted during the video conference in real time. Of course, the device for collecting the audio and video data segments can also be other types of devices.

[0060] Here, the audio and video data segment can include two aspects, namely the video data segment and audio data segment in the video conference. The video data segment and the audio data segment refer to the time series data related to the video and audio respectively. They are generated by different signal processing and encoding methods and transmitted independently in the network; the video data segment contains several video frames, and a single frame of video is a static image, while the audio data segment refers to the continuous audio data block divided in time order in the audio stream. Each audio data segment contains audio information within a period of time, usually an audio waveform of a certain duration, and each frame is the amplitude change of an audio signal; the video and audio data segments need to be synchronized to ensure that the audio and video content can be displayed and played in a coordinated manner. For example, in a video conference, if the video and audio are not synchronized, the picture seen by the participants and the sound heard may not match, affecting the communication effect.

[0061] The frame rate of the video data segment can be set to 30 frames, the bit rate to 10Mbps, and the resolution to 1024*768; the sampling frequency of the audio data segment can be set to 48kHz, the frame rate to 30 frames, the sampling bit number to 16 bits, and the sampling channel number to two-channel stereo. The sampling parameter settings of the video data segment and the audio data segment can be set by the implementer according to the actual situation, and are not specifically limited here.

[0062] So far, this embodiment has obtained the audio data segment and the video data segment corresponding to the audio and video data segment to be transmitted.

[0063] S2, dividing the audio data segment and the video data segment into a plurality of audio data packets and video data packets according to the frame data distribution characteristics of the audio data segment and the video data segment in the audio and video data segment.

[0064] First of all, it should be noted that in a video conference, the participants speak in a relatively orderly process. Every time the speaker finishes a sentence or enters the next stage of the topic, there will be a certain pause. When the speaker pauses, the sound recorded by the microphone will be relatively small. Therefore, the audio and video are divided into several smaller data packets based on the time corresponding to the audio frame with smaller sound.

[0065] The above step S2 can be Figure 2 The steps S201 to S206 shown implement:

[0066] S201, analyzing the voice pause and breathing conditions of the conference speaker according to the audio data segment to determine a voice pause baseline value.

[0067] Here, the speech pause reference value is to analyze the speech pause characteristics of the conference speaker through the fluctuation of the audio amplitude in the audio data segment, which can be used to screen all the audio frame data in the audio data segment to determine the conference speech pause point.

[0068] As a specific implementation, determining the speech pause reference value may include:

[0069] The first step is to fit the audio data segment to obtain a fitted audio sequence.

[0070] Specifically, the audio frame data at all moments in the audio data segment is used as input data, and the input data is processed using a piecewise aggregate approximation algorithm (PAA) to obtain fitting data of the audio data segment, which can be recorded as a fitting audio sequence.

[0071] Among them, obtaining the fitting audio sequence can effectively remove small amplitude fluctuations in the audio and smooth the data; the preset window size in the segmented aggregation approximation algorithm can be 0.05 seconds, which is used to control the fitting accuracy of the audio data segment. The size of the preset window size can be set by the implementer according to the specific actual situation, and no specific limitation is made here; in addition, the detailed implementation process of the segmented aggregation approximation algorithm is a prior art and is not within the protection scope of the present invention, and will not be elaborated here.

[0072] The second step is to extract all the minimum points from the fitting audio sequence and determine the mean, mode and median corresponding to the sound amplitude of all the minimum points.

[0073] Specifically, the minimum point, ie, the moment when the audio signal amplitude is the lowest, is identified from the fitting audio sequence; the average value of the sound amplitude of all the minimum points is calculated, and the mode and median are selected from the sound amplitude of all the minimum points.

[0074] Among them, the minimum points are data points distributed in chronological order; in addition, the length of the fitted audio sequence can be 10 minutes, which is to ensure that the analyzed data is long enough to capture relatively stable pause features.

[0075] The third step is to perform weighted summation on the mean, mode and median with different preset weights to obtain the weighted summation result.

[0076] In this embodiment, the weights of the mean, mode and median are set to 0.5, 0.3 and 0.2 respectively.

[0077] In the fourth step, the weighted sum result is adjusted upward, and the adjusted result is used as the speech pause reference value.

[0078] Specifically, the weighted sum result is adjusted upward by 10% to obtain a sound amplitude at a reasonable pause, and the sound amplitude is used as a speech pause reference value, which is used for pause detection in subsequent audio analysis.

[0079] As an example, the calculation formula of the speech pause reference value may be:

[0080] In the formula, S p Indicates the speech pause baseline value, It represents the mean value of the sound amplitude corresponding to all minimum points in the fitted audio sequence, S m S represents the mode of the sound amplitudes corresponding to all minimum points in the fitted audio sequence. z Represents the median of the sound amplitudes corresponding to all minimum points in the fitted audio sequence.

[0081] In the calculation formula of the speech pause benchmark value, the mean, mode and median are set to weights of 0.5, 0.3 and 0.2 respectively, mainly to comprehensively consider the advantages and limitations of these three statistics through weighted summation. In this weighted processing, the mean, as the indicator that best reflects the overall data trend, obtains a higher weight; the mode and median are respectively given appropriate weights according to their characteristics and application scenarios. This weighting strategy helps to improve the representativeness and reliability of the speech pause benchmark value.

[0082] It is worth mentioning that the purpose of this step is to pre-process the audio data through the PAA algorithm, remove noise and small fluctuations, accurately extract the audio features of the pause moments, and calculate the pause baseline value through statistical methods, so as to facilitate more effective analysis and processing of audio data.

[0083] S202, using the voice pause benchmark value to screen out a number of conference voice pause points, and then determine an initial division index for each conference voice pause point.

[0084] As a specific implementation method, screening conference voice pause points and determining the initial division index of each conference voice pause point includes:

[0085] The first step is to use the voice pause benchmark value to screen out several conference voice pause points.

[0086] Here, the conference voice pause point refers to the time point when the conference speaker pauses or takes a breath while speaking.

[0087] Specifically, determine the sound amplitude of the audio frame at each moment, compare the sound amplitude at each moment with the voice pause benchmark value, and when the sound amplitude for three consecutive moments is less than or equal to the voice pause benchmark value, it means that the conference speaker has paused for breathing here. The audio and video data segment can be divided into two parts here, and the second moment can be specifically recorded as the conference voice pause point.

[0088] The number of consecutive sound amplitudes that meet the voice pause reference value can be a positive integer greater than or equal to 3, and the implementer can set it according to the actual situation. In addition, when there are an odd number of moments when the sound amplitude is less than or equal to the voice pause reference value, the moment corresponding to the median is used as the conference voice pause point; when there are an even number of moments when the sound amplitude is less than or equal to the voice pause reference value, any moment between the two moments in the median is used as the conference voice pause point.

[0089] The second step is to determine the initial division index of each conference speech pause point based on the sound amplitude of the audio frame at each moment and the speech pause baseline value.

[0090] Here, the initial division index refers to the possibility that the conference voice pause point is used to divide the audio and video data segments in terms of audio. At this time, the division index is only quantified by the audio frame distribution characteristics of the audio data segment, and the distribution characteristics of the video frame are not considered. Therefore, it is only used as the initial division index here.

[0091] As a specific implementation method, taking any conference voice pause point as an example, determining the initial division index of the conference voice pause point includes:

[0092] The first sub-step is to calculate the average value of the sound amplitudes of the audio frame corresponding to the conference speech pause point and a first preset number of adjacent audio frames.

[0093] In this embodiment, the first preset number may be set to 2, so that the average value of the sound amplitudes corresponding to the audio frame and its two adjacent audio frames may be obtained.

[0094] In the second sub-step, a negatively correlated normalization process is performed on the difference between the speech pause baseline value and the average value of the sound amplitude, and the normalized value is used as an initial division index for the conference speech pause point.

[0095] In this embodiment, the difference between the speech pause baseline value and the average value of the sound amplitude can characterize the deviation of the sound amplitude of the adjacent audio frames corresponding to the conference speech pause point relative to the speech pause baseline value. The larger the deviation, the worse the data division characteristics of the audio frames at the conference speech pause point, and the smaller the initial division index.

[0096] As an example, the calculation formula for the initial division index of the dth conference speech pause point may be:

[0097] Where V d represents the initial division index of the dth conference speech pause point, tanh represents the hyperbolic tangent function, which is used to normalize the deviation value of the neighborhood amplitude of the audio frame relative to the reference value in positive proportion, that is, || represents the absolute value function, S pIndicates the speech pause baseline value, Represents the average value of the sound amplitudes of the audio frame corresponding to the conference voice pause point and the first preset number of adjacent audio frames.

[0098] S203, determining a view change index of the video frame at each moment according to the grayscale value of each pixel corresponding to the video frame at each moment in the video data segment.

[0099] Here, the visual change index indicates the degree of change of the video frame at any moment relative to the video frame at the previous moment, that is, it is used to indicate the degree of change of the video frame at each moment compared with the video frame at the previous moment. When the speaker is speaking, his body movements or demonstration pictures need to correspond to the content of the speech. When the speaker proceeds to the next stage, he will switch to the next page of demonstration pictures or make related movements such as raising his hand to remind the meeting participants to proceed to the next content, which will cause a large change in the video frames at adjacent moments. Therefore, the visual change index is used as one of the factors to determine the final division index.

[0100] As a specific implementation, determining the visual change index of the video frame at each moment includes:

[0101] In the first step, the pixel points that have changed in the video frame at each moment in the video data segment relative to the video frame at the previous moment are obtained according to the frame difference method, and the pixel points that have changed are recorded as changed pixel points.

[0102] In this embodiment, for a video frame at any time, the grayscale difference between the pixel at the same position in the video frame at this time and the video frame at the previous time is first calculated, and then the grayscale difference is normalized to obtain a normalized value, and finally the pixel whose normalized value is greater than the difference threshold of 0.25 is used as the changed pixel. Of course, the changed pixel can also be determined by other methods, which are not specifically limited here.

[0103] Among them, the implementation process of the frame difference method is a prior art and is not within the protection scope of the present invention, and will not be elaborated here; the difference threshold can be set by the implementer according to the specific actual situation and is not specifically limited; in addition, the normalization implementation method can be a linear normalization function.

[0104] The second step is to determine the visual change index of the video frame at each moment according to the proportion of changed pixels in the video frame at each moment relative to the overall video frame image.

[0105] As an example, the calculation formula of the video change index of the video frame at the cth moment may be:

[0106] In the formula, G c Represents the video frame view change index at the cth moment, Nc,g Indicates the number of changed pixels in the video frame at the cth moment, N c,a Represents the number of all pixels in the video frame at the cth moment.

[0107] In the calculation formula of the visual change index, the more the number of changed pixels in the video frame at any moment is, the greater the degree of visual change of the video frame at that moment is, and the greater the visual change index of the video frame at that moment is.

[0108] S204, determining a final division index for each conference voice pause point according to the initial division index and the visual change index for each conference voice pause point.

[0109] Here, the final division index is used to evaluate the possibility of each conference voice pause point being a data packet point. The greater the possibility, the greater the possibility of being used to divide the audio and video data segments. In video conferencing, although there are natural voice pauses and picture changes, for the conference video as a whole, the speaker may still be speaking when the picture changes, and important pictures may be played when the voice pauses.

[0110] Therefore, in order to ensure that the division points when dividing the audio and video data segments do not contain important information, it is necessary to analyze the distribution characteristics of frame data when the voice pauses and the screen action changes significantly or does not change for a period of time. Among them, frame data refers to audio frames or video frames.

[0111] As a specific implementation method, determining the final division index of any conference speech pause point includes:

[0112] The first step is to obtain the number of changed pixels in the video frame corresponding to the conference voice pause point and a second preset number of adjacent video frames.

[0113] In this embodiment, the more the number of changed pixels in the video frames at several consecutive moments is, the more the video frames at the consecutive moments are in a state of large degree of change, the smaller the possibility that the video frames are still pictures, and the larger the initial division index representing the degree of sound pause at this time, indicating that there may be a pause in the speaker's speech after the picture changes, and the greater the possibility of performing data packetization processing on the audio and video data segments at this time; and when the smaller the number of changed pixels in the video frames at several consecutive moments is, the greater the possibility that the video frames are still pictures, and the larger the initial division index representing the degree of sound pause at this time, indicating that the picture does not change for a period of time when the speaker pauses in his speech, and the greater the possibility of performing data packetization processing on the audio and video data segments at this time.

[0114] Specifically, according to the step of obtaining the changed pixel points recorded in the first step of step S203, the changed pixel points in the video frame corresponding to the conference voice pause point and the second preset number of adjacent video frames can be obtained, and then the number of all changed pixel points in the video frame can be counted. Among them, the size of the second preset number can be 4, and the implementer can set it according to the specific actual situation, and it is not specifically limited here.

[0115] In the second step, the final division index of the conference voice pause point is determined according to the average number of changed pixels corresponding to the video frame corresponding to the conference voice pause point and the second preset number of adjacent video frames, the initial division index and the visual replacement index.

[0116] As an example, the final division index F of the dth conference speech pause point d The calculation formula is:

[0117] In the formula, exp represents an exponential function with a natural constant as the base. represents the average number of changed pixels of the video frame corresponding to the dth conference voice pause point and the second preset number of adjacent video frames, V d represents the initial division index of the dth conference speech pause point, G d Indicates the visual change index of the video frame corresponding to the dth conference voice pause point.

[0118] In the calculation formula of the final partitioning index, It indicates the data performance characteristics when the picture action does not change for a period of time when the voice pauses. Indicates the data performance characteristics when there is a significant change in the picture action during a pause in speech; It is used to represent the situation where the video frames at several moments adjacent to the video frame corresponding to the dth conference voice pause point change continuously. d It is used to represent the degree of image change of the video frame corresponding to the dth conference voice pause point, V d It is used to characterize the degree of speech pause occurring at the dth conference speech pause point; exp(-) is used to realize the inverse proportional normalization processing of the data. Of course, the implementer can also adopt other data processing methods to realize the inverse proportional normalization.

[0119] It should be noted that this embodiment takes two aspects into consideration when determining the final division index, namely, the situation where the speaker may still be speaking when the screen changes, and the situation where important pictures are played when the voice pauses. This improves the numerical accuracy and reliability of the final division index, and further reduces the possibility of losing important information during the data packet division process to a certain extent.

[0120] S205, setting a division threshold, and determining data packet division points from all conference voice pause points based on the final division index.

[0121] In this embodiment, the value range of the final division index is between 0 and 1, so the division threshold can be set to 0.75. The implementer can set the size of the division threshold according to the actual situation, and no specific limitation is made here. The final division index of all conference voice pause points is screened by the set division threshold, specifically:

[0122] If the final division index of a conference voice pause point is greater than or equal to the division threshold, the corresponding moment of the conference voice pause point is determined to be a data subpackaging point; if the final division index of a conference voice pause point is less than the division threshold, the corresponding moment of the conference voice pause point is determined not to be a data subpackaging point.

[0123] S206, using the data packet splitting points to divide the audio data segment and the video data segment to obtain a plurality of audio data packets and video data packets.

[0124] In this embodiment, by dividing the audio data segment and the video data segment, a number of audio data packets corresponding to the audio data segment and a number of video data packets corresponding to the video data segment can be obtained.

[0125] As an example, for any data packet splitting point, the audio data segment or video data segment of the previous moment and the next moment are divided by the data packet splitting point, that is, the next moment is regarded as the beginning of a new data packet, and the previous moment is regarded as the end of the previous data packet.

[0126] It is worth mentioning that compared with the existing method of dividing audio and video data segments into data packets according to fixed time or frame number, this embodiment determines the data packet sub-points in combination with the speaker's speech and the playback of the video screen during the video conference, which can avoid the loss of key parts, key pictures or voice data, and improves the effect of audio and video synchronous playback to a certain extent.

[0127] So far, this embodiment has obtained a number of audio data packets corresponding to the audio data segment and a number of video data packets corresponding to the video data segment.

[0128] S3, first determine the video feature coding matrix corresponding to the video frame at each moment in each video data packet, and then determine the video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission.

[0129] First of all, it should be noted that during a video conference, video data and audio data are transmitted separately in the form of data packets. After the transmission is completed, they are merged in the cache pool to form a long audio and video data before being played synchronously. When the speaker is speaking in a video conference, in order to assist other participants in understanding, some body movements will be made to guide, and a certain number of demonstration pictures will be given to help participants understand what the speaker is saying. The demonstration pictures will also guide the speaker to explain the corresponding plan or plan in the order of the corresponding content, that is, each video frame at each moment in the video data packet has its corresponding audio frame. In order to ensure that there is a one-to-one relationship between the transmitted video frame and the audio frame, that is, to achieve audio and video synchronization, it is necessary to determine the video feature coding matrix corresponding to the audio frame at each moment.

[0130] The above step S3 can be implemented by steps S301 to S302 (not shown):

[0131] S31, determining a video feature coding matrix corresponding to a video frame at each moment in each video data packet.

[0132] Here, the video feature encoding matrix is ​​composed of the serial numbers of the pixel points selected from the video frame and capable of representing the grayscale features of the video frame, and each video frame has its corresponding video feature encoding matrix.

[0133] As a specific implementation, for any video data packet, the above step S31 can be performed by Figure 3 The steps S311 to S313 shown implement:

[0134] S311, when processing the video frame at each moment in the video data packet by using the frame difference method, obtaining each difference feature point in the video frame at each moment.

[0135] Here, a difference feature point refers to a pixel point with a large grayscale change in multiple consecutive frames of images. The difference feature point can be used to characterize the characteristics of the video frame image. It can perform frame difference processing through the pixel difference between the current frame and the previous frame or the next frame. Subsequently, a threshold can be set to filter out pixels with large differences and use the pixels with large differences as difference feature points.

[0136] As a specific implementation, the step of acquiring each difference feature point in the video frame at any moment may include:

[0137] Calculate the pixel difference between the video frame at this moment and the video frame at an adjacent moment, use the threshold to binarize any frame difference image between the two frames, map the part with large difference in the video frame at this moment, and record it as the first difference area. At the same time, use the threshold to binarize the other frame difference image between the two frames, map the part with large difference in the video frame at this moment, and record it as the second difference area. The pixel points corresponding to the first difference area and the second difference area are used as the difference feature points in the video frame at this moment.

[0138] S312, performing a fourth preset number of division processes on the video frame at each moment in the video data packet to determine a division position sequence of each difference feature point in the video frame at each moment.

[0139] As a specific implementation, taking a video frame at any moment as an example, the above step S312 can be implemented by the following steps:

[0140] In the first step, the video frame is divided into a third preset number of first local areas, and each first local area is marked with a corresponding position serial number.

[0141] Specifically, the video frame is evenly divided into 9 first local areas of 3×3, and each first local area is marked with a corresponding position number in a certain marking order, such as 1, 2, 3..., and the marking order can be a Z-shaped mark from bottom to top and from left to right.

[0142] The third preset number can be set to 9, and the implementer can set the size of the third preset number according to the actual situation; in addition, the schematic diagram of each first local area marked is as follows: Figure 4 shown.

[0143] In the second step, the first local area is divided into a third preset number of second local areas, and each second local area is marked with a corresponding position number; the local areas are divided iteratively until the number of divisions meets a fourth preset number, thereby obtaining a divided video frame.

[0144] Specifically, the first local area is evenly divided into 9 second local areas of 3×3, and each second local area is marked with a corresponding position number in the same marking order; the second local area is evenly divided into 9 third local areas of 3×3, and each third local area is marked with a corresponding position number in the same marking order, and the local areas are evenly divided iteratively until the number of divisions is equal to a fourth preset number, and the division of the local areas of the video frame is stopped to obtain the divided video frame.

[0145] The fourth preset number can be set to 5, and the implementer can set the size of the fourth preset number according to the actual situation; in addition, the schematic diagram of each second local area located in any completed mark in the video frame is as shown in FIG. Figure 5 shown.

[0146] The third step is to determine the segmentation position sequence of each difference feature point in the video frame based on the segmented video frame.

[0147] Specifically, each difference feature point in the video frame is first determined, and then the position sequence number of each local area where the difference feature point is located is determined to form a division position sequence.

[0148] Among them, the elements in the division position sequence are the position numbers of each local area where the difference feature point is located, and the position numbers in the division position sequence are constructed in the order of local areas from large to small or from small to large, that is, the position number of the first local area, the position number of the second local area..., the position number of the fourth preset number of local areas, or the position number of the fourth preset number of local areas, the position number of the fourth preset number minus one local area..., the position number of the first local area. For example, the division position sequence of any difference feature point in the video frame is (1, 5, 5, 6, 8), and the division position sequence can also be called sequence coding.

[0149] S313, forming a matrix from the divided position sequences of all the difference feature points in the same video frame, and using the matrix as a video feature coding matrix corresponding to the corresponding video frame.

[0150] In this embodiment, since the smallest unit for dividing a video frame into regions is a region, there are cases where multiple pixels correspond to the same division position sequence. When multiple difference feature points in any video frame correspond to the same division position sequence, only one division position sequence is taken as a representative, and then different division position sequences are arranged to form a matrix, which is used as the video feature coding matrix corresponding to the corresponding video frame. The schematic diagram of the video feature coding matrix is ​​shown in FIG. Figure 6 shown.

[0151] It is worth mentioning that there is a special case, that is, the video feature coding matrix corresponding to the video frames at different times is the same. At this time, the video frame can be evenly divided again on the basis of the original division, and a random value can be added after each row division position sequence in the video feature coding matrix. The random value needs to be bound to the video frame at the corresponding time during subsequent transmission, that is, the random value must be bound to the video frame and transmitted together.

[0152] S32, determining a video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission.

[0153] It should be noted that in a video conference, the collection of video data segments and audio data segments is carried out simultaneously, so there is a time correspondence between the data frames of the two. However, due to the different transmission speeds of audio data and video data and the different clocks of the receiver and sender, the timing correspondence between the audio data and the video data is destroyed after the transmission is completed. Therefore, before transmission, the audio frame at each moment is marked according to the timing correspondence between the audio frame and the video frame, combined with the video feature coding matrix of the video frame at each moment, that is, the video feature coding matrix corresponding to the audio frame at each moment is determined.

[0154] Specifically, for audio frames and video frames at the same moment, the timestamp technology is combined to assign the audio frame with the video feature coding matrix corresponding to the video frame at the same moment.

[0155] In this embodiment, the audio frames and video frames at the same time during the acquisition are first recorded as simultaneous frames, and then combined with the timestamp technology, each audio frame is assigned a video feature coding matrix corresponding to the video frame at the same time, and subsequently during transmission, the audio frames at each moment are transmitted together with the video feature coding matrix of the video frame at the same time.

[0156] Among them, the implementation process of the timestamp technology is a prior art and is not within the protection scope of the present invention, and will not be elaborated here.

[0157] At this point, before transmitting the audio data packet and the video data packet, each audio frame at each moment in the audio data packet has its corresponding video feature encoding matrix composed of encoding the coordinates of feature points in the frame at the same moment.

[0158] As a specific implementation, after determining the video feature coding matrix corresponding to the audio frame at each moment, it also includes image compression processing of the video frame at each moment in each video data packet, so that the compressed video frame that retains more feature points can be transmitted as the key layer in priority.

[0159] It should be noted that the network bandwidth is limited, but the audio and video data segments to be transmitted in the video conference are large, which leads to the need to compress the audio and video data packets. Since most of the content in the video frame at each moment in the video conference remains basically unchanged, that is, the changed part of the video frame occupies a smaller part of the entire picture, in order to ensure that the video conference can still proceed smoothly at a lower bandwidth, the non-feature points in the video frame are compressed, and then the compressed low-resolution layer is used as the key layer. With the help of hierarchical coding technology, the transmission of the key layer video and audio information is prioritized. After ensuring the normal transmission of the key layer video frame, the transmission of high-quality video information is achieved according to the bandwidth.

[0160] Specifically, first, the local area corresponding to the last division where each difference feature point in the video frame is located is used as the feature area, and then the local area other than the feature area in the video frame is used as the non-feature area. Secondly, all non-feature areas in the video frame are compressed to obtain a compressed video frame. In addition, after obtaining the video data packet, the hierarchical coding technology is used for transmission, which is to divide the compressed video frame into hierarchical layers for transmission.

[0161] Among them, the compression method when compressing the non-feature area can use the discrete cosine transform algorithm. The implementation process of the discrete cosine transform algorithm is a prior art, which is not within the protection scope of the present invention and will not be elaborated in detail here; the implementation process of the hierarchical coding technology is also a prior art, which is not within the protection scope of the present invention and will not be elaborated in detail here. In addition, regarding the number of hierarchical coding in the process of implementing the transmission of the video data packet by hierarchical coding, since a video frame is evenly divided H times, that is, a small area in a video frame corresponds to H levels of different sizes, the hierarchical coding can be divided into H layers, and the key layer is a video frame compressed H times, that is, all non-feature areas from small to large are compressed once. The reason is that the difference feature points are contained by H areas from large to small, that is, the non-feature area can have H layers from small to large, so the level of the hierarchical coding can be set to H layers, H represents the fourth preset number, and the empirical value is 5.

[0162] S4, transmit all new audio data packets and video data packets, so that the video frame at each moment corresponds to the audio frame, and obtain the audio and video data segments played synchronously.

[0163] Here, the new audio data packet is composed of an audio frame and a video feature coding matrix corresponding to the audio frame. The sender transmits all the new audio data packets and video data packets obtained, and the receiver decodes the transmitted video and obtains the audio frame to be played based on the difference feature points in the video frame to complete the audio and video synchronous playback.

[0164] Specifically, for the video data packet and the new audio data packet received by the receiver, a video feature coding matrix is ​​obtained in the buffer according to the difference feature points in the video frame at each moment in the video data packet; based on this, the video feature coding matrix of the video frame at any moment is compared with the video feature coding matrix corresponding to the audio frames at all moments, and the audio frame with the same video feature coding matrix as the video frame at that moment is used as the audio frame for synchronous playback; after completing the synchronization of the audio and video data segments for a continuous period of time in the buffer, the audio and video data segment is played. The period of time here can be set to 10 moments, and the implementer can set it according to the specific actual situation without specific limitation.

[0165] An embodiment of the present invention further provides an audio and video synchronization system for video conferencing, including a processor and a memory, wherein the processor is used to process instructions stored in the memory to implement an audio and video synchronization method for video conferencing.

[0166] In summary, the existing real-time video transmission protocol does not take into account the speech problem during video conferencing, resulting in the loss of some key information. Moreover, due to the different transmission speeds of audio and video during transmission, some audio and video cannot correspond to each other during the later audio and video synchronization, resulting in the difficulty of audio and video synchronization during video conferencing, and frequent freezes. The present invention provides an audio and video synchronization method and system for video conferencing. First, based on the frame data distribution characteristics of the audio data segment and the video data segment, the audio data segment and the video data segment are divided into a number of audio data packets and video data packets. Compared with the existing data packet division method based on a fixed time or a fixed number of frames, the present invention makes the information before and after the division point of each data packet less critical, that is, avoids the loss of some key information, which can effectively improve the synchronization effect of subsequent audio and video playback; secondly, the corresponding video feature coding matrix is ​​determined for the audio frame at each moment, which helps to directly decode and obtain the corresponding coding information according to the video frame information after transmission, and then obtain more accurate audio frames corresponding to the video frames, avoiding the audio and video being out of sync due to transmission dislocation and other situations.

[0167] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for synchronizing audio and video for video conferencing, characterized in that: The following steps are involved: Real-time collection of audio and video data segments to be transmitted during a video conference; wherein the audio and video data segments include audio data segments and video data segments; Dividing the audio data segment and the video data segment into a plurality of audio data packets and video data packets according to the frame data distribution characteristics of the audio data segment and the video data segment; First, determine the video feature coding matrix corresponding to the video frame at each moment in each video data packet, and then determine the video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission; Transmit all new audio and video data packets so that the video frame at each moment corresponds to the audio frame, and obtain the audio and video data segments played synchronously; The new audio data packet is composed of an audio frame and a video feature coding matrix corresponding to the audio frame.

2. The method for synchronizing audio and video for video conferencing according to claim 1, characterized in that: The step of dividing the audio data segment and the video data segment into a plurality of audio data packets and video data packets according to the frame data distribution characteristics of the audio data segment and the video data segment comprises: Analyze the voice pauses and breathing conditions of the conference speaker according to the audio data segment to determine the voice pause baseline value; Using the voice pause reference value, a number of conference voice pause points are screened out, and then an initial division index of each conference voice pause point is determined; wherein the conference voice pause point is a moment when the sound amplitude is greater than the pause threshold; Determine the visual change index of the video frame at each moment according to the gray value of each pixel corresponding to the video frame at each moment in the video data segment; wherein the visual change index represents the degree of change of the video frame at any moment relative to the video frame at the previous moment; Determining a final division index for each conference voice pause point according to the initial division index for each conference voice pause point and the visual image replacement index; Set the division threshold and determine the data packet division point from all conference voice pause points based on the final division index; The audio data segment and the video data segment are divided and processed by using the data packet division points to obtain a plurality of audio data packets and video data packets.

3. The method for audio and video synchronization for video conferencing according to claim 2, characterized in that: The step of analyzing the voice pause and breathing conditions of the conference speaker according to the audio data segment to determine the voice pause reference value includes: Fitting the audio data segment to obtain a fitted audio sequence; Extract all minimum points from the fitted audio sequence, and determine the mean, mode and median corresponding to the sound amplitudes of all minimum points; Performing weighted sum processing on the mean, mode and median with different preset weights to obtain a weighted sum result; The weighted sum result is adjusted upward, and the adjusted result is used as the speech pause reference value.

4. The method for audio and video synchronization for video conferencing according to claim 3, characterized in that: The initial division index for determining each conference speech pause point includes: For any conference voice pause point, calculate the average value of the sound amplitudes of the audio frame corresponding to the conference voice pause point and a first preset number of adjacent audio frames; A negatively correlated normalization process is performed on the difference between the speech pause reference value and the average value of the sound amplitude, and the normalized value is used as an initial division index of the conference speech pause point.

5. The method for audio and video synchronization for video conferencing according to claim 2, characterized in that: Determining the final division index of each conference voice pause point according to the initial division index of each conference voice pause point and the visual change index includes: For any conference voice pause point, obtain the number of changed pixels in the video frame corresponding to the conference voice pause point and a second preset number of adjacent video frames; wherein the changed pixels are pixels whose grayscale values ​​change; The final division index F of the dth conference speech pause point d The calculation formula is: In the formula, exp represents an exponential function with a natural constant as the base. represents the average number of changed pixels of the video frame corresponding to the dth conference voice pause point and the second preset number of adjacent video frames, V d represents the initial division index of the dth conference speech pause point, G d Indicates the visual change index of the video frame corresponding to the dth conference voice pause point.

6. The method for audio and video synchronization for video conferencing according to claim 1, characterized in that: The step of determining a video feature coding matrix corresponding to a video frame at each moment in each video data packet includes: For any video data packet, when processing the video frame at each moment in the video data packet using the frame difference method, obtaining each difference feature point in the video frame at each moment; wherein the difference feature point is obtained by comparing the pixel difference between the current frame and the two frames above and below it; Performing a fourth preset number of division processes on the video frame at each moment in the video data packet to determine a division position sequence where each of the difference feature points in the video frame at each moment is located; The division position sequences of all difference feature points in the same video frame are used to form a matrix, and the matrix is ​​used as a video feature coding matrix corresponding to the corresponding video frame.

7. The method for audio and video synchronization for video conferencing according to claim 6, characterized in that: The step of performing a fourth preset number of division processes on the video frame at each moment in the video data packet to determine a division position sequence where each difference feature point in the video frame at each moment is located includes: For any video frame, firstly divide the video frame into a third preset number of first local areas, and mark each first local area with a corresponding position serial number; Then, the first local area is divided into a third preset number of second local areas, and each second local area is marked with a corresponding position serial number; the local areas are divided iteratively until the number of divisions meets a fourth preset number, thereby obtaining a divided video frame; Based on the divided video frames, a division position sequence of each difference feature point in the video frame is determined; wherein the elements in the division position sequence are position serial numbers of each local area where the difference feature point is located.

8. The method for audio and video synchronization for video conferencing according to claim 1, characterized in that: The step of determining the video feature coding matrix corresponding to the audio frame at each moment according to the time sequence correspondence between the audio frame and the video frame before transmission includes: For audio frames and video frames at the same time, the timestamp technology is combined to assign the audio frame with the video feature coding matrix corresponding to the video frame at the same time.

9. The method for audio and video synchronization for video conferencing according to claim 7, characterized in that: After determining the video feature encoding matrix corresponding to the audio frame at each moment, it also includes: The local area corresponding to the last division where each difference feature point in the video frame is located is used as a feature area, and the local area other than the feature area in the video frame is used as a non-feature area; All the non-feature areas in the video frame are compressed to obtain a compressed video frame.

10. An audio and video synchronization system for video conferencing, characterized in that: It comprises a processor and a memory, wherein the processor is used to process instructions stored in the memory to implement an audio and video synchronization method for video conferencing as described in any one of claims 1-9.

Images