An audio and video timestamp synchronization method based on artificial intelligence

Abstract

The invention discloses an audio and video timestamp synchronization method based on artificial intelligence, which belongs to the technical field of audio and video timestamp synchronization. The method comprises: step one: determining a reference clock, establishing a clock monitoring model based on the reference clock, monitoring the reference clock in real time through the clock monitoring model, and obtaining a corresponding clock monitoring curve; step two: in the encoding process of audio and video data, stamping each data block with a timestamp according to the time of the reference clock; the data block comprises a video frame and an audio packet; the timestamp comprises a start time and an end time; step three: performing time verification on each data block according to the clock monitoring curve, and adjusting the timestamp corresponding to the data block according to the verification result; step four: performing network status prediction, and performing audio and video pre-adjustment according to the network prediction result; step five: performing real-time calibration according to the timestamp of each data block, and pre-adjusting the subsequent audio and video according to the calibration record.

Description

Technical Field

[0001] The invention belongs to the technical field of audio and video timestamp synchronization, and specifically is an audio and video timestamp synchronization method based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, the synchronization of audio and video content has become particularly important in the fields of multimedia processing, remote conferencing, and real-time monitoring. As a key parameter to ensure the precise synchronization of audio and video content, the accuracy and stability of timestamps are directly related to user experience and system performance. However, in actual applications, due to various factors such as device clock differences, network delays, and different processing speeds, timestamp asynchrony often occurs, resulting in deviations in audio and video content, affecting viewing effects and communication efficiency.

[0003] Therefore, developing an artificial intelligence-based audio and video timestamp synchronization method to achieve accurate calibration and rapid adjustment of timestamps has become an urgent need in current technological development. Summary of the invention

[0004] In order to solve the problems existing in the above scheme, the present invention provides an audio and video timestamp synchronization method based on artificial intelligence.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] An audio and video timestamp synchronization method based on artificial intelligence, the method comprising:

[0007] Step 1: Determine a reference clock, establish a clock monitoring model based on the reference clock, monitor the reference clock in real time through the clock monitoring model, and obtain a corresponding clock monitoring curve;

[0008] Furthermore, the method for establishing the time monitoring model includes:

[0009] Acquire historical record data of a reference clock and mark it as clock material data; set a time status label, wherein the time status label includes normal timing and abnormal timing; and mark the clock material data with a corresponding time status label;

[0010] The corresponding initial monitoring model is established according to the clock material data. The expression of the initial monitoring module is:

[0011]

[0012] Where: s is the input data, the input data is the timing state data of the reference clock; the output data is the clock monitoring value PA(s);

[0013] According to the clock material data, corresponding clock verification data is set, where the clock verification data includes clock status data and clock monitoring value;

[0014] The initial monitoring model is verified and adjusted through clock verification data, and the initial monitoring model that passes the verification is marked as a time monitoring model.

[0015] Furthermore, the method for real-time monitoring of the reference clock by the clock monitoring model includes:

[0016] Acquire the timing status data of the reference clock in real time, analyze the timing status data through the time monitoring model, and obtain the corresponding clock monitoring value;

[0017] A corresponding clock monitoring curve is generated according to the monitoring time corresponding to each clock monitoring value. The horizontal axis of the clock monitoring curve is the monitoring time, and the vertical axis is the clock monitoring value.

[0018] Step 2: During the encoding process of audio and video data, a timestamp is added to each data block according to the time of the reference clock; the data block includes a video frame and an audio packet; the timestamp includes a start time and an end time;

[0019] Step 3: performing time verification on each of the data blocks according to the clock monitoring curve, and adjusting the timestamp corresponding to the data block accordingly according to the verification result;

[0020] Furthermore, the method for performing time verification on each data block according to the clock monitoring curve includes:

[0021] Obtain a clock monitoring curve, mark the monitoring time corresponding to the clock monitoring value of 1 in the clock monitoring curve as abnormal time, merge the abnormal times according to their continuity, and obtain the abnormal timing segments;

[0022] Integrate the data blocks corresponding to the abnormal timing period into a verification block in chronological order;

[0023] Identify the video content and audio content corresponding to the verification block; perform pairing analysis on the corresponding video content and audio content through a preset content verification model to obtain corresponding pairing analysis results, which include normal pairing, abnormal pairing, and adjustment methods for each abnormality;

[0024] When the pairing analysis result is that the pairing is normal, the verification result is qualified;

[0025] When the analysis result is a pairing anomaly, each abnormal adjustment method is identified, each abnormal adjustment method is screened, and the target adjustment method is determined. The verification result is verification failure.

[0026] Furthermore, the method for screening each abnormal adjustment mode includes:

[0027] Simulate and adjust each abnormal adjustment method to obtain the corresponding estimated duration and audio-visual simulation data;

[0028] Conduct audio-visual evaluation on the audio-visual simulation data to obtain the corresponding adjustment value, and the value range of the adjustment value is (0, 10);

[0029] According to the formula Compare each abnormal adjustment method;

[0030] In the formula: SX is the comparison value; b1 and b2 are both proportionality coefficients, and the value range is 0 < b1 ≤ 1, 0 < b2 ≤ 1; RT1 and RT2 are the adjustment values corresponding to the two abnormal adjustment methods to be compared respectively; T1 and T2 are the estimated durations corresponding to the two abnormal adjustment methods to be compared respectively;

[0031] Determine the target adjustment method according to the obtained comparison value.

[0032] Step Four: Perform network status prediction and conduct audio-video pre-adjustment according to the network prediction result;

[0033] Furthermore, the method for conducting audio-video pre-adjustment according to the network prediction result includes:

[0034] Establish a network deviation library, which is used to store network deviation data corresponding to each network status;

[0035] Input the network prediction result into the network deviation library for matching to obtain the corresponding network deviation data;

[0036] Establish a deviation judgment model, and the expression of the deviation judgment model is:

[0037]

[0038] In the formula: q is the input data, and the input data is the network deviation data; the output data is the deviation judgment value PC(q);

[0039] Analyze each network deviation data through the deviation judgment model to obtain the corresponding deviation judgment value;

[0040] Identify the network status corresponding to the network deviation data with a deviation judgment value of 1, mark it as the network abnormal status, identify the time period corresponding to the network abnormal status, and adjust the timestamps of each data block in the corresponding time period.

[0041] Step Five: Perform real-time calibration according to the timestamps of each data block, and conduct pre-adjustment on subsequent audio-video according to the calibration record.

[0042] Furthermore, the method for conducting pre-adjustment on subsequent audio-video data according to the calibration record includes:

[0043] Identify the time deviation of each data block according to the verification record, and generate a corresponding verification deviation curve according to each time deviation, wherein the horizontal axis of the verification deviation curve is time and the vertical axis is time deviation;

[0044] Perform real-time analysis on the calibration deviation curve to obtain corresponding deviation analysis results, which include normal deviation and abnormal deviation;

[0045] When the deviation analysis result shows that the deviation is normal, no corresponding adjustment is made;

[0046] When the deviation analysis result is abnormal, the timestamps of the data blocks in the subsequent time are adjusted.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention effectively solves the problems of complexity of timestamp calibration verification and lack of real-time adjustment of time asynchrony in traditional audio and video synchronization technology by introducing intelligent calibration verification and real-time adjustment mechanism; it will greatly improve the synchronization quality and user experience of audio and video content, and provide strong support for the development of multimedia processing, remote conferencing, real-time monitoring and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] like Figure 1 As shown, an audio and video timestamp synchronization method based on artificial intelligence, the method includes:

[0053] Step 1: Determine a reference clock, establish a corresponding clock monitoring model based on the reference clock, monitor the reference clock in real time through the clock monitoring model, and obtain a corresponding clock monitoring result;

[0054] The reference clock can be a system clock, a Network Time Protocol (NTP) server, or other high-precision time source.

[0055] The method for establishing the time monitoring model includes:

[0056] Obtain historical record data of the reference clock, which is historical data related to normal timing, abnormal timing, etc. of the clock, and is marked as clock material data; set a time status label, which includes normal timing and abnormal timing; mark the clock material data with a corresponding time status label, and mark it according to the actual timing result;

[0057] According to the clock material data, the corresponding initial monitoring model is established. The expression of the initial monitoring module is: Wherein: s is the input data, which is the timing status data of the reference clock, that is, various data related to whether its timing is abnormal, and the collection items are set according to the actual situation of the reference clock; the output data is the clock monitoring value PA(s); specifically, the clock material data is used as the training set, and the judgment is made in combination with the existing data recognition and judgment technology. For example, the training and judgment can be based on a neural network;

[0058] The corresponding clock verification data is set according to the clock material data to verify and adjust the initial monitoring model, including clock status data and clock monitoring values; the initial monitoring model is verified and adjusted through the clock verification data, and the verified initial monitoring model is marked as a time monitoring model.

[0059] The method of real-time monitoring of the reference clock through the clock monitoring model includes:

[0060] Acquire the timing status data of the reference clock in real time, analyze the timing status data through the time monitoring model, and obtain the corresponding clock monitoring value;

[0061] A corresponding clock monitoring curve is generated according to the monitoring time corresponding to each clock monitoring value. The horizontal axis of the clock monitoring curve is the monitoring time, and the vertical axis is the clock monitoring value.

[0062] Step 2: During the encoding process of audio and video data, a timestamp is added to each data block according to the time of the reference clock; the data block includes a video frame and an audio packet; the timestamp includes a start time and an end time;

[0063] Step 3: Perform time verification on each data block according to the clock monitoring curve, and adjust the timestamp corresponding to the data block accordingly according to the verification result;

[0064] The method for performing time verification on each data block according to the clock monitoring curve includes:

[0065] Obtain a clock monitoring curve, mark the monitoring time corresponding to the clock monitoring value of 1 in the clock monitoring curve as abnormal time, merge the abnormal times according to their continuity, and obtain the abnormal timing segments; that is, merge the abnormal times with continuous time to form a time segment, which is marked as the abnormal timing segment;

[0066] Integrate the data blocks corresponding to the abnormal timing period into a verification block in chronological order;

[0067] Identify the video content and audio content corresponding to the verification block; perform pairing analysis on the corresponding video content and audio content through a preset content verification model to obtain corresponding pairing analysis results, which include normal pairing, abnormal pairing, and various abnormal adjustment methods; the abnormal adjustment method is the measures that can be taken to adjust the pairing abnormality to normal pairing, such as adjusting the size of the playback buffer and the playback speed to correct the deviation, discard or perform compensatory playback, etc.; the content verification model is established based on neural networks such as CNN networks or DNN networks, and a large amount of historical audio and video data is obtained. The corresponding labels are set manually, such as normal pairing and abnormal pairing. For pairing abnormalities, the corresponding abnormal adjustment method is set according to the historical adjustment measures, and the integration setting is used as a training set. Training is performed based on the training set, and analysis is performed through the content verification model after successful training.

[0068] When the pairing analysis result is that the pairing is normal, the verification result is qualified;

[0069] When the analysis result is a pairing anomaly, each abnormal adjustment method is identified, each abnormal adjustment method is screened, and the target adjustment method is determined. The verification result is verification failure.

[0070] Methods for screening various abnormal adjustment methods include:

[0071] Simulate and adjust each abnormal adjustment method to obtain the corresponding estimated duration and audio-visual simulation data; assume that the adjustment is performed according to the abnormal adjustment method, estimate the required duration, and mark the audio and video data after the simulated adjustment as audio-visual simulation data;

[0072] Perform audiovisual evaluation on the audiovisual simulation data to obtain the corresponding adjustment value. The value range of the adjustment value is (0, 10). Evaluate according to the adjustment effect. The adjustment effect is compared with the original audio-visual data, and its smoothness and content completeness are used as the adjustment effect. The adjustment that does not meet the minimum requirement is 0. Since each abnormal adjustment method meets the minimum requirement, it is greater than 0; the original audio-visual data is 10, that is, no adjustment is 10; specifically, it can be evaluated based on the existing technology. For example, an adjustment evaluation model is established based on a CNN network or a DNN network, and a corresponding training set is established through artificial means for training. The training set includes input data and output data. The input data is the audiovisual simulation data and the original audio-visual data; the output data is the adjustment value; analyze through the adjustment evaluation model after successful training to obtain the corresponding adjustment value; in other embodiments, the content verification model can also be adjusted so that the output abnormal adjustment method includes the corresponding adjustment value;

[0073] According to the formula Compare each abnormal adjustment method;

[0074] In the formula: SX is the comparison value; b1 and b2 are both proportionality coefficients, and the value range is 0 < b1 ≤ 1, 0 < b2 ≤ 1; RT1 and RT2 are the adjustment values corresponding to the two abnormal adjustment methods to be compared respectively; T1 and T2 are the estimated durations corresponding to the two abnormal adjustment methods to be compared respectively;

[0075] Determine the target adjustment method according to the obtained comparison value; that is, perform cyclic comparison to determine the best abnormal adjustment method.

[0076] Step Four: Perform network state prediction and perform audio-visual pre-adjustment according to the network prediction result;

[0077] Use the existing network prediction technology to perform network state prediction to obtain the corresponding network prediction result.

[0078] The methods for performing audio-visual pre-adjustment according to the network prediction result include:

[0079] Obtain various possible network states, perform playback simulation according to each network state, that is, perform audio-visual data playback simulation in the corresponding network state, obtain the timestamp deviation situation of each data block of the audio-visual data in each network state, specifically tested and statistically analyzed by the platform side, or can be statistically analyzed based on the existing conclusion data to obtain the time deviation of the audio-visual data in each network state, integrated into network deviation data, and mark the corresponding network state label; establish a network deviation library according to each network deviation data;

[0080] Match the corresponding network deviation data according to the network prediction results; analyze each network deviation data to determine whether audio and video adjustment is required, that is, whether the deviation exceeds the preset standard; specifically, a deviation judgment model can be established, and the deviation judgment model is used to analyze the network deviation data to determine whether audio and video adjustment is required, and the need for audio and video adjustment or exceeding the preset standard is considered to meet the deviation judgment requirements; the expression of the deviation judgment model is Where: q is the input data, the input data is the network deviation data; the output data is the deviation judgment value PC(q);

[0081] Analyze the deviation data of each network through the deviation judgment model to obtain the corresponding deviation judgment value;

[0082] Identify the network status corresponding to the network deviation data with a deviation judgment value of 1, mark it as a network abnormal status, identify the time period corresponding to the network abnormal status, and adjust the timestamps of each data block in the corresponding time period; the adjustment method can be determined through the content verification model, and then screened and adjusted.

[0083] Step 5: Perform real-time calibration based on the timestamp of each data block, and pre-adjust the subsequent audio and video based on the calibration record.

[0084] Real-time calibration is performed according to the timestamp of each data block, that is, calibration is performed during playback to understand the deviation of the timestamp of each data block.

[0085] The calibration record is the deviation record of the timestamp of each data block.

[0086] The method for pre-adjusting subsequent audio and video data according to the calibration record includes:

[0087] Identify the time deviation of each data block according to the verification record, and generate a corresponding verification deviation curve according to each time deviation, wherein the horizontal axis of the verification deviation curve is time and the vertical axis is time deviation;

[0088] Perform real-time analysis on the calibration deviation curve to determine whether the deviation exceeds the preset standard. Because time deviation has a cumulative effect, it can be judged based on the time deviation of the current time and set according to whether it affects the audiovisual of audio and video data. It is generally set according to field standards; obtain the corresponding deviation analysis results, including normal deviation and abnormal deviation;

[0089] When the deviation analysis result shows that the deviation is normal, no corresponding adjustment is made;

[0090] When the deviation analysis result is abnormal, the timestamps of the data blocks in the subsequent time are adjusted; the adjustment is performed according to the above adjustment method; or the analysis and adjustment can be performed based on the existing technology.

[0091] The above formulas are all calculated by removing dimensions and taking numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0092] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An audio and video timestamp synchronization method based on artificial intelligence, characterized in that: Methods include: Step 1: Determine a reference clock, establish a clock monitoring model based on the reference clock, monitor the reference clock in real time through the clock monitoring model, and obtain a corresponding clock monitoring curve; Step 2: During the encoding process of audio and video data, a timestamp is added to each data block according to the time of the reference clock; the data block includes a video frame and an audio packet; the timestamp includes a start time and an end time; Step 3: performing time verification on each of the data blocks according to the clock monitoring curve, and adjusting the timestamp corresponding to the data block accordingly according to the verification result; Step 4: Predict the network status and pre-adjust the audio and video according to the network prediction results; Step 5: Perform real-time calibration based on the timestamps of each data block, and pre-adjust the subsequent audio and video based on the calibration records; The method for establishing the time monitoring model includes: Acquire historical record data of a reference clock and mark it as clock material data; set a time status label, wherein the time status label includes normal timing and abnormal timing; and mark the clock material data with a corresponding time status label; The corresponding initial monitoring model is established according to the clock material data. The expression of the initial monitoring module is: ; Where: s is the input data, the input data is the timing state data of the reference clock; the output data is the clock monitoring value PA(s); According to the clock material data, corresponding clock verification data is set, where the clock verification data includes clock status data and clock monitoring value; Verify and adjust the initial monitoring model through clock verification data, and mark the initial monitoring model that passes the verification as a time monitoring model; The method for performing time verification on each data block according to the clock monitoring curve includes: Obtain a clock monitoring curve, mark the monitoring time corresponding to the clock monitoring value of 1 in the clock monitoring curve as abnormal time, merge the abnormal times according to their continuity, and obtain the abnormal timing segments; Integrate the data blocks corresponding to the abnormal timing period into a verification block in chronological order; Identify the video content and audio content corresponding to the verification block; perform pairing analysis on the corresponding video content and audio content through a preset content verification model to obtain corresponding pairing analysis results, which include normal pairing, abnormal pairing, and adjustment methods for each abnormality; When the pairing analysis result is that the pairing is normal, the verification result is qualified; When the analysis result is a pairing anomaly, each abnormal adjustment method is identified, each abnormal adjustment method is screened, and the target adjustment method is determined. The verification result is verification failure.

2. According to the artificial intelligence-based audio and video timestamp synchronization method of claim 1, it is characterized in that: The method of real-time monitoring of the reference clock through the clock monitoring model includes: Acquire the timing status data of the reference clock in real time, analyze the timing status data through the time monitoring model, and obtain the corresponding clock monitoring value; A corresponding clock monitoring curve is generated according to the monitoring time corresponding to each clock monitoring value. The horizontal axis of the clock monitoring curve is the monitoring time, and the vertical axis is the clock monitoring value.

3. The method for synchronizing audio and video timestamps based on artificial intelligence according to claim 1, characterized in that: Methods for screening various abnormal adjustment methods include: Simulate and adjust each abnormal adjustment method to obtain the corresponding estimated duration and audio-visual simulation data; Perform audio-visual evaluation on the audio-visual simulation data to obtain corresponding adjustment values, where the adjustment values ​​range from (0 to 10); According to the formula Compare each abnormal adjustment method; Where: SX is the comparison value; b1 and b2 are both proportionality coefficients, and the value ranges are 0 < b1 ≤ 1, 0 < b2 ≤ 1; RT1 and RT2 are the adjustment values corresponding to the two abnormal adjustment methods for comparison respectively; T1 and T2 are the estimated durations corresponding to the two abnormal adjustment methods for comparison respectively. Determine the target adjustment method according to the obtained comparison value.

4. The method for synchronizing audio and video timestamps based on artificial intelligence according to claim 1, characterized in that: The method for performing audio-video pre-adjustment according to the network prediction result includes: Establish a network deviation library, which is used to store network deviation data corresponding to each network state; Input the network prediction result into the network deviation library for matching to obtain the corresponding network deviation data; A deviation judgment model is established, and the expression of the deviation judgment model is: ; Where: q is the input data, and the input data is network deviation data; the output data is the deviation judgment value PC(q); Analyze each network deviation data through a deviation judgment model to obtain the corresponding deviation judgment value; Identify the network state corresponding to the network deviation data with a deviation judgment value of 1, mark it as the network abnormal state, identify the time period corresponding to the network abnormal state, and adjust the timestamps of each data block in the corresponding time period.

5. The method for synchronizing audio and video timestamps based on artificial intelligence according to claim 1, characterized in that: The method for performing pre-adjustment on subsequent audio-video data according to the calibration record includes: Identify the time deviation of each data block according to the verification record, generate a corresponding verification deviation curve based on each time deviation, the horizontal axis of the verification deviation curve is time, and the vertical axis is the time deviation; Perform real-time analysis on the verification deviation curve to obtain the corresponding deviation analysis result, and the deviation analysis result includes normal deviation and abnormal deviation; When the deviation analysis result is normal deviation, no corresponding adjustment is made; When the deviation analysis result is abnormal deviation, adjust the timestamps of the data blocks in the subsequent time.

Images