Audio and video real-time optimization processing method based on television station live broadcast

Abstract

The application discloses a real-time audio and video optimization processing method based on television station live broadcast, and relates to the technical field of audio and video optimization. The application solves the problem of poor coherence between the inserted frame picture and the original frame picture caused by not positioning the frame insertion interval in combination with key information such as audio frequency band characteristics and spectral peak points, optimizes the audio quality through denoising processing, accurately positions the frame insertion interval based on the frequency band characteristics, selects a standard value segment through the value segment density, locks the position of the frame to be supplemented in combination with the spectral peak point time interval, generates the pixel to be supplemented based on the pixel change characteristics of the front and rear frames in the frame insertion process, ensures the coherence and consistency between the inserted frame picture and the original frame picture, and avoids the picture quality blur caused by blind frame insertion. For the scene in which the video duration is greater than the audio duration, the lost signal is directly generated and returned for completion, the process is simple and efficient, the single-scene processing duration can be controlled within the delay range allowed by the live broadcast, and the processing quality and real-time performance are considered.

Description

Technical Field

[0001] This invention relates to the field of audio and video optimization technology, specifically a method for real-time audio and video optimization processing based on live television broadcasts. Background Technology

[0002] Television live broadcasts place extremely high demands on audio-visual synchronization, data integrity, and real-time performance. In scenarios such as news broadcasts, sports event broadcasts, and gala broadcasts, issues like audio-visual asynchrony, data loss, and missing frames directly impact broadcast quality and viewer experience, representing core technical challenges that the industry urgently needs to address. Currently, traditional audio-visual live broadcast processing solutions have significant shortcomings in time synchronization correction and data completion, making it difficult to meet the stringent requirements of live broadcast scenarios.

[0003] In terms of time synchronization processing, traditional methods often use a single-dimensional time verification approach or make rough adjustments by manually setting a delay threshold. They lack the dual recognition logic of "duration comparison + key moment verification", which can easily lead to synchronization misjudgment due to a single judgment dimension. Moreover, the correction process often results in over-adjustment or under-adjustment, which cannot accurately adapt to the inherent delay range of 110-180ms in the entire live broadcast link. This can lead to problems such as "lip-syncing" and "audio-visual disconnection", affecting the professionalism of the live broadcast.

[0004] In the data completion and frame optimization stages, traditional solutions lack targeted strategies for scenarios where audio and video durations do not match. When the audio duration is longer than the video duration, they often use blind frame interpolation or simple frame repetition to make up the duration without combining key information such as audio frequency band characteristics and spectral peaks to locate the interpolation interval. This results in poor continuity between the interpolated frame and the original frame, and blurry image quality. At the same time, they ignore audio noise reduction processing before frame interpolation, causing noise to be output synchronously with the completed frame, further reducing the viewing experience. When the video duration is longer than the audio duration, they simply discard some video frames or extend the video playback time without establishing a mechanism for returning and completing lost audio signals. This can easily lead to missing audio information and damage the integrity of the live broadcast content.

[0005] Furthermore, traditional solutions generally lack end-to-end association and labeling of audio and video data. During data transmission and processing, if data is lost or confused, it is difficult to quickly locate and retrieve it, which can easily lead to playback errors and low recovery efficiency. At the same time, some optimization solutions rely on complex algorithms and hardware upgrades, and the processing time exceeds the allowable latency range for live streaming, resulting in live streaming stuttering. Alternatively, due to cumbersome operation logic, a large amount of manual intervention is required, which increases the operation and maintenance costs and operational difficulty, limiting their industrial applicability. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a real-time audio and video optimization processing method based on live television broadcasts. This method solves the problem of poor continuity between interpolated frames and original frames due to the failure to locate interpolation intervals by combining key information such as audio frequency band characteristics and spectral peaks.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time audio and video optimization processing method based on live television broadcasts, comprising the following steps: Step 1: Verify the frequency synchronization of interrelated audio and video data sets. Based on the temporal characteristics associated with the two sets of data, identify the temporal differences between them. If no temporal difference exists, proceed to Step 2. If a temporal difference exists, proceed to Step 3. The specific method is as follows: Based on the live streaming process, confirm the interrelated audio and video data, identify the time length associated with the audio data and record it as the audio duration, and then confirm the time length associated with the video data and record it as the video duration. If the audio duration equals the video duration, proceed to step two; If the audio duration is not equal to the video duration, proceed to step three; Step 2: Analyze whether the temporal characteristics associated with the audio and video data are synchronized. If they are synchronized, no processing is required. If they are not synchronized, lag the earlier data to ensure complete synchronization of the temporal characteristics associated with the two sets of data. The specific method is as follows: The end time of the corresponding audio data is taken as the time feature T1 associated with the audio data, and the end time of the video data is taken as the time feature T2 associated with the video data. If the two sets of times associated with T1 and T2 are completely synchronized, no processing is required. If T1 is earlier and T2 is later, the audio data will be output with a delay until the time characteristics T1 and T2 associated with the audio data are completely consistent, at which point the delay processing will stop. If T1 is later and T2 is earlier, the video data will be output with a delay until the time characteristics associated with the video data T2 and T1 are completely consistent, at which point the delay processing will stop. Step 3: Analyze the specific differences in the temporal characteristics of audio data and video data. If the temporal characteristics of audio data are longer than those of video data, noise removal processing is performed on the audio data first, and then frame interpolation processing is performed on the video data to make the temporal characteristics associated with the video data consistent with those of the audio data. If the temporal characteristics of video data are longer than those of audio data, an audio loss signal is generated and retransmitted back to the output end. The processing method for audio data time feature T1 being longer than video data time feature T2 is as follows: The audio data is first denoised, and the different frequency bands associated with different sound data in the current audio data are identified. The multiple sets of different frequency bands are sorted in ascending order of their values ​​to confirm the frequency band sequence. Based on the time characteristics of video and audio data, a set of time intervals is generated. Different value segments are selected sequentially from the frequency band sequence, and the value segment characteristics associated with a single set of value segments are confirmed. The sound data corresponding to different frequency bands within a single set of value segments are recorded as associated data, and the sound duration corresponding to the associated data is extracted. The sound durations of several sets of associated data are summed to obtain the value segment characteristics belonging to the corresponding value segment. The relevant value segments that satisfy the condition that the value segment characteristics ∈ the time interval are recorded as undetermined value segments. If there is only one set of undetermined value segments, then this undetermined value segment is directly recorded as the standard value segment, and the video data is processed by frame interpolation based on the standard value segment; If there are multiple sets of undetermined value segments, a standard value segment is selected from the multiple sets of undetermined value segments. The value segment density of each set of undetermined value segments is identified, the value segment characteristics associated with the corresponding undetermined value segments are confirmed, and the maximum and minimum frequency bands of the corresponding undetermined value segments are confirmed. The difference frequency band is locked, where the difference frequency band = maximum frequency band - minimum frequency band. The value segment density is calculated as: value segment characteristics ÷ difference frequency band. The undetermined value segment with the maximum value density is taken as the standard value segment. Then, the video data is processed by frame interpolation based on the standard value segment. The specific method for interpolating video data based on standard value ranges is as follows: Identify the spectral peaks associated with each frequency band within the standard value range, and identify the zero points associated with the two sides of the spectral peaks. Based on the corresponding times associated with the two sets of zero points, identify the time interval associated with the corresponding spectral peaks. Then identify the associated times of different frames in the video data and confirm whether there are frames with associated times ∈ time intervals. If they exist, it means that this time interval does not need to be interpolated. If they do not exist, it means that this time interval needs to be interpolated, and this time interval is marked as the interval to be interpolated. Using the interval to be supplemented as a reference, identify the associated frame at the front of the corresponding time, and then identify the preceding frame at the front of the associated frame at the corresponding time. Using the preceding frame and the associated frame as references, identify the change characteristics of pixels at the same position, and denote the pixel value of the corresponding pixel in the preceding frame as Q. i The pixel value of the corresponding pixel in the associated frame is denoted as G. i Where i represents different positions, using (Q i -G i =C i Confirm the change feature C associated with the corresponding location. i ; Then, take the characteristic of change Ci Generate the pixel to be supplemented, where the pixel to be supplemented = G i -C i The system identifies the pixels to be supplemented associated with the corresponding position, and generates the corresponding frames to be supplemented for the interval based on the different pixels to be supplemented associated with different positions. The system then performs frame interpolation, with the specific time of the frame interpolation being the time corresponding to the peak point of the spectrum.

[0008] Preferably, when the time feature T2 of the video data is longer than the time feature T1 of the audio data, an audio loss signal is directly generated, and the corresponding audio data is marked as audio to be supplemented and retransmitted to the output terminal for retransmission processing.

[0009] This invention provides a real-time audio and video optimization processing method based on live television broadcasts. Compared with existing technologies, it has the following advantages: This invention employs a dual time feature recognition method of "duration comparison + end time verification." First, it uses the consistency between audio and video durations to initially screen for time differences. Then, it uses the end times T1 and T2 as the core to verify synchronization, avoiding misjudgments caused by a single-dimensional judgment. For asynchronous scenarios, it adopts a precise control method of "early data lag processing," which can quickly align audio and video time features without complex algorithm iterations. Combined with a preset delay range of 110-180ms, it ensures that synchronization correction does not exceed the live streaming delay threshold and completely solves the problems of "lip-sync misalignment" and "audio-visual disconnection." By adding unique association tags to associated audio and video data, a "one-to-one" binding is achieved throughout the entire data transmission and processing process. This not only enables rapid location and retrieval of lost data to avoid playback errors, but also allows for precise differentiation between data loss and frame missing scenarios during the time difference analysis stage, providing a clear basis for subsequent completion processing. For scenarios where the audio duration is longer than the video duration, the audio quality is first optimized through noise reduction processing. Then, the interval between frames to be interpolated is accurately located based on frequency band characteristics. Standard value segments are selected by value segment density, and the position of the frame to be interpolated is locked by combining the frequency band peak time interval. During the interpolation process, the pixels to be interpolated are generated based on the pixel change characteristics of the previous and next frames. This ensures the continuity and consistency between the interpolated frame and the original frame, while avoiding the blurring of image quality caused by blind interpolation. For scenarios where the video duration is longer than the audio duration, the lost signal is directly generated and sent back to complete it. The process is simple and efficient, and the processing time of a single scenario can be controlled within the latency range allowed for live streaming, balancing processing quality and real-time performance. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Please see Figure 1 This application provides a method for real-time audio and video optimization processing based on live television broadcasts, including the following steps: Step 1: Verify the synchronization of the interrelated audio and video data sets. Based on the time characteristics associated with the two sets of data, identify the time differences between them. If there are no time differences, proceed to Step 2. If there are time differences, proceed to Step 3. Specifically, when there is a deviation in the time relationship between the two sets of data, it means that the two sets of data are out of sync, that is, audio and video are out of sync. To solve this problem, it is necessary to correct this audio and video asynchrony. The specific method for identifying the time difference associated with the two sets of data is as follows: Based on the live streaming process, identify the interrelated audio and video data (with corresponding association tags; the system will mark the associated data during the data transmission process to facilitate quick retrieval if data is lost later, and to avoid errors during playback). Identify the time length associated with the audio data and record it as the audio duration. Then identify the time length associated with the video data and record it as the video duration. If the audio duration equals the video duration, it means there is no time difference between the audio and video data. In this case, proceed to step two to analyze whether time adjustment is needed for the audio or video data. If the audio duration is not equal to the video duration, it means that there is a time difference between the audio data and the video data. Then, proceed to step three to determine whether the audio data is lost or to perform frame trimming on the video data to fill in the associated video frames. Step 2: Analyze whether the time characteristics associated with the audio data and video data are synchronized. If they are synchronized, no processing is required. If they are not synchronized, lag the data that comes first to make the time characteristics associated with the two sets of data completely synchronized. The specific method for analyzing whether time characteristics are synchronized is as follows: The end time of the corresponding audio data is taken as the time feature T1 associated with the audio data, and the end time of the video data is taken as the time feature T2 associated with the video data. If the two sets of times associated with T1 and T2 are completely synchronized, no processing is required. If T1 is earlier and T2 is later, the audio data will be output with a delay until the time characteristics T1 and T2 associated with the audio data are completely consistent, at which point the delay processing will stop. If T1 is later and T2 is earlier, the video data will be output with a delay until the time characteristics associated with the video data T2 and T1 are completely consistent, at which point the delay processing will stop. Specifically, the associated audio and video data are subject to time delay during the live broadcast. The time delay range is generally set between 110-180ms. The specific setting standard is determined in advance by the relevant operators based on their experience and is a preset range. Step 3: Analyze the specific differences in the temporal characteristics of audio data and video data. If the temporal characteristics of audio data are longer than those of video data, noise removal processing is performed on the audio data first, and then frame interpolation processing is performed on the video data to make the temporal characteristics associated with the video data consistent with those of the audio data. If the temporal characteristics of video data are longer than those of audio data, an audio loss signal is generated and retransmitted back to the output end. Regarding step three: When the temporal feature T1 of the audio data is longer than the temporal feature T2 of the video data, i.e., T1 > T2: The audio data is first denoised, and the different frequency bands associated with different sound data in the current audio data are identified. The multiple sets of different frequency bands are sorted in ascending order of their values ​​to confirm the frequency band sequence. Based on the time characteristics of video and audio data, a set of time intervals, [T2, T1], is generated. Different value segments are selected sequentially from the frequency band sequence. The frequency bands associated with each value segment are continuously increasing and belong to the frequency band sequence. The value segment characteristics associated with each set of value segments are confirmed. The sound data corresponding to different frequency bands within each set of value segments are recorded as associated data, and the sound duration corresponding to the associated data is extracted. The sound durations of several sets of associated data are summed to obtain the value segment characteristics belonging to the corresponding value segments. The relevant value segments that satisfy the condition that the value segment characteristics ∈ the time interval are recorded as undetermined value segments. If there is only one set of undetermined value segments, then this undetermined value segment is directly recorded as the standard value segment, and the video data is processed by frame interpolation based on the standard value segment; If there are multiple sets of undetermined value segments, a standard value segment is selected from the multiple sets of undetermined value segments. The value segment density of each set of undetermined value segments is identified, the value segment characteristics associated with the corresponding undetermined value segments are confirmed, and the maximum and minimum frequency bands of the corresponding undetermined value segments are confirmed. The difference frequency band is locked, where the difference frequency band = maximum frequency band - minimum frequency band. The value segment density is calculated as: value segment characteristics ÷ difference frequency band. The undetermined value segment with the maximum value density is taken as the standard value segment. Then, the video data is processed by frame interpolation based on the standard value segment. The specific method for interpolating video data based on standard value ranges is as follows: Identify the spectral peaks associated with each frequency band within the standard value range, and identify the zero points associated with the two sides of the spectral peaks. Based on the corresponding times associated with the two sets of zero points, identify the time interval associated with the corresponding spectral peaks. Then identify the associated times of different frames in the video data and confirm whether there are frames with associated times ∈ time intervals. If they exist, it means that this time interval does not need to be interpolated. If they do not exist, it means that this time interval needs to be interpolated, and this time interval is marked as the interval to be interpolated. The specific method for frame interpolation is as follows: Using the interval to be supplemented as a reference, identify the associated frame at the front end at the corresponding time (that is, the first set of frames; for example, if each frame is one second apart, and the interval to be supplemented is 3-4, then the corresponding time of the associated frame at the front end is 2). Then identify the preceding frame at the front end of the associated frame at the corresponding time. Using the preceding frame and the associated frame as references, identify the change characteristics of pixels at the same position, and denote the pixel value of the corresponding pixel in the preceding frame as Q. i The pixel value of the corresponding pixel in the associated frame is denoted as G. i Where i represents different positions, using (Q i -G i =C i Confirm the change feature C associated with the corresponding location. i ; Then, take the characteristic of change C i Generate the pixel to be supplemented, where the pixel to be supplemented = G i -C i The system identifies the pixels to be supplemented associated with the corresponding position, and generates the corresponding frames to be supplemented for the interval based on the different pixels to be supplemented associated with different positions. The system then performs frame interpolation, with the specific time of the frame interpolation being the time corresponding to the peak point of the spectrum.

[0013] When the temporal feature T2 of the video data is longer than the temporal feature T1 of the audio data, that is, T2 > T1; The system directly generates lost audio signals, marks the corresponding audio data as audio to be supplemented, and retransmits it to the output terminal for retransmission processing to ensure audio synchronization.

[0014] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0015] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A method for real-time audio and video optimization processing based on live television broadcasts, characterized in that, Includes the following steps: Step 1: Verify the synchronization of the single set of interrelated audio and video data, and identify the time difference between the two sets of data based on the time characteristics associated with them. If there is no time difference, proceed to Step 2; if there is a time difference, proceed to Step 3. Step 2: Analyze whether the time characteristics associated with the audio data and video data are synchronized. If they are synchronized, no processing is required. If they are not synchronized, lag the data that comes first to make the time characteristics associated with the two sets of data completely synchronized. Step 3: Analyze the specific differences in the temporal characteristics of audio and video data. If the temporal characteristics of audio data are longer than those of video data, noise removal is performed on the audio data first, and then frame interpolation is performed on the video data to make the temporal characteristics associated with the video data consistent with those of the audio data. If the temporal characteristics of video data are longer than those of audio data, an audio loss signal is generated and retransmitted back to the output.

2. The real-time audio and video optimization processing method based on live TV broadcast as described in claim 1, characterized in that, In step one, the specific method for identifying the time difference associated with the two sets of data is as follows: Based on the live streaming process, confirm the interrelated audio and video data, identify the time length associated with the audio data and record it as the audio duration, and then confirm the time length associated with the video data and record it as the video duration. If the audio duration equals the video duration, proceed to step two; If the audio duration is not equal to the video duration, proceed to step three.

3. The real-time audio and video optimization processing method based on live TV broadcast as described in claim 1, characterized in that, In step two, the specific method for analyzing whether the temporal characteristics associated with the audio data and video data are synchronized is as follows: The end time of the corresponding audio data is taken as the time feature T1 associated with the audio data, and the end time of the video data is taken as the time feature T2 associated with the video data. If the two sets of times associated with T1 and T2 are completely synchronized, no processing is required. If T1 is earlier and T2 is later, the audio data will be output with a delay until the time characteristics T1 and T2 associated with the audio data are completely consistent, at which point the delay processing will stop. If T1 is later and T2 is earlier, the video data will be output with a delay until the time characteristics associated with the video data T2 and T1 are completely consistent, at which point the delay processing will stop.

4. The real-time audio and video optimization processing method based on live TV broadcast as described in claim 1, characterized in that, In step three, the processing method for the audio data time feature T1 being longer than the video data time feature T2 is as follows: The audio data is first denoised, and the different frequency bands associated with different sound data in the current audio data are identified. The multiple sets of different frequency bands are sorted in ascending order of their values ​​to confirm the frequency band sequence. Based on the time characteristics of video and audio data, a set of time intervals is generated. Different value segments are selected sequentially from the frequency band sequence, and the value segment characteristics associated with a single set of value segments are confirmed. The sound data corresponding to different frequency bands within a single set of value segments are recorded as associated data, and the sound duration corresponding to the associated data is extracted. The sound durations of several sets of associated data are summed to obtain the value segment characteristics belonging to the corresponding value segment. The relevant value segments that satisfy the condition that the value segment characteristics ∈ the time interval are recorded as undetermined value segments. If there is only one set of undetermined value segments, then this undetermined value segment is directly recorded as the standard value segment, and the video data is processed by frame interpolation based on the standard value segment.

5. The real-time audio and video optimization processing method based on live TV broadcast as described in claim 4, characterized in that, If there are multiple sets of undetermined value segments, a standard value segment is selected from the multiple sets of undetermined value segments. The value segment density of each set of undetermined value segments is identified, the value segment characteristics associated with the corresponding undetermined value segments are confirmed, and the maximum and minimum frequency bands of the corresponding undetermined value segments are confirmed. The difference frequency band is locked, where the difference frequency band = maximum frequency band - minimum frequency band. The value segment density is calculated as: value segment characteristics ÷ difference frequency band. The undetermined value segment with the maximum value density is taken as the standard value segment. Then, the video data is processed by frame interpolation based on the standard value segment.

6. The real-time audio and video optimization processing method based on live TV broadcast as described in claim 5, characterized in that, In step three, the specific method for interpolating video data based on standard value segments is as follows: Identify the spectral peaks associated with each frequency band within the standard value range, and identify the zero points associated with the two sides of the spectral peaks. Based on the corresponding times associated with the two sets of zero points, identify the time interval associated with the corresponding spectral peaks. Then identify the associated times of different frames in the video data and confirm whether there are frames with associated times ∈ time intervals. If they exist, it means that this time interval does not need to be interpolated. If they do not exist, it means that this time interval needs to be interpolated, and this time interval is marked as the interval to be interpolated. Using the interval to be supplemented as a reference, identify the associated frame at the front of the corresponding time, and then identify the preceding frame at the front of the associated frame at the corresponding time. Using the preceding frame and the associated frame as references, identify the change characteristics of pixels at the same position, and denote the pixel value of the corresponding pixel in the preceding frame as Q. i The pixel value of the corresponding pixel in the associated frame is denoted as G. i Where i represents different positions, using (Q i -G i =C i Confirm the change feature C associated with the corresponding location. i ; Then, take the characteristic of change C i Generate the pixel to be supplemented, where the pixel to be supplemented = G i -C i The system identifies the pixels to be supplemented associated with the corresponding position, and generates the corresponding frames to be supplemented for the interval based on the different pixels to be supplemented associated with different positions. The system then performs frame interpolation, with the specific time of the frame interpolation being the time corresponding to the peak point of the spectrum.

7. The real-time audio and video optimization processing method based on live television broadcast as described in claim 1, characterized in that, In step three, when the video data time feature T2 is longer than the audio data time feature T1, an audio loss signal is directly generated, and the corresponding audio data is marked as audio to be supplemented and retransmitted to the output terminal for retransmission processing.

Images