Video playback method based on digital audio

Abstract

The invention relates to the technical field of video playback, in particular to a video playback method based on digital audio, comprising: calibrating the starting point of audio and video playback and starting synchronization monitoring; adaptively adjusting the audio and video quality according to network conditions; dynamically optimizing synchronization and quality adjustment effects according to user evaluations; the invention calibrates the starting playback point of audio and video by timestamps, and dynamically adjusts the progress of audio and video playback by real-time synchronization monitoring during playback to ensure audio and video synchronization playback; according to changes in network conditions, adopts bandwidth prediction technology, combines a time series model to predict network bandwidth fluctuations, and ensures the smoothness and stability of audio and video playback; dynamically optimizes playback strategies by users' ratings of audio and video synchronization effects and quality adjustment effects, and ensures that audio and video synchronization and quality can be automatically adjusted according to user feedback.

Description

Technical Field

[0001] The present invention relates to the technical field of video playback, and in particular to a video playback method based on digital audio. Background Art

[0002] With the rapid development of digital audio and video technology, the video playback method based on digital audio has become an important part of multimedia applications. However, in practical applications, this method still faces many technical difficulties, including:

[0003] When the network environment is unstable or the latency is high, the synchronization between digital audio and video is often affected, resulting in a poor user experience. This phenomenon of audio and video asynchrony is particularly evident in live broadcasts and real-time playback.

[0004] The compression process of digital audio often results in a degradation of sound quality, especially when low bit rate encoding is used. The audio details and dynamic range may be weakened, affecting the user's listening experience. Summary of the invention

[0005] In view of the above problems, a video playback method based on digital audio is proposed, which mainly solves the problems of video audio and video asynchrony and low audio and video quality under network fluctuation conditions.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a video playback method based on digital audio, comprising the following steps:

[0007] Calibrate the start point of audio and video playback and start synchronization monitoring, including,

[0008] Calibrate the start play point of audio and video, and perform synchronization monitoring based on the calibration results, specifically:

[0009] Use timestamps to calibrate the start play point of audio and video, use real-time play progress to analyze synchronization status, then use audio feature analysis algorithm to analyze audio status, and dynamically adjust the synchronization of audio and video based on the audio status analysis results;

[0010] Adaptively adjust audio and video quality based on network conditions, including:

[0011] Collect network data to predict bandwidth fluctuations in the future time period, and adaptively adjust the audio and video quality based on the prediction results, specifically:

[0012] Use network acquisition equipment to collect network status data in real time, and use time series models to predict bandwidth. Adaptively adjust audio and video quality based on bandwidth prediction results, and adjust audio and video based on user preferences.

[0013] Dynamically optimize synchronization and quality adjustment based on user evaluation, including:

[0014] A user rating model is constructed to rate the synchronization effect and quality adjustment effect, and the synchronization and quality adjustment effects are dynamically optimized based on the rating results.

[0015] As a preferred solution of the digital audio-based video playback method of the present invention, the calibration of the start playback point of the audio and video using the timestamp is specifically as follows:

[0016] Use the server to obtain the initial timestamp of the audio stream and the initial timestamp of the video stream , and set the local time deviation ;

[0017] Calculate the time of audio playback and video playback at the video playback location, then we have:

[0018] Audio playback time, ;

[0019] Video playback time, in, Indicates the initial timestamp of the audio stream. Indicates the initial timestamp of the video stream. Indicates the local time deviation, Indicates the audio playback timestamp. Indicates the video playback timestamp;

[0020] Using the calculated playback timestamp for calibration, we have,

[0021] in, Indicates the audio playback timestamp. Indicates the video playback timestamp. Indicates the time difference between audio and video playback. By adjusting the time difference, the start point of audio and video playback can be calibrated. Then,

[0022] If the calculated time difference between audio playback and video playback satisfies the formula , indicating that the calibration of the starting point of audio and video playback is completed. Otherwise, the timestamps of the audio and video sent by the server are adjusted until the calibration conditions are met.

[0023] As a preferred solution of the digital audio-based video playback method of the present invention, the synchronization status analysis using the real-time playback progress is specifically as follows:

[0024] in, Indicates the real-time playback progress of the audio. Indicates the real-time playback progress of the video. Indicates the difference between the audio playback progress and the video playback progress, which is used to determine whether the audio and video synchronization needs to be dynamically adjusted. If yes,

[0025] If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it indicates that the current audio playback is synchronized with the video playback;

[0026] If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it means that the current audio playback is not synchronized with the video playback.

[0027] As a preferred solution of the digital audio-based video playback method of the present invention, the dynamic adjustment of the synchronization between audio and video according to the audio state analysis result is as follows:

[0028] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is a silent state, a delay operation is performed on the current audio to achieve synchronization of audio and video;

[0029] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is normal, the playback speed of the current audio is adjusted to achieve synchronization of audio and video;

[0030] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is a noise state, the energy of the current audio is adjusted until the current video state is a normal state, and then the playback speed of the current audio is adjusted to achieve synchronization of audio and video;

[0031] The insertion delay operation is to insert a silent delay into the audio in a muted state to adjust the audio playback timestamp, which is specifically implemented as follows:

[0032] in, Indicates the inserted silence delay time. Indicates the maximum allowed silent delay time. Indicates the current playback rate of the audio. Indicates the adjusted audio playback progress.

[0033] The adjustment of the playback speed is for audio in a normal state, and the audio playback timestamp is changed by adjusting the audio playback rate, as follows:

[0034] in, Indicates the current playback rate of the audio. represents the rate adjustment factor, Indicates the time difference of audio and video playback. Indicates the current timestamp of the audio. Indicates the adjusted audio playback progress.

[0035] As a preferred solution of the digital audio-based video playback method of the present invention, the bandwidth prediction by the time series model is specifically as follows:

[0036] Construct a time series model to predict bandwidth, then we have:

[0037] in, Indicates prediction The network bandwidth at the time, Represents the autoregressive coefficient, which is the weight of the impact of historical bandwidth data on bandwidth prediction. Represents the initial autoregressive coefficient, which is set by the implementer according to the actual application scenario. represents the order of the autoregressive term, represents the moving average coefficient, which is the weight of the error term on the bandwidth prediction. represents the order of the moving average, represents the error term;

[0038] To predict the bandwidth change rate, we have,

[0039] in, Indicates that bandwidth will The rate of change over time, Represents prediction The network bandwidth at the time, Indicates the network bandwidth data after smoothing;

[0040] Using historical data to determine bandwidth fluctuations, we have:

[0041] The historical network bandwidth data of the day before the prediction time node is used as the input of the time series model, and the output of the model is used as the bandwidth fluctuation state judgment threshold , the specific judgment is as follows:

[0042] When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula When , it indicates that the current bandwidth fluctuation state is rising;

[0043] When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula , it indicates that the current bandwidth fluctuation status is decreasing.

[0044] As a preferred solution of the digital audio-based video playback method of the present invention, the audio and video adjustment according to user preferences is specifically as follows:

[0045] When the bandwidth fluctuation state is in an increasing state, the audio and video quality is reduced according to the user's preference.

[0046] When the user prefers audio quality, the adaptive adjustment of audio and video quality is achieved by reducing the video quality parameters, specifically:

[0047] ;

[0048] The audio and video after adaptive adjustment are:

[0049] in, Indicates the video quality at the current moment. Indicates the adjusted video quality. Indicates the adjustment parameters of video quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the audio quality at the current moment. Indicates the adjusted audio and video quality;

[0050] When the user prefers video quality, the audio and video quality is adaptively adjusted by reducing the audio quality parameters. Specifically:

[0051] ;

[0052] The audio and video after adaptive adjustment are:

[0053] in, Indicates the audio quality at the current moment. Indicates the adjusted audio quality. Indicates the adjustment parameters of audio quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the video quality at the current moment. Indicates the adjusted audio and video quality; Indicates the actual bandwidth at the current moment. Indicates the actual bandwidth at the moment;

[0054] When the bandwidth fluctuation state is in a decreasing state, the video quality is improved according to the user preference, then,

[0055] Comprehensively adjust the audio and video quality based on the user's preference weight, specifically:

[0056] in, Indicates the adjusted video quality. Indicates the adjusted audio quality. represents the video preference weight, represents the audio preference weight, Indicates the adjusted audio and video quality.

[0057] As a preferred solution of the digital audio-based video playback method of the present invention, the user scoring model is constructed to score the synchronization effect and the quality adjustment effect as follows:

[0058] User ratings for Utilize Sync Effects , User ratings for video quality And objective observation results For comprehensive scoring, we have:

[0059] Rating of synchronization effect,

[0060] in, Indicates Users' ratings of synchronization performance, Indicates the total number of users who participated in the rating. represents the objective observation of the synchronization effect, Indicates the user rating of the synchronization effect. Represent the weight coefficients of user ratings and objective observation results respectively;

[0061] Rating of video quality,

[0062] in, Indicates Users' ratings of video quality, Indicates the total number of users who participated in the rating. Represents the objective observation result of video quality. Indicates the user rating of the video quality. Represent the weight coefficients of user ratings and objective observation results respectively;

[0063] Based on the comprehensive score of synchronization effect and video quality, there are:

[0064] in, Indicates the user rating of the synchronization effect. Indicates the user rating of the video quality. Respectively represent the weight coefficients of synchronization effect and video quality.

[0065] As a preferred solution of the digital audio-based video playback method of the present invention, the dynamic optimization of synchronization and quality adjustment effect according to the scoring result is specifically as follows:

[0066] Set the scoring thresholds for synchronization and video quality as well as ;

[0067] Dynamic optimization is performed based on the comparison between user ratings and rating thresholds, then:

[0068] When the comparison result between the synchronization effect and the scoring threshold satisfies the formula When the current period is maintained, the synchronous adjustment is maintained;

[0069] When the comparison result between the synchronization effect and the scoring threshold satisfies the formula When the audio and video synchronization is adjusted again;

[0070] When the comparison result between the video quality and the scoring threshold satisfies the formula When playing, maintain the current video quality;

[0071] When the comparison result between the video quality and the scoring threshold satisfies the formula Re-adjust the audio and video quality of the video.

[0072] The beneficial effects of the present invention are as follows: the present invention calibrates the start playback point of audio and video through timestamps, and dynamically adjusts the progress of audio and video playback through real-time synchronous monitoring during playback to ensure synchronous playback of audio and video; reduces the phenomenon of audio and video being out of sync, and improves the user's viewing experience; according to changes in network conditions, adopts bandwidth prediction technology, combined with a time series model to predict network bandwidth fluctuations, to ensure the smoothness and stability of audio and video playback; dynamically optimizes the playback strategy through users' ratings of audio and video synchronization effects and quality adjustment effects, to ensure that audio and video synchronization and quality can be automatically adjusted according to user feedback; through an audio feature analysis algorithm, the state of the audio can be dynamically analyzed, and the audio playback speed or delay can be adaptively adjusted according to the analysis results, making the synchronization of audio and video more intelligent and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0074] Figure 1 The figure is a schematic diagram of the overall method steps of the digital audio-based video playback method of the present invention.

[0075] Figure 2 The present invention is a schematic diagram of the logic judgment for realizing the audio and video synchronization optimization of the video playback method based on digital audio.

[0076] Figure 3 The present invention is a schematic diagram of logic judgment for realizing dynamic optimization of synchronization and quality adjustment effect of the video playback method based on digital audio. DETAILED DESCRIPTION

[0077] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0078] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0079] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0080] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0081] Meanwhile, in the description of the present invention, it should be noted that the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0082] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0083] Example 1: Reference Figure 1 to Figure 3 , as an embodiment of the present invention, provides a video playback method based on digital audio, comprising the following steps:

[0084] S1: Calibrate the start point of audio and video playback and start synchronous monitoring.

[0085] Specifically, calibrating the starting point of audio and video playback is to calibrate the starting playback point of the audio and video, and continuously monitor the synchronization status of the two during the playback process. The synchronization monitoring is to calculate the starting delay of the audio and video by comparing the server timestamp and the local time, and then adjust the playback timing, and track the actual playback progress of the audio and video in real time, calculate the difference between the two, and dynamically adjust the delay according to the calculated difference results to keep the video and audio synchronized.

[0086] Furthermore, the start playback point of the audio and video is calibrated as follows:

[0087] When playback starts, use the server to obtain the initial timestamp of the audio stream and the initial timestamp of the video stream , and set the local time deviation ;

[0088] Calculate the time of audio playback and video playback at the video playback location, then we have:

[0089] Audio playback time, ;

[0090] Video playback time, in, Indicates the initial timestamp of the audio stream. Indicates the initial timestamp of the video stream. Indicates the local time deviation, Indicates the audio playback timestamp. Indicates the video playback timestamp;

[0091] Using the calculated playback timestamp for calibration, we have,

[0092] in, Indicates the audio playback timestamp. Indicates the video playback timestamp. Indicates the time difference between audio and video playback. By adjusting the time difference, the start point of audio and video playback can be calibrated. Then,

[0093] If the calculated time difference between audio playback and video playback satisfies the formula , indicating that the calibration of the starting point of audio and video playback is completed. Otherwise, the timestamps of the audio and video sent by the server are adjusted until the calibration conditions are met.

[0094] Furthermore, real-time tracking of the real-time playback progress of audio and video is to obtain the real-time playback progress of audio and video using the playback time length, and calculate the difference between the two based on the obtained playback progress, then,

[0095] in, Indicates the real-time playback progress of the audio. Indicates the real-time playback progress of the video. Indicates the difference between the audio playback progress and the video playback progress, which is used to determine whether the audio and video synchronization needs to be dynamically adjusted. If yes,

[0096] If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it indicates that the current audio playback is synchronized with the video playback. There is no need to dynamically adjust the synchronization between audio and video, and the current playback settings are maintained.

[0097] If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it means that the current audio playback is not synchronized with the video playback, and the synchronization between audio and video needs to be dynamically adjusted to achieve synchronization between audio and video.

[0098] Dynamically adjusting the synchronization of audio and video is to analyze the audio status through the audio feature analysis algorithm, and adaptively adjust the audio playback speed and delay according to the analysis results. The specific implementation is as follows:

[0099] The characteristics of the audio signal are used to complete the audio feature extraction, and the extracted features are constructed as the feature vector of the audio feature analysis, then,

[0100] By framing and windowing the original audio signal, we have:

[0101] Divide the original audio signal into Multiple frame signals,

[0102] Window the divided frame signal.

[0103] in, Indicates the length of each frame, Indicates frame shift, which is the time difference between the starting positions of two adjacent frames. represents the original audio signal, Indicates The frame signal, represents the windowing function;

[0104] Perform Fourier transform on each frame and convert the transform result into energy calculation, then we have:

[0105]

[0106] in, Indicates The energy of the frame, Indicates The Fourier transform result of the frame. The Fourier transform is an existing public technology that is easy to associate with those skilled in the art. In this embodiment, only this algorithm is used to realize the calculation of energy, and no further details are given here.

[0107] Extract features including energy, Mel frequency cepstral coefficients and spectral entropy, and construct feature vectors based on the extracted features, then we have,

[0108]

[0109] in, Indicates The energy of the frame, Indicates Mel-frequency cepstral coefficients of the frame, It represents spectral entropy, Mel frequency cepstral coefficient and spectral entropy are all conventional feature extraction, which is an existing public technology and will not be described in detail in this embodiment. Represents the constructed feature vector, which is used to judge the audio state, then,

[0110] Set audio energy threshold And the spectral entropy threshold , the audio state is determined according to the set threshold and feature vector parameter value, as follows:

[0111] When the parameter value of the feature vector meets the set threshold, the formula , indicating that the current audio status is muted;

[0112] When the parameter value of the feature vector meets the set threshold, the formula , it indicates that the current audio status is normal;

[0113] When the parameter value of the feature vector meets the set threshold, the formula , it indicates that the current audio state is noise state.

[0114] Adaptively adjust the audio playback speed and delay based on the analysis results. This is to combine the audio status analysis results and the audio and video playback progress difference, dynamically adjust the audio playback speed and insert delay to ensure synchronization between the two. The specific adjustments are as follows:

[0115] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is a silent state, a delay operation is performed on the current audio to achieve synchronization of audio and video;

[0116] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is normal, the playback speed of the current audio is adjusted to achieve synchronization of audio and video;

[0117] When the difference in audio and video playback progress satisfies the formula When the audio state analysis result is a noise state, the energy of the current audio is adjusted until the current video state is a normal state, and then the playback speed of the current audio is adjusted to achieve synchronization of audio and video.

[0118] Furthermore, the insertion delay operation is to insert a silent delay into the audio in a muted state to adjust the playback timestamp of the audio. The specific implementation is as follows:

[0119] in, Indicates the inserted silence delay time. Indicates the maximum allowed silent delay time. Indicates the current playback rate of the audio. Indicates the adjusted audio playback progress.

[0120] The playback speed adjustment is for audio in a normal state. By adjusting the audio playback rate, the audio playback timestamp is changed. The details are as follows:

[0121]

[0122] in, Indicates the current playback rate of the audio. represents the rate adjustment factor, Indicates the time difference of audio and video playback. Indicates the current timestamp of the audio. Indicates the adjusted audio playback progress.

[0123] It should be noted that by adjusting the audio playback timestamp and monitoring the adjusted audio and video playback progress in real time, we have:

[0124]

[0125] And according to the real-time monitoring results, the dilution rate is optimized and the delay insertion time is adjusted.

[0126]

[0127] in, Indicates the new audio and video playback progress difference. The optimization factor representing the adjustment coefficient is obtained by reversely adjusting the adjusted audio and video playback progress result. Indicates the optimized rate adjustment coefficient.

[0128] S2: Adaptively adjust audio and video quality based on network conditions.

[0129] Specifically, adaptively adjusting the audio and video quality according to network conditions is achieved by monitoring the changing status of the network bandwidth in real time, and predicting the bandwidth fluctuation in the future time period based on the changing trend of the network bandwidth, and adaptively adjusting the audio and video quality based on the prediction results to maintain the clarity and continuity of the audio and video.

[0130] Furthermore, real-time monitoring of the changing state of network bandwidth is to use network acquisition equipment to collect network status data in real time, including network bandwidth, network request packet loss rate, and acquisition delay. Then,

[0131] Bandwidth collection,

[0132]

[0133] Packet loss rate of network requests,

[0134]

[0135] Collection delay,

[0136]

[0137] in, Indicates time The network bandwidth at the time, Indicates time The amount of data transferred within Indicates the time interval, Indicates time The packet loss rate at the time, Indicates time The number of packets lost within Indicates time The total number of packets sent within Indicates time The average delay at time, Indicates the number of data packets collected. Indicates The delay time of each data packet;

[0138] Perform data smoothing on the collected bandwidth data, then we have:

[0139]

[0140] in, Indicates the network bandwidth data after smoothing. represents the average moving window size, Indicates The network bandwidth at the moment.

[0141] Furthermore, the bandwidth fluctuation in the future time period is predicted by using the collected bandwidth data and the time series model to predict the bandwidth, as follows:

[0142] Construct a time series model to predict bandwidth, then we have:

[0143]

[0144] in, Represents prediction The network bandwidth at the time, Represents the autoregressive coefficient, which is the weight of the impact of historical bandwidth data on bandwidth prediction. Represents the initial autoregressive coefficient, which is set by the implementer according to the actual application scenario. represents the order of the autoregressive term, represents the moving average coefficient, which is the weight of the error term on the bandwidth prediction. represents the order of the moving average, represents the error term;

[0145] To predict the bandwidth change rate, we have,

[0146]

[0147] in, Indicates that bandwidth will The rate of change over time, Represents prediction The network bandwidth at the time, Indicates the network bandwidth data after smoothing;

[0148] Using historical data to determine bandwidth fluctuations, we have:

[0149] The historical network bandwidth data of the day before the prediction time node is used as the input of the time series model, and the output of the model is used as the bandwidth fluctuation state judgment threshold , the specific judgment is as follows:

[0150] When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula When , it indicates that the current bandwidth fluctuation state is rising;

[0151] When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula , it indicates that the current bandwidth fluctuation status is decreasing.

[0152] Furthermore, the audio and video quality is adaptively adjusted according to the prediction results, that is, the audio and video quality is adaptively adjusted according to the fluctuation of the bandwidth. The specific adjustments are as follows:

[0153] When the bandwidth fluctuation is in an increasing state, the audio and video quality is reduced;

[0154] When the bandwidth fluctuation state is in a decreasing state, the audio and video quality is improved.

[0155] Specifically, the adaptive adjustment of audio and video quality is to dynamically adjust the audio quality and video quality according to user preferences, collect user preference data, determine the user's preference weight for audio and video quality, and adaptively adjust the audio and video quality, as follows:

[0156] According to the user's historical browsing data, the user's preference weight for audio and video is set, then:

[0157] Audio preference weight ,

[0158]

[0159] Video preference weight ,

[0160]

[0161] in, Respectively represent the user's preference coefficients for audio and video, Indicates the user's audio preference weight, Indicates the user's preference weight for the video;

[0162] Based on the user's preference weight for audio and video, the audio and video quality is adaptively adjusted as follows:

[0163] When the user's audio and video weight comparison results meet the formula When , it indicates that the user prefers audio quality, and the audio and video quality is adaptively adjusted on the basis of ensuring audio quality;

[0164] When the user's audio and video weight comparison results meet the formula , it indicates that the user prefers video quality, and the audio and video quality is adaptively adjusted on the basis of ensuring video quality.

[0165] Furthermore, adaptive adjustment is based on comprehensive analysis and adjustment according to the fluctuation status of network bandwidth and user preferences. The specific analysis is as follows:

[0166] When the bandwidth fluctuation state is in an increasing state, the audio and video quality is reduced according to the user's preference.

[0167] When the user prefers audio quality, the adaptive adjustment of audio and video quality is achieved by reducing the video quality parameters, specifically:

[0168]

[0169] The audio and video after adaptive adjustment are:

[0170]

[0171] in, Indicates the video quality at the current moment. Indicates the adjusted video quality. Indicates the adjustment parameters of video quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the audio quality at the current moment. Indicates the adjusted audio and video quality. Indicates the actual bandwidth at the current moment. Indicates The actual bandwidth at the moment;

[0172] When the user prefers video quality, the audio and video quality is adaptively adjusted by reducing the audio quality parameters. Specifically:

[0173]

[0174] The audio and video after adaptive adjustment are:

[0175]

[0176] in, Indicates the audio quality at the current moment. Indicates the adjusted audio quality. Indicates the adjustment parameters of audio quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the video quality at the current moment. Indicates the adjusted audio and video quality;

[0177] When the bandwidth fluctuation state is in a decreasing state, the video quality is improved according to the user preference, then,

[0178] Comprehensively adjust the audio and video quality based on the user's preference weight, specifically:

[0179]

[0180] in, Indicates the adjusted video quality. Indicates the adjusted audio quality. represents the video preference weight, represents the audio preference weight, Indicates the adjusted audio and video quality.

[0181] S3: Dynamically optimize synchronization and quality adjustment effects based on user evaluation.

[0182] Specifically, dynamically optimizing the synchronization and quality adjustment effects according to user evaluations is to formulate a user rating model, and dynamically optimizing the synchronization and quality adjustment effects according to the output results of the user rating model.

[0183] Furthermore, the user rating model collects user evaluations of the adjusted audio and video, and dynamically optimizes the synchronization and quality adjustment of the audio and video based on the user's evaluation scores, as follows:

[0184] Collect users' subjective feedback data in real time, including audio-visual synchronization feedback and picture quality satisfaction ratings, and standardize the users' subjective feedback data;

[0185] User ratings for Utilize Sync Effects , User ratings for video quality And objective observation results For comprehensive scoring, we have:

[0186] Rating of synchronization effect,

[0187]

[0188] in, Indicates Users' ratings of synchronization performance, Indicates the total number of users who participated in the rating. represents the objective observation of the synchronization effect, Indicates the user rating of the synchronization effect. Represent the weight coefficients of user ratings and objective observation results respectively;

[0189] Rating of video quality,

[0190]

[0191] in, Indicates Users' ratings of video quality, Indicates the total number of users who participated in the rating. Represents the objective observation result of video quality. Indicates the user rating of the video quality. Represent the weight coefficients of user ratings and objective observation results respectively;

[0192] Based on the comprehensive score of synchronization effect and video quality, there are:

[0193]

[0194] in, Indicates the user rating of the synchronization effect. Indicates the user rating of the video quality. Respectively represent the weight coefficients of synchronization effect and video quality.

[0195] Furthermore, the synchronization and quality adjustment effects are dynamically optimized according to the user rating results, as follows:

[0196] Set the scoring thresholds for synchronization and video quality as well as ;

[0197] Dynamic optimization is performed based on the comparison between user ratings and rating thresholds, then:

[0198] When the comparison result between the synchronization effect and the scoring threshold satisfies the formula , it means that the user is satisfied with the audio and video synchronization effect of the current video and maintains the current synchronization adjustment;

[0199] When the comparison result between the synchronization effect and the scoring threshold satisfies the formula , it means that the user is not satisfied with the audio and video synchronization effect of the current video, and the audio and video synchronization is adjusted again until the user is satisfied;

[0200] When the comparison result between the video quality and the scoring threshold satisfies the formula When , it means that the user is satisfied with the quality of the current video and the video quality is maintained for playback;

[0201] When the comparison result between the video quality and the scoring threshold satisfies the formula , it means that the user is not satisfied with the quality of the current video, and the audio and video quality of the video will be adjusted again until the user is satisfied.

[0202] It should be noted that the dynamic optimization of synchronization effect and video quality is further limited by comprehensive scoring, as follows:

[0203] For situations where users are dissatisfied with the synchronization effect or video quality, further qualifies based on the comprehensive score, then:

[0204] For the secondary adjustment of audio and video synchronization and audio and video quality, the comprehensive score must satisfy the formula , it means that the current audio and video synchronization and quality adjustment are completed, otherwise the audio and video synchronization and quality are readjusted until the formula is satisfied. The setting of the comprehensive scoring formula is limited to 0.8 in the present invention and is only used for illustration in this embodiment. The implementer can set it according to the actual application scenario.

[0205] Furthermore, if the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0206] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0207] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk case (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0208] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment may not be described (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those features that are not relevant to implementing the invention).

[0209] It should be understood that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but for those of ordinary skill having the benefit of this disclosure, the development effort will be a routine task of design, fabrication, and production without undue experimentation.

[0210] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A video playback method based on digital audio, characterized in that: The following steps are included: Calibrate the start point of audio and video playback and start synchronization monitoring, including, Calibrate the start play point of audio and video, and perform synchronization monitoring based on the calibration results, specifically: Use timestamps to calibrate the start play point of audio and video, use real-time play progress to analyze synchronization status, then use audio feature analysis algorithm to analyze audio status, and dynamically adjust the synchronization of audio and video based on the audio status analysis results; Adaptively adjust audio and video quality based on network conditions, including: Collect network data to predict bandwidth fluctuations in the future time period, and adaptively adjust the audio and video quality based on the prediction results, specifically: Use network acquisition equipment to collect network status data in real time, and use time series models to predict bandwidth. Adaptively adjust audio and video quality based on bandwidth prediction results, and adjust audio and video based on user preferences. Dynamically optimize synchronization and quality adjustment effects based on user reviews. include, A user rating model is constructed to rate the synchronization effect and quality adjustment effect, and the synchronization and quality adjustment effects are dynamically optimized based on the rating results.

2. The video playback method based on digital audio according to claim 1, characterized in that: The calibration of the start play point of the audio and video using the timestamp is specifically as follows: Use the server to obtain the initial timestamp of the audio stream and the initial timestamp of the video stream , and set the local time deviation ; Calculate the time of audio playback and video playback at the video playback location, then we have: Audio playback time, ; Video playback time, in, Indicates the initial timestamp of the audio stream. Indicates the initial timestamp of the video stream. Indicates the local time deviation, Indicates the audio playback timestamp. Indicates the video playback timestamp; Using the calculated playback timestamp for calibration, we have, in, Indicates the audio playback timestamp. Indicates the video playback timestamp. Indicates the time difference between audio and video playback. By adjusting the time difference, the start point of audio and video playback can be calibrated. Then, If the calculated time difference between audio playback and video playback satisfies the formula , indicating that the calibration of the starting point of audio and video playback is completed. Otherwise, the timestamps of the audio and video sent by the server are adjusted until the calibration conditions are met.

3. The video playback method based on digital audio as claimed in claim 2, characterized in that: The synchronization status analysis using real-time playback progress is specifically as follows: in, Indicates the real-time playback progress of the audio. Indicates the real-time playback progress of the video. Indicates the difference between the audio playback progress and the video playback progress, which is used to determine whether the audio and video synchronization needs to be dynamically adjusted. If yes, If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it indicates that the current audio playback is synchronized with the video playback; If the calculated difference between the audio playback progress and the video playback progress satisfies the formula , it means that the current audio playback is not synchronized with the video playback.

4. The video playback method based on digital audio as claimed in claim 3, characterized in that: The dynamic adjustment of audio and video synchronization according to the audio state analysis result is specifically as follows: When the audio and video playback progress difference satisfies the formula When the audio state analysis result is a silent state, a delay operation is performed on the current audio to achieve synchronization of audio and video; When the audio and video playback progress difference satisfies the formula When the audio state analysis result is normal, the playback speed of the current audio is adjusted to achieve synchronization of audio and video; When the audio and video playback progress difference satisfies the formula When the audio state analysis result is a noise state, the energy of the current audio is adjusted until the current video state is a normal state, and then the playback speed of the current audio is adjusted to achieve synchronization of audio and video; The insertion delay operation is to insert a silent delay into the audio in a muted state to adjust the audio playback timestamp, which is specifically implemented as follows: in, Indicates the inserted silence delay time. Indicates the maximum allowed silent delay time. Indicates the current playback rate of the audio. Indicates the adjusted audio playback progress. The adjustment of the playback speed is for audio in a normal state, and the audio playback timestamp is changed by adjusting the audio playback rate, as follows: in, Indicates the current playback rate of the audio. represents the rate adjustment factor, Indicates the time difference of audio and video playback. Indicates the current timestamp of the audio. Indicates the adjusted audio playback progress.

5. The video playback method based on digital audio as claimed in claim 4, characterized in that: The bandwidth prediction by the time series model is specifically as follows: Construct a time series model to predict bandwidth, then we have: in, Indicates prediction The network bandwidth at the time, Represents the autoregressive coefficient, which is the weight of the impact of historical bandwidth data on bandwidth prediction. Represents the initial autoregressive coefficient, which is set by the implementer according to the actual application scenario. represents the order of the autoregressive term, represents the moving average coefficient, which is the weight of the error term on the bandwidth prediction. represents the order of the moving average, represents the error term; To predict the bandwidth change rate, we have, in, Indicates that bandwidth will The rate of change over time, Indicates prediction The network bandwidth at the time, Indicates the network bandwidth data after smoothing; Using historical data to determine bandwidth fluctuations, we have: The historical network bandwidth data of the day before the prediction time node is used as the input of the time series model, and the output of the model is used as the bandwidth fluctuation state judgment threshold , the specific judgment is as follows: When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula When , it indicates that the current bandwidth fluctuation state is rising; When the predicted bandwidth change rate is compared with the judgment threshold, the result satisfies the formula , it indicates that the current bandwidth fluctuation status is decreasing.

6. The video playback method based on digital audio as claimed in claim 5, characterized in that: The audio and video adjustment according to user preferences is specifically as follows: When the bandwidth fluctuation state is in an increasing state, the audio and video quality is reduced according to the user's preference. When the user prefers audio quality, the adaptive adjustment of audio and video quality is achieved by reducing the video quality parameters, specifically: ； The audio and video after adaptive adjustment are: in, Indicates the video quality at the current moment. Indicates the adjusted video quality. Indicates the adjustment parameters of video quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the audio quality at the current moment. Indicates the adjusted audio and video quality. Indicates the actual bandwidth at the current moment. Indicates The actual bandwidth at the moment; When the user prefers video quality, the audio and video quality is adaptively adjusted by reducing the audio quality parameters. Specifically: ； The audio and video after adaptive adjustment are: in, Indicates the audio quality at the current moment. Indicates the adjusted audio quality. Indicates the adjustment parameters of audio quality. Indicates the bandwidth change ratio, represents the video preference weight, represents the audio preference weight, Indicates the video quality at the current moment. Indicates the adjusted audio and video quality; When the bandwidth fluctuation state is in a decreasing state, the video quality is improved according to the user preference, then, Comprehensively adjust the audio and video quality based on the user's preference weight, specifically: in, Indicates the adjusted video quality. Indicates the adjusted audio quality. represents the video preference weight, represents the audio preference weight, Indicates the adjusted audio and video quality.

7. The video playback method based on digital audio as claimed in claim 6, characterized in that: The user scoring model is constructed to score the synchronization effect and the quality adjustment effect as follows: User ratings for Utilize Sync Effects , User ratings for video quality And objective observation results For comprehensive scoring, we have: Rating of synchronization effect, in, Indicates Users' ratings of synchronization performance, Indicates the total number of users who participated in the rating. represents the objective observation of the synchronization effect, Indicates the user rating of the synchronization effect. Represent the weight coefficients of user ratings and objective observation results respectively; Rating of video quality, in, Indicates Users' ratings of video quality, Indicates the total number of users who participated in the rating. Represents the objective observation result of video quality. Indicates the user rating of the video quality. Represent the weight coefficients of user ratings and objective observation results respectively; Based on the comprehensive score of synchronization effect and video quality, there are: in, Indicates the user rating of the synchronization effect. Indicates the user rating of the video quality. Respectively represent the weight coefficients of synchronization effect and video quality.

8. The video playback method based on digital audio as claimed in claim 7, characterized in that: The dynamic optimization of synchronization and quality adjustment effects according to the scoring results is specifically as follows: Set the scoring thresholds for synchronization and video quality as well as ; Dynamic optimization is performed based on the comparison between user ratings and rating thresholds, then: When the comparison result between the synchronization effect and the scoring threshold satisfies the formula When the current period is maintained, the synchronous adjustment is maintained; When the comparison result between the synchronization effect and the scoring threshold satisfies the formula When the audio and video synchronization is adjusted again; When the comparison result between the video quality and the scoring threshold satisfies the formula When playing, maintain the current video quality; When the comparison result between the video quality and the scoring threshold satisfies the formula Re-adjust the audio and video quality of the video.

Images