Audio frame insertion method, device, equipment, medium and program product

Abstract

The invention discloses an audio frame insertion method and device, equipment, a medium and a program product. The method comprises the steps of obtaining audio data; wherein the audio data comprises a plurality of audio frames; recognizing lost audio frame information based on the multiple audio frames through a preset audio frame recognition model; wherein the lost audio frame information comprises the position and the number of audio frames; predicting a to-be-inserted audio frame based on the multi-frame audio frame and the lost audio frame information through a preset prediction model; and performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information. By adopting the method provided by the embodiment of the invention, the frame insertion processing speed and precision can be improved, and the problem of sound and picture desynchrony during video playing is effectively solved.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia technology, and in particular to an audio frame interpolation method, device, equipment, medium and program product. Background Art

[0002] In the related technology, audio frame loss during multimedia transmission and storage is a common phenomenon. Especially when the network transmission is unstable, audio frame loss will directly lead to the asynchrony of audio and video during video playback, which greatly affects the user experience. Frame insertion is the current mainstream repair method for solving audio frame loss. However, the current frame insertion method is usually a simple delay adjustment and linear interpolation scheme, which is difficult to ensure the precise synchronization of audio and video, especially in the face of continuous frame loss or random loss of multiple frames. As a result, the generated audio will have noise, lack of smooth transition, and it will be difficult to accurately synchronize audio and video, which will affect the user experience. Summary of the invention

[0003] The present disclosure discloses an audio frame insertion method, device, equipment, medium and program product. The technical solution of the present disclosure is as follows:

[0004] In a first aspect, the present disclosure discloses an audio frame interpolation method, comprising:

[0005] Acquire audio data; wherein the audio data includes multiple audio frames;

[0006] Identifying lost audio frame information based on the multiple audio frames through a preset audio frame recognition model; wherein the lost audio frame information includes the position and number of audio frames;

[0007] Predicting the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model;

[0008] Perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

[0009] In a possible implementation manner, after acquiring the audio data, the method further includes:

[0010] Performing data preprocessing on the audio data to obtain audio data after data preprocessing; wherein the data preprocessing includes noise reduction and sampling rate conversion;

[0011] The step of identifying lost audio frame information based on the multiple audio frames by using a preset audio frame includes:

[0012] The lost audio frame information is identified by presetting the audio frame to identify the audio data preprocessed based on the data.

[0013] In a possible implementation, the preset audio frame is identified as an autoregressive difference moving average model.

[0014] In a possible implementation, the preset prediction model is implemented based on a long short-term memory network; the audio frame to be inserted is feature information of the predicted audio frame;

[0015] The performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information includes:

[0016] Converting the feature information of the predicted audio frame back into an audio signal;

[0017] Frame interpolation processing is performed according to the lost audio frame information and the audio signal.

[0018] In a possible implementation, the performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information includes:

[0019] When the number of lost audio frames is less than a preset number and the change of audio frames before and after the lost audio frame is less than a preset change, inserting the audio frame to be inserted by using a still frame replication method;

[0020] When the number of lost audio frames is less than the preset number and the change amount of audio frames before and after the lost audio frame is greater than or equal to the preset change amount, interpolating the audio frame to be inserted by using a linear interpolation method;

[0021] When the number of lost audio frames is greater than or equal to the preset number, and the change in the audio frames before and after the lost audio frames is greater than or equal to the preset change, the motion compensated interpolation method is used to interpolate the audio frames to be inserted.

[0022] In a possible implementation, it further includes:

[0023] Monitor the audio and video synchronization status after frame insertion;

[0024] The audio frame to be inserted is adjusted according to the audio and video synchronization state.

[0025] In a second aspect, the present disclosure discloses an audio frame interpolation device, comprising:

[0026] An acquisition module, used to acquire audio data; wherein the audio data includes multiple audio frames;

[0027] An identification module, configured to identify lost audio frame information based on the multiple audio frames by using a preset audio frame identification model; wherein the lost audio frame information includes the position and number of audio frames;

[0028] A prediction module, configured to predict the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model;

[0029] The frame insertion module is used to perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

[0030] In a third aspect, the present disclosure discloses an electronic device, including:

[0031] processor;

[0032] a memory for storing instructions executable by the processor;

[0033] The processor is configured to execute the instructions to implement the method described in the first aspect.

[0034] In a fourth aspect, the present disclosure discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0035] In a fifth aspect, the present disclosure discloses a computer program product, including a computer program / instructions, characterized in that when the computer program / instructions are executed by a processor, the method described in the first aspect is implemented.

[0036] The technical solution disclosed in this disclosure brings at least the following beneficial effects:

[0037] In the technical solution disclosed in the present disclosure, audio data is obtained; wherein the audio data includes multiple audio frames; the lost audio frame information is identified based on the multiple audio frames through a preset audio frame recognition model; wherein the lost audio frame information includes the position and number of audio frames; the audio frames to be inserted are predicted based on the multiple audio frames and the lost audio frame information through a preset prediction model; and the audio frames to be inserted are inserted according to the lost audio frame information. In this way, on the one hand, a deep learning network model can be combined to realize an efficient process from detection to prediction, significantly improving the processing speed and accuracy. On the other hand, based on the automatic detection of audio frame loss and accurate prediction of lost frames, the lost frames can be automatically and seamlessly inserted into the original audio stream, thereby effectively solving the problem of audio and video asynchrony during video playback.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0040] Figure 1 A flowchart of an audio frame interpolation method provided by an embodiment of the present disclosure;

[0041] Figure 2 is a flowchart of an audio frame interpolation system provided by an embodiment of the present disclosure;

[0042] Figure 3 is a structural schematic diagram of an audio frame interpolation device provided by an embodiment of the present disclosure;

[0043] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0046] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0047] The acquisition, storage, use, and processing of data in the technical solution disclosed in this disclosure are in compliance with the relevant provisions of national laws and regulations.

[0048] It should be noted that in the embodiments of the present disclosure, there may be existing solutions in the industry such as certain software, components, models, etc., which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

[0049] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0050] Figure 1 The present invention provides a flowchart of an audio frame interpolation method, which can be applied to a server, such as a single server or a server cluster. Figure 1 As shown, the audio frame interpolation method may include the following steps:

[0051] S101, obtaining audio data.

[0052] The audio data includes multiple audio frames.

[0053] In the embodiments of the present disclosure, audio data may be acquired first. Exemplarily, the audio data may include multiple audio frames, that is, acquiring audio data may be acquiring multiple audio frames. For example, a recording data stream may be collected in the form of audio frames as raw data. As an example, the recording data stream may be acquired through a microphone or an audio or video file.

[0054] S102: Identify lost audio frame information based on multiple audio frames using a preset audio frame recognition model.

[0055] The lost audio frame information includes the position and number of audio frames.

[0056] In an embodiment of the present disclosure, after acquiring multiple audio frames, the multiple audio frames can be input into a preset audio frame recognition model, and the multiple audio frames can be analyzed and processed by the preset audio frame recognition model to identify the lost audio frame information, such as identifying the location and number of lost audio frames. As an example, the preset audio frame recognition model can extract useful features such as energy, zero-crossing rate, etc. from the audio signal (audio data, i.e., the multiple audio frames acquired as described above), and perform real-time monitoring based on the preset audio frame recognition model to identify any anomalies that deviate from normal behavior. As a specific example, the preset audio frame recognition model can predict the behavior pattern of normal audio frames through time domain analysis, identify audio frames that deviate from the expected pattern based on prediction errors, and accurately identify the loss of audio frames, including the location and number of lost audio frames.

[0057] In some possible implementations, the preset audio frame recognition may be an autoregressive differential moving average model, ARIMA model, which is also known as the Autoregressive Integrated Moving Average Model. Through time domain analysis, a statistical model is used to establish a time series model ARIMA (autoregressive integrated moving average) to predict the behavior pattern of normal audio frames. Frames that deviate from the expected pattern can be identified based on the prediction error, and the loss of audio frames can be accurately identified, including the location and number of lost frames.

[0058] S103: predicting the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model.

[0059] In an embodiment of the present disclosure, after identifying the lost audio frame information, multiple audio frames and the lost audio frame information can be input into a preset prediction model, and the preset prediction model is used to perform prediction analysis on the multiple audio frames and the lost audio frame information to predict and obtain new audio frames, that is, audio frames to be inserted. As an example, the preset prediction model can be implemented based on LSTM (Long Short-Term Memory). As a specific example, the training process of the preset prediction model can be as follows:

[0060] 1. Dataset construction and standardization, including standardization and normalization of audio frame features for easy training. This process can, for example, use the time series model generated by the intelligent monitoring module to create an input sequence containing the previous N frames and a target output sequence for the next frame for each audio frame. Then, a long short-term memory model can be constructed. The long short-term memory network LS2.TM has the ability to process time series data. By using historical data as an input sequence, LSTM can predict the missing audio frames. When training the model, you can first build the model, determine the shape of the input sequence, such as (batch_size, sequence_length, feature_dim), and select the appropriate output layer according to the type of features to be predicted. Then, train the model, select a suitable loss function, such as mean square error, select an optimizer, divide the data into batches for training, and regularly verify the model performance. After that, predict the new audio frame, use the trained model (that is, the preset prediction model) to predict the new audio frame (that is, the audio frame to be inserted), and convert the predicted features back to the original audio signal (using overlap-addition and other methods to merge the predicted audio frames into a continuous audio stream).

[0061] S104: Perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

[0062] In an embodiment of the present disclosure, after the audio frame to be inserted is predicted, frame insertion can be performed. Exemplarily, each audio frame to be inserted can be inserted into a corresponding position according to the lost audio frame information, such as the position of the audio frame, and the audio information of the audio frame to be inserted, such as the position of the audio frame, to obtain complete audio data. In this way, the missing audio frame can be filled.

[0063] In an embodiment of the present disclosure, audio data is acquired; wherein the audio data includes multiple audio frames; lost audio frame information is identified based on the multiple audio frames through a preset audio frame recognition model; wherein the lost audio frame information includes the position and number of audio frames; the audio frames to be inserted are predicted based on the multiple audio frames and the lost audio frame information through a preset prediction model; and the audio frames to be inserted are inserted according to the lost audio frame information. In this way, on the one hand, a deep learning network model can be combined to realize an efficient process from detection to prediction, significantly improving processing speed and accuracy. On the other hand, based on the automatic detection of audio frame loss and accurate prediction of lost frames, the lost frames can be automatically and seamlessly inserted into the original audio stream, thereby effectively solving the problem of audio and video asynchrony during video playback.

[0064] In some possible implementations, after acquiring the audio data, the method further includes:

[0065] Performing data preprocessing on the audio data to obtain audio data after data preprocessing; wherein the data preprocessing includes noise reduction and sampling rate conversion;

[0066] Accordingly, the lost audio frame information is identified based on multiple audio frames through the preset audio frame identification, including:

[0067] The lost audio frame information is identified by presetting the audio frame recognition based on the audio data after data preprocessing.

[0068] In an embodiment of the present disclosure, in order to improve the quality of the voice signal, the audio data may be preprocessed. Exemplarily, after the audio data is acquired, the audio data may be preprocessed, for example, noise reduction processing, sampling rate conversion processing, noise reduction, for example, may use filtering techniques, such as Gaussian filtering or adaptive filtering, to remove background noise in the recording and improve the quality of the voice signal; sampling rate conversion, for example, may be achieved by adopting an audio resampling strategy. Then, the audio frame recognition may be preset to identify the lost audio frame information based on the audio data after data preprocessing. For example, useful features such as energy, zero crossing rate, etc. are extracted from the audio signal (audio frame), and real-time monitoring is performed based on the model to identify any anomalies that deviate from normal behavior.

[0069] In some possible implementations, the preset prediction model is implemented based on a long short-term memory network; the audio frame to be inserted is feature information of the predicted audio frame;

[0070] Accordingly, according to the lost audio frame information, the audio frame to be inserted is subjected to frame insertion processing, including:

[0071] Converting the feature information of the predicted audio frame back into an audio signal;

[0072] Perform frame interpolation processing based on the lost audio frame information and the audio signal.

[0073] In the embodiments of the present disclosure, considering that the audio frames to be inserted predicted by the preset prediction model are usually in the form of feature information, when performing frame insertion processing on the audio frames to be inserted based on the lost audio frame information, the feature information of the predicted audio frames can be first converted back to audio signals, for example, by using overlap-addition and other methods to merge the predicted audio frames into a continuous audio stream, and then performing frame insertion processing based on the lost audio frame information and the audio signal.

[0074] In a further possible implementation, performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information includes:

[0075] When the number of lost audio frames is less than a preset number and the change amount of audio frames before and after the lost audio frame is less than a preset change amount, a still frame replication method is used to perform frame insertion processing on the audio frame to be inserted;

[0076] When the number of lost audio frames is less than a preset number and the change amount of the audio frames before and after the lost audio frame is greater than or equal to the preset change amount, a linear interpolation method is used to perform interpolation processing on the audio frame to be inserted;

[0077] When the number of lost audio frames is greater than or equal to a preset number, and the change in audio frames before and after the lost audio frames is greater than or equal to a preset change, a motion compensation interpolation method is used to perform interpolation processing on the audio frames to be inserted.

[0078] In an embodiment of the present disclosure, different interpolation methods can be used according to different situations of lost audio frames. Exemplarily, if the number of lost audio frames is less than a preset number, and the change in the audio frames before and after the lost audio frame is less than the preset change, that is, if only a small number of frames are lost and the change in the previous and next frames is not large, then the still frame replication method can be used to interpolate the inserted audio frame. If the number of lost audio frames is less than a preset number, and the change in the audio frames before and after the lost audio frame is greater than or equal to the preset change, that is, a small number of frames are lost and there is a large change in the previous and next frames, then the linear interpolation method can be used to interpolate the inserted audio frame. If the number of lost audio frames is greater than or equal to the preset number, and the change in the audio frames before and after the lost audio frame is greater than or equal to the preset change, that is, multiple frames are lost or there is complex motion in the previous and next frames, then the motion compensation interpolation method can be used to interpolate the inserted audio frame. As an example, the short-time Fourier transform (STFT) can be used to obtain the spectrum information, the spectrum envelope can be obtained by smoothing the amplitude value of the spectrum, the spectrum information can be modified as needed, and finally the short-time inverse Fourier transform (ISTFT) can be used to reconstruct the audio signal from the spectrum information. In this way, a smoother and clearer audio frame can be synthesized by processing the source audio stream.

[0079] In some possible implementations, the following further includes:

[0080] Monitor the audio and video synchronization status after frame insertion;

[0081] Adjust the audio frame to be inserted according to the audio and video synchronization status.

[0082] In the embodiments of the present disclosure, a feedback mechanism can also be introduced to continuously monitor the audio and video synchronization state, and fine-tune the interpolation position and parameters according to the actual performance during playback to further optimize the audio and video synchronization effect. Exemplarily, the audio and video synchronization state after interpolation processing can be monitored. For example, the video can be started to play, and the synchronization of audio and video can be monitored in real time. For example, a feature matching algorithm can be used to monitor the synchronization error between the speaker's voice and lip shape. Then, according to the audio and video synchronization state, the audio frame to be inserted is adjusted. For example, if the synchronization error is found to exceed the threshold (for example, it can be set to ±50 milliseconds, or set to other values ​​as needed), the cause of the error is analyzed; if it is caused by insufficient video frame rate, the interpolation frequency is increased. If it is caused by some specific scenes, the interpolation position and algorithm parameters are adjusted for the scene. Continue to play after adjusting the parameters and monitor the synchronization state. If necessary, repeat the adjustment process until the requirements are met.

[0083] To make the audio frame interpolation method provided by the embodiment of the present disclosure clearer, an example is used below to illustrate. The audio frame interpolation method provided by the embodiment can be executed by an audio frame interpolation system, see Figure 2 The audio frame interpolation system may include an intelligent detection module, a deep learning prediction module, an adaptive interpolation module, and a synchronization optimization and feedback control module. Among them:

[0084] Smart detection module for performing:

[0085] 1. Audio Data Collection

[0086] Function description: Collect recorded data streams in the form of audio frames as raw data.

[0087] Technical implementation: Use microphone or audio and video files.

[0088] 2. Preprocessing

[0089] Function description: Preprocess the collected audio data, such as noise reduction, sampling rate conversion, etc.

[0090] Technical implementation: Noise reduction uses filtering techniques, such as Gaussian filtering or adaptive filtering, to remove background noise from recordings and improve the quality of speech signals. Sampling rate conversion can be achieved by adopting an audio resampling strategy.

[0091] 3. Monitoring

[0092] Functional description: Extract useful features from audio signals, such as energy, zero-crossing rate, etc., perform real-time monitoring based on the model, and identify any anomalies that deviate from normal behavior.

[0093] Technical implementation: Through time domain analysis, a statistical model is used to establish the time series model ARIMA (autoregressive integrated moving average) to predict the behavior pattern of normal audio frames. Frames that deviate from the expected pattern can be identified based on the prediction error, and the loss of audio frames can be accurately identified, including the location and number of lost frames.

[0094] Deep learning prediction module for performing:

[0095] 1. Dataset construction and standardization

[0096] Function description: Standardize and normalize audio frame features for easy training.

[0097] Technical implementation: Using the time series model generated by the intelligent monitoring module, for each audio frame, an input sequence containing the previous N frames and a target output sequence for the next frame is created.

[0098] 2. Building a long short-term memory model

[0099] Function description: The long short-term memory network LS2.TM has the ability to process time series data. By taking historical data as the input sequence, LSTM can predict the missing audio frames.

[0100] Technical implementation:

[0101] (1) Model building

[0102] Determine the shape of the input sequence, such as (batch_size, sequence_length, feature_dim), and select the appropriate output layer based on the type of features that need to be predicted.

[0103] (2) Training Model

[0104] Choose an appropriate loss function such as mean squared error, select an optimizer, split the data into batches for training, and validate the model performance regularly.

[0105] (3) Predicting new audio frames

[0106] Use the trained model to predict new audio frames and convert the predicted features back into the original audio signal (using methods such as overlap-add to merge the predicted audio frames into a continuous audio stream).

[0107] Adaptive interpolation module, used to perform:

[0108] 1. Generate high-quality audio frames

[0109] Function description: Process the source audio stream and synthesize smoother and clearer audio frames.

[0110] Technical implementation: Use short-time Fourier transform (STFT) to obtain spectrum information. Obtain spectrum envelope by smoothing the amplitude value of the spectrum, modify spectrum information as needed, and finally use short-time inverse Fourier transform (ISTFT) to reconstruct audio signal from spectrum information.

[0111] 2. Adaptive interpolation

[0112] Function description: According to the specific situation of the lost frame (such as the degree of loss, the correlation between the previous and next frames, etc.), dynamically decide which method to use to fill the lost frame.

[0113] Technical implementation: If only a small number of frames are lost and the previous and next frames do not change much, you can choose to copy the still frame.

[0114] If a small number of frames are missing and there is a large change between the previous and next frames, you can try linear interpolation.

[0115] If multiple frames are lost or there is complex motion between previous and next frames, motion compensated interpolation is used.

[0116] Synchronous optimization feedback control module for performing:

[0117] Function description: Introduce a feedback mechanism to continuously monitor the audio and video synchronization status, and fine-tune the insertion position and parameters according to the actual performance during playback to further optimize the audio and video synchronization effect.

[0118] Technical implementation:

[0119] (1) Initialization: Set the synchronization error threshold (e.g. ±50 milliseconds).

[0120] (2) Play and monitor: Start playing the video and monitor the synchronization between the audio and video in real time. Use a feature matching algorithm to monitor the synchronization error between the speaker's voice and lip shape.

[0121] (3) Analysis and adjustment: If the synchronization error is found to exceed the threshold, the cause of the error is analyzed. If it is caused by insufficient video frame rate, the frequency of interpolation is increased. If it is caused by certain specific scenes, the interpolation position and algorithm parameters are adjusted for the scene.

[0122] (4) Repeat loop: Continue playing after adjusting the parameters and monitor the synchronization status. If necessary, repeat the adjustment process until the requirements are met.

[0123] It can be seen that, on the one hand, the audio frame interpolation method of the embodiment of the present disclosure can realize the integration of intelligent detection and prediction, and combined with deep learning, it can realize an efficient process from detection to prediction, significantly improving the processing speed and accuracy. On the other hand, the audio frame interpolation method of the embodiment of the present disclosure can implement an adaptive interpolation strategy, and can flexibly adjust the interpolation algorithm according to the loss situation in different scenarios to ensure the sound quality while minimizing the interference to the original audio stream. At the same time, the audio frame interpolation method of the embodiment of the present disclosure can dynamically synchronize optimization and feedback control to continuously adjust the interpolation strategy to ensure that the synchronization of audio and video is stable and reliable throughout the playback cycle.

[0124] The specific implementation and technical effects of each step of this embodiment are similar to those of the above method embodiments, and will not be repeated here.

[0125] Based on the same inventive concept, the embodiment of the present disclosure also provides an audio frame interpolation device. Figure 3 As shown, the audio frame interpolation device 300 includes:

[0126] The acquisition module 310 is used to acquire audio data; wherein the audio data includes multiple audio frames;

[0127] An identification module 320, configured to identify lost audio frame information based on the multiple audio frames by using a preset audio frame identification model; wherein the lost audio frame information includes the position and number of audio frames;

[0128] A prediction module 330, configured to predict the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model;

[0129] The frame insertion module 340 is used to perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

[0130] In a possible implementation manner, the audio frame interpolation device 300 further includes:

[0131] A preprocessing module, used to perform data preprocessing on the audio data to obtain audio data after data preprocessing; wherein the data preprocessing includes noise reduction and sampling rate conversion;

[0132] The identification module 320 is used to:

[0133] The lost audio frame information is identified by presetting the audio frame to identify the audio data preprocessed based on the data.

[0134] In a possible implementation, the preset audio frame is identified as an autoregressive difference moving average model.

[0135] In a possible implementation, the preset prediction model is implemented based on a long short-term memory network; the audio frame to be inserted is feature information of the predicted audio frame;

[0136] The frame insertion module 340 is used to:

[0137] Converting the feature information of the predicted audio frame back into an audio signal;

[0138] Frame interpolation processing is performed according to the lost audio frame information and the audio signal.

[0139] In a possible implementation manner, the frame insertion module 340 is further configured to:

[0140] When the number of lost audio frames is less than a preset number and the change of audio frames before and after the lost audio frame is less than a preset change, inserting the audio frame to be inserted by using a still frame replication method;

[0141] When the number of lost audio frames is less than the preset number and the change amount of audio frames before and after the lost audio frame is greater than or equal to the preset change amount, interpolating the audio frame to be inserted by using a linear interpolation method;

[0142] When the number of lost audio frames is greater than or equal to the preset number, and the change in the audio frames before and after the lost audio frames is greater than or equal to the preset change, the motion compensated interpolation method is used to interpolate the audio frames to be inserted.

[0143] In a possible implementation manner, the audio frame interpolation device 300 further includes:

[0144] Monitoring module, used to monitor the audio and video synchronization status after frame insertion processing;

[0145] The adjustment module is used to adjust the audio frame to be inserted according to the audio and video synchronization state.

[0146] The specific implementation method and technical effect of the device provided in the embodiment of the present disclosure are similar to those of the above-mentioned method embodiment, and will not be repeated here.

[0147] According to an embodiment of the present disclosure, the present disclosure also discloses an electronic device, a computer-readable storage medium, and a computer program product.

[0148] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0150] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the audio frame interpolation method. For example, in some embodiments, the audio frame interpolation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the audio frame interpolation method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the audio frame interpolation method in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code of the computer program product for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a computer-readable storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may be a machine-readable signal medium or a machine-readable storage medium. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An audio frame interpolation method, characterized in that: include: Acquire audio data; wherein the audio data includes multiple audio frames; Identifying lost audio frame information based on the multiple audio frames through a preset audio frame recognition model; wherein the lost audio frame information includes the position and number of audio frames; Predicting the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model; Perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

2. The audio frame interpolation method according to claim 1, characterized in that: After obtaining the audio data, the method further includes: Performing data preprocessing on the audio data to obtain audio data after data preprocessing; wherein the data preprocessing includes noise reduction and sampling rate conversion; The step of identifying lost audio frame information based on the multiple audio frames by using a preset audio frame includes: The lost audio frame information is identified by presetting the audio frame to identify the audio data preprocessed based on the data.

3. The audio frame interpolation method according to claim 1, characterized in that: The preset audio frame recognition is an autoregressive difference moving average model.

4. The audio frame interpolation method according to claim 1, characterized in that: The preset prediction model is implemented based on a long short-term memory network; the audio frame to be inserted is feature information of the predicted audio frame; The performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information includes: Converting the feature information of the predicted audio frame back into an audio signal; Frame interpolation processing is performed according to the lost audio frame information and the audio signal.

5. The audio frame interpolation method according to claim 4, characterized in that: The performing frame insertion processing on the audio frame to be inserted according to the lost audio frame information includes: When the number of lost audio frames is less than a preset number and the change of audio frames before and after the lost audio frame is less than a preset change, inserting the audio frame to be inserted by using a still frame replication method; When the number of lost audio frames is less than the preset number and the change amount of audio frames before and after the lost audio frame is greater than or equal to the preset change amount, interpolating the audio frame to be inserted by using a linear interpolation method; When the number of lost audio frames is greater than or equal to the preset number, and the change in the audio frames before and after the lost audio frames is greater than or equal to the preset change, the motion compensated interpolation method is used to interpolate the audio frames to be inserted.

6. The audio frame interpolation method according to any one of claims 1 to 5, characterized in that: Also includes: Monitor the audio and video synchronization status after frame insertion; The audio frame to be inserted is adjusted according to the audio and video synchronization state.

7. An audio frame interpolation device, characterized in that: include: An acquisition module, used to acquire audio data; wherein the audio data includes multiple audio frames; An identification module, configured to identify lost audio frame information based on the multiple audio frames by using a preset audio frame identification model; wherein the lost audio frame information includes the position and number of audio frames; A prediction module, configured to predict the audio frame to be inserted based on the multiple audio frames and the lost audio frame information by using a preset prediction model; The frame insertion module is used to perform frame insertion processing on the audio frame to be inserted according to the lost audio frame information.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio frame interpolation method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio frame interpolation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the audio frame interpolation method according to any one of claims 1 to 6 is implemented.

Images