Method, device and server for real-time AV1 transcoding of audio and video based on network speed

Abstract

The present invention provides a method, device and server for real-time AV1 transcoding of audio and video based on network speed, which relates to the technical field of AV1 video encoding and decoding, including: obtaining audio and video data, and performing integrity check processing on the audio and video data through a preset cyclic redundancy check model; if the integrity check of the audio and video data is qualified, determining the real-time network speed by monitoring the size of the transmission data packet, and determining the network status monitoring result based on the real-time network speed; when the mobile network is in a network bandwidth-limited state, performing key information extraction processing on the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, and determining the data key frames and data non-key frames of the audio and video data; sending the real-time network speed to a preset dynamic transcoding model to determine the target compression rate of AV1 encoding, so as to preferentially encode and store the key information of the audio and video data. The present invention can significantly improve the quality and stability of audio and video transmission.

Description

Technical Field

[0001] The present invention relates to the technical field of AV1 video encoding and decoding, and in particular to a method, device, and server for real-time AV1 transcoding of audio and video based on network speed. Background Art

[0002] At present, relevant technologies have proposed that audio and video data need to be encoded and compressed before transmission during the encoding and transmission process, and the bit rate of the mobile network (that is, the compression ability of audio and video data) is positively correlated with the network speed. When the network bandwidth is tight at low speed, the bit rate is low, and the compressed audio and video files are large, and the network transmission capacity is poor at low speed. When the network bandwidth is abundant at high speed, the bit rate is high, and the compressed audio and video files are small. Therefore, if the bit rate is adjusted only according to the network speed, it will cause data transmission delays and reception end jams when the network bandwidth is tight. When the network bandwidth is abundant, broadband resources will be wasted, which will affect the quality and stability of audio and video transmission. In addition, this solution is difficult to respond to changes quickly in a complex and changeable network environment, and has poor adaptability and flexibility. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, device and server for real-time AV1 transcoding of audio and video based on network speed, which can significantly improve the quality and stability of audio and video transmission.

[0004] In a first aspect, an embodiment of the present invention provides a real-time AV1 transcoding method for audio and video based on network speed, the method comprising: obtaining audio and video data, and performing integrity check processing on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; if the integrity check of the audio and video data passes, determining the real-time network speed by monitoring the number of bytes of packets received and sent by the mobile network and the size of the transmitted data packets within a preset time interval, and determining a network status monitoring result based on the real-time network speed, wherein the network status monitoring result includes: a sufficient network bandwidth state and a limited network bandwidth state; when the mobile network is in the limited network bandwidth state, extracting key information from the audio and video data using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate for AV1 encoding, and encoding and transmitting the data key frames and data non-key frames in sequence at the target compression rate to prioritize encoding and storing key information of the audio and video data.

[0005] In one embodiment, the step of performing integrity check processing on audio and video data through a preset cyclic redundancy check model includes: performing cyclic redundancy check on the audio and video data through a preset cyclic redundancy check model to determine a CRC check code; when the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

[0006] In one embodiment, after the step of determining whether the integrity check of the audio and video data is qualified, it includes: performing format parsing processing on the header information of the audio and video data to extract the format information of the audio and video data, wherein the format information includes: encoding format, resolution and frame rate; comparing the format information with the list of standard formats supported by the system, and when the comparison results are consistent, determining that the format compatibility check has passed.

[0007] In one embodiment, the step of determining the real-time network speed by monitoring the number of bytes of packets received and sent and the size of transmitted data packets of the mobile network within a preset time interval includes: dividing the difference in the number of bytes of packets received and sent in adjacent time intervals by the time difference between adjacent time intervals to determine the first real-time network speed, and determining the second real-time network speed through a data transmission test of a standard data packet; and adding the product of the first preset weight and the first real-time network speed, and the product of the second preset weight and the second real-time network speed, respectively, to determine the target real-time network speed.

[0008] In one embodiment, the step of determining the network status monitoring result based on the real-time network speed includes: when the real-time network speed is less than a preset network speed threshold, determining that the current network is in a network bandwidth limited state, and reducing the data transmission volume by increasing the file compression rate of audio and video data to ensure smooth transmission of audio and video data; when the real-time network speed is not less than the preset network speed threshold, determining that the current network is in a network bandwidth sufficient state, and reducing the file compression rate of audio and video data to retain complete audio and video details.

[0009] In one embodiment, a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model are used to extract key information from audio and video data, and the steps of determining data key frames of the audio and video data include: converting the audio stream in the audio and video data into a mono mode, and performing signal segmentation processing on the converted audio stream to determine the audio frames contained in the audio and video data; using the preset bidirectional long short-term memory model to perform criticality scoring on each audio frame, determining the audio score corresponding to each audio frame, and determining the frame whose audio score is higher than the preset audio score threshold as the audio key frame.

[0010] In one embodiment, the method also includes: performing format conversion processing on the video stream in the audio and video data, and constructing multiple video cubes using continuous multiple frames of images in the video stream, wherein the video cube contains image frames of multiple time dimensions; using a preset 3D convolutional neural network model to perform criticality scoring on each video cube and determine the image score corresponding to each image frame; statistically analyzing the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determining the video cube whose probability is greater than the preset probability threshold as a video key frame.

[0011] In one embodiment, the step of sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate for AV1 encoding includes: obtaining a preset compression rate calculation parameter and a base compression rate when the network speed is 0; and determining the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed, and the sum of the base compression rate.

[0012] In one embodiment, after the step of encoding and transmitting the data key frames and the data non-key frames in sequence at a target compression rate, the method includes: encoding the data key frames and the data non-key frames in sequence, sending them to the AV1 decoder through the fast transmission protocol for decoding processing, and determining the decoded audio and video data; after encapsulating the decoded audio and video data in the preset transmission format of the transmission channel, transmitting it to the external audio and video data receiving end.

[0013] In the second aspect, an embodiment of the present invention further provides a real-time AV1 transcoding device for audio and video based on network speed, the device comprising: a data verification module, which obtains audio and video data and performs integrity verification processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; a network speed monitoring module, which, if the integrity verification of the audio and video data is qualified, determines the real-time network speed by monitoring the number of bytes of the mobile network received and sent packets and the size of the transmitted data packets within a preset time interval, and determines the network status monitoring result based on the real-time network speed, wherein the network status monitoring result includes: sufficient network bandwidth and network The network bandwidth is limited in the state; the key information extraction module, when the mobile network is in the network bandwidth limited state, uses the preset bidirectional long short-term memory model and the preset 3D convolutional neural network model to extract the key information of the audio and video data, and determine the data key frames and data non-key frames of the audio and video data, where the data key frames include: audio key frames and video key frames; the transcoding transmission module sends the real-time network speed to the preset dynamic transcoding model, determines the target compression rate of AV1 encoding, and encodes and transmits the data key frames and data non-key frames in sequence at the target compression rate, so as to give priority to encoding and storing the key information of the audio and video data.

[0014] In a third aspect, an embodiment of the present invention further provides a server, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] Embodiments of the present invention provide a method, device, and server for real-time AV1 transcoding of audio and video based on network speed. After acquiring audio and video data, the method performs integrity checking on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission. If the integrity check of the audio and video data passes, the method determines the real-time network speed by monitoring the number of bytes of packets received and sent by the mobile network and the size of the transmitted data packets within a preset time interval. The network status monitoring result is then determined based on the real-time network speed. When the network status monitoring result indicates that the mobile network is in a bandwidth-limited state, key information is extracted from the audio and video data using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data. Finally, the real-time network speed is sent to a preset dynamic transcoding model to determine the target compression rate of the AV1 encoding. The data key frames and data non-key frames are then encoded and transmitted sequentially at the target compression rate, thereby prioritizing the encoding and storage of key information of the audio and video data. Embodiments of the present invention can significantly improve the quality and stability of audio and video transmission.

[0018] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a method for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a method for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a flow chart of an adaptive AV1 encoding method provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the structure of a device for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] Currently, most existing audio and video codecs, such as H.264 / AVC, HEVC / H.265, and VP9, ​​have compression rates insufficient to meet the current demands of codec system development. Furthermore, the compression rates of most audio and video codec systems are fixed and cannot be adaptively adjusted in real time based on the current network speed. This can lead to the loss of key image or audio information, especially when network speeds are low. Specifically, related technologies suggest that audio and video data must be encoded and compressed before transmission. The bitrate of a mobile network (i.e., its ability to compress audio and video data) is positively correlated with network speed. When bandwidth is limited at low speeds, the bitrate is low, resulting in larger compressed audio and video files. Furthermore, network transmission capacity is poor at low speeds. When bandwidth is abundant at high speeds, the bitrate is high, resulting in smaller compressed audio and video files. Therefore, adjusting the bitrate based solely on network speed can lead to data transmission delays and lag at the receiving end when network bandwidth is limited, while wasting bandwidth resources when network bandwidth is abundant, impacting the quality and stability of audio and video transmission. Furthermore, this solution struggles to respond quickly to changes in complex and changing network environments, resulting in poor adaptability and flexibility.

[0028] Based on this, the network speed-based real-time AV1 transcoding method, device and server for audio and video provided by the present invention can monitor the network status in real time, and dynamically adjust the encoding compression rate and output bit rate according to the actual situation of the current network bandwidth. When the network bandwidth is tight, the data transmission volume is reduced by lowering the bit rate, thereby alleviating the network pressure and ensuring the smoothness of audio and video transmission; when the network bandwidth is sufficient, the video quality is improved by increasing the bit rate, and the bandwidth resources are fully utilized. This adaptive adjustment mechanism not only improves the stability and quality of audio and video transmission, but also optimizes the user experience. In addition, it can be flexibly applied in different application scenarios to meet diverse needs, and has broad market prospects and application value.

[0029] See also Figure 1 The flowchart of a method for real-time AV1 transcoding of audio and video based on network speed is shown. The method mainly includes the following steps S102 to S108:

[0030] In step S102, the audio and video data is obtained, and an integrity check is performed on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission. In one embodiment, after receiving the audio and video data from an external source, an integrity check is performed to ensure that the data is not damaged or lost during transmission, and the format of the received audio and video stream is analyzed to verify whether it meets the requirements of the codec format supported by the system, especially focusing on the compatibility of AV1 encoding. The encoding parameters, frame rate, resolution and other information of the audio and video stream can be preliminarily analyzed to evaluate its basic quality level, providing a reference for subsequent encoding and decoding processing.

[0031] In step S104, if the integrity check of the audio and video data passes, the real-time network speed is determined by monitoring the number of bytes of packets sent and received and the size of the transmitted data packets of the mobile network within a preset time interval, and the network status monitoring result is determined based on the real-time network speed. The network status monitoring result includes: sufficient network bandwidth status and limited network bandwidth status. In one embodiment, key parameters such as bandwidth and latency of the network where the system is located can be monitored in real time to accurately assess the current network status. Based on the network status monitoring result, it is determined whether the compression rate of the audio and video stream needs to be adjusted. When the network bandwidth is sufficient, the compression rate can be appropriately reduced to retain more audio and video details; when the network bandwidth is limited, the compression rate can be increased to reduce the data transmission volume to ensure smooth playback. In addition, in the process of adjusting the compression rate, intelligent algorithms (BiLSTM, 3D CNN) can be used to ensure that key information in the audio and video (such as voice, face, etc.) is preferentially retained to avoid the loss of important information.

[0032] In step S106, when the mobile network is in a state of limited network bandwidth, key information extraction and processing are performed on the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine the data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames. In one embodiment, the AV1 codec can be selected and configured according to the characteristics of the audio and video streams and system requirements to prepare for efficient audio and video compression processing. Specifically, the advanced algorithm of the AV1 encoder is used to compress the audio and video streams with high quality, significantly reducing the size of the audio and video files while maintaining the video quality unchanged, thereby reducing storage requirements and transmission bandwidth.

[0033] In step S108, the real-time network speed is sent to a preset dynamic transcoding model to determine the target compression rate of AV1 encoding. The key frames and non-key frames of the data are encoded and transmitted in sequence at the target compression rate to prioritize encoding and storing key information of the audio and video data. In one embodiment, the compressed data stream can be transmitted via UDP and then AV1 decoded. After that, the audio and video stream processed by AV1 adaptive encoding and decoding is encapsulated and output to the target storage medium or transmission channel.

[0034] The above-mentioned network speed-based real-time AV1 transcoding method for audio and video provided in the embodiment of the present invention can significantly improve the quality and stability of audio and video transmission.

[0035] See also Figure 2 The figure shows a schematic diagram of a method for real-time AV1 transcoding of audio and video based on network speed. This embodiment of the present invention also provides an implementation method for real-time AV1 transcoding of audio and video based on network speed. For details, see (A) to (C) below:

[0036] (A) After acquiring the audio and video data, perform integrity check and format verification. For details on integrity check and format verification, see (a) to (b) below:

[0037] (a) Performing a cyclic redundancy check on the audio and video data using a preset cyclic redundancy check model to determine a CRC check code: When the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data has passed. In one embodiment, a cyclic redundancy check (CRC) calculates a check code by polynomial division, which can detect and correct errors in data transmission: CRC is used to perform an integrity check on the received audio and video data packets. The receiving end receives the audio and video data and a preset check code calculated in advance by long division, and after adding the preset check code to the audio and video data, the same CRC calculation is performed to determine the CRC check code, which is compared with the preset check code. If they are the same, the audio and video data may not be damaged. If they are different, the audio and video data may be damaged.

[0038] (b) Performing format parsing on the header information of the audio and video data to extract the format information of the audio and video data, and comparing the format information with the list of standard formats supported by the system. When the comparison results are consistent, it is determined that the format compatibility check has passed, where the format information includes: encoding format, resolution, and frame rate.

[0039] (B) See Figure 3 The flowchart of an adaptive AV1 encoding method is shown in FIG. The steps for performing adaptive AV1 encoding are as follows (1) to (4):

[0040] (1) Divide the difference in the number of bytes of packets sent and received in adjacent time intervals by the time difference between adjacent time intervals to determine the first real-time network speed, and determine the second real-time network speed through a data transmission test of a standard data packet. Then, add the product of the first preset weight and the first real-time network speed, and the product of the second preset weight and the second real-time network speed, respectively, to determine the target real-time network speed. When the real-time network speed is less than the preset network speed threshold, it is determined that the current network is in a state of limited network bandwidth. By increasing the file compression rate of audio and video data, the data transmission volume is reduced, so that the audio and video data can be transmitted smoothly. When the real-time network speed is not less than the preset network speed threshold, it is determined that the current network is in a state of sufficient network bandwidth. By reducing the file compression rate of audio and video data, the complete audio and video details are retained.

[0041] Specifically, the real-time network speed can be estimated by counting the number of bytes sent and received and the size of the transmitted data packets within a specific time interval. The mathematical formula for expressing the real-time network speed using the number of bytes sent and received is:

[0042]

[0043] in, Indicates the current time Number of bytes sent and received. Indicates the next moment Number of bytes sent and received. Indicates the real-time byte transfer speed.

[0044] Send a test message (a standard TCP / IP packet size, denoted as bytes), the intermediate monitoring station immediately sends back the test message, corrects the sending and receiving timestamps of the test message and calculates the total duration of the sending and receiving cycle (Since the transmission network can be regarded as bidirectionally symmetrical, the one-way transmission time is approximately half of the total time. ), then the network speed based on message transmission It can be expressed as:

[0045]

[0046]

[0047] Afterwards, a weighted average is used to comprehensively evaluate the current real-time network speed, where and They are and The weight is the optimal parameter obtained from multiple experiments, and ,To summarize, the current real-time network speed can be expressed as:

[0048]

[0049] (2) The audio stream in the audio and video data is converted into a mono mode, and the converted audio stream is subjected to signal segmentation processing to determine the various audio frames contained in the audio and video data. The preset bidirectional long short-term memory model is used to perform a criticality score on each audio frame, determine the audio score corresponding to each audio frame, and determine the frame with an audio score higher than the preset audio score threshold as the audio key frame. Among them, the long short-term memory network (LSTM) has demonstrated a strong sequence modeling capability in processing audio data, especially in retaining key information in audio under limited network speed. The bidirectional long short-term memory network (BiLSTM) has been improved on this basis and can simultaneously capture the past and future information of the sequence data, so that the contextual content of the audio can be analyzed more accurately. When processing the sequence, the input of each time step will be passed to the two LSTM layers separately, and then their outputs will be merged, thereby improving the accuracy of the information obtained.

[0050] Specifically, under low network speed conditions, it is particularly important to utilize limited bandwidth and network processing capabilities to prioritize and retain key audio and video information. When using BiLSTM to extract and retain key segments of the audio stream, first, the audio stream needs to be converted to mono and sampled at an appropriate sampling rate (16kHz). The audio signal is then segmented into shorter frames (120-150ms) and a window function is applied to reduce edge effects. After that, the fast Fourier transform (FFT) is used to extract audio features from each frame, including short-time energy, zero-crossing rate, MFCC (Mel-frequency cepstral coefficients), for subsequent feature analysis and model training.

[0051] Furthermore, it is necessary to expand the audio stream to build a data set for training the model. After the model training is completed, the audio stream before expansion is sent to the model to extract key frames. Specifically, key segments in the audio data are labeled based on the audio content (human voice, music climax, specific sound events, etc.) or information volume (energy, frequency changes, etc.), and the diversity of the audio data set is increased by adding noise, changing the volume, etc.

[0052] After obtaining the dataset, the BiLSTM model is trained and evaluated: the processed audio feature sequence is used as the input of the model, the bidirectional LSTM layer is used to capture the bidirectional information of the audio sequence, the output layer is designed as a fully connected layer, and the critical score (information percentage) of each time step is output. The dataset is then divided into training set, validation set and test set. The model is trained using the training set data, and the model parameters are updated using the backpropagation algorithm. The backpropagation algorithm is used to update the model parameters so that the model can be trained smoothly. The validation set is mainly used to adjust the model's hyperparameters. During the training process, the validation set is used to evaluate the model performance under different hyperparameter configurations, thereby helping to select the best model configuration. The results of the validation set directly affect the selection and adjustment of the model; the test set is used to finally evaluate the performance of the model. Once the model training and parameter adjustment are completed, the test set provides an independent sample set to measure the model's generalization ability, that is, the model's performance on unseen data.

[0053] Finally, the trained model is used to predict the audio data and output the scores of key audio clips. Audio clips with scores above a certain level are defined as key frames of the audio stream. When the network speed slows down or the transmission bandwidth is limited, the audio key frames are prioritized and retained.

[0054] (3) By converting the format of the video stream in the audio and video data, multiple video cubes are constructed using continuous multi-frame images in the video stream, wherein the video cube contains image frames of multiple time dimensions. Then, a preset 3D convolutional neural network model is used to perform key scoring on each video cube, determine the image score corresponding to each image frame, and statistically analyze the probability that the image score corresponding to each image frame in the video cube is higher than the preset image score threshold. The video cube with a probability greater than the preset probability threshold is determined as the video key frame. Among them, under low network speed conditions, the ability of convolutional neural network (CNN) to process video frames and retain key information of the image is particularly prominent. 3D convolutional neural network (3D CNN) is an extension of CNN. It adds a time dimension on the basis of traditional two-dimensional convolution, so that it can extract features in three-dimensional space and better capture the temporal and spatial feature information in the video.

[0055] Specifically, when using 3D CNN to process video streams, it is first necessary to sample a series of video frames from the video stream at fixed and random time intervals, then convert each frame image into a size and format suitable for network input, and then stack the consecutive multiple frames into a video cube as the input of the 3D CNN. Each video cube contains multiple frames in the time dimension. After all video cubes are divided into training set, validation set and test set, they can be used for subsequent model training.

[0056] Furthermore, when training and evaluating the 3D CNN model, the video cube needs to be used as input, and the video cube needs to be convolved using a 3D convolution kernel to capture spatial and temporal features at the same time. 3D pooling is used to reduce the dimension of the feature map while retaining important information of the image. The output layer is used to predict whether each video cube contains a key frame. After that, the model needs to be trained using the training set, and the model parameters are updated through the backpropagation algorithm and the SGD optimizer, so that the trained model can be used to predict key frames for the video data, and the probability of each image frame in the video cube being a key frame is output. A threshold is set based on the probability so that frames above the threshold are considered key frames. In addition, before the video is sent to the AV1 encoder, the video is marked according to the predicted key frames so that the key frames can be encoded more finely during the encoding process to ensure their quality. The 3D CNN model can provide dynamic key frames for subsequent AV1 encoding to ensure priority processing and retention of key video information when network speed is limited.

[0057] (4) Obtaining a preset compression rate calculation parameter and a basic compression rate when the network speed is 0, and determining the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed and the sum of the basic compression rate. In one embodiment, the compression rate of audio and video encoding is dynamically adjusted based on the real-time measurement value V of the network speed, and there is a positive correlation between them:

[0058]

[0059] Here, C represents the AV1 encoding compression ratio, V is the real-time network speed obtained in step 1, a is the optimal parameter obtained through multiple experiments, and b represents the basic compression ratio when the network speed is zero. When the real-time network speed is fast, the encoding compression ratio is large, which can ensure high compression efficiency. When the network speed is slow, the encoding compression ratio is small, and the system will prioritize encoding and retaining key audio and video information.

[0060] (C) Perform AV1 encoding, UDP transmission, AV1 decoding, and finally output the results. That is, after encoding the data key frames and data non-key frames in sequence, they are sent to the AV1 decoder for decoding via the fast transmission protocol, and the decoded audio and video data is determined. After the decoded audio and video data is encapsulated in the preset transmission format of the transmission channel, it is transmitted to the external audio and video data receiving end.

[0061] In one embodiment, the sending end creates a UDP socket and encapsulates the encoded bit stream into multiple UDP data packets for transmission. During the UDP transmission process, a forward error correction (FEC) strategy is used, that is, some redundant information is embedded in the transmitted data packets, so that the receiving end can still recover the original data through the redundant information when some data packets are lost. UDP is a connectionless and fast transmission protocol. Each data packet is sent independently, with the advantages of low resource consumption and fast processing speed. It is suitable for application scenarios such as audio and video streaming transmission, which have high real-time requirements and can tolerate a certain amount of packet loss.

[0062] In another embodiment, the UDP packet transmission rate and packet ID are counted over a period of time. When a packet ID is detected to have changed from an old ID to a new ID, the number of packets with the old ID received by the AV1 decoder is counted. If the ratio reaches 80% of the transmission rate, the UDP packet transmission is considered successful. The remaining untransmitted packets are padded with zeros and frames, and the bitstream data in the received multiple packets is reassembled into a complete bitstream. The complete bitstream is then sent to the AV1 decoder for decoding. The decoded audio and video data can be used for playback or further processing.

[0063] In summary, the present invention can overcome the limitations of insufficient and unchanging audio and video compression rates in existing codec systems. While maintaining the same audio and video quality, AV1-encoded audio and video files can occupy less storage space and bandwidth. Furthermore, the compression rate can be adaptively changed based on real-time network speed, preserving key audio and video information.

[0064] Regarding the method for real-time AV1 transcoding of audio and video based on network speed provided in the above embodiment, an embodiment of the present invention provides a device for real-time AV1 transcoding of audio and video based on network speed, see Figure 4 The structure diagram of a real-time AV1 transcoding device for audio and video based on network speed is shown, and the device includes the following parts:

[0065] The data verification module 402 obtains the audio and video data and performs integrity verification on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission;

[0066] The network speed monitoring module 404 determines the real-time network speed by monitoring the number of bytes of packets sent and received and the size of transmitted data packets over the mobile network within a preset time interval if the integrity check of the audio and video data passes. The network status monitoring result is determined based on the real-time network speed, where the network status monitoring result includes: sufficient network bandwidth status and limited network bandwidth status.

[0067] Key information extraction module 406, when the mobile network is in a limited network bandwidth state, performs key information extraction processing on the audio and video data using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data, where data key frames include: audio key frames and video key frames;

[0068] The transcoding transmission module 408 sends the real-time network speed to a preset dynamic transcoding model to determine the target compression rate of AV1 encoding. It then encodes and transmits the key frames and non-key frames of data in sequence at the target compression rate, thereby giving priority to encoding and storing key information of the audio and video data.

[0069] The above-mentioned network speed-based real-time AV1 audio and video transcoding device provided in the embodiment of the present application can significantly improve the quality and stability of audio and video transmission.

[0070] In one embodiment, when performing the step of performing integrity check processing on the audio and video data through a preset cyclic redundancy check model, the above-mentioned data verification module 402 is also used to: perform cyclic redundancy check on the audio and video data through a preset cyclic redundancy check model to determine the CRC check code; when the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

[0071] In one embodiment, after the step of determining whether the integrity check of the audio and video data is qualified, the above-mentioned data verification module 402 is also used to: perform format parsing processing on the header information of the audio and video data to extract the format information of the audio and video data, wherein the format information includes: encoding format, resolution and frame rate; compare the format information with the list of standard formats supported by the system, and when the comparison results are consistent, determine that the format compatibility check is passed.

[0072] In one embodiment, when performing the step of determining the real-time network speed by monitoring the number of bytes of packets received and sent in the mobile network and the size of the transmitted data packets within a preset time interval, the above-mentioned network speed monitoring module 404 is also used to: divide the difference in the number of bytes of packets received and sent in adjacent time intervals by the time difference between adjacent time intervals to determine the first real-time network speed, and determine the second real-time network speed through a data transmission test of a standard data packet; and add the product of the first preset weight and the first real-time network speed, and the product of the second preset weight and the second real-time network speed, respectively, to determine the target real-time network speed.

[0073] In one embodiment, when performing the step of determining the network status monitoring result based on the real-time network speed, the above-mentioned network speed monitoring module 404 is also used to: when the real-time network speed is less than the preset network speed threshold, determine that the current network is in a network broadband limited state, and increase the file compression rate of audio and video data to reduce the data transmission volume, so that the audio and video data are transmitted smoothly; when the real-time network speed is not less than the preset network speed threshold, determine that the current network is in a network broadband sufficient state, and reduce the file compression rate of audio and video data to retain complete audio and video details.

[0074] In one embodiment, when performing key information extraction processing on audio and video data and determining data key frames of the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, the above-mentioned key information extraction module 406 is also used to: convert the audio stream in the audio and video data into a mono mode, and perform signal segmentation processing on the converted audio stream to determine the audio frames contained in the audio and video data; use the preset bidirectional long short-term memory model to perform criticality scoring on each audio frame, determine the audio score corresponding to each audio frame, and determine the frame with an audio score higher than a preset audio score threshold as an audio key frame.

[0075] In one embodiment, the key information extraction module 406 is further used to: perform format conversion processing on the video stream in the audio and video data, and construct multiple video cubes using continuous multiple-frame images in the video stream, wherein the video cube contains image frames of multiple time dimensions; use a preset 3D convolutional neural network model to perform criticality scoring on each video cube and determine the image score corresponding to each image frame; and perform statistics on the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determine the video cube with a probability greater than the preset probability threshold as a video key frame.

[0076] In one embodiment, when sending the real-time network speed to a preset dynamic transcoding model to determine the target compression rate for AV1 encoding, the transcoding transmission module 408 is further configured to: obtain a preset compression rate calculation parameter and a base compression rate when the network speed is 0; and determine the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed, and the sum of the base compression rate.

[0077] In one embodiment, after encoding and transmitting the data key frames and the data non-key frames in sequence at a target compression rate, the above-mentioned transcoding transmission module 408 is further used to: encode the data key frames and the data non-key frames in sequence, and then send them to the AV1 decoder for decoding processing through the fast transmission protocol to determine the decoded audio and video data; after encapsulating the decoded audio and video data in a preset transmission format of the transmission channel, transmit it to an external audio and video data receiving end.

[0078] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0079] An embodiment of the present invention provides a server. Specifically, the server includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes any one of the methods in the above embodiments.

[0080] Figure 5 A structural diagram of a server provided in an embodiment of the present invention, wherein the server 100 includes: a processor 50, a memory 51, a bus 52 and a communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected via the bus 52; the processor 50 is used to execute an executable module stored in the memory 51, such as a computer program.

[0081] The memory 51 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. The system network element communicates with at least one other network element via at least one communication interface 53 (which may be wired or wireless), and may utilize the Internet, a wide area network, a local area network, a metropolitan area network, or the like.

[0082] The bus 52 may be an ISA bus, a PCI bus, or an EISA bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0083] Among them, the memory 51 is used to store programs, and the processor 50 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0084] The processor 50 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be completed by hardware integrated logic circuits or software instructions in the processor 50. The processor 50 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 51 , and the processor 50 reads the information in the memory 51 and completes the steps of the above method in combination with its hardware.

[0085] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the previous method embodiment and will not be repeated here.

[0086] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0087] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for real-time AV1 transcoding of audio and video based on network speed, characterized in that: The method comprises: Acquire audio and video data and perform integrity check on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; If the integrity check of the audio and video data passes, the real-time network speed is determined by monitoring the number of bytes of packets sent and received and the size of the transmitted data packets of the mobile network within a preset time interval, and a network status monitoring result is determined based on the real-time network speed, wherein the network status monitoring result includes: a sufficient network bandwidth state and a limited network bandwidth state; When the mobile network is in the network bandwidth limited state, performing key information extraction processing on the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; Sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate for AV1 encoding, and encoding and transmitting the key frames and non-key frames of the data in sequence at the target compression rate, so as to preferentially encode and store key information of the audio and video data; The step of determining the real-time network speed by monitoring the number of bytes of packets sent and received and the size of transmitted data packets in a preset time interval of the mobile network includes: dividing the difference in the number of bytes of packets sent and received in adjacent time intervals by the time difference between the adjacent time intervals to determine a first real-time network speed, and determining a second real-time network speed by a data transmission test of a standard data packet; and respectively adding the product of a first preset weight and the first real-time network speed, and the product of a second preset weight and the second real-time network speed to determine a target real-time network speed; The step of determining the network status monitoring result according to the real-time network speed includes: when the real-time network speed is less than a preset network speed threshold, determining that the current network is in a network bandwidth-limited state, and increasing the file compression rate of the audio and video data to reduce the data transmission volume, so that the audio and video data are transmitted smoothly; when the real-time network speed is not less than the preset network speed threshold, determining that the current network is in a network bandwidth-sufficient state, and decreasing the file compression rate of the audio and video data to retain complete audio and video details; The step of extracting key information from the audio and video data and determining the data key frames of the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model includes: converting the audio stream in the audio and video data into a mono mode, and performing signal segmentation processing on the converted audio stream to determine the various audio frames contained in the audio and video data; using the preset bidirectional long short-term memory model to perform a criticality score on each of the audio frames, determining the audio score corresponding to each of the audio frames, and determining the frame whose audio score is higher than a preset audio score threshold as the audio key frame; The method further includes: performing format conversion processing on the video stream in the audio and video data, and constructing multiple video cubes using multiple consecutive frames of images in the video stream, wherein the video cubes contain image frames of multiple time dimensions; using a preset 3D convolutional neural network model, performing criticality scoring on each of the video cubes to determine an image score corresponding to each of the image frames; statistically analyzing the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determining the video cube with a probability greater than the preset probability threshold as the video key frame; The real-time network speed is expressed by counting the number of bytes sent and received and the size of the transmitted data packets within a preset time interval: ； in, Indicates the current time Number of bytes sent and received. Indicates the next moment Number of bytes sent and received. Indicates the real-time byte transmission speed; The server sends a test message to the intermediate monitoring station, receives the test message immediately sent back by the intermediate monitoring station, corrects the sending and receiving timestamps of the test message, and calculates the total duration of the sending and receiving cycle. , to determine the network speed based on message transmission : in, ; in, Used to represent a byte, using and The target real-time network speed is comprehensively evaluated by taking the weighted average value of 2. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of performing integrity check processing on the audio and video data using a preset cyclic redundancy check model includes: Performing a cyclic redundancy check on the audio and video data using a preset cyclic redundancy check model to determine a CRC check code; When the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; When the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

3. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 2, characterized in that: After determining that the integrity check of the audio and video data is qualified, the method further includes: Performing format parsing on the header information of the audio and video data to extract format information of the audio and video data, wherein the format information includes: encoding format, resolution and frame rate; The format information is compared with a list of standard formats supported by the system, and when the comparison results are consistent, it is determined that the format compatibility check has passed.

4. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate for AV1 encoding includes: Get the preset compression ratio calculation parameters and the basic compression ratio when the network speed is 0; The target compression ratio is determined as the sum of the product of the compression ratio calculation parameter and the target real-time network speed and the basic compression ratio.

5. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: After the step of sequentially encoding and transmitting the data key frame and the data non-key frame at the target compression rate, the method includes: After encoding the data key frame and the data non-key frame in sequence, the encoded data is sent to an AV1 decoder via a fast transport protocol for decoding, thereby determining decoded audio and video data; After the decoded audio and video data is encapsulated in the preset transmission format of the transmission channel, it is transmitted to the external audio and video data receiving end.

6. A real-time AV1 transcoding device for audio and video based on network speed, characterized in that: The device comprises: A data verification module obtains audio and video data and performs integrity verification on the audio and video data using a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; a network speed monitoring module, which, if the integrity check of the audio and video data passes, determines the real-time network speed by monitoring the number of bytes of packets sent and received and the size of transmitted data packets of the mobile network within a preset time interval, and determines a network status monitoring result based on the real-time network speed, wherein the network status monitoring result includes: a sufficient network bandwidth state and a limited network bandwidth state; a key information extraction module, which, when the mobile network is in the network bandwidth limited state, performs key information extraction processing on the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; a transcoding transmission module that sends the real-time network speed to a preset dynamic transcoding model, determines a target compression rate for AV1 encoding, and sequentially encodes and transmits the key frames and non-key frames at the target compression rate, so as to preferentially encode and store key information of the audio and video data; The step of determining the real-time network speed by monitoring the number of bytes of packets sent and received and the size of transmitted data packets in a preset time interval of the mobile network includes: dividing the difference in the number of bytes of packets sent and received in adjacent time intervals by the time difference between the adjacent time intervals to determine a first real-time network speed, and determining a second real-time network speed by a data transmission test of a standard data packet; and respectively adding the product of a first preset weight and the first real-time network speed, and the product of a second preset weight and the second real-time network speed to determine a target real-time network speed; The step of determining the network status monitoring result according to the real-time network speed includes: when the real-time network speed is less than a preset network speed threshold, determining that the current network is in a network bandwidth-limited state, and increasing the file compression rate of the audio and video data to reduce the data transmission volume, so that the audio and video data are transmitted smoothly; when the real-time network speed is not less than the preset network speed threshold, determining that the current network is in a network bandwidth-sufficient state, and decreasing the file compression rate of the audio and video data to retain complete audio and video details; The step of extracting key information from the audio and video data and determining the data key frames of the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model includes: converting the audio stream in the audio and video data into a mono mode, and performing signal segmentation processing on the converted audio stream to determine the various audio frames contained in the audio and video data; using the preset bidirectional long short-term memory model to perform a criticality score on each of the audio frames, determining the audio score corresponding to each of the audio frames, and determining the frame whose audio score is higher than a preset audio score threshold as the audio key frame; The device further includes: performing format conversion processing on a video stream in the audio and video data, and constructing multiple video cubes using multiple consecutive frames of images in the video stream, wherein the video cubes contain image frames of multiple time dimensions; performing criticality scoring on each of the video cubes using a preset 3D convolutional neural network model, and determining an image score corresponding to each of the image frames; performing statistics on the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determining the video cube with a probability greater than the preset probability threshold as the video key frame; The real-time network speed is expressed by counting the number of bytes sent and received and the size of the transmitted data packets within a preset time interval: ； in, Indicates the current time Number of bytes sent and received. Indicates the next moment Number of bytes sent and received. Indicates the real-time byte transmission speed; The server sends a test message to the intermediate monitoring station, receives the test message immediately sent back by the intermediate monitoring station, corrects the sending and receiving timestamps of the test message, and calculates the total duration of the sending and receiving cycle. , to determine the network speed based on message transmission : in, ; in, Used to represent a byte, using and The target real-time network speed is comprehensively evaluated by taking the weighted average value of 7. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 5.

Images