Audio and video real-time AV1 transcoding method and device based on network speed and server

Abstract

The invention provides an audio and video real-time AV1 transcoding method and device based on network speed and a server, and relates to the technical field of AV1 video coding and decoding, and the method comprises the steps: obtaining audio and video data, and carrying out the integrity verification of the audio and video data through a preset cyclic redundancy verification model; if the integrity verification of the audio and video data is qualified, determining a real-time network speed by monitoring the size of a transmission data packet, and determining a network state monitoring result according to the real-time network speed; when the mobile network is in a network broadband limited state, key information extraction processing is carried out on the audio and video data through a preset bidirectional long-short-term memory model and a preset 3D convolutional neural network model, and data key frames and data non-key frames of the audio and video data are determined; and sending the real-time network speed to a preset dynamic variable code model, and determining a target compression rate of AV1 coding so as to preferentially code and store key information of the audio and video data. According to the invention, the quality and stability of audio and video transmission can be obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AV1 video encoding and decoding, and in particular to a method, device and server for real-time AV1 transcoding of audio and video based on network speed. Background Art

[0002] At present, relevant technologies have proposed that audio and video data need to be encoded and compressed before transmission during the encoding and transmission process, and the bit rate of the mobile network (that is, the compression capability of audio and video data) is positively correlated with the network speed. When the network bandwidth is tight at low speed, the bit rate is low, and the compressed audio and video files are large, and the network transmission capability is poor at low speed. When the network bandwidth is abundant at high speed, the bit rate is high, and the compressed audio and video files are small. Therefore, if the bit rate is adjusted only according to the network speed, it will cause data transmission delays and reception end jams when the network bandwidth is tight, and waste broadband resources when the network bandwidth is abundant, thereby affecting the quality and stability of audio and video transmission. In addition, this solution is difficult to respond to changes quickly in a complex and changeable network environment, and has poor adaptability and flexibility. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a method, device and server for real-time AV1 transcoding of audio and video based on network speed, which can significantly improve the quality and stability of audio and video transmission.

[0004] In a first aspect, an embodiment of the present invention provides a real-time AV1 transcoding method for audio and video based on network speed, the method comprising: obtaining audio and video data, and performing integrity check processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; if the integrity check of the audio and video data is qualified, the real-time network speed is determined by monitoring the number of bytes of the received and sent packets of the mobile network and the size of the transmission data packet within a preset time interval, and the network status monitoring result is determined according to the real-time network speed, wherein the network status monitoring result includes: sufficient network bandwidth state and limited network bandwidth state; when the mobile network is in the limited network bandwidth state, key information extraction processing is performed on the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine the data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; the real-time network speed is sent to the preset dynamic transcoding model to determine the target compression rate of AV1 encoding, and the data key frames and data non-key frames are encoded and transmitted in sequence at the target compression rate to preferentially encode and store the key information of the audio and video data.

[0005] In one embodiment, the step of performing integrity check processing on audio and video data through a preset cyclic redundancy check model includes: performing cyclic redundancy check on the audio and video data through a preset cyclic redundancy check model to determine a CRC check code; when the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

[0006] In one embodiment, after the step of determining whether the integrity check of the audio and video data is qualified, the step includes: performing format parsing processing on the header information of the audio and video data to extract the format information of the audio and video data, wherein the format information includes: encoding format, resolution and frame rate; comparing the format information with the list of standard formats supported by the system, and when the comparison results are consistent, determining that the format compatibility check has passed.

[0007] In one embodiment, the step of determining the real-time network speed by monitoring the number of bytes of packets received and sent in the mobile network and the size of the transmitted data packets within a preset time interval includes: dividing the difference in the number of bytes of packets received and sent in adjacent time intervals by the time difference between adjacent time intervals to determine the first real-time network speed, and determining the second real-time network speed through a data transmission test of a standard data packet; and adding the product of the first preset weight and the first real-time network speed, and the product of the second preset weight and the second real-time network speed, respectively, to determine the target real-time network speed.

[0008] In one embodiment, the step of determining the network status monitoring result according to the real-time network speed includes: when the real-time network speed is less than a preset network speed threshold, determining that the current network is in a network bandwidth limited state, and reducing the data transmission amount by increasing the file compression rate of the audio and video data to ensure smooth transmission of the audio and video data; when the real-time network speed is not less than the preset network speed threshold, determining that the current network is in a network bandwidth sufficient state, and reducing the file compression rate of the audio and video data to retain the complete audio and video details.

[0009] In one embodiment, a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model are used to extract key information from audio and video data, and the step of determining data key frames of the audio and video data includes: converting the audio stream in the audio and video data into a mono mode, and performing signal segmentation processing on the converted audio stream to determine each audio frame contained in the audio and video data; using the preset bidirectional long short-term memory model to perform criticality scoring on each audio frame, determine the audio score corresponding to each audio frame, and determine the frame with an audio score higher than a preset audio score threshold as an audio key frame.

[0010] In one embodiment, the method also includes: performing format conversion processing on the video stream in the audio and video data, and constructing multiple video cubes using continuous multiple frames of images in the video stream, wherein the video cube contains image frames of multiple time dimensions; using a preset 3D convolutional neural network model to perform critical scoring on each video cube and determine the image score corresponding to each image frame; and statistically analyzing the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determining the video cube with a probability greater than the preset probability threshold as a video key frame.

[0011] In one embodiment, the step of sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate of AV1 encoding includes: obtaining a preset compression rate calculation parameter and a basic compression rate when the network speed is 0; and determining the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed and the sum of the basic compression rate.

[0012] In one embodiment, after the step of encoding and transmitting the data key frames and the data non-key frames in sequence at the target compression rate, the method includes: after encoding the data key frames and the data non-key frames in sequence, sending them to the AV1 decoder through the fast transmission protocol for decoding processing to determine the decoded audio and video data; after encapsulating the decoded audio and video data in the preset transmission format of the transmission channel, transmitting them to the external audio and video data receiving end.

[0013] In a second aspect, an embodiment of the present invention further provides an audio and video real-time AV1 transcoding device based on network speed, the device comprising: a data verification module, which obtains audio and video data, and performs integrity verification processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; a network speed monitoring module, if the integrity verification of the audio and video data is qualified, then the real-time network speed is determined by monitoring the number of bytes of the received and sent packets of the mobile network and the size of the transmitted data packets within a preset time interval, and the network status monitoring result is determined according to the real-time network speed, wherein the network status monitoring result includes: the sufficient network bandwidth status and the network The network bandwidth is limited; the key information extraction module, when the mobile network is in a network bandwidth limited state, performs key information extraction and processing on the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, and determines the data key frames and data non-key frames of the audio and video data, where the data key frames include: audio key frames and video key frames; the transcoding transmission module sends the real-time network speed to the preset dynamic transcoding model, determines the target compression rate of AV1 encoding, and encodes and transmits the data key frames and data non-key frames in sequence at the target compression rate, so as to give priority to encoding and storing the key information of the audio and video data.

[0014] In a third aspect, an embodiment of the present invention further provides a server, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement any one of the methods provided in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0016] The embodiments of the present invention bring the following beneficial effects: The embodiment of the present invention provides a method, device and server for real-time AV1 transcoding of audio and video based on network speed. After acquiring audio and video data, the method performs integrity check processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission. If the integrity check of the audio and video data is qualified, the real-time network speed is determined by monitoring the number of bytes of the received and sent packets of the mobile network and the size of the transmission data packet within a preset time interval, and the network status monitoring result is determined according to the real-time network speed. When the network status monitoring result shows that the mobile network is in a network bandwidth-limited state, the key information of the audio and video data is extracted and processed through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine the data key frames and data non-key frames of the audio and video data. Finally, the real-time network speed is sent to the preset dynamic transcoding model to determine the target compression rate of AV1 encoding, and the data key frames and data non-key frames are encoded and transmitted in sequence at the target compression rate, so as to preferentially encode and store the key information of the audio and video data. The embodiment of the present invention can significantly improve the quality and stability of audio and video transmission.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a method for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention; Figure 2 A schematic diagram of a method for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention; Figure 3 A schematic diagram of a flow chart of an adaptive AV1 encoding method provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a device for real-time AV1 transcoding of audio and video based on network speed provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] At present, the existing audio and video coding and decoding methods are mostly H.264 / AVC, HEVC / H.265 and VP9, ​​etc. The compression rate is not enough to meet the needs of the current actual coding and decoding system development. In addition, the compression rate of most audio and video coding and decoding systems is fixed and cannot be adjusted adaptively in real time according to the current network speed. Especially when the network speed is poor, the key information of the image or audio will be lost. Specifically, the relevant technology proposes that the audio and video data needs to be encoded and compressed before transmission during the encoding and transmission process, and the bit rate of the mobile network (that is, the compression ability of the audio and video data) is positively correlated with the network speed. When the network bandwidth is tight at a low speed, the bit rate is low, the compressed audio and video files are large, and the network transmission capacity is poor at a low speed. When the network bandwidth is abundant at a high speed, the bit rate is high, and the compressed audio and video files are small. Therefore, if the bit rate is adjusted only according to the network speed, it will cause data transmission delays and jamming at the receiving end when the network bandwidth is tight, and waste broadband resources when the network bandwidth is abundant, thereby affecting the quality and stability of audio and video transmission. In addition, this solution is difficult to respond to changes quickly in a complex and changeable network environment, and has poor adaptability and flexibility.

[0023] Based on this, the network speed-based audio and video real-time AV1 transcoding method, device and server provided by the present invention can monitor the network status in real time, dynamically adjust the encoding compression rate and output bit rate according to the actual situation of the current network bandwidth, and when the network bandwidth is tight, reduce the data transmission volume by lowering the bit rate, thereby alleviating the network pressure and ensuring the smoothness of audio and video transmission; when the network bandwidth is abundant, improve the video quality by increasing the bit rate and make full use of bandwidth resources. The adaptive adjustment mechanism not only improves the stability and quality of audio and video transmission, but also optimizes the user experience. In addition, it can be flexibly applied in different application scenarios to meet diverse needs, and has broad market prospects and application value.

[0024] See also Figure 1 The flowchart of a method for real-time AV1 transcoding of audio and video based on network speed is shown, and the method mainly includes the following steps S102 to S108: Step S102, obtain audio and video data, and perform integrity check processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission. In one embodiment, after receiving audio and video data from an external source, an integrity check is performed to ensure that the data is not damaged or lost during transmission, and the received audio and video stream is format analyzed to verify whether it meets the requirements of the codec format supported by the system, especially focusing on the compatibility of AV1 encoding. The encoding parameters, frame rate, resolution and other information of the audio and video stream can be preliminarily analyzed to evaluate its basic quality level, providing a reference for subsequent encoding and decoding processing.

[0025] Step S104, if the integrity check of the audio and video data is qualified, the real-time network speed is determined by monitoring the number of bytes of the received and sent packets of the mobile network and the size of the transmission data packets within the preset time interval, and the network status monitoring result is determined according to the real-time network speed, wherein the network status monitoring result includes: sufficient network bandwidth status and limited network bandwidth status. In one embodiment, the bandwidth, delay and other key parameters of the network where the system is located can be monitored in real time to accurately evaluate the current network status, and according to the network status monitoring result, it is determined whether the compression rate of the audio and video stream needs to be adjusted. When the network bandwidth is sufficient, the compression rate can be appropriately reduced to retain more audio and video details; when the network bandwidth is limited, the compression rate can be increased to reduce the amount of data transmission to ensure smooth playback. In addition, in the process of adjusting the compression rate, intelligent algorithms (BiLSTM, 3D CNN) can be used to ensure that key information in the audio and video (such as voice, face, etc.) is retained first to avoid the loss of important information.

[0026] Step S106, when the mobile network is in a state of limited network bandwidth, the audio and video data are subjected to key information extraction processing by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, and data key frames and data non-key frames of the audio and video data are determined, wherein the data key frames include: audio key frames and video key frames. In one implementation, the AV1 codec can be selected and configured according to the characteristics of the audio and video streams and system requirements, and efficient audio and video compression processing can be prepared. Specifically, the advanced algorithm of the AV1 encoder is used to perform high-quality compression on the audio and video streams, and the size of the audio and video files can be significantly reduced while maintaining the video quality unchanged, thereby reducing storage requirements and transmission bandwidth.

[0027] Step S108, sending the real-time network speed to a preset dynamic transcoding model, determining the target compression rate of AV1 encoding, and encoding and transmitting the data key frames and data non-key frames in sequence at the target compression rate, so as to preferentially encode and store the key information of the audio and video data. In one implementation, the compressed data stream can be transmitted via UDP, and then AV1 decoding is performed on it, and then the audio and video stream processed by AV1 adaptive encoding and decoding is encapsulated and output to the target storage medium or transmission channel.

[0028] The above-mentioned real-time AV1 transcoding method for audio and video based on network speed provided in the embodiment of the present invention can significantly improve the quality and stability of audio and video transmission.

[0029] See also Figure 2 The schematic diagram of a method for real-time AV1 transcoding of audio and video based on network speed is shown. The embodiment of the present invention also provides an implementation method of real-time AV1 transcoding of audio and video based on network speed. For details, see (A) to (C) below: (A) After obtaining the audio and video data, integrity check and format verification are performed. The integrity check and format verification are specifically shown in (a) to (b) below: (a) Performing a cyclic redundancy check on the audio and video data through a preset cyclic redundancy check model to determine a CRC check code: when the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified. In one embodiment, the cyclic redundancy check (CRC) calculates the check code by polynomial division, which can detect and correct errors in data transmission: using CRC to perform integrity check on the received audio and video data packets, receiving the audio and video data and the preset check code calculated in advance by long division at the receiving end, and performing the same CRC calculation to determine the CRC check code after adding the preset check code to the audio and video data, and comparing it with the preset check code. If they are the same, the audio and video data may not be damaged. If they are different, the audio and video data may be damaged.

[0030] (b) Format information of the audio and video data is extracted by performing format parsing on the header information of the audio and video data, and the format information is compared with the list of standard formats supported by the system. When the comparison results are consistent, it is determined that the format compatibility check has passed, wherein the format information includes: encoding format, resolution and frame rate.

[0031] (B) See Figure 3 FIG. 1 is a flow chart of an adaptive AV1 encoding method, wherein the steps of performing adaptive AV1 encoding are specifically as follows (1) to (4): (1) The first real-time network speed is determined by dividing the difference in the number of bytes of the packets sent and received in adjacent time intervals by the time difference between the adjacent time intervals, and the second real-time network speed is determined by a data transmission test of a standard data packet. Then, the product of the first preset weight and the first real-time network speed and the product of the second preset weight and the second real-time network speed are added together to determine the target real-time network speed. When the real-time network speed is less than the preset network speed threshold, it is determined that the current network is in a state of limited network bandwidth. By increasing the file compression rate of the audio and video data, the data transmission volume is reduced to ensure smooth transmission of the audio and video data. When the real-time network speed is not less than the preset network speed threshold, it is determined that the current network is in a state of sufficient network bandwidth. By reducing the file compression rate of the audio and video data, the complete audio and video details are retained.

[0032] Specifically, the real-time network speed can be estimated by counting the number of bytes of sent and received packets and the size of the transmitted data packets within a specific time interval. The mathematical formula for expressing the real-time network speed using the number of bytes of sent and received packets is:

[0033] in, Indicates the current time Number of bytes sent and received. Indicates the next moment Number of bytes sent and received. Indicates the real-time byte transfer speed.

[0034] Send a test message (a standard TCP / IP packet size, denoted as bytes), the intermediate monitoring station immediately sends back the test message, corrects the sending and receiving timestamps of the test message, and counts the total duration of the sending and receiving cycle. (Since the transmission network can be regarded as bidirectionally symmetrical, the one-way transmission time is approximately half of the total time. ), then the network speed based on message transmission It can be expressed as:

[0035]

[0036] Afterwards, a weighted average is used to comprehensively evaluate the current real-time network speed, where and They are and The weight is the optimal parameter obtained from multiple experiments, and , in summary, the current real-time network speed can be expressed as:

[0037] (2) The audio stream in the audio and video data is converted into mono mode, and the converted audio stream is subjected to signal segmentation processing to determine the audio frames contained in the audio and video data. The preset bidirectional long short-term memory model is used to perform a criticality score on each audio frame, determine the audio score corresponding to each audio frame, and determine the frame with an audio score higher than the preset audio score threshold as the audio key frame. Among them, the long short-term memory network (LSTM) has demonstrated a strong sequence modeling capability when processing audio data, especially in retaining key information in the audio under limited network speed. The bidirectional long short-term memory network (BiLSTM) has been improved on this basis and can simultaneously capture the past and future information of the sequence data, so as to more accurately analyze the contextual content of the audio. When processing the sequence, the input of each time step will be passed to two LSTM layers separately, and then their outputs will be merged, thereby improving the accuracy of the acquired information.

[0038] Specifically, under low network speed conditions, it is particularly important to use limited bandwidth and network processing capabilities to prioritize and retain key audio and video information. When using BiLSTM to extract and retain key segments of the audio stream, first, the audio stream needs to be converted to mono and sampled at an appropriate sampling rate (16kHz). The audio signal is then divided into shorter frames (120-150ms) and a window function is applied to reduce edge effects. The fast Fourier transform (FFT) is then used to extract audio features from each frame, including short-time energy, zero-crossing rate, MFCC (Mel frequency cepstral coefficients), etc., for subsequent feature analysis and model training.

[0039] Furthermore, it is necessary to expand the audio stream to build a data set for training the model, so that after the model training is completed, the audio stream before expansion is sent to the model to extract key frames. Specifically, the key segments in the audio data are labeled based on the audio content (human voice, music climax, specific sound events, etc.) or the amount of information (energy, frequency changes, etc.), and the diversity of the audio data set is increased by adding noise, changing the volume, etc.

[0040] After obtaining the data set, the BiLSTM model is trained and evaluated: the processed audio feature sequence is used as the input of the model, the bidirectional LSTM layer is used to capture the bidirectional information of the audio sequence, the output layer is designed as a fully connected layer, and the critical score (percentage of information) of each time step is output. Then the data set is divided into training set, validation set and test set. The model is trained with the training set data, and the model parameters are updated through the back propagation algorithm. The back propagation algorithm is used to update the model parameters so that the model can be trained smoothly. The validation set is mainly used to adjust the hyperparameters of the model. During the training process, the validation set is used to evaluate the model performance under different hyperparameter configurations, thereby helping to select the best model configuration. The results of the validation set directly affect the selection and adjustment of the model; the test set is used to finally evaluate the performance of the model. Once the model training and parameter adjustment are completed, the test set provides an independent sample set to measure the generalization ability of the model, that is, the performance of the model on unseen data.

[0041] Finally, the trained model is used to predict the audio data and output the scores of key audio clips. Audio clips with scores above a certain level are defined as key frames of the audio stream. When the network speed slows down or the transmission bandwidth is limited, the audio key frames are prioritized and retained.

[0042] (3) By performing format conversion processing on the video stream in the audio and video data, multiple video cubes are constructed using continuous multiple frames of images in the video stream, wherein the video cube contains image frames of multiple time dimensions. Then, a preset 3D convolutional neural network model is used to perform critical scoring on each video cube, determine the image score corresponding to each image frame, and statistically analyze the probability that the image score corresponding to each image frame in the video cube is higher than the preset image score threshold. The video cube with a probability greater than the preset probability threshold is determined as the video key frame. In the case of low network speed, the ability of the convolutional neural network (CNN) to process video frames and retain key information of the image is particularly prominent. The 3D convolutional neural network (3D CNN) is an extension of CNN. It adds a time dimension to the traditional two-dimensional convolution, so that it can extract features in three-dimensional space and better capture the temporal and spatial feature information in the video.

[0043] Specifically, when using 3D CNN to process video streams, you first need to sample a series of video frames from the video stream at fixed and random time intervals, then convert each frame image into a size and format suitable for network input, and then stack multiple consecutive frames into a video cube as the input of the 3D CNN. Each video cube contains multiple frames in the time dimension. After all video cubes are divided into training sets, validation sets, and test sets, they can be used for subsequent model training.

[0044] Furthermore, when training and evaluating the 3D CNN model, the video cube needs to be used as input, and the video cube needs to be convolved using a 3D convolution kernel to capture spatial and temporal features at the same time. 3D pooling is used to reduce the dimension of the feature map while retaining important information of the image. The output layer is used to predict whether each video cube contains a key frame. After that, the model needs to be trained using the training set, and the model parameters are updated through the back-propagation algorithm and the SGD optimizer, so that the trained model can be used to predict key frames for the video data, and the probability of each picture frame in the video cube being a key frame is output. A threshold is set based on the probability so that frames above the threshold are regarded as key frames. In addition, before the video is sent to the AV1 encoder, the video is marked according to the predicted key frames so that the key frames can be more finely encoded during the encoding process to ensure their quality. The 3D CNN model can provide dynamic key frames for subsequent AV1 encoding to ensure that key video information is prioritized and retained when network speed is limited.

[0045] (4) Obtaining a preset compression rate calculation parameter and a basic compression rate when the network speed is 0, and determining the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed and the sum of the basic compression rate. In one embodiment, the compression rate of audio and video encoding is dynamically adjusted based on the real-time measurement value V of the network speed, and there is a positive correlation between them:

[0046] Among them, C represents the compression rate of AV1 encoding, V is the real-time network speed obtained in step 1, a is the optimal parameter obtained through multiple experiments, and b represents the basic compression rate when the network speed is zero. When the real-time network speed is faster, the encoding compression rate is larger, which can ensure high compression efficiency. When the network speed is slower, the encoding compression rate is smaller, and the system will give priority to encoding and retaining key audio and video information.

[0047] (C) Perform AV1 encoding, UDP transmission, AV1 decoding and finally output the result, that is, after encoding the data key frames and data non-key frames in sequence, send them to the AV1 decoder through the fast transmission protocol for decoding processing, determine the decoded audio and video data, and after encapsulating the decoded audio and video data in the preset transmission format of the transmission channel, transmit it to the external audio and video data receiving end.

[0048] In one implementation, the sender creates a UDP socket and encapsulates the encoded bit stream into multiple UDP data packets for transmission. During the UDP transmission process, a forward error correction (FEC) strategy is used, that is, some redundant information is embedded in the transmitted data packets, so that the receiving end can still restore the original data through the redundant information when some data packets are lost. UDP is a connectionless and fast transmission protocol. Each data packet is sent independently. It has the advantages of low resource consumption and fast processing speed. It is suitable for application scenarios such as audio and video stream transmission, which have high real-time requirements and can tolerate a certain amount of packet loss.

[0049] In another embodiment, the transmission rate of UDP data packets and data packet IDs within a period of time are counted. When it is detected that the data packet changes from the old ID to the new ID, the number of data packets with the old ID received by the AV1 decoder is counted. If the ratio reaches 80% of the transmission, it is considered that the UDP data packet is successfully transmitted, and zero-filling and frame-filling operations are performed on the remaining untransmitted data packets. The bit stream data in the received multiple data packets are reassembled into a complete bit stream, and then sent to the AV1 decoder for decoding. The decoded audio and video data can be used for playback or further processing.

[0050] In summary, the present invention can overcome the limitations of insufficient and single audio and video compression rate in existing coding and decoding systems. Under the same audio and video quality, the AV1-encoded audio and video files occupy less storage space and bandwidth, and the compression rate is adaptively changed based on the real-time network speed to retain the key information of the audio and video.

[0051] For the method for real-time AV1 transcoding of audio and video based on network speed provided in the above-mentioned embodiment, an embodiment of the present invention provides a device for real-time AV1 transcoding of audio and video based on network speed, see Figure 4The structure diagram of a real-time AV1 transcoding device for audio and video based on network speed is shown, and the device includes the following parts: The data verification module 402 obtains the audio and video data and performs integrity verification on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during the transmission process; The network speed monitoring module 404 determines the real-time network speed by monitoring the number of bytes of the received and sent packets and the size of the transmitted data packets of the mobile network within a preset time interval if the integrity check of the audio and video data is qualified, and determines the network status monitoring result according to the real-time network speed, wherein the network status monitoring result includes: sufficient network bandwidth status and limited network bandwidth status; The key information extraction module 406 performs key information extraction processing on the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model when the mobile network is in a network bandwidth limited state, and determines data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; The transcoding transmission module 408 sends the real-time network speed to a preset dynamic transcoding model, determines the target compression rate of AV1 encoding, and encodes and transmits the data key frames and data non-key frames in sequence at the target compression rate, so as to preferentially encode and store the key information of the audio and video data.

[0052] The above-mentioned network speed-based audio and video real-time AV1 transcoding device provided in the embodiment of the present application can significantly improve the quality and stability of audio and video transmission.

[0053] In one embodiment, when performing the step of performing integrity check processing on the audio and video data through a preset cyclic redundancy check model, the above-mentioned data verification module 402 is also used to: perform cyclic redundancy check on the audio and video data through a preset cyclic redundancy check model to determine the CRC check code; when the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; when the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

[0054] In one embodiment, after the step of determining whether the integrity check of the audio and video data is qualified, the above-mentioned data verification module 402 is also used to: perform format parsing processing on the header information of the audio and video data to extract the format information of the audio and video data, wherein the format information includes: encoding format, resolution and frame rate; compare the format information with the list of standard formats supported by the system, and when the comparison results are consistent, determine that the format compatibility check has passed.

[0055] In one embodiment, when performing the step of determining the real-time network speed by monitoring the number of bytes of received and sent packets and the size of transmitted data packets of the mobile network within a preset time interval, the network speed monitoring module 404 is also used to: determine the first real-time network speed by dividing the difference in the number of bytes of received and sent packets in adjacent time intervals by the time difference between adjacent time intervals, and determine the second real-time network speed by a data transmission test of a standard data packet; and respectively add the product of the first preset weight and the first real-time network speed, and the product of the second preset weight and the second real-time network speed to determine the target real-time network speed.

[0056] In one embodiment, when performing the step of determining the network status monitoring result according to the real-time network speed, the above-mentioned network speed monitoring module 404 is also used for: when the real-time network speed is less than a preset network speed threshold, determining that the current network is in a network bandwidth limited state, and increasing the file compression rate of the audio and video data to reduce the data transmission volume, so that the audio and video data are transmitted smoothly; when the real-time network speed is not less than the preset network speed threshold, determining that the current network is in a network bandwidth sufficient state, and reducing the file compression rate of the audio and video data to retain the complete audio and video details.

[0057] In one embodiment, when performing key information extraction processing on audio and video data and determining data key frames of the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, the above-mentioned key information extraction module 406 is also used to: convert the audio stream in the audio and video data into a mono mode, and perform signal segmentation processing on the converted audio stream to determine each audio frame contained in the audio and video data; use the preset bidirectional long short-term memory model to perform criticality scoring on each audio frame, determine the audio score corresponding to each audio frame, and determine the frame with an audio score higher than a preset audio score threshold as an audio key frame.

[0058] In one embodiment, the key information extraction module 406 is further used to: perform format conversion processing on the video stream in the audio and video data, and construct multiple video cubes using continuous multiple frames of images in the video stream, wherein the video cube contains image frames of multiple time dimensions; perform criticality scoring on each video cube using a preset 3D convolutional neural network model, and determine the image score corresponding to each image frame; perform statistics on the probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold, and determine the video cube with a probability greater than the preset probability threshold as a video key frame.

[0059] In one embodiment, when sending the real-time network speed to a preset dynamic transcoding model to determine the target compression rate of AV1 encoding, the transcoding transmission module 408 is further used to: obtain a preset compression rate calculation parameter and a basic compression rate when the network speed is 0; and determine the target compression rate as the product of the compression rate calculation parameter and the target real-time network speed and the sum of the basic compression rate.

[0060] In one embodiment, after performing the step of encoding and transmitting the data key frames and the data non-key frames in sequence at a target compression rate, the above-mentioned transcoding transmission module 408 is also used to: after encoding the data key frames and the data non-key frames in sequence, send them to the AV1 decoder for decoding processing through the fast transmission protocol to determine the decoded audio and video data; after encapsulating the decoded audio and video data in the preset transmission format of the transmission channel, transmit them to the external audio and video data receiving end.

[0061] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0062] An embodiment of the present invention provides a server. Specifically, the server includes a processor and a storage device. The storage device stores a computer program, and when the computer program is run by the processor, it executes any one of the methods in the above implementation modes.

[0063] Figure 5 A structural diagram of a server provided in an embodiment of the present invention, the server 100 includes: a processor 50, a memory 51, a bus 52 and a communication interface 53, wherein the processor 50, the communication interface 53 and the memory 51 are connected via the bus 52; the processor 50 is used to execute an executable module stored in the memory 51, such as a computer program.

[0064] The memory 51 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0065] The bus 52 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0066] Among them, the memory 51 is used to store programs, and the processor 50 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 50 or implemented by the processor 50.

[0067] The processor 50 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 50. The above processor 50 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51 and completes the steps of the above method in combination with its hardware.

[0068] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0069] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0070] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for real-time AV1 transcoding of audio and video based on network speed, characterized in that: The method comprises: Acquire audio and video data, and perform integrity check processing on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; If the integrity check of the audio and video data is qualified, the real-time network speed is determined by monitoring the number of bytes of the sent and received packets and the size of the transmitted data packets of the mobile network within a preset time interval, and the network status monitoring result is determined according to the real-time network speed, wherein the network status monitoring result includes: a sufficient network bandwidth state and a limited network bandwidth state; When the mobile network is in the network bandwidth limited state, the audio and video data are subjected to key information extraction processing by a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; The real-time network speed is sent to a preset dynamic transcoding model to determine a target compression rate of AV1 encoding, and the data key frames and the data non-key frames are encoded and transmitted in sequence at the target compression rate, so as to preferentially encode and store key information of the audio and video data.

2. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of performing integrity check processing on the audio and video data by using a preset cyclic redundancy check model includes: Performing a cyclic redundancy check on the audio and video data using a preset cyclic redundancy check model to determine a CRC check code; When the CRC check code is different from the preset check code, it is determined that the audio and video data has been damaged; When the CRC check code is the same as the preset check code, it is determined that the integrity check of the audio and video data is qualified.

3. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 2, characterized in that: After the step of determining that the integrity check of the audio and video data is qualified, the method further comprises: By performing format parsing processing on the header information of the audio and video data, the format information of the audio and video data is extracted, wherein the format information includes: encoding format, resolution and frame rate; The format information is compared with a list of standard formats supported by the system, and when the comparison results are consistent, it is determined that the format compatibility check has passed.

4. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The steps of determining the real-time network speed by monitoring the number of bytes of packets sent and received and the size of the transmitted data packets of the mobile network within a preset time interval include: The first real-time network speed is determined by dividing the difference in the number of bytes of the sent and received packets in adjacent time intervals by the time difference of the adjacent time intervals, and the second real-time network speed is determined by a data transmission test of a standard data packet; The product of the first preset weight and the first real-time network speed and the product of the second preset weight and the second real-time network speed are added together to determine the target real-time network speed.

5. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of determining the network status monitoring result according to the real-time network speed includes: When the real-time network speed is less than the preset network speed threshold, it is determined that the current network is in a network bandwidth-limited state, and the audio and video data are transmitted smoothly by increasing the file compression rate of the audio and video data to reduce the data transmission volume; When the real-time network speed is not less than the preset network speed threshold, it is determined that the current network is in a state of sufficient network bandwidth, and the file compression rate of the audio and video data is reduced to retain complete audio and video details.

6. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of performing key information extraction processing on the audio and video data by using a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model to determine the data key frames of the audio and video data includes: Converting the audio stream in the audio and video data into a mono mode, and performing signal segmentation processing on the converted audio stream to determine each audio frame included in the audio and video data; Using a preset bidirectional long short-term memory model, each of the audio frames is scored for criticality, an audio score corresponding to each of the audio frames is determined, and a frame having an audio score higher than a preset audio score threshold is determined as the audio key frame.

7. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 6, characterized in that: The method further comprises: By performing format conversion processing on the video stream in the audio and video data, multiple video cubes are constructed using continuous multiple frames of images in the video stream, wherein the video cube contains image frames of multiple time dimensions; Using a preset 3D convolutional neural network model, critical scoring is performed on each of the video cubes to determine an image score corresponding to each of the image frames; The probability that the image score corresponding to each image frame in the video cube is higher than a preset image score threshold is counted, and the video cube with a probability greater than the preset probability threshold is determined as the video key frame.

8. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: The step of sending the real-time network speed to a preset dynamic transcoding model to determine a target compression rate of AV1 encoding includes: Get the preset compression ratio calculation parameters and the basic compression ratio when the network speed is 0; The target compression rate is determined as the sum of the product of the compression rate calculation parameter and the target real-time network speed and the basic compression rate.

9. The method for real-time AV1 transcoding of audio and video based on network speed according to claim 1, characterized in that: After the step of sequentially encoding and transmitting the data key frame and the data non-key frame at the target compression rate, the method includes: After encoding the data key frame and the data non-key frame in sequence, the data key frame and the data non-key frame are sent to an AV1 decoder for decoding through a fast transport protocol to determine decoded audio and video data; After the decoded audio and video data is encapsulated in a preset transmission format of the transmission channel, it is transmitted to the external audio and video data receiving end.

10. A real-time AV1 transcoding device for audio and video based on network speed, characterized in that: The device comprises: A data verification module obtains audio and video data and performs integrity verification on the audio and video data through a preset cyclic redundancy check model to ensure that the data is not damaged or lost during transmission; A network speed monitoring module, if the integrity check of the audio and video data is qualified, determines the real-time network speed by monitoring the number of bytes of the received and sent packets and the size of the transmitted data packets of the mobile network within a preset time interval, and determines the network status monitoring result according to the real-time network speed, wherein the network status monitoring result includes: sufficient network bandwidth status and limited network bandwidth status; A key information extraction module, when the mobile network is in the network bandwidth limited state, performs key information extraction processing on the audio and video data through a preset bidirectional long short-term memory model and a preset 3D convolutional neural network model, and determines data key frames and data non-key frames of the audio and video data, wherein the data key frames include: audio key frames and video key frames; The transcoding transmission module sends the real-time network speed to a preset dynamic transcoding model, determines a target compression rate of AV1 encoding, and sequentially encodes and transmits the data key frames and the data non-key frames at the target compression rate, so as to preferentially encode and store key information of the audio and video data.

11. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 9.

Images