Audio and video wireless optimization transmission system and method based on content awareness

Abstract

The invention belongs to the technical field of wireless transmission, and discloses an audio and video wireless optimization transmission system and method based on content awareness. Constructing a wireless transmission platform taking a client and a server as a basic framework, wherein the client comprises a sending end and a receiving end; the method comprises the following steps: acquiring audio and video data needing to be transmitted based on a sending end, and performing frame rate division on the audio and video data to obtain a plurality of single-frame frame rate audios and videos; performing feature extraction on the obtained frame rate audios and videos to obtain corresponding audio and video feature information, and performing key frame marking on the frame rate audios and videos in the corresponding audio and video data based on the audio and video feature information; uploading the audio and video data marked by the key frame to a server, and constructing a corresponding wireless transmission strategy by the server based on the received audio and video data and executing the wireless transmission strategy; the receiving end is used for receiving the transmitted audio and video data, performing integrity verification on the audio and video data, and displaying the audio and video data to a user after the verification is passed; the invention improves the transmission efficiency and optimizes the use of network resources.

Description

Technical Field

[0001] The present invention relates to the field of wireless transmission technology, and more specifically, to a content-aware based audio and video wireless optimization transmission system and method. Background Art

[0002] In the current digital age, wireless transmission of audio and video content has become an indispensable part of people's daily life and work; with the popularity of smartphones, tablets and other mobile devices, users' demand for accessing and sharing audio and video content anytime and anywhere is growing; however, due to the complexity and resource limitations of the wireless network environment itself, traditional audio and video wireless transmission methods face a series of challenges.

[0003] Compared with existing technologies, traditional audio and video wireless transmission methods mostly use fixed coding and unified bandwidth allocation strategies, which are difficult to adapt to real-time bandwidth changes and cannot accurately meet the dynamic transmission needs of audio and video, resulting in idle resources or transmission congestion;

[0004] The inherent characteristics of audio and video data also exacerbate the transmission difficulties; large amounts of data and much redundant information, if transmitted directly without optimization processing, will greatly increase the network load; for example, the massive pixel data of high-definition video and the dense sampling points of high-fidelity audio will quickly exhaust the limited bandwidth if they are not effectively compressed and filtered. In addition, audio and video with different frame rates, resolutions, and content complexity have different transmission requirements. The traditional "one-size-fits-all" model cannot take into account diversity, resulting in the obstruction of key information transmission or excessive resource occupation of secondary content, reducing transmission efficiency and quality;

[0005] Furthermore, the inherent characteristics of wireless channels bring challenges such as high bit error rate and unstable delay. Signals are easily affected by multipath fading and interference noise, resulting in data loss or erroneous reception, affecting the integrity and accuracy of audio and video. Delay jitter destroys the synchronization of audio and video, causing problems such as asynchrony between the picture and sound, freezes and frame skipping, which is particularly prominent in real-time interactive scenarios, greatly reducing user experience satisfaction.

[0006] In view of this, the present invention proposes a content-aware based audio and video wireless optimization transmission system and method to solve the above problems. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0008] The content-aware audio and video wireless optimization transmission method includes:

[0009] Step 1: Building a wireless transmission platform based on a client and a server, wherein the client includes a sending end and a receiving end;

[0010] Step 2: Based on the sending end, the audio and video data to be transmitted is obtained, and the frame rate is divided to obtain several single-frame audio and video frames;

[0011] Step 3: extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information, and based on the feature information, marking key frames of the frame rate audio and video in the corresponding audio and video data;

[0012] Step 4: uploading the audio and video data marked with key frames to the server, and the server constructs and executes a corresponding wireless transmission strategy based on the received audio and video data;

[0013] Step 5: The receiving end is used to receive the transmitted audio and video data, and perform integrity verification on it, and then display it to the user after the verification passes.

[0014] Furthermore, the client is composed of a receiving end and a sending end, and each client is provided with a unique device terminal number, wherein the client corresponding to each user can serve as a receiving end and a sending end; the sending end is used to obtain the audio and video data required to be transmitted by the user, and perform data processing on it; the receiving end is used to receive the audio and video data transmitted by the sending end; the server end is used to obtain the network channel status between the sending end and the receiving end, and formulate and execute corresponding transmission strategies in combination with the audio and video data after data processing.

[0015] Furthermore, based on the transmitting end, the audio and video data to be transmitted is obtained, and the frame rate is divided, and the process of obtaining the frame rate audio and video of several single frames includes:

[0016] The transmitting end is provided with a collection unit, and when the collection unit identifies that an audio and video transmission task is generated in the corresponding transmitting end, the collection unit collects data of the audio and video streams required to be transmitted in the corresponding audio and video transmission task to obtain corresponding audio and video data;

[0017] The collected audio and video data is divided according to the video frame rate to obtain several single-frame audio and video frame rates.

[0018] Furthermore, the process of extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information includes:

[0019] Extract data from the obtained frame rate audio and video to obtain corresponding frame rate audio and frame rate image; at the same time, assign corresponding timestamps to the corresponding frame rate audio and frame rate image based on the acquisition time of the audio and video;

[0020] Then, the obtained frame rate audio is preprocessed to obtain a corresponding frame rate spectrogram;

[0021] According to the time sequence corresponding to the timestamps, the corresponding frame rate spectrogram and frame rate image are input into the pre-built audio and video feature model to obtain the corresponding audio and video feature information.

[0022] Further, the obtained frame rate audio is sampled to obtain a number of data sampling points, and then N1 continuous data sampling points are merged to obtain a number of sampled audios (i.e., the corresponding frame rate audio is divided into unequal audio parts to obtain a number of continuous sampled audios); wherein N1 is a constant, and the total number of data sampling points N1 corresponding to different sampled audios is not exactly the same;

[0023] The obtained sampled audio is subjected to adaptive signal transformation to obtain a corresponding sampled spectrum; the corresponding adaptive signal transformation formula is as follows: Where ni represents the i-th sampled audio corresponding to the n-th frame rate audio; FFT() represents the fast Fourier transform operation; δ n represents the windowing function, which is adaptively selected by the signal frequency and length of the corresponding frame rate audio; P ni represents the sampling spectrum of the i-th sampled audio corresponding to the n-th frame rate audio; X ni Indicates the total number of data sampling points in the i-th sampled audio corresponding to the n-th frame rate audio; * describes the convolution operation;

[0024] Get the spectrum frequency in the corresponding sampling spectrum, perform logarithmic operation on it, and obtain the corresponding logarithmic spectrum line; the formula for logarithmic operation is: Where, F(t) represents the corresponding logarithmic spectrum line; DCT() represents the discrete cosine transform operation, and pf represents the spectrum frequency in the corresponding sampling spectrum; wherein, one spectrum frequency corresponds to one logarithmic spectrum line;

[0025] Furthermore, based on the obtained logarithmic spectrum lines, the spectral coefficients corresponding to the corresponding collected audio are constructed, and the spectral coefficients can be used to capture the spectrum envelope information of the speech signal, reduce the redundant information in the original speech signal, and improve the efficiency and accuracy of the speech recognition system;

[0026] The formula for obtaining the corresponding spectral coefficient is:

[0027] Wherein, i is the index of the sampled audio, a is the logarithmic spectrum corresponding to the corresponding sampled audio, m represents the index of the number of pre-selected filters, and M is the total number of pre-selected filters; SP (i,a) represents the spectral coefficient corresponding to the i-th sampled audio; S(i, m) represents the i-th sampled audio after filtering by the m-th filter;

[0028] The spectral coefficients corresponding to all sampled audios corresponding to the corresponding frame rate audios are obtained, and the corresponding frame rate spectrogram is constructed based on the spectral coefficients.

[0029] Furthermore, the process of constructing the audio and video feature model includes:

[0030] The backbone network of the audio and video feature model is defined as an improved convolutional neural network; the basic architecture of the improved convolutional neural network is an input layer, a convolution layer, a pooling layer, a feature fusion layer and an output layer;

[0031] The input layer is used to receive an input vector and perform image normalization on it so that it meets the model requirements; the input vector includes a frame rate spectrogram and a frame rate image; wherein the image normalization refers to scaling the image size of the input vector so that it meets the model input requirements, and performing image dimension upscaling on the scaled input vector, and the image dimension upscaling refers to mapping the image from the original pixel space to a higher-dimensional feature space, so as to better represent the features of the image;

[0032] The convolution layer uses multi-scale convolution kernels to perform convolution operations on the input vector processed by the input layer to obtain the corresponding audio features and image features;

[0033] The pooling layer is used to perform an adaptive pooling operation on the output vector of the convolutional layer;

[0034] The feature fusion layer is used to perform feature fusion on the audio features and image features after the pooling operation to obtain corresponding audio and video fusion features;

[0035] The output layer is provided with a fully connected layer, which uses the flatten algorithm to expand the audio and video fusion features to meet the dimension requirements of the fully connected layer. Then the fully connected layer is used to reduce the dimension of the data; after the dimensionality reduction process is completed, the result is output;

[0036] Acquire several groups of historical audio and video data and audio and video feature information corresponding to the corresponding audio and video data; and construct a corresponding training data set based on the data; the training data set consists of several training samples;

[0037] The obtained training data set is divided into a training set and a test set, and the corresponding audio and video feature model is iteratively trained based on the training set until the loss function tends to converge, the model parameters are saved, and the corresponding audio and video feature model is verified based on the test set. If the verification passes, the corresponding audio and video feature model is output and put into use. If the verification fails, the iterative training continues until it meets the requirements.

[0038] Furthermore, the operation formula of the convolution operation is: In the formula, Z l1+1 and Z l1They represent the output vector and input vector of the l1+1th convolutional layer respectively; (i, j) represents the dynamic area position during the convolution operation; ω l1+1 refers to the connection weight between layer l1 and layer l1+1, b l1+1 represents the bias term, Represents the convolution operation;

[0039] The formula for the pooling operation is: Where s0 represents the step size, which is used to determine the pooling speed during the pooling operation; p∈(1,∞) is used to indicate the degree of pooling, and when p takes the maximum value, it indicates the maximum pooling operation; It represents the dynamic area position after the pooling operation in the feature map corresponding to the k-th input vector in the l2-th pooling layer, k is a natural number; x, y refer to the horizontal and vertical local offsets in the corresponding feature map; B1 and B2 are natural numbers; they are used for the size of the pooling window in the horizontal and vertical directions during the pooling operation respectively; l2 represents the index of the pooling layer;

[0040] The corresponding feature fusion process includes:

[0041] The audio features and image features completed by the pooling operation are obtained, and two-dimensional convolution processing is performed on them respectively to obtain the corresponding convolution audio Ka and Va and the convolution image Qv; the convolution audio Ka and Qv obtained are subjected to dot product operation to obtain the corresponding dot product weight, and the weight is assigned to the convolution audio Va to obtain the corresponding initial fusion feature Fav; the acquisition formula of the corresponding initial fusion feature is: Where Fv represents the image feature, β represents the learning parameter, softmax is the pre-selected activation function; d represents the feature dimension of the corresponding convolution audio and convolution image; ⊙ represents the dot product operation; T represents the matrix transpose;

[0042] Perform two-dimensional convolution processing on the obtained initial fusion features, and repeat the above initial fusion feature acquisition process to obtain corresponding audio and video fusion features;

[0043] Among them, the formula for two-dimensional convolution processing is: Ka=Fa*Wa; Qv=Fv*Wv; Va=Fa*Wb; Wa, Wv and Wb respectively represent the network weight matrices in the network model training process; Fa represents the audio feature.

[0044] Further, based on the obtained audio and video feature information, the frame rate audio and video in the corresponding audio and video data are key-frame marked, and the process of constructing and executing the corresponding wireless transmission strategy based on the received audio and video data includes:

[0045] Obtain audio and video feature information corresponding to each frame rate audio and video in the corresponding audio and video data; randomly select audio and video feature information corresponding to N2 frame rates of audio and video as the corresponding initial clustering center; N2 is a natural number;

[0046] Obtain the Euclidean distances between other audio and video feature information and the corresponding initial cluster centers respectively; obtain the audio and video feature information with the smallest Euclidean distance from the corresponding initial cluster center, and construct corresponding feature cluster pairs based on the audio and video feature information;

[0047] Obtain the Euclidean distances between different feature cluster pairs, and based on the above cluster pair acquisition process, perform secondary clustering on the corresponding feature cluster pairs, and so on, until several feature cluster sets are obtained;

[0048] Arbitrarily obtain an audio and video feature information in the feature cluster set, and obtain the average value of the Euclidean distance between other frame rate audio and video; and obtain the frame rate audio and video image corresponding to the audio and video feature information with the minimum average value, and perform frame difference operation on it and the adjacent frame rate audio and video image to obtain the corresponding frame difference factor, and determine whether it is a key frame based on the frame difference factor, and if so, mark the corresponding frame rate audio and video as a key frame;

[0049] After the key frame marking is completed, the audio and video data after the key frame marking is encoded and compressed based on a preset second encoding scheme to obtain corresponding second encoded audio; and the audio and video data and the corresponding audio and video transmission task are uploaded to the server;

[0050] When the server receives the corresponding second encoded audio, the server acquires transmission data during the transmission of the corresponding second encoded audio;

[0051] The server obtains the device terminal numbers of the corresponding sending end and the transmitting end respectively based on the received audio and video transmission tasks, and builds a corresponding temporary wireless transmission channel based on the device terminal numbers; and collects the status data of each network channel in the corresponding temporary wireless transmission channel to obtain corresponding channel network data;

[0052] Pre-allocating audio and video network bandwidths for various frame rates in corresponding audio and video data based on the acquired transmission data;

[0053] Based on the obtained transmission data corresponding to the audio of each frame rate and the pre-allocated network bandwidth, and in combination with the corresponding channel network data, network channel allocation is performed to obtain a corresponding initial channel transmission strategy;

[0054] After the allocation is completed, if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is not greater than the pre-allocated network bandwidth, the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set third encoding scheme; if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is greater than the pre-allocated network bandwidth; the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set first encoding scheme;

[0055] After the change is completed, the corresponding initial channel transmission strategy is updated and returned to the sender for execution.

[0056] Furthermore, the receiving end is used to receive the transmitted audio and video data, and perform integrity verification on it. After the verification is passed, the process of displaying it to the user includes:

[0057] When the receiving end identifies that the corresponding temporary wireless transmission channel is constructed, the receiving end constructs a cache node; a cache slot is set in the cache node; when the receiving end receives the corresponding encoded audio, it decodes it, obtains the timestamp in the decoded encoded audio, and verifies whether the same timestamp exists in the existing cache slot in the corresponding cache node, if not, it is stored in the corresponding cache slot and a new cache slot is constructed; if it exists, the corresponding encoded audio is ignored, wherein the encoded audio is one of the first encoded audio, the second encoded audio and the third encoded audio;

[0058] After the transmission is completed, the cache node will merge the decoded encoded audio based on the timestamp and key frame mark to obtain the corresponding first audio and video, and perform frame rate sequence verification and frame rate error verification on it; if there is no corresponding frame rate sequence error and frame rate error verification, the cache node will play the frame rate audio and video in the received first audio and video according to the audio and video before and after the timestamp. If there is an error, the frame rate position of the missing frame rate audio and video is obtained and fed back to the server, and the server retransmits the corresponding frame rate audio and video, and so on.

[0059] The wireless optimized audio and video transmission system based on content perception includes:

[0060] The data acquisition module is used to capture the audio and video data to be transmitted and divide the frame rate thereof to obtain audio and video of several frame rates;

[0061] A data processing module is used to process the obtained frame rate audio and video, and obtain corresponding audio and video feature information based on the data processing result;

[0062] A transmission strategy module, used to mark the key frames of audio and video at each frame rate in the corresponding audio and video data based on the audio and video feature information, and to construct a corresponding channel transmission strategy based on the key frame marking;

[0063] The user display module is used to receive the transmitted audio and video data, and perform integrity verification on it. After the verification is passed, it is displayed to the user.

[0064] The technical effects and advantages of the content-aware audio and video wireless optimization transmission system and method of the present invention are as follows:

[0065] 1. By obtaining the network channel status between the sender and the receiver, and formulating corresponding transmission strategies based on the processed audio and video data, it is possible to achieve reasonable allocation and optimal utilization of network resources and avoid resource waste;

[0066] 2. Through preprocessing, feature extraction and encoding compression of audio and video data, as well as building a temporary wireless transmission channel and allocating network channels on the server side, it can effectively deal with problems such as network fluctuations and packet loss, and improve the robustness and stability of the system;

[0067] 3. By building a wireless transmission platform based on the client and server, and adopting frame rate division, feature extraction and key frame marking technologies, it is possible to effectively compress audio and video data, reduce data transmission volume, and thus improve the efficiency of wireless transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of the content-aware based audio and video wireless optimization transmission method of the present invention;

[0069] Figure 2 It is a schematic diagram of the content-aware based audio and video wireless optimization transmission system of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] See also Figure 1 As shown, the content-aware audio and video wireless optimization transmission method of this embodiment includes:

[0073] Step 1: Build a wireless transmission platform based on the client and server architecture. The client includes the sender and the receiver.

[0074] Step 2: Based on the sending end, the audio and video data to be transmitted is obtained, and the frame rate is divided to obtain several single-frame audio and video frames;

[0075] Step 3: extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information, and based on the feature information, marking key frames of the frame rate audio and video in the corresponding audio and video data;

[0076] Step 4: Upload the key-frame marked audio and video data to the server, and the server constructs and executes the corresponding wireless transmission strategy based on the received audio and video data;

[0077] Step 5: The receiving end is used to receive the transmitted audio and video data, and perform integrity verification on it, and then display it to the user after passing the verification;

[0078] It should be further explained that, in the specific implementation process, the client is composed of a receiving end and a sending end, and each client is set with a unique device terminal number, wherein the client corresponding to each user can be used as a receiving end and a sending end; the sending end is used to obtain the audio and video data that the user needs to transmit, and perform data processing on it; the receiving end is used to receive the audio and video data transmitted by the sending end; the server end is used to obtain the network channel status between the sending end and the receiving end, and formulate and execute corresponding transmission strategies based on the audio and video data after data processing.

[0079] It should be further explained that, in the specific implementation process, the process of obtaining the audio and video data to be transmitted based on the sending end and dividing the frame rate thereof to obtain the frame rate audio and video of several single frames includes:

[0080] The transmitting end is provided with a collection unit. When the collection unit identifies that an audio and video transmission task is generated in the corresponding transmitting end, the collection unit collects data of the audio and video streams required to be transmitted in the corresponding audio and video transmission task to obtain corresponding audio and video data;

[0081] Then, the collected audio and video data are divided according to the video frame rate to obtain several single-frame audio and video frames.

[0082] It should be further explained that, in the specific implementation process, the process of extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information includes:

[0083] Extract data from the obtained frame rate audio and video to obtain corresponding frame rate audio and frame rate image; at the same time, assign corresponding timestamps to the corresponding frame rate audio and frame rate image based on the acquisition time of the audio and video;

[0084] Then, the obtained frame rate audio is preprocessed to obtain a corresponding frame rate spectrogram;

[0085] According to the time sequence corresponding to the timestamps, the corresponding frame rate spectrogram and frame rate image are input into the pre-built audio and video feature model to obtain the corresponding audio and video feature information.

[0086] It should be further explained that, in a specific implementation process, the process of preprocessing the obtained frame rate audio to obtain the corresponding frame rate spectrogram includes:

[0087] The obtained frame rate audio is sampled to obtain a number of data sampling points, and then N1 continuous data sampling points are merged to obtain a number of sampled audios (i.e., the corresponding frame rate audio is divided into unequal parts to obtain a number of continuous sampled audios); wherein N1 is a constant, and the total number of data sampling points N1 corresponding to different sampled audios is not completely the same;

[0088] The obtained sampled audio is subjected to adaptive signal transformation to obtain a corresponding sampled spectrum; the corresponding adaptive signal transformation formula is as follows: Where ni represents the i-th sampled audio corresponding to the n-th frame rate audio; FFT() represents the fast Fourier transform operation; δ n represents the windowing function, which is adaptively selected by the signal frequency and length of the corresponding frame rate audio; P ni represents the sampling spectrum of the i-th sampled audio corresponding to the n-th frame rate audio; X ni Indicates the total number of data sampling points in the i-th sampled audio corresponding to the n-th frame rate audio; * describes the convolution operation;

[0089] Get the spectrum frequency in the corresponding sampling spectrum, perform logarithmic operation on it, and obtain the corresponding logarithmic spectrum line; the formula for logarithmic operation is: Where, F(t) represents the corresponding logarithmic spectrum line; DCT() represents the discrete cosine transform operation, and pf represents the spectrum frequency in the corresponding sampling spectrum; wherein, one spectrum frequency corresponds to one logarithmic spectrum line;

[0090] Furthermore, based on the obtained logarithmic spectrum, the spectral coefficients corresponding to the corresponding collected audio are constructed. The spectral coefficients can be used to capture the spectrum envelope information of the speech signal, reduce the redundant information in the original speech signal, and improve the efficiency and accuracy of the speech recognition system.

[0091] The formula for obtaining the corresponding spectral coefficient is:

[0092] Wherein, i is the index of the sampled audio, a is the logarithmic spectrum corresponding to the corresponding sampled audio, m represents the index of the number of pre-selected filters, and M is the total number of pre-selected filters; SP (i,a)represents the spectral coefficient corresponding to the i-th sampled audio; S(i, m) represents the i-th sampled audio after filtering by the m-th filter;

[0093] Obtaining spectral coefficients corresponding to all sampled audios corresponding to the corresponding frame rate audio, and constructing a corresponding frame rate spectrogram based on the spectral coefficients;

[0094] It should be further explained that, in the specific implementation process, the construction process of the audio and video feature model includes:

[0095] The backbone network of the audio and video feature model is defined as an improved convolutional neural network; the basic architecture of the improved convolutional neural network is an input layer, a convolution layer, a pooling layer, a feature fusion layer, and an output layer;

[0096] The input layer is used to receive input vectors and perform image normalization on them to meet the model requirements; the input vectors include frame rate spectrograms and frame rate images; image normalization refers to scaling the image size of the input vector to meet the model input requirements, and performing image dimension upscaling on the scaled input vector, which refers to mapping the image from the original pixel space to a higher-dimensional feature space to better represent the features of the image;

[0097] The convolution layer uses multi-scale convolution kernels to perform convolution operations on the input vector processed by the input layer to obtain the corresponding audio features and image features;

[0098] The operation formula of the convolution operation is: In the formula, Z l1+1 and Z l1 They represent the output vector and input vector of the l1+1th convolutional layer respectively; (i, j) represents the dynamic area position during the convolution operation; ω l1+1 refers to the connection weight between layer l1 and layer l1+1, b l1+1 represents the bias term, Represents the convolution operation;

[0099] The pooling layer is used to perform adaptive pooling operations on the output vectors of the convolutional layer. The formula for the pooling operation is: Where s0 represents the step size, which is used to determine the pooling speed during the pooling operation; p∈(1,∞) is used to indicate the degree of pooling (i.e., the pooling type), and when p takes the maximum value, it indicates the maximum pooling operation; It represents the dynamic area position after the pooling operation in the feature map corresponding to the k-th input vector in the l2-th pooling layer, k is a natural number; x, y refer to the horizontal and vertical local offsets in the corresponding feature map; B1 and B2 are natural numbers; they are used for the size of the pooling window in the horizontal and vertical directions during the pooling operation respectively; l2 represents the index of the pooling layer;

[0100] The feature fusion layer is used to fuse the audio features and image features after the pooling operation; the corresponding fusion process is:

[0101] The audio features and image features completed by the pooling operation are obtained, and two-dimensional convolution processing is performed on them respectively to obtain the corresponding convolution audio Ka and Va and the convolution image Qv; the convolution audio Ka and Qv obtained are subjected to dot product operation to obtain the corresponding dot product weight, and the weight is assigned to the convolution audio Va to obtain the corresponding initial fusion feature Fav; the acquisition formula of the corresponding initial fusion feature is: Wherein, Fv represents image features, β represents learning parameters, softmax is a pre-selected excitation function; d represents the feature dimension of the corresponding convolution audio and convolution image; ⊙ represents dot product operation; T represents matrix transposition, and the convolution audio and convolution image in the present invention are feature matrices obtained after two-dimensional convolution processing of audio features and image features;

[0102] The formula for two-dimensional convolution processing is: Ka = Fa*Wa; Qv = Fv*Wv; Va = Fa*Wb; Wa, Wv and Wb represent the network weight matrix in the network model training process; Fa represents the audio feature;

[0103] Then, the obtained initial fusion features are subjected to two-dimensional convolution processing, and the above-mentioned initial fusion feature acquisition process is repeated to obtain corresponding audio and video fusion features;

[0104] The output layer is equipped with a fully connected layer, which uses the flatten algorithm to expand the audio and video fusion features to meet the dimension requirements of the fully connected layer. Then the fully connected layer is used to reduce the dimension of the data; after the dimensionality reduction process is completed, the result is output;

[0105] The loss function of the audio and video feature model is defined as Where N3 is the total number of training samples input into the audio and video feature model, h is an index variable used to traverse all training samples, T`(h) is the output of the audio and video feature model corresponding to the h-th training sample, and T(h) is the label of the h-th training sample;

[0106] Acquire several groups of historical audio and video data and audio and video feature information corresponding to the corresponding audio and video data; and construct a corresponding training data set based on the data; the training data set consists of several training samples;

[0107] The obtained training data set is divided into a training set and a test set, and the corresponding audio and video feature model is iteratively trained based on the training set until the loss function tends to converge, the model parameters are saved, and the corresponding audio and video feature model is verified based on the test set. If the verification passes, the corresponding audio and video feature model is output and put into use. If the verification fails, the iterative training continues until it meets the requirements.

[0108] It should be further explained that, in a specific implementation process, the process of marking key frames of the frame rate audio and video in the corresponding audio and video data based on the obtained audio and video feature information includes:

[0109] Obtain audio and video feature information corresponding to audio and video at each frame rate in the corresponding audio and video data; randomly select audio and video feature information corresponding to audio and video at N2 frame rates as the corresponding initial clustering center;

[0110] Obtain the Euclidean distances between other audio and video feature information and the corresponding initial cluster centers respectively; obtain the audio and video feature information with the smallest Euclidean distance from the corresponding initial cluster centers, and construct corresponding feature cluster pairs based on the audio and video feature information; obtain other audio and video feature information except the feature cluster pairs, and repeat the process of obtaining the corresponding feature cluster pairs until all audio and video feature information has corresponding feature cluster pairs; or the Euclidean distance between the audio and video feature information and other audio and video feature information is not less than a preset distance threshold;

[0111] Then, the Euclidean distances between different feature cluster pairs are obtained, and based on the above cluster pair acquisition process, the corresponding feature cluster pairs are clustered again, and so on, until several feature cluster sets are obtained;

[0112] Arbitrarily obtain an audio and video feature information in the feature clustering set, and obtain the average value of the Euclidean distance between other frame rate audio and video; and obtain the frame rate audio and video image corresponding to the audio and video feature information with the minimum average value, and perform frame difference operation on it and the adjacent frame rate audio and video image to obtain the corresponding frame difference factor; if the corresponding frame difference factor is not less than a preset frame difference threshold, it indicates that the corresponding frame rate audio and video is a key frame, and then the corresponding frame rate audio and video is marked as a key frame in the corresponding audio and video data; if the corresponding frame difference factor is less than the preset frame difference threshold, it indicates that the corresponding frame rate audio and video is not a key frame; then obtain the frame rate audio and video corresponding to the audio and video with the second minimum average value, and so on;

[0113] After the key frame marking is completed, the audio and video data after the key frame marking is encoded and compressed based on a preset second encoding scheme to obtain the corresponding second encoded audio; and the audio and video data and the corresponding audio and video transmission task are uploaded to the server.

[0114] It should be further explained that, in the specific implementation process, the process of constructing and executing a corresponding wireless transmission strategy based on the received audio and video data includes:

[0115] When the server receives the corresponding second encoded audio, the server obtains the transmission data of the corresponding second encoded audio during the transmission process; the transmission data includes parameters such as transmission space, transmission rate, transmission delay, and transmission packet loss rate;

[0116] Furthermore, the server obtains the device terminal numbers of the corresponding sending end and the transmitting end respectively based on the received audio and video transmission tasks, and builds the corresponding temporary wireless transmission channel based on them; and collects the status data of each network channel in the corresponding temporary wireless transmission channel to obtain the corresponding channel network data; the channel network data includes parameters such as channel bandwidth and channel delay;

[0117] Based on the obtained transmission data, the audio and video network bandwidth of each frame rate in the corresponding audio and video data is pre-allocated; the formula for pre-allocation is: In the formula, C g C represents the network bandwidth pre-allocated for the g-th frame rate audio and video; total is the total idle network bandwidth in the corresponding temporary wireless transmission channel; R g Y represents the transmission space required for the g-th frame rate audio and video; g represents the audio weight of the g-th frame rate audio and video; it is determined by whether the corresponding frame rate audio is a key frame and the Euclidean distance between it and the nearest key frame; g represents the index of the frame rate audio and video; wherein the constraint condition for the network bandwidth allocation in the above frame rate audio and video is to use the second encoding scheme for encoding compression;

[0118] Then, based on the transmission data corresponding to the obtained audios of each frame rate and the pre-allocated network bandwidth, and in combination with the corresponding channel network data, network channel allocation is performed to obtain the corresponding initial channel transmission strategy;

[0119] After the allocation is completed, if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is not greater than the pre-allocated network bandwidth, the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set third encoding scheme; if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is greater than the pre-allocated network bandwidth; the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set first encoding scheme;

[0120] After the change is completed, the corresponding initial channel transmission strategy is updated and returned to the sender for execution; the updated initial channel transmission strategy is not the final transmission strategy, and the corresponding transmission strategy excludes the frame rate audio and video being transmitted, and the transmission strategies corresponding to the remaining frame rate audio and video will be updated in real time according to the channel network information;

[0121] It should be further explained that, in the specific implementation process, when changing the corresponding coding scheme, it is necessary to consider both the idle bandwidth of the network channel and the idle total network bandwidth in the audio and video wireless transmission channel. If the total network bandwidth is sufficient, the frame rate of audio and video with key frame markers should be increased first.

[0122] Among them, the pre-set first coding scheme, second coding scheme and third coding scheme refer to the coding processing process using different modulation modes and coding methods, wherein the size of the coding processing effect is: the first coding scheme> the second coding scheme> the third coding scheme.

[0123] It should be further explained that, in the specific implementation process, the receiving end is used to receive the transmitted audio and video data, and perform integrity verification on it. After the verification is passed, the process of displaying it to the user includes:

[0124] When the receiving end recognizes that the corresponding temporary wireless transmission channel is constructed, the receiving end constructs a cache node; a cache slot is set in the cache node; when the receiving end receives the corresponding encoded audio, it decodes it, obtains the timestamp in the decoded encoded audio, and verifies whether the same timestamp exists in the existing cache slot in the corresponding cache node, if not, it is stored in the corresponding cache slot and a new cache slot is constructed; if it exists, the corresponding encoded audio is ignored, wherein the encoded audio is one of the first encoded audio, the second encoded audio and the third encoded audio;

[0125] After the transmission is completed, the cache node will merge the decoded encoded audio based on the timestamp and key frame mark to obtain the corresponding first audio and video, and perform frame rate sequence verification and frame rate error verification on it; if there is no corresponding frame rate sequence error and frame rate error verification, the cache node will play the frame rate audio and video in the received first audio and video according to the timestamp before and after. For problems such as buffering and freezing during audio and video playback, the cache center will adaptively select appropriate bit rate stream and resolution and other parameters for playback based on the current network status of the receiving end; users can also set bit rate stream, resolution and other parameters based on the receiving end for local cache playback;

[0126] If there is an error, the frame rate position of the missing frame rate audio and video is obtained and fed back to the server, and the server retransmits the corresponding frame rate audio and video, and so on;

[0127] The frame rate error verification refers to error verification based on a preset automatic repeat request (ARQ) strategy; the automatic repeat request (ARQ) strategy is a prior art, and the inventor will not elaborate on it in detail;

[0128] The present invention constructs a wireless transmission platform; in terms of transmission efficiency, through frame rate division and key frame marking, key information is accurately screened, the transmission volume is reduced, and the coding scheme and channel bandwidth are flexibly allocated according to network conditions to ensure smoothness, reduce peaks and fill valleys, alleviate congestion, and improve overall efficiency and stability; in terms of quality assurance, the feature extraction model deeply mines the audio spectrum and image features and integrates them, key frame screening ensures integrity and continuity, the receiving end verifies and corrects errors, and multiple mechanisms work together to protect, presenting users with an orderly, high-quality audio-visual experience, so that wirelessly transmitted audio and video have both efficiency and quality.

[0129] Example 2

[0130] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides a content-aware audio and video wireless optimization transmission system, including:

[0131] The data acquisition module is used to obtain the audio and video data to be transmitted and divide the frame rate thereof to obtain audio and video of several frame rates;

[0132] A data processing module is used to process the obtained frame rate audio and video, and obtain corresponding audio and video feature information based on the data processing result;

[0133] A transmission strategy module, used to mark the key frames of audio and video at each frame rate in the corresponding audio and video data based on the audio and video feature information, and to construct a corresponding channel transmission strategy based on the key frame marking;

[0134] The user display module is used to receive the transmitted audio and video data, perform integrity verification on it, and display it to the user after passing the verification;

[0135] The modules are connected to each other via wired and / or wireless means to achieve data transmission between the modules.

[0136] Example 3

[0137] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the above-mentioned content-aware-based audio and video wireless optimization transmission system and method is implemented.

[0138] Since the electronic device introduced in this embodiment is an electronic device used to implement the audio and video wireless optimization transmission system and method based on content perception in the embodiment of this application, based on the audio and video wireless optimization transmission system and method based on content perception introduced in the embodiment of this application, the technical personnel of this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not introduced in detail here. As long as the technical personnel of this field implement the electronic device used by the audio and video wireless optimization transmission system and method based on content perception in the embodiment of this application, it belongs to the scope of protection of this application.

[0139] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0140] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A content-aware audio and video wireless optimization transmission method, characterized in that: include: Step 1: Building a wireless transmission platform based on a client and a server, wherein the client includes a sending end and a receiving end; Step 2: Based on the transmitting end, the audio and video data to be transmitted is obtained, and the frame rate is divided to obtain the frame rate audio and video of several single frames; Step 3: extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information, and based on the feature information, marking key frames of the frame rate audio and video in the corresponding audio and video data; Step 4: uploading the audio and video data marked with key frames to the server, and the server constructs and executes a corresponding wireless transmission strategy based on the received audio and video data; Step 5: The receiving end is used to receive the transmitted audio and video data, and perform integrity verification on it, and then display it to the user after the verification passes.

2. The method for wireless optimized transmission of audio and video based on content perception according to claim 1, characterized in that: The client terminal is composed of a receiving terminal and a sending terminal, and each client terminal is provided with a unique device terminal number; The sending end is used to obtain the audio and video data that the user needs to transmit and perform data processing on it; the receiving end is used to receive the audio and video data transmitted by the sending end; the service end is used to obtain the network channel status between the sending end and the receiving end, and formulate and execute corresponding transmission strategies based on the audio and video data after data processing.

3. The method for wireless optimized transmission of audio and video based on content perception according to claim 2 is characterized in that: The process of obtaining the audio and video data to be transmitted based on the sending end and dividing the frame rate thereof to obtain the frame rate audio and video of several single frames includes: The transmitting end is provided with a collection unit, and when the collection unit identifies that an audio and video transmission task is generated in the corresponding transmitting end, the collection unit collects data of the audio and video streams required to be transmitted in the corresponding audio and video transmission task to obtain corresponding audio and video data; The collected audio and video data is divided according to the video frame rate to obtain several single-frame audio and video frame rates.

4. The method for wireless optimized transmission of audio and video based on content perception according to claim 3 is characterized in that: The process of extracting features from the obtained frame rate audio and video to obtain corresponding audio and video feature information includes: Extract data from the obtained frame rate audio and video to obtain corresponding frame rate audio and frame rate image; at the same time, assign corresponding timestamps to the corresponding frame rate audio and frame rate image based on the acquisition time of the audio and video; Preprocessing the obtained frame rate audio to obtain a corresponding frame rate spectrogram; According to the time sequence corresponding to the timestamps, the corresponding frame rate spectrogram and frame rate image are input into the pre-built audio and video feature model to obtain the corresponding audio and video feature information.

5. The method for wireless optimized transmission of audio and video based on content perception according to claim 4 is characterized in that: The process of preprocessing the obtained frame rate audio includes: Performing unequal audio division on the corresponding frame rate audio to obtain a plurality of continuous sampled audios, wherein the sampled audios are composed of a plurality of continuous data sampling points; The obtained sampled audio is subjected to adaptive signal transformation to obtain a corresponding sampled spectrum; the corresponding adaptive signal transformation formula is as follows: Where ni represents the i-th sampled audio corresponding to the n-th frame rate audio; FFT() represents the fast Fourier transform operation; δ n represents the windowing function; P ni represents the sampling spectrum of the i-th sampled audio corresponding to the n-th frame rate audio; X ni Indicates the total number of data sampling points in the i-th sampled audio corresponding to the n-th frame rate audio; * describes the convolution operation; Get the spectrum frequency in the corresponding sampling spectrum, perform logarithmic operation on it, and obtain the corresponding logarithmic spectrum line; the formula for logarithmic operation is: Where, F(t) represents the corresponding logarithmic spectrum; DCT() represents the discrete cosine transform operation, and pf represents the spectrum frequency in the corresponding sampled spectrum; The spectral coefficients corresponding to the corresponding collected audio are constructed based on the obtained logarithmic spectrum lines. The formula for obtaining the corresponding spectral coefficients is: Wherein, i is the index of the sampled audio, a is the logarithmic spectrum corresponding to the corresponding sampled audio, m represents the index of the number of pre-selected filters, and M is the total number of pre-selected filters; SP (i,a) represents the spectral coefficient corresponding to the i-th sampled audio; S(i, m) represents the i-th sampled audio after filtering by the m-th filter; The spectral coefficients corresponding to all sampled audios corresponding to the corresponding frame rate audios are obtained, and the corresponding frame rate spectrogram is constructed based on the spectral coefficients.

6. The method for wireless optimized transmission of audio and video based on content perception according to claim 4, characterized in that: The construction process of the audio and video feature model includes: The backbone network of the audio and video feature model is defined as an improved convolutional neural network; the basic architecture of the improved convolutional neural network is an input layer, a convolution layer, a pooling layer, a feature fusion layer and an output layer; The input layer is used to receive the input vector and perform image normalization on it; The convolution layer uses a multi-scale convolution kernel to perform a convolution operation on the input vector processed by the input layer to obtain the corresponding audio features and image features; The pooling layer is used to perform an adaptive pooling operation on the output vector of the convolutional layer; The feature fusion layer is used to perform feature fusion on the audio features and image features after the pooling operation to obtain corresponding audio and video fusion features; The output layer is provided with a fully connected layer, which uses a flatten algorithm to expand the audio and video fusion features, and then uses the fully connected layer to perform dimensionality reduction processing on the data; after the dimensionality reduction processing is completed, the result is output; Acquire several groups of historical audio and video data and audio and video feature information corresponding to the corresponding audio and video data; and construct a corresponding training data set based on the data; the training data set consists of several training samples; The obtained training data set is divided into a training set and a test set, and the corresponding audio and video feature model is iteratively trained based on the training set until the loss function tends to converge, the model parameters are saved, and the corresponding audio and video feature model is verified based on the test set. If the verification passes, the corresponding audio and video feature model is output and put into use. If the verification fails, the iterative training continues until it meets the requirements.

7. The method for wireless optimized transmission of audio and video based on content perception according to claim 6, characterized in that: The operation formula of the convolution operation is: In the formula, z l1+1 and z l1 They represent the output vector and input vector of the l1+1th convolutional layer respectively; (i, j) represents the dynamic area position during the convolution operation; ω l1+1 refers to the connection weight between layer l1 and layer l1+1, b l1+1 represents the bias term, Represents the convolution operation; The formula for the pooling operation is: In the formula, s0 represents the step length; represents the dynamic area position after pooling in the feature map corresponding to the k-th input vector in the l2-th pooling layer, where k is a natural number; x and y refer to the horizontal and vertical local offsets in the corresponding feature map; B1 and B2 are used for the size of the pooling window in the horizontal and vertical directions during the pooling operation, respectively; l2 represents the index of the pooling layer; p∈(1,∞), used to indicate the degree of pooling; The corresponding feature fusion process includes: The audio features and image features completed by the pooling operation are obtained, and two-dimensional convolution processing is performed on them respectively to obtain the corresponding convolution audio Ka and Va and the convolution image Qv; the convolution audio Ka and Qv obtained are subjected to dot product operation to obtain the corresponding dot product weight, and the weight is assigned to the convolution audio Va to obtain the corresponding initial fusion feature Fav; the acquisition formula of the corresponding initial fusion feature is: Where Fv represents the image feature, β represents the learning parameter, softmax is the pre-selected activation function; d represents the feature dimension of the corresponding convolution audio and convolution image; ⊙ represents the dot product operation; T represents the matrix transpose; Perform two-dimensional convolution processing on the obtained initial fusion features, and repeat the above initial fusion feature acquisition process to obtain corresponding audio and video fusion features; Among them, the formula for two-dimensional convolution processing is: Ka=Fa*Wa; Qv=Fv*Wv; Va=Fa*Wb; Wa, Wv and Wb respectively represent the network weight matrices in the network model training process; Fa is the audio feature.

8. The method for wireless optimized transmission of audio and video based on content perception according to claim 7, characterized in that: The process of marking the frame rate audio and video in the corresponding audio and video data as key frames based on the obtained audio and video feature information, and constructing and executing the corresponding wireless transmission strategy based on the received audio and video data includes: Obtain audio and video feature information corresponding to each frame rate audio and video in the corresponding audio and video data; randomly select audio and video feature information corresponding to N2 frame rates of audio and video as the corresponding initial clustering center; N2 is a natural number; Obtain the Euclidean distances between other audio and video feature information and the corresponding initial cluster centers respectively; obtain the audio and video feature information with the smallest Euclidean distance from the corresponding initial cluster center, and construct corresponding feature cluster pairs based on the audio and video feature information; Obtain the Euclidean distances between different feature cluster pairs, and based on the above cluster pair acquisition process, perform secondary clustering on the corresponding feature cluster pairs, and so on, until several feature cluster sets are obtained; Arbitrarily obtain an audio and video feature information in the feature cluster set, and obtain the average value of the Euclidean distance between other frame rate audio and video; and obtain the frame rate audio and video image corresponding to the audio and video feature information with the minimum average value, and perform frame difference operation on it and the adjacent frame rate audio and video image to obtain the corresponding frame difference factor, and determine whether it is a key frame based on the frame difference factor, and if so, mark the corresponding frame rate audio and video as a key frame; After the key frame marking is completed, the audio and video data after the key frame marking is encoded and compressed based on a preset second encoding scheme to obtain corresponding second encoded audio; and the audio and video data and the corresponding audio and video transmission task are uploaded to the server; When the server receives the corresponding second encoded audio, the server acquires transmission data during the transmission of the corresponding second encoded audio; The server obtains the device terminal numbers of the corresponding sending end and the transmitting end respectively based on the received audio and video transmission tasks, and builds a corresponding temporary wireless transmission channel based on the device terminal numbers; and collects the status data of each network channel in the corresponding temporary wireless transmission channel to obtain corresponding channel network data; Pre-allocating audio and video network bandwidths for various frame rates in corresponding audio and video data based on the acquired transmission data; Based on the obtained transmission data corresponding to the audio of each frame rate and the pre-allocated network bandwidth, and in combination with the corresponding channel network data, network channel allocation is performed to obtain a corresponding initial channel transmission strategy; After the allocation is completed, if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is not greater than the pre-allocated network bandwidth, the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set third encoding scheme; if the idle network bandwidth of the network channel allocated to the corresponding frame rate audio and video is greater than the pre-allocated network bandwidth; the encoding compression scheme of the corresponding frame rate audio and video is changed to the pre-set first encoding scheme; After the change is completed, the corresponding initial channel transmission strategy is updated and returned to the sender for execution.

9. The method for wireless optimized transmission of audio and video based on content perception according to claim 8, characterized in that: The receiving end is used to receive the transmitted audio and video data, and verify its integrity. After the verification is passed, the process of displaying it to the user includes: When the receiving end identifies that the corresponding temporary wireless transmission channel is constructed, the receiving end constructs a cache node; a cache slot is set in the cache node; when the receiving end receives the corresponding encoded audio, it decodes it, obtains the timestamp in the decoded encoded audio, and verifies whether the same timestamp exists in the existing cache slot in the corresponding cache node, if not, it is stored in the corresponding cache slot and a new cache slot is constructed; if it exists, the corresponding encoded audio is ignored, wherein the encoded audio is one of the first encoded audio, the second encoded audio and the third encoded audio; After the transmission is completed, the cache node will merge the decoded encoded audio based on the timestamp and key frame mark to obtain the corresponding first audio and video, and perform frame rate sequence verification and frame rate error verification on it; if there is no corresponding frame rate sequence error and frame rate error verification, the cache node will play the frame rate audio and video in the received first audio and video according to the audio and video before and after the timestamp. If there is an error, the frame rate position of the missing frame rate audio and video is obtained and fed back to the server, and the server retransmits the corresponding frame rate audio and video, and so on.

10. A content-aware audio and video wireless optimization transmission system, which is used to implement the content-aware audio and video wireless optimization transmission method according to any one of claims 1 to 9, characterized in that: include: The data acquisition module is used to obtain the audio and video data to be transmitted and divide the frame rate thereof to obtain audio and video of several frame rates; A data processing module is used to process the obtained frame rate audio and video, and obtain corresponding audio and video feature information based on the data processing result; A transmission strategy module, used to mark the key frames of audio and video at each frame rate in the corresponding audio and video data based on the audio and video feature information, and to construct a corresponding channel transmission strategy based on the key frame marking; The user display module is used to receive the transmitted audio and video data, and perform integrity verification on it. After the verification is passed, it is displayed to the user.

Images