Video transmission method, electronic device and computer storage medium

Abstract

The present disclosure provides a video transmission method, comprising: utilizing a final encoding network in a pre-trained autoencoding network to encode prediction residual data of a video frame to be transmitted to obtain video feature data; the autoencoding network includes the final encoding network and a final decoding network; and transmitting the video feature data to a decoding device. This method improves both the video transmission bitrate and the encoding and decoding accuracy of video frames, while also having the advantages of low computational effort, low computational complexity, and short encoding and decoding times. The present disclosure also provides an electronic device and a computer storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of image communication, and in particular to a video transmission method, an electronic device and a computer storage medium. Background Art

[0002] With the advancement of video technology and display hardware performance, high resolution has become the future direction of video technology development. To efficiently and quickly transmit high-resolution video, traditional video codec technologies typically perform efficient intra-frame and inter-frame prediction on video data to obtain prediction residual data. To reduce the amount of video encoding data, that is, to reduce data redundancy, the prediction residual data is further subjected to discrete cosine or sine transforms to improve the data energy distribution, thereby achieving data sparsification. The transformed data is then quantized to effectively reduce the data value space and achieve better compression effects. Finally, the data is dequantized and inversely transformed to achieve data recovery.

[0003] However, the traditional video coding and decoding technology, which first transforms the prediction residual data, then quantizes the transformed data, and finally dequantizes and inversely transforms the data, has many disadvantages, such as low data sparsity, large amount of calculation, and long encoding time.

[0004] Specifically, during the transformation and quantization processes, discrete cosine transforms, sine transforms, and inverse transforms are often performed using orthogonal matrices. This does not inherently reduce data accuracy, but its effectiveness in improving the energy distribution of data is limited, meaning that data sparsification still has room for improvement. Quantization, the process of mapping continuous values ​​to a finite number of discrete amplitudes, can improve data compression, but excessively large quantization amplitudes can lead to increased data loss. Consequently, there is a conflict between reducing bitrate and improving accuracy. Currently, rate-distortion optimization (RDO) quantization is employed to address this conflict, improving encoding performance. However, this quantization requires traversing multiple optional quantization values ​​and incurring a rate-distortion penalty. This computationally intensive approach is not conducive to fast hardware implementation and can result in extended encoding times. Therefore, improvements to video encoding and decoding technologies are needed. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present disclosure provides a video transmission method, an electronic device and a computer storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a video transmission method, the method comprising:

[0007] Using a final encoding network in a pre-trained autoencoding network, encoding prediction residual data of a video frame to be transmitted to obtain video feature data, the autoencoding network including the final encoding network and the final decoding network;

[0008] The video feature data is transmitted to a decoding end device.

[0009] In some embodiments, the method further comprises the step of training the autoencoder network:

[0010] Adjust the initial network according to the transmission scenario parameters of the sample video frame to obtain the intermediate network;

[0011] The intermediate network is trained according to the prediction residual data of the sample video frame to obtain the autoencoding network; wherein the transmission scene parameters of the sample video frame are the same as the transmission scene parameters of the video frame to be transmitted.

[0012] In some embodiments, the initial network includes an initial encoding network and an initial decoding network, and the intermediate network includes an intermediate encoding network and an intermediate decoding network; and training the intermediate network according to the prediction residual data of the sample video frame to obtain the autoencoding network includes:

[0013] Inputting the prediction residual data of the sample video frame into the intermediate coding network to obtain intermediate feature data output by the intermediate coding network;

[0014] Inputting the intermediate feature data into the intermediate decoding network to obtain intermediate prediction residual data output by the intermediate decoding network;

[0015] When the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data do not meet the preset optimization stop condition, the parameters of the intermediate network are adjusted until the prediction residual data of the sample video frame and the intermediate feature data and intermediate prediction residual data obtained after adjustment meet the preset optimization stop condition.

[0016] In some embodiments, the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data satisfying a preset optimization stop condition includes:

[0017] The data dimension of the intermediate feature data is smaller than a preset dimension threshold, and the average variance between the prediction residual data of the sample video frame and the intermediate prediction residual data is smaller than a preset variance threshold.

[0018] In some embodiments, the initial encoding network includes an initial encoding input layer, an initial encoding output layer, and multiple initial encoding intermediate layers; the initial decoding network includes an initial decoding input layer, multiple initial decoding intermediate layers, an initial decoding output layer, and multiple initial decoding intermediate layers; the intermediate encoding network includes an intermediate encoding input layer, an intermediate encoding output layer, and multiple intermediate encoding intermediate layers; the intermediate decoding network includes an intermediate decoding input layer, an intermediate decoding output layer, and multiple intermediate decoding intermediate layers; the final encoding network includes a final encoding input layer, a final encoding output layer, and multiple final encoding intermediate layers; the final decoding network includes a final decoding input layer, a final decoding output layer, and multiple final decoding intermediate layers;

[0019] The types of the initial encoding intermediate layer, the initial decoding intermediate layer, the intermediate encoding intermediate layer, the intermediate decoding intermediate layer, the final encoding intermediate layer, and the final decoding intermediate layer all include the following three types: convolutional layer, pooling layer, and activation layer;

[0020] The type of the initial encoding output layer, the type of the initial decoding input layer, the type of the intermediate encoding output layer, the type of the intermediate decoding input layer, the type of the final encoding output layer, and the type of the final decoding input layer are all fully connected layers;

[0021] The type of the initial encoding input layer, the type of the initial decoding output layer, the type of the intermediate encoding input layer, the type of the intermediate decoding output layer, the type of the final encoding input layer, and the type of the final decoding output layer are all the same.

[0022] In some embodiments, adjusting the parameters of the intermediate network includes:

[0023] The number of the intermediate encoding intermediate layers and the number of the intermediate decoding intermediate layers are adjusted, as well as the number of neurons in the intermediate encoding intermediate layers and the number of neurons in the intermediate decoding intermediate layers.

[0024] In some embodiments, the transmission scenario parameters include a scene category identifier and a transmission bandwidth; and adjusting the initial network according to the transmission scenario parameters of the sample video frame to obtain the intermediate network includes:

[0025] According to the scene category identifier and the transmission bandwidth, the number of the initial coding intermediate layers and the number of the initial decoding intermediate layers are adjusted, and the dimension of the output data of the initial coding output layer is adjusted.

[0026] In a second aspect, an embodiment of the present disclosure provides a video transmission method, the method comprising:

[0027] Receiving video feature data transmitted by an encoding end device, wherein the video feature data is obtained by the encoding end device encoding prediction residual data of a to-be-transmitted video frame using a final encoding network in a pre-trained autoencoding network;

[0028] Decoding the video feature data using a final decoding network in the autoencoder network to obtain decoded prediction residual data;

[0029] A transmitted video frame is determined according to the decoded prediction residual data.

[0030] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0031] one or more processors;

[0032] a storage device having one or more programs stored thereon;

[0033] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the video transmission method as described above.

[0034] In a fourth aspect, an embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon, which implements the video transmission method as described above when the program is executed.

[0035] The encoding end device uses the final encoding network in the pre-trained self-encoding network to encode the prediction residual data of the video frame to be transmitted to obtain video feature data, and transmits the video feature data to the decoding end device, so that the decoding end device can use the final decoding network in the self-encoding network to decode the video feature data and obtain the transmitted video frame. Compared with traditional video encoding and decoding technology, this video transmission method can not only improve the video transmission bit rate but also improve the video frame encoding and decoding accuracy. It also has the advantages of small computational amount, low computational complexity and short encoding and decoding time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a video transmission method applied to an encoding terminal device provided by an embodiment of the present disclosure;

[0037] Figure 2 This is a schematic diagram of the process of training the autoencoding network provided by the embodiment of the present disclosure. Figure 1 ;

[0038] Figure 3 This is a schematic diagram of the process of training the autoencoding network provided by the embodiment of the present disclosure. Figure 2 ;

[0039] Figure 4 This is a flow chart of a video transmission method applied to a decoding end device provided by an embodiment of the present disclosure;

[0040] Figure 5 Schematic diagram of the principle of the video transmission method provided by the embodiment of the present disclosure;

[0041] Figure 6 This is a schematic diagram of an autoencoding network provided by an embodiment of the present disclosure;

[0042] Figure 7 This is another schematic diagram of an autoencoding network provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.

[0044] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0045] The terms used herein are used only to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof is not excluded.

[0046] The embodiments described herein may be described with reference to plan views and / or cross-sectional views, with the aid of idealized schematic diagrams of the present disclosure. Thus, the example illustrations may be modified based on manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings are schematic in nature, and the shapes of the regions shown in the drawings illustrate specific shapes of the regions of the elements, but are not intended to be limiting.

[0047] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0048] In order to transmit high-resolution video, traditional video encoding and decoding technology usually first transforms and quantizes the video frames by the encoding end device, and then dequantizes and inversely transforms the data by the decoding end device. However, this process has many disadvantages, such as the contradiction between reducing the bit rate and improving the accuracy, large amount of calculation, and long encoding and decoding time. In view of this, the embodiment of the present disclosure proposes that the prediction residual data of the sample video frame can be used to train a self-encoding network that meets the bit rate and accuracy requirements, including the final encoding network and the final decoding network. The encoding end device and the decoding end device then use the final encoding network and the final decoding network to process the video data respectively. Compared with traditional video encoding and decoding technology, this method has the advantages of reducing the bit rate and improving the accuracy, small amount of calculation, and short encoding and decoding time.

[0049] Correspondingly, such as Figure 1 As shown, the embodiment of the present disclosure provides a video transmission method, which may include the following steps:

[0050] In step S11, the prediction residual data of the video frame to be transmitted is encoded using the final encoding network in the pre-trained autoencoding network to obtain video feature data, wherein the autoencoding network includes the final encoding network and the final decoding network;

[0051] In step S12, the video feature data is transmitted to a decoding end device.

[0052] Among them, the predicted residual data is the difference between the actual value and the predicted value. The actual value refers to the actual pixel value of the video frame to be transmitted, and the predicted value refers to the predicted pixel value obtained by predicting the pixel value of the video frame to be transmitted using the encoded video frame. Determining the predicted residual data of the video frame belongs to the existing technology and will not be repeated here.

[0053] In the disclosed embodiment, steps S11 and S12 are performed by the encoding device, while the step of pre-training the autoencoding network can be performed by either the encoding device or the decoding device, or by a device other than the encoding device and the decoding device. However, after the autoencoding network is trained, the encoding device at least stores the final encoding network of the autoencoding network, and the decoding device at least stores the final decoding network of the autoencoding network.

[0054] After transmitting the video feature data to the decoding end device, the decoding end device uses the final decoding network in the self-encoding network to decode the video feature data to obtain decoded prediction residual data, and determines the transmitted video frame based on the decoded prediction residual data. The transmitted video frame refers to the video frame restored by the decoding end device, which is consistent with the video frame to be transmitted, or the difference between the video frame to be transmitted and the video frame to be transmitted is very small. In other words, what the encoding end device transmits to the decoding end device is the encoded video frame to be transmitted, rather than directly transmitting the video frame to be transmitted to the decoding end device. The decoding end device decodes the video feature data to obtain the restored video frame to be transmitted, that is, the transmitted video frame, rather than directly receiving the video frame to be transmitted transmitted by the encoding end device.

[0055] It can be seen from the above steps S11-S12 that, using the video transmission method provided by the embodiment of the present disclosure, the encoding end device encodes the prediction residual data of the video frame to be transmitted by using the final encoding network in the pre-trained self-encoding network to obtain video feature data, and transmits the video feature data to the decoding end device, so that the decoding end device can use the final decoding network in the self-encoding network to decode the video feature data and perform other processing to obtain the transmitted video frame. Compared with traditional video encoding and decoding technology, this video transmission method can not only improve the video transmission bit rate but also improve the encoding and decoding accuracy of the video frame, and also has the advantages of small computational amount, low computational complexity and short encoding and decoding time.

[0056] In the process of pre-training the autoencoder network, the initial network can be initialized and adjusted to obtain an intermediate network, and then the prediction residual data of the sample video frame can be used to train the intermediate network to obtain the autoencoder network. Figure 2 As shown, the method further includes the step of training to obtain the autoencoding network:

[0057] In step S21, the initial network is adjusted according to the transmission scenario parameters of the sample video frame to obtain an intermediate network;

[0058] In step S22, the intermediate network is trained according to the prediction residual data of the sample video frame to obtain the autoencoding network; wherein the transmission scene parameters of the sample video frame are the same as the transmission scene parameters of the video frame to be transmitted.

[0059] In the disclosed embodiments, different autoencoder networks can be trained based on different transmission scenario parameters. That is, when transmitting a video frame with certain transmission scenario parameters, the autoencoder network used is trained using sample video frames with the same transmission scenario parameters. When transmitting a video frame with different transmission scenario parameters, a different autoencoder network is used.

[0060] The intermediate network is obtained by optimizing the initial network according to the transmission scenario parameters of the sample video frame, and the autoencoding network is obtained by optimizing the intermediate network according to the prediction residual data of the sample video frame. The initial network, the intermediate network, and the autoencoding network can all include two network parts: a network for encoding processing and a network for decoding processing. Accordingly, in some embodiments, the initial network includes an initial encoding network and an initial decoding network, and the intermediate network includes an intermediate encoding network and an intermediate decoding network; Figure 3 As shown, the training of the intermediate network based on the prediction residual data of the sample video frame to obtain the autoencoder network (i.e., step S22) may include the following steps:

[0061] In step S221, the prediction residual data of the sample video frame is input into the intermediate coding network to obtain intermediate feature data output by the intermediate coding network;

[0062] In step S222, the intermediate feature data is input into the intermediate decoding network to obtain intermediate prediction residual data output by the intermediate decoding network;

[0063] In step S223, when the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data do not meet the preset optimization stop condition, the parameters of the intermediate network are adjusted until the prediction residual data of the sample video frame and the intermediate feature data and intermediate prediction residual data obtained after adjustment meet the preset optimization stop condition.

[0064] It should be understood that if the prediction residual data of the sample video frame is first input into the intermediate encoding network to obtain intermediate feature data and then the intermediate feature data is input into the intermediate decoding network to obtain intermediate prediction residual data, the prediction residual data, intermediate feature data and intermediate prediction residual data of the sample video frame have met the preset optimization stop condition, then step S223 does not need to be executed, and the intermediate network can be directly used as the self-encoding network.

[0065] If the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data do not meet the preset optimization stop condition, the parameters of the intermediate network are adjusted. After adjusting the parameters of the intermediate network, steps S221 and S222 are continued. Specifically, the prediction residual data of the sample video frame is input into the intermediate encoding network after the parameter adjustment to obtain intermediate prediction residual data output by the intermediate encoding network after the parameter adjustment. The intermediate prediction residual data is input into the intermediate decoding network after the parameter adjustment to obtain intermediate prediction residual data output by the intermediate decoding network after the parameter adjustment. It is then determined again whether the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data obtained after the parameter adjustment meet the preset optimization stop condition. If so, adjusting the parameters of the intermediate network is stopped, and the current intermediate network is determined to be an autoencoder network. If not, adjusting the parameters of the intermediate network is continued, and steps S221 and S222 are repeated until the prediction residual data of the sample video frame, the adjusted intermediate feature data, and the intermediate prediction residual data meet the preset optimization stop condition.

[0066] Specifically, satisfying the preset optimization stop condition may include two aspects. On the one hand, the intermediate coding network at this time has reduced the bit rate to a sufficiently low level, which can be specifically reflected in that the dimension of the output data of the intermediate coding network is less than the preset dimension threshold; on the other hand, the intermediate decoding network at this time has been able to restore the video frame to be transmitted to a certain extent, that is, the restoration degree of the intermediate decoding network is already high enough, which can be specifically reflected in that the average variance between the predicted residual data of the sample video frame and the intermediate predicted residual data is less than the preset variance threshold. At this time, the average variance between the predicted residual data of the sample video frame and the intermediate predicted residual data is used as the loss function. The smaller the value of the loss function, the higher the restoration degree. When the loss function is already less than the preset variance threshold, it means that the restoration degree is already high enough.

[0067] Accordingly, in some embodiments, the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data satisfying the preset optimization stop condition (i.e., described in step S223) may include: the data dimension of the intermediate feature data is less than the preset dimension threshold, and the average variance between the prediction residual data of the sample video frame and the intermediate prediction residual data is less than the preset variance threshold.

[0068] The intermediate feature data is typically expressed as a feature vector, whose data dimension is the number of elements in the pointer. When the preset optimization stopping conditions are met, it indicates that the autoencoder network has been able to compress the bitrate of the video frame to be transmitted to a sufficiently low level and also generate prediction residual data with a sufficiently high degree of restoration (compared to the prediction residual data of the video frame to be transmitted) based on the intermediate feature data.

[0069] In some embodiments, the initial encoding network includes an initial encoding input layer, an initial encoding output layer, and multiple initial encoding intermediate layers; the initial decoding network includes an initial decoding input layer, multiple initial decoding intermediate layers, an initial decoding output layer, and multiple initial decoding intermediate layers; the intermediate encoding network includes an intermediate encoding input layer, an intermediate encoding output layer, and multiple intermediate encoding intermediate layers; the intermediate decoding network includes an intermediate decoding input layer, an intermediate decoding output layer, and multiple intermediate decoding intermediate layers; the final encoding network includes a final encoding input layer, a final encoding output layer, and multiple final encoding intermediate layers; the final decoding network includes a final decoding input layer, a final decoding output layer, and multiple final decoding intermediate layers;

[0070] The types of the initial encoding intermediate layer, the initial decoding intermediate layer, the intermediate encoding intermediate layer, the intermediate decoding intermediate layer, the final encoding intermediate layer, and the final decoding intermediate layer all include the following three types: convolutional layer, pooling layer, and activation layer;

[0071] The type of the initial encoding output layer, the type of the initial decoding input layer, the type of the intermediate encoding output layer, the type of the intermediate decoding input layer, the type of the final encoding output layer, and the type of the final decoding input layer are all fully connected layers;

[0072] The type of the initial encoding input layer, the type of the initial decoding output layer, the type of the intermediate encoding input layer, the type of the intermediate decoding output layer, the type of the final encoding input layer, and the type of the final decoding output layer are all the same.

[0073] Accordingly, in some embodiments, adjusting the parameters of the intermediate network (i.e., as described in step S223) may include the following steps: adjusting the number of the intermediate encoding intermediate layers and the number of the intermediate decoding intermediate layers, and adjusting the number of neurons in the intermediate encoding intermediate layer and the number of neurons in the intermediate decoding intermediate layer.

[0074] As shown above, the intermediate network includes an intermediate encoding network and an intermediate decoding network. The intermediate encoding network includes multiple intermediate encoding intermediate layers, and the intermediate decoding network also includes multiple intermediate decoding intermediate layers. The types of intermediate encoding intermediate layers and intermediate decoding intermediate layers both include convolutional layers, pooling layers, and activation layers. Therefore, when adjusting the parameters of the intermediate network, the number of intermediate decoding intermediate layers and the number of intermediate decoding intermediate layers can be adjusted, that is, the number of convolutional layers, pooling layers, and activation layers specifically included in the intermediate encoding network can be determined, and the number of convolutional layers, pooling layers, and activation layers specifically included in the intermediate decoding network can be determined. In addition, each layer includes multiple neurons, and the number of neurons in each intermediate encoding intermediate layer and the number of neurons in each intermediate decoding intermediate layer can also be adjusted.

[0075] In some embodiments, the transmission scene parameters include a scene category identifier and a transmission bandwidth; adjusting the initial network according to the transmission scene parameters of the sample video frame to obtain an intermediate network (i.e., step S21) may include the following steps: adjusting the number of the initial encoding intermediate layers and the number of the initial decoding intermediate layers according to the scene category identifier and the transmission bandwidth, and adjusting the dimension of the output data of the initial encoding output layer.

[0076] Among them, the scene category identifier is used to characterize the application scenario of the video frame, and the application scenario may include environmental monitoring, video browsing services, video communication, etc. The transmission bandwidth refers to the rated bandwidth when transmitting the video frame. By pre-configuring the mapping relationship between the transmission scenario parameters and the parameters of the initial network, the initial network can be adjusted according to different transmission scenario parameters to obtain different intermediate networks. The parameters of the initial network refer to the number of initial encoding intermediate layers, the number of initial decoding intermediate layers, and the dimension of the output data of the initial encoding output layer. The dimension of the output data of the initial encoding output layer refers to the dimension of the initial feature data output by the initial encoding output layer.

[0077] The traditional video encoding and decoding process includes multiple sub-processes. The embodiment of the present disclosure adopts an autoencoding network to replace the entire traditional video encoding and decoding process, which is also conducive to simplifying the deployment process of hardware and software.

[0078] Furthermore, after the autoencoding network is trained, the prediction residual data of the sample video frame can be processed using traditional video coding and decoding technologies, namely traditional transform processing, quantization processing, inverse quantization processing and inverse transform processing, to obtain restored prediction residual data. The encoding bit rate of the autoencoding network (i.e., the dimension of the final encoding network output data) is compared with the encoding bit rate of the traditional video coding and decoding technology (i.e., the dimension of the data after transform processing and quantization processing), and the PSNR (Peak Signal to Noise Ratio) value of the sample video frame restored using the autoencoding network is compared with the PSNR value of the sample video frame restored using the traditional video coding and decoding technology. The parameters of the autoencoding network are further adjusted based on the comparison results.

[0079] like Figure 4 As shown, when the video transmission method provided by the embodiment of the present disclosure is applied to a decoding end device, it may include the following steps:

[0080] In step S31, video feature data transmitted by an encoding end device is received, wherein the video feature data is obtained by the encoding end device encoding prediction residual data of a video frame to be transmitted using a final encoding network in a pre-trained autoencoding network;

[0081] In step S32, the video feature data is decoded using the final decoding network in the autoencoder network to obtain decoded prediction residual data;

[0082] In step S33, a transmitted video frame is determined according to the decoded prediction residual data.

[0083] It can be seen from the above steps S31-S33 that, using the video transmission method provided by the embodiment of the present disclosure, the decoding end device receives the video feature data transmitted by the encoding end device; wherein, the video feature data is obtained by the encoding end device using the final encoding network in the pre-trained self-encoding network to encode the prediction residual data of the video frame to be transmitted; the decoding end device uses the final decoding network in the self-encoding network to decode the video feature data to obtain the decoded prediction residual data. Compared with traditional video encoding and decoding technologies, this video transmission method can not only improve the video transmission bit rate but also improve the video frame encoding and decoding accuracy. It also has the advantages of small computational complexity, low computational complexity, and short encoding and decoding time.

[0084] like Figure 5As shown, in the video transmission method provided by the embodiment of the present disclosure, the final encoding network in the self-encoding network is equivalent to a transform / quantization network. The encoding end device can output an entropy-coded bitstream (i.e., video feature data) by inputting the prediction residual data block of the video frame to be transmitted into the transform / quantization network. The final decoding network in the self-encoding network is equivalent to an inverse transform / inverse quantization network. The decoding end device can output decoded data (i.e., decoded prediction residual data) by inputting the entropy-coded bitstream transmitted by the encoding end device into the inverse transform / inverse quantization network.

[0085] like Figure 6 As shown in FIG. 1 , a schematic diagram of an autoencoding network provided by an embodiment of the present disclosure is shown, wherein the input layer of the final encoding network (encoder) inputs the prediction residual data of the video frame to be transmitted, represented by x1, x2...x6, and the hidden layers of the autoencoding network include the final encoding intermediate layer, the final encoding output layer, the final decoding input layer, and the final decoding intermediate layer. Figure 6 The final encoding output layer and the final decoding input layer are shown as the same layer, with a1, a2, and a3 representing the video feature data. The output layer of the final decoding network (decoder) outputs the decoded prediction residual data, The convolutional layers, pooling layers, and activation layers included in the final encoding intermediate layer, as well as the convolutional layers, pooling layers, and activation layers included in the final decoding intermediate layer, are not shown in the figure.

[0086] like Figure 7 As shown, another schematic diagram of an autoencoding network provided by an embodiment of the present disclosure shows the input layer (Input) of the final encoding network, the output layer (Onput) of the final decoding network, the convolution layer (Conv) of the final encoding network, the pooling layer (Pool) of the final encoding network, the activation layer (Active) of the final encoding network, the convolution layer (Conv) of the final decoding network, the activation layer (Active) of the final decoding network, the sampling layer (Upsampling) of the final encoding network, the final encoding output layer and the final decoding input layer (FC, fully connected layer). The intermediate layer of the final encoding network and the intermediate layer of the final decoding network both include N unit structures.

[0087] It should be noted that the autoencoding network in the embodiment of the present disclosure can also be replaced by other deep networks such as GAN (Generative Adversarial Net, Generative Adversarial Neural Network), which also uses a deep network to replace the entire video encoding and decoding process of transformation processing, quantization processing, inverse transformation processing and inverse quantization processing, and the method of training the deep network can be the same as the method of training the autoencoding network. Whether it is an autoencoding network or other deep networks, it is necessary to support some basic operations of deep learning, such as: Convolution, Deconvolution, ReLU (activation), Sigmoid (prediction and output function of logistic regression), Full-Connection, Reshape (a function that can re-adjust the number of rows, columns, and dimensions of a matrix), etc.

[0088] In addition, an embodiment of the present disclosure further provides an electronic device, which may include:

[0089] one or more processors;

[0090] a storage device having one or more programs stored thereon;

[0091] When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the video transmission method as described above.

[0092] In addition, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, wherein when the program is executed, the video transmission method as described above is implemented.

[0093] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, and the functional modules / units in the device can be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0094] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A video transmission method, characterized in that: The method comprises: Using a final encoding network in a pre-trained autoencoding network, encoding prediction residual data of a video frame to be transmitted to obtain video feature data, the autoencoding network including the final encoding network and the final decoding network; Transmitting the video feature data to a decoding end device; The method further comprises the step of training the autoencoding network: Adjust the initial network according to the transmission scenario parameters of the sample video frame to obtain the intermediate network; The intermediate network is trained based on the prediction residual data of the sample video frame to obtain the autoencoding network; wherein the transmission scene parameters of the sample video frame are the same as the transmission scene parameters of the video frame to be transmitted; the transmission scene parameters include a scene category identifier and a transmission bandwidth.

2. The method according to claim 1, characterized in that The initial network includes an initial encoding network and an initial decoding network, and the intermediate network includes an intermediate encoding network and an intermediate decoding network; and the step of training the intermediate network according to the prediction residual data of the sample video frame to obtain the autoencoding network includes: Inputting the prediction residual data of the sample video frame into the intermediate coding network to obtain intermediate feature data output by the intermediate coding network; Inputting the intermediate feature data into the intermediate decoding network to obtain intermediate prediction residual data output by the intermediate decoding network; When the prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data do not meet the preset optimization stop condition, the parameters of the intermediate network are adjusted until the prediction residual data of the sample video frame and the intermediate feature data and intermediate prediction residual data obtained after adjustment meet the preset optimization stop condition.

3. The method according to claim 2, characterized in that The prediction residual data of the sample video frame, the intermediate feature data, and the intermediate prediction residual data satisfying a preset optimization stop condition include: The data dimension of the intermediate feature data is smaller than a preset dimension threshold, and the average variance between the prediction residual data of the sample video frame and the intermediate prediction residual data is smaller than a preset variance threshold.

4. The method according to claim 2, characterized in that The initial encoding network includes an initial encoding input layer, an initial encoding output layer, and a plurality of initial encoding intermediate layers; the initial decoding network includes an initial decoding input layer, an initial decoding output layer, and a plurality of initial decoding intermediate layers; the intermediate encoding network includes an intermediate encoding input layer, an intermediate encoding output layer, and a plurality of intermediate encoding intermediate layers; the intermediate decoding network includes an intermediate decoding input layer, an intermediate decoding output layer, and a plurality of intermediate decoding intermediate layers; The final encoding network includes a final encoding input layer, a final encoding output layer and a plurality of final encoding intermediate layers, and the final decoding network includes a final decoding input layer, a final decoding output layer and a plurality of final decoding intermediate layers; The types of the initial encoding intermediate layer, the initial decoding intermediate layer, the intermediate encoding intermediate layer, the intermediate decoding intermediate layer, the final encoding intermediate layer, and the final decoding intermediate layer all include the following three types: convolutional layer, pooling layer, and activation layer; The type of the initial encoding output layer, the type of the initial decoding input layer, the type of the intermediate encoding output layer, the type of the intermediate decoding input layer, the type of the final encoding output layer, and the type of the final decoding input layer are all fully connected layers; The type of the initial encoding input layer, the type of the initial decoding output layer, the type of the intermediate encoding input layer, the type of the intermediate decoding output layer, the type of the final encoding input layer, and the type of the final decoding output layer are all the same.

5. The method according to claim 4, characterized in that The adjusting the parameters of the intermediate network includes: The number of the intermediate encoding intermediate layers and the number of the intermediate decoding intermediate layers are adjusted, as well as the number of neurons in the intermediate encoding intermediate layers and the number of neurons in the intermediate decoding intermediate layers.

6. The method according to claim 4, characterized in that The adjusting of the initial network according to the transmission scenario parameters of the sample video frame to obtain the intermediate network includes: According to the scene category identifier and the transmission bandwidth, the number of the initial coding intermediate layers and the number of the initial decoding intermediate layers are adjusted, and the dimension of the output data of the initial coding output layer is adjusted.

7. A video transmission method, characterized in that: The method comprises: Receiving video feature data transmitted by an encoding end device, wherein the video feature data is obtained by the encoding end device encoding prediction residual data of a to-be-transmitted video frame using a final encoding network in a pre-trained autoencoding network; Decoding the video feature data using a final decoding network in the autoencoder network to obtain decoded prediction residual data; Determining a transmitted video frame according to the decoded prediction residual data; The self-encoding network is a network obtained by the encoding end device adjusting the initial network according to the transmission scenario parameters of the sample video frame, and training the intermediate network according to the prediction residual data of the sample video frame; The transmission scene parameters of the sample video frame are the same as the transmission scene parameters of the video frame to be transmitted; the transmission scene parameters include a scene category identifier and a transmission bandwidth.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the video transmission method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, wherein: When the program is executed, the video transmission method according to any one of claims 1 to 7 is implemented.

Images