Video transmission method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video transmission method, electronic equipment and a storage medium, and the method comprises the steps: determining key frames and non-key frames of at least one picture group of a to-be-transmitted video; determining sampling rates corresponding to the key frame and the non-key frame; and according to each sampling rate, compressing and encoding the corresponding picture group block into transmission data, and sending the transmission data to a decoding end. According to the embodiment of the invention, compressed sampling of the video is realized, the video compression effect can be improved, the image quality in the video data reconstruction process is enhanced, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a video transmission method, electronic equipment and storage medium. Background Art

[0002] With the booming development of the Internet, more and more application scenarios such as online video and video conferencing are associated with video, and the video resolution is also significantly improved. However, the emergence of 4K and 8K ultra-high-definition video, VR video, etc. has led to an explosive growth in the amount of data, which consumes a huge amount of storage space for video data storage and also poses a great challenge to the transmission bandwidth. At present, the Nyquist-Shannong sampling theorem has always been the golden rule in the field of signal processing. It requires a sampling rate of no less than twice the signal bandwidth to obtain dense sampling values, which also puts extremely high demands on the sampling system, greatly affects the compression effect, and consumes more storage space and bandwidth. Therefore, how to compress the original video and obtain a better quality reconstructed video at the same time, so that the video can break through the limitations of the Nyquist-Shannong sampling theorem at the encoding end and improve the compression ratio of the video has become an urgent problem to be solved. Summary of the invention

[0003] The embodiments of the present application aim to provide a video transmission method, an electronic device and a storage medium, the purpose of which is to achieve compressed sampling of the video, improve the video compression effect, enhance the image quality of the video data reconstruction process, and improve the user experience.

[0004] The embodiment of the present application provides a video transmission method, which is applied to an encoding end, wherein the method comprises: determining key frames and non-key frames of at least one group of picture groups of a video to be transmitted;

[0005] Determine the sampling rates corresponding to the key frames and the non-key frames;

[0006] The corresponding picture group blocks are compressed and encoded into transmission data according to each sampling rate, and the transmission data is sent to a decoding end.

[0007] The embodiment of the present application also provides a video transmission method, which is applied to a decoding end, wherein the method comprises: receiving transmission data sent by an encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to key frames and non-key frames;

[0008] Reconstructing the transmission data into an initial key frame and an initial non-key frame;

[0009] Reconstructing the initial key frame into a key frame based on a preset deep reconstruction network, and guiding the initial non-key frame to be reconstructed into a non-key frame according to the key frame;

[0010] A picture group of a transmission video is constructed based on the key frame and the non-key frame.

[0011] The present application also provides an electronic device, which includes:

[0012] one or more processors;

[0013] A memory for storing one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement any method described in the embodiments of the present application.

[0015] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs are executed by one or more processors to implement any method as described in the embodiments of the present application.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is an example diagram of the process of video compression sensing calculation in the prior art;

[0019] Figure 2 is a flow chart of a video transmission method provided by an embodiment of the present application;

[0020] Figure 3 is a flow chart of another video transmission method provided by an embodiment of the present application;

[0021] Figure 4 is a flow chart of another video transmission method provided by an embodiment of the present application;

[0022] Figure 5 is a flow chart of another video transmission method provided by an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of a preset deep reconstruction network provided in an embodiment of the present application;

[0024] Figure 7 is a flow chart of another video transmission method provided by an embodiment of the present application;

[0025] Figure 8 is an example diagram of a video transmission system architecture provided by an embodiment of the present application;

[0026] Fig. 9 This is an example diagram of data encryption using compressed sensing provided by an embodiment of the present application;

[0027] Fig.10 is a schematic diagram of a structure of an interpretable reconstruction network provided in an embodiment of the present application;

[0028] Fig.11 It is a structural diagram of a key frame deep reconstruction network provided in an embodiment of the present application;

[0029] Fig.12 This is an example diagram of a basic module structure of a key frame deep reconstruction network provided in an embodiment of the present application;

[0030] Fig.13 is a structural schematic diagram of a video transmission device provided in an embodiment of the present application;

[0031] Fig.14 is a structural schematic diagram of another video transmission device provided in an embodiment of the present application;

[0032] Fig.15 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] It should be understood that the specific implementations described herein are only used to explain the present application and are not used to limit the present application.

[0034] In the subsequent description, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning themselves. Therefore, "module", "component" or "unit" may be used interchangeably.

[0035] The original video data is composed of multiple video frames, which are played frame by frame to form a continuous picture. The original video data can be represented by a matrix. The length and width of the matrix represent the resolution of the original video data respectively. Each frame image in the original video data has the same resolution.

[0036] At present, the flowchart of video compression sensing is as follows Figure 1As shown, x represents the original video, which can be represented by the multiplication of the sparse basis matrix ψ and the sparse coefficient α, φ represents the observation matrix, and the length and width of the two-dimensional observation matrix are M and N respectively. Through matrix multiplication, the measurement value M×1-dimensional measurement value y can be obtained. If M≤N, the measurement value y will occupy less storage space. Unlike the Nyquist-Shannong theorem, which requires a sampling rate twice the signal bandwidth to obtain dense sampling values, since real video data often meets the sparsity condition, the above process can use sampling values ​​far lower than the requirements of the Nyquist-Shannong theorem to restore the original input. At the same time, unlike the sampling and compression process of the traditional processing flow of video data, compressed sensing combines the sampling and compression processes into one through the observation matrix. The sampling value is the compressed value. The simplification of the process significantly reduces the amount of sampling data at the sampling end and reduces the pressure on the equipment at the sampling end. Therefore, compressed sensing can generally be divided into two stages: sampling and reconstruction, providing a new solution for video compression and reconstruction.

[0037] Figure 2 is a flowchart of a video transmission method provided by an embodiment of the present application. The present application can be applied to the case of video compression sampling in a video transmission scenario. The method can be executed by a video transmission device, which can be implemented by software and / or hardware. The device is generally integrated in an encoding end, which can include a server, a personal terminal, a cloud service platform, etc. See Figure 2 The method provided in the embodiment of the present application specifically includes the following steps:

[0038] Step 110: Determine key frames and non-key frames of at least one set of picture groups of the video to be transmitted.

[0039] The video to be transmitted may be a video to be transmitted, and the video to be transmitted may be compressed and sampled by the encoder in the embodiment of the present application. A group of pictures (GOP) may be a basic unit for processing by the encoder, and a group of pictures may be composed of a group of key frames and non-key frames. The video to be transmitted may be divided into a plurality of groups of pictures, and a key frame may be an intra-frame coded image frame in the video to be transmitted, and the encoding process of the key frame does not need to refer to other video frames for encoding, and a non-key frame may be a predictive coded frame in the video to be transmitted, and the non-key frame is predictively coded by intra-frame coded frames and other predictive coded frames.

[0040] In an embodiment of the present application, the encoding end can process the video to be transmitted and convert the video to be transmitted into one or more picture groups, each of which may include key frames and non-key frames, wherein the picture group may include a single key frame picture group and a double key frame picture group according to the different numbers of key frames in the picture group.

[0041] Step 120: Determine the sampling rates corresponding to the key frames and non-key frames.

[0042] The sampling rate may be a ratio of sampled data in a video frame of the video to be transmitted to original data of the video frame.

[0043] In an embodiment of the present application, the corresponding sampling rates are determined for the key frames and non-key frames in each picture group, and the determination method may include pre-configuring fixed sampling rates for the key frames and non-key frames respectively, or adaptively determining the sampling rates of the key frames and non-key frames according to the software and hardware performance of the encoding end. In some exemplary embodiments, the sampling rates of the key frames and non-key frames may be different, and the sampling rate of the key frames may be greater than the sampling rate of the non-key frames.

[0044] Step 130: compress and encode the corresponding picture group blocks into transmission data according to each sampling rate, and send the transmission data to the decoding end.

[0045] Among them, block compression can be a process of dividing the key frames or non-key frames in the picture group into multiple block data, and compressing and encoding each block data according to its corresponding sampling rate. It can be understood that the sampling rate of each block data can be determined by the key frame or non-key frame to which it belongs.

[0046] In an embodiment of the present application, for each picture group, the key frames and non-key frames therein are compressed in blocks according to their corresponding sampling rates, and the data generated after the block compression of each picture group can be used as transmission data, and the transmission data can be transmitted to the decoding end, so that the decoding end can reconstruct the picture based on the transmission data. In other application embodiments, the encoding end can also store the generated transmission data.

[0047] In some application embodiments, determining the sampling rates corresponding to key frames and non-key frames includes: pre-configuring fixed sampling rates for the key frames and the non-key frames, respectively; and determining the sampling rates corresponding to the key frames and non-key frames based on hardware constraints of the encoding end.

[0048] In an embodiment of the present application, the sampling rates corresponding to the key frames and non-key frames can be determined by the hardware constraints of the encoding end, and the hardware constraints may include but are not limited to the computing performance, storage space, remaining power, etc. of the encoding end. It can be understood that the higher the hardware constraints of the encoding end, the higher the sampling rate values ​​of the processed key frames and non-key frames can be. In other application embodiments, the key frames and non-key frames can also be pre-configured with fixed sampling rates, and the pre-configured sampling rates of the corresponding key frames and non-key frames can be read respectively.

[0049] Figure 3is a flowchart of another video transmission method provided by an embodiment of the present application. The present application embodiment is a specific embodiment based on the above application embodiment, and introduces the compression encoding process of the picture group. Figure 3 The method provided in the embodiment of the present application specifically includes the following steps:

[0050] Step 210: Determine key frames and non-key frames of at least one set of picture groups of the video to be transmitted.

[0051] Step 220: Determine the sampling rates corresponding to the key frames and non-key frames.

[0052] Step 230: Divide the key frames and non-key frames in the picture group into block data according to a preset scale.

[0053] The preset scale may be a scale for dividing key frames and non-key frames. For example, if the image size of key frames and non-key frames is 32*32 and the preset scale is 4*4, the key frames and non-key frames may be divided into 64 block data respectively, and the image size in each block data may be 4*4. The preset scale may be determined according to the business scenario and the software and hardware capabilities of the encoding end. Each block data may be data formed by the key frame and non-key frame blocks, and each block data may be a portion of the video data of a key frame or a non-key frame.

[0054] In an embodiment of the present application, each key frame and non-key frame in a picture group can be divided into block data according to a preset scale, and each block data can be a part of a key frame or a non-key frame. In some exemplary embodiments, the block data division process can be implemented by a sampling matrix, the dimension of the sampling matrix can correspond to the preset scale, and the key frames and non-key frames in the picture group can be processed by the sampling matrix, so as to generate block data corresponding to each key frame and non-key frame.

[0055] Step 240: Determine the block sampling rates corresponding to the block data of the key frame and the non-key frame respectively according to the sampling rates.

[0056] The block sampling rate may be a sampling rate corresponding to each block of data. It is understandable that the block sampling rates corresponding to different block data may be different.

[0057] In an embodiment of the present application, corresponding block sampling rates can be allocated to block data belonging to key frames or non-key frames based on the sampling rates corresponding to each of the key frames and non-key frames. It can be understood that the block sampling rates allocated to multiple block data belonging to the same key frame or non-key frame can conform to the sampling rates of the corresponding key frames or non-key frames. For example, the conformity may include that the average value of all block sampling rates belonging to the same key frame is the same as the sampling rate of the key frame, or that the distribution of the sampling data collected according to the block sampling rate is the same as the distribution of the sampling data of the key frame collected according to the sampling rate of the key frame.

[0058] Step 250: Sample the corresponding block data according to the block sampling rate to obtain sampled block data.

[0059] In the embodiment of the present application, data collection can be performed for each block data using its corresponding sampling rate, and the collected data can be used as sampled block data.

[0060] Step 260: Use each sample block data as transmission data.

[0061] Specifically, all the block data corresponding to the sampling block data of each picture group may be used as the transmission data.

[0062] In some application embodiments, the block sampling rates corresponding to the block data of the key frame and the non-key frame are determined according to the sampling rates, including:

[0063] The information difference of the pixel points in each block data is determined according to the difference between the mean filtering results of the key frame and the non-key frame and the original data; the number of sampling values ​​of the pixel points in the corresponding block data is determined according to the information difference; the iteration error is determined according to the number of sampling values ​​of all the pixel points in the block data; when the iteration error meets the preset conditions, the sampling rate corresponding to the number of sampling values ​​is used as the block sampling rate of the corresponding block data; when the iteration error does not meet the preset conditions, the number of sampling values ​​is reallocated according to the preset allocation formula and the iteration error is re-determined until the block sampling rate of all the block data is determined.

[0064] Among them, the mean filtering result can also be the result generated by filtering the key frame and the non-key frame, each pixel point of the key frame and the non-key frame can determine its corresponding mean filtering result, the original data can be the data value of each pixel point of the key frame and the non-key frame before the mean filtering, the iteration error can be the error between the current sampling rate of the key frame or the non-key frame and the sampling rate determined in the above steps caused by the sampling values ​​allocated to all the block data in a key frame or non-key frame, the preset condition can be the judgment condition for judging whether the sampling values ​​allocated to all the block data in a key frame or non-key frame meet the sampling rate corresponding to the key frame or non-key frame, and the preset condition can include but is not limited to the pre-configured iteration error threshold and the iteration allocation rule of the block sampling value of each key frame or non-key frame block data, etc. The number of sampling values ​​can be the number of collected pixel points in the block data, and the ratio of the number of sampling values ​​to the total number of pixels in the block data can be used as the sampling rate.

[0065] In an embodiment of the present application, mean filtering can be performed on the key frames and non-key frames in each group of picture groups, and the mean filtering results corresponding to each pixel point of the key frames and non-key frames can be determined. The difference between the corresponding mean filtering results and the original data can be determined for each pixel point, and the difference can be used as the information difference of the corresponding pixel point. The corresponding information difference is determined for the pixels contained in the block data of each key frame or non-key frame, and the number of sampling values ​​can be determined for each pixel point according to the information difference. For example, when the information difference is less than a threshold value, the number of sampling values ​​of the pixel point can be 1, otherwise, the number of sampling values ​​of the pixel point can be 0. After determining the number of sampling values ​​of the pixel points in each block data, the iteration error corresponding to each block data can be determined. For example, the difference between the sum of the number of sampling values ​​of all pixel points and the number of sampling values ​​corresponding to the sampling rate of the block data can be used as the iteration error of the block data; the iteration error can be compared with the preset conditions for each block data. When it is determined that the iteration error meets the preset conditions, the block sampling rate can be determined using the current number of sampling values ​​of the block data. When it is determined that the iteration error does not meet the preset conditions, the number of sampling values ​​can be adjusted according to the preset allocation formula, and the iteration error can be re-determined according to the above process and compared with the preset conditions. This process can be repeated until the corresponding block sampling rate is determined for the block data.

[0066] In some other application embodiments, the information difference of the pixel points in each block data is determined according to the difference between the mean filtering results of the key frame and the non-key frame and the original data, including:

[0067] Intra-frame mean filtering is performed on each video frame in the key frame and non-key frame to determine the mean filtering result corresponding to each pixel in each video frame; the difference between the original data of the pixel point in each block data and the mean filtering result is determined, and the product of the difference and the preset self-selected hyperparameter is used as the information difference.

[0068] Among them, the preset self-selected hyperparameter can be a pre-configured weight coefficient, and the preset self-selected hyperparameter is used to adjust the order of magnitude of the information difference.

[0069] In an embodiment of the present application, each frame of video image in the key frame and non-key frame can be used as input for mean filtering processing, and each pixel point can determine its corresponding mean filtering result after each frame of video image is mean filtered; the difference between the mean filtering result of each pixel point and the corresponding original data can be determined, and the product of the difference and the preset self-selected hyperparameter is used as the information difference of the pixel point.

[0070] In some other application embodiments, determining the number of sampling values ​​of pixel points in the corresponding block data according to the information difference includes:

[0071] When the video image to which the block data belongs is a key frame, the information difference of the pixels in the block data is respectively intercepted according to the first sampling rate of the key frame to determine the number of sampling values; when the video image to which the block data belongs is a non-key frame, the information difference of the pixels in the block data is respectively intercepted according to the second sampling rate of the non-key frame to determine the number of sampling values.

[0072] In an embodiment of the present application, the number of sampling values ​​can be allocated to the pixels in the block data based on the information difference. The number of sampling values ​​can be the number of sampling times or the sampling frequency of each pixel. The number of sampling values ​​can be used to determine whether the pixel is sampled. When the video image to which the block data belongs is a key frame, that is, the block data belongs to a part of a key frame, the first sampling value can be determined according to the first sampling rate of the key frame, and the information difference of the pixel of the block data can be intercepted so that the value range of the intercepted information difference is within the range of the first sampling value. The interception of the information difference can include subtracting a part of the value from the information difference so that the remaining value is within the range of the sampling value corresponding to the first sampling rate. The remaining value of the intercepted information difference can be used as the number of sampling values. Similarly, when the video image to which the block data belongs is a non-key frame, the block data belongs to a part of a non-key frame, and the second sampling value corresponding to the second sampling rate of the non-key frame can be determined. The second sampling value can be used to intercept the value of the information difference corresponding to the block data, and the value of the intercepted information difference can be used as the number of sampling values.

[0073] In some other application embodiments, determining the iteration error according to the number of sampling values ​​of all pixel points in the block data includes:

[0074] Determine the sum of the number of sampling values ​​of all pixels in the block data; and use the difference between the sum and the preset number of sampling values ​​corresponding to the block data as an iteration error.

[0075] In an embodiment of the present application, for each block data, the sum of the number of sampling values ​​of each pixel point in the block data can be determined, and the sum is subtracted from a preset sampling number. The preset sampling number can be determined by the first sampling rate of the key frame corresponding to the block data or the second sampling rate of the non-key frame, and the determined difference can be used as an iterative error.

[0076] Based on the above application embodiment, the preset allocation formula includes at least one of the following:

[0077]

[0078] Q = Q-sign(δ)×D;

[0079] Among them, Q represents the number of sampling values, L represents the number of channels of block data, δ represents the iteration error, D represents the iteration matrix randomly selected from the multinomial distribution, and the sign function represents the positive and negative value judgment function.

[0080] In some application embodiments, when reallocating the number of sampling values ​​of block data, it can be determined according to a preset allocation formula, which can include, for each block data, subtracting the quotient of the iteration error and the number of channels from the current number of sampling values ​​of the block data as a new initial block sampling value, or subtracting a modification value from the current number of sampling values ​​of the block data, which is determined by the positive or negative value of the iteration error and an iteration matrix randomly selected from a polynomial distribution.

[0081] In some other application embodiments, the sampling rate corresponding to the number of sampling values ​​is used as the corresponding block sampling rate of the block data, including:

[0082] The corresponding block data dimension and the number of sampling values ​​are determined for each block data; and the ratio of the number of sampling values ​​to the block data dimension is used as the block sampling rate corresponding to the block data.

[0083] The block data dimension may be the data size of the block data, and the block data dimension may be determined by the number of channels of the block data and a preset scale. For example, if the number of channels is l and the preset scale is B, then the block data dimension is lB. 2 .

[0084] In the embodiment of the present application, the block data dimension and the number of sampling values ​​of each block data can be determined, and the ratio of the number of sampling values ​​to the block data dimension is used as the block sampling rate corresponding to the block data.

[0085] Based on the above application embodiment, the first sampling rate of the key frame is greater than the second sampling rate of the non-key frame.

[0086] Figure 4 is a flowchart of another video transmission method provided by an embodiment of the present application. The present application can be applied to the case of video compression sampling in a video transmission scenario. The method can be executed by a video transmission device, which can be implemented by software and / or hardware. The device is generally integrated in the decoding end. The encoding end may include a server, a personal terminal, a cloud service platform, etc. See Figure 4 The method provided in the embodiment of the present application specifically includes the following steps:

[0087] Step 310: Receive transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames.

[0088] The transmission data may be compressed and sampled video data transmitted by the decoding end, and the transmission data may be generated by compression encoding using different sampling rates according to the key frames or non-key frames to which they belong.

[0089] In an embodiment of the present application, the encoding end may compress and encode the video image in blocks according to the sampling rates corresponding to the key frames and non-key frames to generate transmission data, and the decoding end may receive the transmission data sent by the encoding end.

[0090] Step 320: Reconstruct the transmission data into an initial key frame and an initial non-key frame.

[0091] In an embodiment of the present application, the transmission data can be reconstructed, and the reconstruction process can include operations such as upsampling and decompression. The transmission data can be reconstructed into initial key frames and initial non-key frames respectively, wherein the initial key frames can be reconstructed and generated by the transmission data corresponding to the key frames, and the initial non-key frames can be reconstructed and generated by the transmission data corresponding to the non-key frames.

[0092] Step 330: reconstruct the initial key frame into a key frame based on a preset deep reconstruction network, and reconstruct the initial non-key frame into a non-key frame according to the guidance of the key frame.

[0093] Among them, the preset deep reconstruction network can be a pre-trained neural network, and the preset deep reconstruction network can improve image quality. In some application embodiments, the preset deep reconstruction network can include a convolutional neural network.

[0094] In an embodiment of the present application, a preset deep reconstruction network can be called to perform deep reconstruction on the initial key frame, the depth reconstruction result can be used as a key frame, the initial non-key frame can be guided and reconstructed based on the key frame processed by the preset deep reconstruction network, and the initial non-key frame can be reconstructed into a non-key frame. The reconstruction process of the non-key frame can be guided by the key frame. For example, reconstruction can be performed based on the feature data of the key frame and the initial non-key frame.

[0095] Step 340: construct a picture group for transmitting the video based on the key frames and non-key frames.

[0096] In an embodiment of the present application, the generated key frames and non-key frames can be reconstructed into a picture group. Depending on the number of key frames included in the picture group, the picture group can be constructed as a single key frame picture group, a double key frame picture group, etc.

[0097] Figure 5 is a flowchart of another video transmission method provided by an embodiment of the present application. The embodiment of the present application is a specific embodiment based on the above-mentioned embodiment. Figure 5 The method provided in the embodiment of the present application specifically includes the following steps:

[0098] Step 410: Receive transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames.

[0099] Step 420: Reconstruct the transmission data into an initial key frame and an initial non-key frame.

[0100] Step 430: Process the initial key frame according to the preset deep reconstruction network, and obtain the key frame output result of the deep reconstruction network.

[0101] In an embodiment of the present application, the initial key frame can be input into a preset deep reconstruction network for processing, and the initial key frame can be reconstructed by the preset deep reconstruction network to improve the image quality of the initial key frame. The output result of the preset deep reconstruction network can be collected as the key frame output result. It can be understood that the input data of the preset deep reconstruction network can be the initial key frame that has been preliminarily reconstructed, and the key frame output result that has been processed by the preset deep reconstruction network can also be input.

[0102] Step 440: input the key frame output result and the initial non-key frame into a preset reconstruction network to generate a non-key frame output result.

[0103] In an embodiment of the present application, the key frame output result of the deep reconstruction network output can be extracted, and the key frame output result can be input into a preset reconstruction network together with the initial non-key frame for processing, and the result output by the preset reconstruction network can be used as a non-key frame output result, wherein the preset reconstruction network can be a convolutional neural network for reconstructing non-key frames in the prior art, and the output channel of the preset reconstruction network includes at least data channels corresponding to the initial non-key frame and the key frame output result.

[0104] Step 450: Repeat the above process until the key frame output result and the non-key frame output result meet the depth reconstruction condition, and use the key frame output result as the key frame and the non-key frame output result as the non-key frame.

[0105] The depth reconstruction condition may be a condition for determining whether the depth reconstruction of key frames and non-key frames is completed, and the depth reconstruction condition may include the number of repeated executions or the picture quality requirements of key frames and non-key frames.

[0106] In an embodiment of the present application, it can be determined whether the key frame output result and the non-key frame output result meet the preset depth reconstruction conditions. If so, the key frame output result is used as the key frame for depth reconstruction and the non-key frame output result is used as the non-key frame for depth reconstruction. If not, the key frame output result is re-input into the preset depth reconstruction network for depth reconstruction, and the new key frame output result and the non-key frame output result corresponding to the key frame output result are used as the input of the preset reconstruction network, so as to obtain the new non-key frame output result, and re-determine whether the new non-key frame output result and the new key frame output result meet the depth reconstruction conditions. This process can be repeated until the determined non-key frame output result and the key frame output result meet the depth reconstruction conditions.

[0107] Further, based on the above application embodiment, the depth reconstruction condition includes at least one of the following:

[0108] The preset depth reconstruction network and the number of calls of the preset reconstruction network are greater than or equal to the preset threshold; the image quality of the key frame output result and the non-key frame output result meets the preset quality requirement.

[0109] In an embodiment of the present application, the depth conditions corresponding to the depth reconstruction satisfied by the key frame output results and the non-key frame output results may include a preset depth reconstruction network and the number of calls of the preset reconstruction network being greater than or equal to a preset threshold, and the image quality of the key frame output results and the non-key frame output results meeting the preset quality requirements.

[0110] Step 460: construct a picture group of the transmitted video based on the key frames and non-key frames.

[0111] In some other application embodiments, the initial key frame is processed according to a preset deep reconstruction network, and the key frame output result of the deep reconstruction network is obtained, including:

[0112] The iteration step of the iterative optimization module of the preset deep reconstruction network is obtained, and the optimization result corresponding to the initial key frame is obtained according to the iteration step and the preset formula, wherein the preset formula includes:

[0113] r (k) =x (k) -ρφ T (φx (k-1) -y), where r (k) represents the optimization result of the initial key frame k, ρ represents the iteration step, φ represents the sampling matrix of the initial key frame, and φ T represents the transpose of the sampling matrix of the initial keyframe, x (k) Represents the initial key frame, y represents the transmission data corresponding to the initial key frame; calling the feature extraction module of the preset deep reconstruction network to extract feature data from the optimization result as the input to be convolved; processing the input to be convolved as the output to be convolved according to the convolution reconstruction module of the preset deep reconstruction network; taking the remaining data after the feature data is extracted by the feature extraction module in the optimization result and the sum of the output to be convolved as the key frame output result.

[0114] In the embodiment of the present application, the preset deep reconstruction network can be composed of an iterative optimization module, a feature extraction module, and a convolutional reconstruction module. In the iterative optimization module, the output initial key frame can be optimized according to the iteration step size. The optimization process can be performed by the preset formula r (k) =x (k) -ρφ T (φx (k-1) -y) implementation, preset formula r (k) represents the optimization result of the initial key frame k, ρ represents the iteration step, φ represents the sampling matrix of the initial key frame, and φ T represents the transpose of the sampling matrix of the initial keyframe, x (k) Represents the initial key frame, and y represents the transmission data corresponding to the initial key frame. The iterative optimization module can input the optimization result of the optimized initial key frame into the feature extraction module, and the feature extraction module extracts the feature data in the optimization result, and the feature data can be used as the output of the convolution reconstruction module, thereby reducing the data processing amount of the convolution reconstruction model. The convolution reconstruction module can process the feature data to generate the output to be convolved, and the remaining data that does not belong to the feature data after the output to be convolved and the feature extraction module process the optimization result can be summed, and the sum value can be used as the key frame output result.

[0115] In some application embodiments, the convolution reconstruction module of the preset deep reconstruction network includes at least a first convolution unit, a Soft function, a second convolution unit and a π convolution unit; the first convolution unit and the second convolution unit are connected through a Soft function, and the π convolution unit receives the output of the second convolution unit as input; wherein the first convolution unit and the second convolution unit respectively include two layers of convolution kernels and one layer of activation function, and the π convolution unit includes two layers of convolution kernels and two layers of activation functions.

[0116] In the examples of this application, see Figure 6 , the convolutional reconstruction module of the preset deep reconstruction model can be composed of a first convolutional unit, a Soft function, a second convolutional unit and a π convolutional unit, wherein the first convolutional unit and the second convolutional unit can process the input video data through a convolutional kernel, the first convolutional unit and the second convolutional unit can be connected through a Soft function, and the π convolutional unit can be connected to the second convolutional unit, and can receive the output of the second convolutional unit and use the output as the input of the π convolutional unit, the first convolutional unit and the second convolutional unit respectively include two layers of convolutional kernels and one layer of activation function, and the π convolutional unit includes two layers of convolutional kernels and two layers of activation functions. It can be understood that the preset deep reconstruction model can include multiple convolutional reconstruction modules, and the output of one convolutional reconstruction module in two adjacent convolutional reconstruction modules can be used as the input of another convolutional reconstruction module.

[0117] In some other application embodiments, the key frame output result and the initial non-key frame are input into a preset reconstruction network to generate a non-key frame output result, including:

[0118] Extract the key frame output results corresponding to at least one initial key frame respectively; use the key frame output results and the initial non-key frame as input parameters of the preset reconstruction network; extract the output parameters of the preset reconstruction network as non-key frame output results; wherein the preset convolutional network includes at least one of the following: convolutional neural network, LeNet-5 convolutional neural network, AlexNet convolutional neural network, VGG-16 convolutional neural network, ResNet convolutional neural network.

[0119] In an embodiment of the present application, the preset reconstruction network can be specifically a preset convolutional network, which can include at least one of a convolutional neural network, a LeNet-5 convolutional neural network, an AlexNet convolutional neural network, a VGG-16 convolutional neural network, and a ResNet convolutional neural network. The key frame output result and the initial non-key frame can be used as the output of the preset reconstruction network. It can be understood that for a single key frame picture group and a double key frame picture group, the number of channels of the input layer of the preset reconstruction network can be different. For example, in a double key frame picture group, the number of input channels of the preset reconstruction network can correspond to two groups of key frame output results and the initial non-key frame of the key frame. The output parameters of the preset reconstruction network can be extracted as the non-key frame output result.

[0120] Figure 7 is a flowchart of another video transmission method provided by an embodiment of the present application. The embodiment of the present application is a specific embodiment based on the above-mentioned embodiment. Figure 7 The method provided in the embodiment of the present application specifically includes the following steps:

[0121] Step 510: Generate a preset deep reconstruction network based on a preset training set training.

[0122] The preset training set may be a video image set used to train a preset deep reconstruction network.

[0123] In an embodiment of the present application, a preset deep reconstruction network can be pre-trained using a video image set so that the preset deep reconstruction network can reconstruct the video image and solve the problem of reduced picture quality caused by encoding during image transmission.

[0124] Step 520: Receive transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames.

[0125] Step 530: Reconstruct the transmission data into an initial key frame and an initial non-key frame.

[0126] Step 540: reconstruct the initial key frame into a key frame based on a preset deep reconstruction network, and reconstruct the initial non-key frame into a non-key frame according to the guidance of the key frame.

[0127] Step 550: construct a picture group of the transmitted video based on the key frames and non-key frames.

[0128] Further, based on the above application embodiment, the preset deep reconstruction network is generated based on the preset training set training, including:

[0129] Obtain a preset training set, wherein the preset training set includes at least Akiyo video sequence, Coastguard video sequence, Foreman video sequence, Mother_daughter video sequence, Paris video sequence and Silent video sequence; input the preset training set into a pre-constructed initial deep reconstruction network, wherein the initial deep reconstruction network includes at least an iterative optimization module, a feature extraction module and a convolutional reconstruction module; obtain the training result of the initial deep reconstruction network, and determine the loss value of the training result according to a preset loss function; when the loss value meets the preset condition, use the initial reconstruction network as the preset deep reconstruction network; when the loss value does not meet the preset condition, adjust the attribute parameters of the initial reconstruction network and continue to obtain the loss value of the training result until the training of the preset deep reconstruction network is completed.

[0130] In an embodiment of the present application, a preset training set including an Akiyo video sequence, a Coastguard video sequence, a Foreman video sequence, a Mother_daughter video sequence, a Paris video sequence and a Silent video sequence can be obtained, and the initial deep reconstruction network is trained using the preset training set. The initial deep reconstruction network can iteratively optimize the module, the feature extraction module and the convolution reconstruction module, obtain the training result of each initial deep reconstruction network, and determine the loss value corresponding to the training result according to the preset loss function, compare the loss value with the preset condition, and determine that the initial reconstruction network training is completed when the loss value meets the preset condition, and use the initial reconstruction network as the preset deep reconstruction network, otherwise, adjust the attribute parameters of the initial reconstruction network, and re-use the preset training set to train the adjusted initial deep reconstruction network until the initial deep reconstruction network training is completed. Specifically, the preset conditions may include the number of training iterations and the loss value threshold, etc.

[0131] Further, based on the above application embodiment, it is assumed that the loss function includes at least one of the following:

[0132] When the picture group of the preset training set is a single key frame, the preset loss function is:

[0133] When the preset training set has two keyframes, the preset loss function is:

[0134] Where N is the number of picture groups in the preset training set, x key Represents the original key frame data in the preset training set, I key represents the reconstruction result of the key frames in the preset training set, K is the number of non-key frames in the picture group, x k Represents the original key frame data in the preset training set, Ik Represents the reconstruction result of key frames in the preset training set, N b is the number of key frame blocks, N k Then it is the key frame image block size N p , N p is the number of convolutional reconstruction modules, Represents the output result of j trainings.

[0135] In an exemplary embodiment, Figure 8 This is an example diagram of a video transmission system architecture provided by an embodiment of the present application. Based on the method provided by an embodiment of the present application, a compressed sensing network framework is proposed, which mainly includes a sampling end and a reconstruction end. The sampling end performs adaptive sampling rate allocation sampling for key frames and non-key frames respectively, and the reconstruction end reconstructs the original video based on the transmitted measurement values. Figure 8 As shown, since video frames often contain a lot of redundant information, we can use different sampling rates to reduce the size of the total measurement value for the key frames and non-key frames of the video to achieve the purpose of better reducing bandwidth occupancy. For the reconstruction end, it can be subdivided into two stages: initial reconstruction and deep reconstruction. In the initial reconstruction stage, according to the compressed sensing theory, we obtain the initial reconstruction result, and then optimize the initial reconstructed image through the deep reconstruction network, and finally output the video result reconstructed by the reconstruction end. In the framework, the sampling rate for key frames and non-key frames can be independently selected at the sampling end based on actual hardware limitations such as storage space and bandwidth. At the same time, an adaptive sampling rate allocation method can be adopted for each frame of video data to better retain the original data information, so as to better improve the reconstruction quality. At the reconstruction end, the embodiment of the present application provides an interpretable network for the reconstruction of the video. The use of an interpretable network improves the interpretability of the network while improving the reconstruction quality.

[0136] Specifically, the specific processing flow of each stage of compressed sensing based on the above framework includes the following steps:

[0137] (1) Based on the actual needs of the user and the physical hardware limitations, different sampling rates α are selected for key frames and non-key frames at the sampling end. k and α n The corresponding sampling rate can obtain the sampling matrices of the key frame and non-key frame respectively S k and S n , the sampling process is based on the sampling matrix S k , S n The original video key frame X k and non-keyframe X n Perform matrix multiplication operations, and for each frame of data, adaptive sampling rate allocation can be used to allocate the sampled data, and finally the key frame sampling values ​​Y can be obtained respectivelyk and non-keyframe sample value Y n .

[0138] Y k =S k *X k (1)

[0139] Y n =S n *X n (2)

[0140] (2) The sample value Y obtained from the sampling end k and Y n As the main data, it is necessary to further integrate the video information and other data without changing the sampling data to form a standardized measurement value, which needs to include the following:

[0141] Original video sampling value: As the main body of the transmitted data, after sampling at the sampling end, different sampling values ​​with different overall data compression ratios can be obtained according to different sampling rates. Its data volume is often much smaller than the original video data volume, which can reduce the bandwidth pressure. At the same time, note that Fig. 9 As shown in the figure, if the sampling matrix is ​​not obtained, the compressed data often cannot intuitively obtain semantic information, especially at low sampling rates. This is because the sampling matrix is ​​random and realizes the mapping from the video domain to the sampling domain. It is difficult to directly obtain its inverse mapping without the sampling matrix. This makes it impossible for attackers to restore the original video data, which plays a role in protecting the user's data privacy.

[0142] Sampling matrix: According to the theory of compressed sensing, the sampling and compression process is completed through the sampling matrix. Depending on the different networks at the reconstruction end, it is necessary to choose whether to transmit the sampling matrix to the reconstruction end to obtain a reconstructed video with better quality.

[0143] Basic video information: basic information of the original video such as resolution and number of video frames is obtained according to the original video file format, which is used as the header file of the measurement value, and a reconstructed video with the same format as the original video is obtained based on the basic video information.

[0144] (3) The reconstruction end needs to reconstruct the original video based on the obtained measurement values. The network implementations are different. The commonly used methods generally divide it into two sub-stages: initial reconstruction and deep reconstruction. On this basis, the embodiment of the present application also proposes an interpretable network, which can be specifically implemented based on the overall framework.

[0145] In the embodiment of the present application, taking the sampling end implemented based on the above framework as an example, the compressed sensing sampling process of the sampling end adopts matrix multiplication, which can be summarized as the following process. After the original data x is expanded, its dimension can be expressed as N×1, that is, Let φ represent the sampling matrix, and the dimension of the sampling matrix can be expressed as M×N. Then its compressed sensing process can be expressed as the following formula, where y represents the sampling value. According to the knowledge related to matrix multiplication, the dimension of the sampling value y can be expressed as M×1, that is, For compressed sensing tasks, the sampling rate can be expressed as Generally speaking, M is much smaller than N, and ultimately the functions of sampling and compression are achieved simultaneously.

[0146] y=φ*x (3)

[0147] Based on the compressed sensing sampling process, in order to be more flexible in applying data of different scales, the block compressed sensing algorithm BCS came into being. Block compressed sensing usually divides the original image or video frame data into blocks according to a certain scale B, and finally obtains block data composed of B×B blocks. Considering that the data may have multiple channels, assuming that the number of channels is l, the dimension of the block data is B×B×l, and the sampling matrix φ needs to correspond to the dimension according to the block scale, that is, its dimension M×N needs to be limited to M×lB 2 , we call the block compressed sensing sampling matrix φ′, that is The sampling process can be expressed as the following formula:

[0148] y=φ′*x (4)

[0149] Traditional block compressed sensing algorithms often use the same sampling rate for each block, which is not conducive to retaining the original data information and reconstructing it. Therefore, on the basis of traditional compressed sensing, assigning different sampling rates to blocks is an extremely effective means to improve the reconstruction quality, thereby performing adaptive sampling rate allocation. Blur can be denoted as a mean filter function, and β is a self-selected hyperparameter. Since the complexity of the texture of the original data should be closely related to the sampling rate allocation, the original video data can be subjected to intra-frame mean filtering, and the information difference can be obtained after the residual absolute value between it and the original data is weighted. Finally, the final sampling rate allocation needs to be obtained through the information difference to ensure that the overall data sampling rate does not change after the block sampling. We denote the final information difference as S, and abs as the absolute value operation.

[0150] S = abs(x-blur(x))*β (5)

[0151] In the embodiment of the present application, the above information difference can be intercepted by using the sampling values ​​of the sampling rates corresponding to the key frame and the non-key frame to obtain the corresponding sampling values;

[0152] Specifically, the process of allocating the sampling rate by information difference is as follows, which uses formula (6) as the loop exit condition, and formulas (7) and (8) are the main algorithms of the iterative loop. First, the iteration error is obtained according to formula (6). If the iteration error is 0, the allocation result Q is output. If the iteration error is not 0 and the number of iterations is less than the threshold, the formula (7) is iterated. If the error is not 0 and the number of iterations is greater than the threshold, it is necessary to iterate according to formula (8). After formulas (7) and (8) select one to perform an operation, they jump back to formula (6) to calculate δ and determine whether the loop needs to be terminated.

[0153] In the formula, the sum function represents summation, q represents the number of sampling values ​​corresponding to the sampling rate, sign is a positive and negative value judgment function, L represents the number of channels of the block data, and D is an iterative matrix randomly obtained from the multinomial distribution. The output result Q corresponds one-to-one to the number of blocks of the original data to represent the number of sampling values ​​of each block of data.

[0154] δ=sum(Q)-q (6)

[0155]

[0156] Q=Q-sign(δ)×D (8)

[0157] Finally, for any block of samples, we assume that the assigned sampling rate is s i ,s i The value of is the corresponding sampling value Q of the current sampling block in Q i The only certainty is The sampling process is calculated according to the formula, where is the sampling matrix corresponding to the sampled values ​​and The amount of data after sampling is m i It can be expressed as That is, the total amount of data in each block is calculated at the sampling rate and rounded down.

[0158]

[0159] Finally, we will all the sample data y i The data transmitted from the sampling end to the reconstruction end is composed of the information described in the first embodiment and other related information.

[0160] In the embodiment of the present application, the reconstruction end based on the above framework is taken as an example. Although with the rise of the deep learning wave, the compressed sensing network based on deep neural network has achieved a great improvement in reconstruction effect, it is often troubled by the "black box network" and its network structure lacks interpretability. In order to solve this problem, the present application proposes an interpretable reconstruction network, which takes into account the inter-frame information and the guidance of key frames to non-key frames while using deep learning methods to simulate the traditional soft threshold iterative algorithm ISTA process, thereby achieving better video reconstruction quality while improving the performance of the video compression sensing network. The network structure is as follows: Fig.10 As shown, it is also divided into a sampling end and a reconstruction end. At the reconstruction end, a depth-expanded iterative shrinkage threshold algorithm is used to reconstruct the key frames, and the information in the key frame iteration is used to know the iterative optimization process of the non-key frames.

[0161] In the initial reconstruction stage, we directly reconstruct the initial reconstruction result with the same size as the original video through the sampling matrix, which is used as the input of the deep reconstruction stage. The reconstruction result is gradually optimized through the deep reconstruction stage, and finally the video frame with better reconstruction quality is output.

[0162] In the deep reconstruction stage, for each frame in the GOP, the reconstruction process uses modules with consistent structure to be connected in series. The quality of the video frame is gradually and iteratively optimized through the connected modules. However, in order to reduce the amount of calculation, based on the premise that key frames have a greater impact on the quality of video reconstruction, we use an interpretable network for key frames, and a simple connected convolutional CNN network with less calculation for non-key frames.

[0163] The structure of the key frame deep reconstruction network is as follows Fig.11 As shown, it is mainly divided into two stages. The first stage obtains the value r after gradient iteration. (k) , and then extract the features through the D module as the input of the subsequent work, and the unprocessed r (k) The second stage is to use the soft connected H and The module memorizes the proximal iteration and obtains g through the fusion of the final features. (k) , and finally the output of the current stage is obtained through the residual as x (k)

[0164] The reconstruction process in the embodiment of the present application includes the following stages: In the first stage, the input value x is reconstructed by learning the iterative step length ρ of the neural network. (k) Iterate, φ and φ T denote the sampling matrix and its transpose respectively.

[0165] r (k) =x (k) -ρφ T(φx (k-1) -y) (10)

[0166] In the second stage of the proximal iteration, H and Use soft to connect and iteratively optimize through neural network

[0167]

[0168] The core modules H, And the π module structure used to fuse features is as follows Fig.12 As shown, H, It is mainly composed of two layers of convolution kernels and one layer of activation function PReLU. The π module is composed of two layers of PReLU activation function and two layers of convolution kernels. The output channel of the last layer of convolution kernel in the π module is 1 to achieve the purpose of fusion features. The parameter k3 is the scale of the convolution kernel, p1 means that the convolution kernel is filled with one bit, and 32 means the number of input channels.

[0169] In the embodiment of the present application, the training process of the interpretable reconstruction network provided is as follows:

[0170] The training set uses HEVC test video sequences. In order to generalize the training data, common data enhancement methods such as random cropping and flipping are used. The random cropping size is 96*96, and the size of the video frame sequence group GOP is set to 8. The test set uses 6 CIF video sequences, namely Akiyo, Coastguard, Foreman, Mother_daughter, Paris, and Silent.

[0171] As for the parameters of compressed sensing, the sampling part of compressed sensing uses block sampling, the block size BlockSize is set to 32*32, the sampling rate is set to 0.5 for key frames, and to 0.01 and 0.1 for non-key frames.

[0172] For single keyframe GOP and double keyframe GOP, two loss functions are used respectively, which are essentially L2 Loss of the original image and the reconstructed image. The formula is as follows, and the parameters are explained as follows:

[0173] N: The number of GOP groups in the training set.

[0174] x key and I key : Represent the original key frame data and the reconstructed result respectively. Considering the difference between double key frames and single key frames in a GOP group, the subscripts (1) and (2) in the upper right corner are used to distinguish them.

[0175] K: The number of non-key frames.

[0176] x k and I k : Similar to the key frames, they represent the original data of non-key frames and the reconstructed results respectively. Since a GOP contains multiple non-key frames, the formula is presented in a summation manner. The value of k starts from 1 and traverses to K. At the same time, the subscript (i) is used in the upper right corner to distinguish non-key frames.

[0177] N b 、N k With N p :N b is the number of key frame blocks, N k Then it is the key frame image block size N p , N p is the number of iteration stages, and the depth reconstruction image generated by each iteration is Add loss function to better improve the reconstruction quality.

[0178]

[0179]

[0180] The training environment is Ubuntu 20.04 LTS system, the deep learning framework uses Pytorch: 1.8.0 CUDA: 10.0, and 4 GeForce RTX 3090s are used for neural network training.

[0181] Fig.13 1 is a schematic diagram of the structure of a video transmission device provided in an embodiment of the present application. The device can execute the video transmission method provided in any embodiment of the present application and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and / or hardware. Fig.13 As shown, the device provided in the embodiment of the present application specifically includes:

[0182] The frame acquisition module 601 is used to determine key frames and non-key frames of at least one picture group of the video to be transmitted.

[0183] The sampling rate module 602 is used to determine the sampling rates corresponding to the key frames and non-key frames.

[0184] The sampling transmission module 603 is used to compress and encode the corresponding picture group blocks into transmission data according to each sampling rate, and send the transmission data to the decoding end.

[0185] In some application embodiments, the sampling rate module 602 is specifically used to: pre-configure fixed sampling rates for the key frames and the non-key frames respectively; and determine the sampling rates corresponding to the key frames and non-key frames based on the hardware constraints of the encoding end.

[0186] In some application embodiments, the sampling transmission module 603 includes:

[0187] The data block unit is used to divide the key frames and non-key frames in the picture group into block data according to a preset scale.

[0188] The block sampling rate unit is used to determine the block sampling rates corresponding to the block data of the key frame and the non-key frame respectively according to each sampling rate.

[0189] A data sampling unit, used for sampling corresponding block data according to the block sampling rate as sampled block data;

[0190] The data encoding unit is used to use each sampling block data as transmission data.

[0191] In some application embodiments, the block sampling rate unit is specifically used to: determine the information difference of the pixel points in each block data according to the difference between the mean filtering results of the key frames and non-key frames and the original data; determine the number of sampling values ​​of the pixel points in the corresponding block data based on the information difference; determine the iteration error according to the number of sampling values ​​of all pixels in the block data; when the iteration error meets the preset conditions, use the sampling rate corresponding to the number of sampling values ​​as the block sampling rate of the corresponding block data; when the iteration error does not meet the preset conditions, reallocate the number of sampling values ​​according to the preset allocation formula and redetermine the iteration error until the block sampling rate of all block data is determined.

[0192] In some application embodiments, the information difference of the pixel points in each block data is determined in the block sampling rate unit according to the difference between the mean filtering results of the key frames and non-key frames and the original data, including: performing intra-frame mean filtering on each frame of video image in the key frames and non-key frames to determine the mean filtering results corresponding to each pixel point in each frame of video image; determining the difference between the original data of the pixel points in each block data and the mean filtering result, and taking the product of the difference and the preset optional hyperparameter as the information difference.

[0193] In some application embodiments, the block sampling rate unit determines the number of sampling values ​​of the pixel points in the corresponding block data according to the information difference, including:

[0194] When the video image to which the block data belongs is a key frame, information differences of pixel points in the block data are respectively intercepted according to the first sampling rate of the key frame to determine the number of sampling values;

[0195] When the video image to which the block data belongs is a non-key frame, the information difference of the pixel points in the block data is respectively intercepted according to the second sampling rate of the non-key frame to determine the number of sampling values.

[0196] In some application embodiments, determining the iteration error in the block sampling rate unit according to the initial block sampling values ​​of all pixels in the block data includes:

[0197] Determine the sum of the number of sample values ​​of all pixels in the block data;

[0198] The difference between the summed value and the preset number of samples corresponding to the block data is taken as the iteration error.

[0199] In some application embodiments, the preset allocation formula in the block sampling rate unit includes at least one of the following;

[0200]

[0201] Q = Q-sign(δ)×D;

[0202] Among them, Q represents the number of sampling values, L represents the number of channels of block data, δ represents the iteration error, D represents the iteration matrix randomly selected from the multinomial distribution, and the sign function represents the positive and negative value judgment function.

[0203] In some application embodiments, the block sampling rate unit uses the sampling rate corresponding to the number of sampling values ​​as the block sampling rate of the corresponding block data, including:

[0204] Determine the corresponding block data dimension and the number of sampling values ​​for each block data;

[0205] The ratio of the number of sample values ​​to the dimension of the block data is taken as the block sampling rate corresponding to the block data.

[0206] In some application embodiments, a first sampling rate of key frames in the device is greater than a second sampling rate of non-key frames.

[0207] Fig.14 1 is a schematic diagram of the structure of a video transmission device provided in an embodiment of the present application. The device can execute the video transmission method provided in any embodiment of the present application and has the corresponding functional modules and beneficial effects of the execution method. The device can be implemented by software and / or hardware. Fig.14 As shown, the device provided in the embodiment of the present application specifically includes:

[0208] The data receiving module 701 is used to receive the transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames.

[0209] The initial reconstruction module 702 is used to reconstruct the transmission data into an initial key frame and an initial non-key frame.

[0210] The depth reconstruction module 703 is used to reconstruct the initial key frame into a key frame based on a preset depth reconstruction network, and reconstruct the initial non-key frame into a non-key frame according to the guidance of the key frame.

[0211] The picture group module 704 is used to construct a picture group of the transmitted video based on the key frames and non-key frames.

[0212] In some application embodiments, the depth reconstruction module 703 includes:

[0213] The key frame reconstruction unit is used to process the initial key frame according to the preset deep reconstruction network and obtain the key frame output result of the deep reconstruction network.

[0214] The non-key frame reconstruction unit is used to input the key frame output result and the initial non-key frame into a preset reconstruction network to generate a non-key frame output result.

[0215] The iterative execution unit is used to repeatedly execute the above process until the key frame output result and the non-key frame output result meet the depth reconstruction condition, and the key frame output result is used as the key frame and the non-key frame output result is used as the non-key frame.

[0216] In some application embodiments, the key frame reconstruction unit is specifically used to: obtain the iteration step of the iterative optimization module of the preset deep reconstruction network, and obtain the optimization result corresponding to the initial key frame according to the iteration step and the preset formula, wherein the preset formula includes:

[0217] r (k) =x (k) -ρφ T (φx (k-1) -y), where r (k) represents the optimization result of the initial key frame k, ρ represents the iteration step, φ represents the sampling matrix of the initial key frame, and φ T represents the transpose of the sampling matrix of the initial keyframe, x (k) represents the initial key frame, and y represents the transmission data corresponding to the initial key frame;

[0218] Call the feature extraction module of the preset deep reconstruction network to extract feature data from the optimization result as the input to be convolved;

[0219] Processing the input to be convolved into the output to be convolved according to the convolution reconstruction module of the preset deep reconstruction network;

[0220] The remaining data after the feature data is extracted by the feature extraction module in the optimization result and the sum of the output to be convolved are used as the key frame output result.

[0221] In some application embodiments, the convolutional reconstruction module of the preset deep reconstruction network in the key frame reconstruction unit includes at least a first convolutional unit, a Soft function, a second convolutional unit, and a π convolutional unit;

[0222] The first convolution unit is connected to the second convolution unit through a Soft function, and the π convolution unit receives the output of the second convolution unit as input;

[0223] Among them, the first convolution unit and the second convolution unit respectively include two layers of convolution kernels and one layer of activation function, and the π convolution unit includes two layers of convolution kernels and two layers of activation functions.

[0224] In some application embodiments, the non-key frame reconstruction unit is specifically used to: respectively extract the key frame output results corresponding to at least one initial key frame;

[0225] The output results of each key frame and the initial non-key frame are used as input parameters of the preset reconstruction network;

[0226] Extracting the output parameters of the preset reconstruction network as the non-key frame output result;

[0227] Among them, the preset convolutional network includes at least one of the following: convolutional neural network, LeNet-5 convolutional neural network, AlexNet convolutional neural network, VGG-16 convolutional neural network, ResNet convolutional neural network.

[0228] In some application embodiments, the depth reconstruction condition includes at least one of the following:

[0229] The preset deep reconstruction network and the number of calls of the preset reconstruction network are greater than or equal to a preset threshold;

[0230] The image quality of the key frame output result and the non-key frame output result meets the preset quality requirements.

[0231] In some application embodiments, a model training module is also included for generating a preset deep reconstruction network based on a preset training set training.

[0232] In some application embodiments, the model training module is specifically used to: obtain a preset training set, wherein the preset training set at least includes an Akiyo video sequence, a Coastguard video sequence, a Foreman video sequence, a Mother_daughter video sequence, a Paris video sequence, and a Silent video sequence;

[0233] Inputting a preset training set into a pre-constructed initial deep reconstruction network, wherein the initial deep reconstruction network at least includes an iterative optimization module, a feature extraction module, and a convolutional reconstruction module;

[0234] Obtaining the training results of the initial deep reconstruction network, and determining the loss value of the training results according to a preset loss function;

[0235] When the loss value meets the preset condition, the initial reconstruction network is used as the preset deep reconstruction network;

[0236] When the loss value does not meet the preset conditions, the attribute parameters of the initial reconstruction network are adjusted and the loss value of the training result is continued to be obtained until the training of the preset deep reconstruction network is completed.

[0237] In some application embodiments, the preset loss function of the model training module includes at least one of the following:

[0238] When the picture group of the preset training set is a single key frame, the preset loss function is:

[0239] When the preset training set has two keyframes, the preset loss function is:

[0240] Where N is the number of picture groups in the preset training set, x key Represents the original key frame data in the preset training set, I key represents the reconstruction result of the key frames in the preset training set, K is the number of non-key frames in the picture group, x k Represents the original key frame data in the preset training set, I k Represents the reconstruction result of key frames in the preset training set, N b is the number of key frame blocks, N k Then it is the key frame image block size N p , N p is the number of convolutional reconstruction modules, Represents the output result of j trainings.

[0241] Fig.15 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a processor 10, a memory 11, an input device 12, and an output device 13. The number of processors 10 in the electronic device may be one or more. Fig.15 A processor 10 is taken as an example; the processor 10, the memory 11, the input device 12 and the output device 13 in the electronic device can be connected by a bus or other means. Fig.15 The example of connecting through bus is taken in the following.

[0242] The memory 11, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the modules corresponding to the device in the embodiment of the present application (frame acquisition module 601, sampling rate module 602 and sampling transmission module 603, and / or, data receiving module 701, initial reconstruction module 702, depth reconstruction module 703 and picture group module 704). The processor 10 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 11, that is, implementing the above method.

[0243] The memory 11 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 11 may further include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0244] The input device 12 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 13 may include a display device such as a display screen.

[0245] The embodiment of the present application further provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to perform a video transmission method, the method comprising:

[0246] Determine key frames and non-key frames of at least one set of picture groups of the video to be transmitted;

[0247] Determine the sampling rates corresponding to the key frames and the non-key frames;

[0248] The corresponding picture group blocks are compressed and encoded into transmission data according to each sampling rate, and the transmission data is sent to a decoding end.

[0249] and / or,

[0250] The computer executable instructions, when executed by a computer processor, are used to perform a video transmission method, the method also comprising:

[0251] Receiving transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames;

[0252] Reconstructing the transmission data into an initial key frame and an initial non-key frame;

[0253] Reconstructing the initial key frame into a key frame based on a preset deep reconstruction network, and guiding the initial non-key frame to be reconstructed into a non-key frame according to the key frame;

[0254] A picture group of a transmission video is constructed based on the key frame and the non-key frame.

[0255] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0256] It is worth noting that in the embodiment of the above-mentioned device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0257] Those skilled in the art will appreciate that all or some of the steps, devices, and functional modules / units in the above disclosed methods may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0258] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. The corresponding software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0259] The above content illustrates the preferred embodiments of the present application with reference to the accompanying drawings, but does not limit the scope of the present application. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application shall be within the scope of the present application.

Claims

1. A video transmission method, characterized in that: Applied to the encoding end, the method includes: Determine key frames and non-key frames of at least one set of picture groups of the video to be transmitted; Determine the sampling rates corresponding to the key frames and the non-key frames; The corresponding picture group blocks are compressed and encoded into transmission data according to each sampling rate, and the transmission data is sent to a decoding end.

2. The method according to claim 1, characterized in that: Determining the sampling rates corresponding to the key frames and the non-key frames includes at least one of the following: Pre-configuring fixed sampling rates for the key frame and the non-key frame respectively; The sampling rates corresponding to the key frames and the non-key frames are determined based on the hardware constraints of the encoding end.

3. The method according to claim 1, characterized in that: The step of compressing and encoding the corresponding picture group blocks into transmission data according to each sampling rate comprises: Dividing the key frames and the non-key frames in the picture group into block data according to a preset scale; Determine the block sampling rates corresponding to the block data of the key frame and the non-key frame respectively according to the sampling rates; The block data corresponding to the sampling according to the block sampling rate is sampled as the sampled block data; Each of the sampled block data is used as the transmission data.

4. The method according to claim 3, characterized in that: The determining the block sampling rates corresponding to the block data of the key frame and the non-key frame respectively according to the sampling rates includes: Determine the information difference of the pixel points in each of the block data according to the difference between the mean filtering results of the key frame and the non-key frame and the original data; Determine the number of sampling values ​​of the pixel points in the corresponding block data according to the information difference; Determine an iteration error according to the number of sampling values ​​of all the pixel points in the block data; When the iteration error satisfies a preset condition, the sampling rate corresponding to the number of sampling values ​​is used as the block sampling rate of the corresponding block data; When the iteration error does not meet the preset condition, the number of sampling values ​​is redistributed according to a preset allocation formula and the iteration error is redetermined until the block sampling rate of all the block data is determined.

5. The method according to claim 4, characterized in that: The determining the information difference of the pixel points in each block data according to the difference between the mean filtering result of the key frame and the non-key frame and the original data includes: Performing intra-frame mean filtering on each frame of the video image in the key frame and the non-key frame to determine a mean filtering result corresponding to each pixel point in each frame of the video image; Determine the difference between the original data of the pixel point in each block of data and the mean filtering result, and use the product of the difference and the preset self-selected hyperparameter as the information difference.

6. The method according to claim 4, characterized in that: The determining the number of sampling values ​​of the pixel points in the corresponding block data according to the information difference comprises: When the video image to which the block data belongs is the key frame, intercepting the information difference of the pixel points in the block data respectively according to the first sampling rate of the key frame to determine the number of sampling values; When the video image to which the block data belongs is the non-key frame, the information differences of the pixels in the block data are respectively intercepted according to the second sampling rate of the non-key frame to determine the number of sampling values.

7. The method according to claim 4, characterized in that: The determining of the iteration error according to the number of sampling values ​​of all the pixel points in the block data includes: Determine a sum of the number of sampling values ​​of all the pixels in the block data; The difference between the sum value and the preset sampling quantity corresponding to the block data is used as the iteration error.

8. The method according to claim 4, characterized in that: The preset allocation formula includes at least one of the following: Q = Q-sign(δ)×D; Among them, the Q represents the number of sampling values, the L represents the number of channels of the block data, the δ represents the iteration error, the D represents the iteration matrix randomly selected from the polynomial distribution, and the sign function represents the positive and negative value judgment function.

9. The method according to claim 4, characterized in that: The using the sampling rate corresponding to the number of sampling values ​​as the corresponding block sampling rate of the block data includes: Determine the corresponding block data dimension and the number of sampling values ​​for each block data; The ratio of the number of sampling values ​​to the dimension of the block data is used as the block sampling rate corresponding to the block data.

10. The method according to claim 1, characterized in that: The first sampling rate of the key frame is greater than the second sampling rate of the non-key frame.

11. A video transmission method, characterized in that: Applied to a decoding end, the method comprises: Receiving transmission data sent by the encoding end, wherein the transmission data is generated by block compression encoding according to the sampling rates corresponding to the key frames and non-key frames; Reconstructing the transmission data into an initial key frame and an initial non-key frame; Reconstructing the initial key frame into a key frame based on a preset deep reconstruction network, and guiding the initial non-key frame to be reconstructed into a non-key frame according to the key frame; A picture group of a transmission video is constructed based on the key frame and the non-key frame.

12. The method according to claim 11, characterized in that: The reconstructing the initial key frame into a key frame based on a preset deep reconstruction network, and guiding the initial non-key frame to be reconstructed into a non-constructed frame according to the key frame, includes: Processing the initial key frame according to the preset deep reconstruction network, and obtaining the key frame output result of the deep reconstruction network; Inputting the key frame output result and the initial non-key frame into a preset reconstruction network to generate a non-key frame output result; The above process is repeatedly performed until the key frame output result and the non-key frame output result meet the depth reconstruction condition, and the key frame output result is used as the key frame and the non-key frame output result is used as the non-key frame.

13. The method according to claim 12, characterized in that: The processing of the initial key frame according to the preset deep reconstruction network and obtaining the key frame output result of the deep reconstruction network includes: Obtain an iteration step of an iterative optimization module of the preset deep reconstruction network, and obtain an optimization result corresponding to the initial key frame according to the iteration step and a preset formula, wherein the preset formula includes: r (k) =x (k) -ρφ T (φx (k-1) -y), wherein said r (k) represents the optimization result of the initial key frame k, ρ represents the iteration step, φ represents the sampling matrix of the initial key frame, and φ T represents the transpose of the sampling matrix of the initial key frame, the x (k) represents the initial key frame, and y represents the transmission data corresponding to the initial key frame; Calling the feature extraction module of the preset deep reconstruction network to extract feature data from the optimization result as input to be convolved; Processing the input to be convolved into an output to be convolved according to the convolution reconstruction module of the preset deep reconstruction network; The remaining data in the optimization result after the feature data is extracted by the feature extraction module and the sum of the output to be convolved are used as the key frame output result.

14. The method according to claim 12, characterized in that: The convolutional reconstruction module of the preset deep reconstruction network includes at least a first convolution unit, a Soft function, a second convolution unit and a π convolution unit; The first convolution unit and the second convolution unit are connected via the Soft function, and the π convolution unit receives the output of the second convolution unit as input; Among them, the first convolution unit and the second convolution unit respectively include two layers of convolution kernels and one layer of activation function, and the π convolution unit includes two layers of convolution kernels and two layers of activation functions.

15. The method according to claim 12, characterized in that: The step of inputting the key frame output result and the initial non-key frame into a preset reconstruction network to generate a non-key frame output result includes: Respectively extracting the key frame output results corresponding to at least one of the initial key frames; Using the key frame output results and the initial non-key frame as input parameters of the preset reconstruction network; Extracting output parameters of the preset reconstruction network as the non-key frame output result; Among them, the preset convolutional network includes at least one of the following: convolutional neural network, LeNet-5 convolutional neural network, AlexNet convolutional neural network, VGG-16 convolutional neural network, ResNet convolutional neural network.

16. The method according to claim 12, characterized in that: The depth reconstruction condition includes at least one of the following: The preset deep reconstruction network and the number of calls of the preset reconstruction network are greater than or equal to a preset threshold; The image quality of the key frame output result and the non-key frame output result meets the preset quality requirement.

17. The method according to claim 11, characterized in that: Also includes: The preset deep reconstruction network is generated based on the preset training set training.

18. The method according to claim 17, characterized in that: The generating the preset deep reconstruction network based on the preset training set training includes: Acquire the preset training set, wherein the preset training set at least includes an Akiyo video sequence, a Coastguard video sequence, a Foreman video sequence, a Mother_daughter video sequence, a Paris video sequence, and a Silent video sequence; Inputting the preset training set into a pre-constructed initial deep reconstruction network, wherein the initial deep reconstruction network at least includes an iterative optimization module, a feature extraction module and a convolutional reconstruction module; Obtaining a training result of the initial deep reconstruction network, and determining a loss value of the training result according to a preset loss function; When the loss value meets a preset condition, using the initial reconstruction network as the preset deep reconstruction network; When the loss value does not meet the preset condition, the attribute parameters of the initial reconstruction network are adjusted and then the loss value of the training result is continued to be obtained until the training of the preset deep reconstruction network is completed.

19. The method according to claim 18, characterized in that: The preset loss function includes at least one of the following: When the picture group of the preset training set is a single key frame, the preset loss function is: When the picture group of the preset training set is a double key frame, the preset loss function is: Wherein, N is the number of picture groups in the preset training set, and the x key represents the original key frame data in the preset training set, the I key represents the reconstruction result of the key frames in the preset training set, K is the number of non-key frames in the picture group, and x k represents the original key frame data in the preset training set, the I k Represents the reconstruction result of key frames in the preset training set, N b is the number of key frame blocks, N k Then it is the key frame image block size N p , N p is the number of the convolutional reconstruction modules, Represents the output result of j trainings.

20. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video transmission method as described in any one of claims 1-10 or 11-19.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs are executed by one or more processors to implement the video transmission method as described in any one of claims 1-10 or 11-19.

Images