Video encoding and decoding and transmission method, device, equipment and storage medium
By generating comprehensive feature vectors and using generative adversarial networks to simulate video transmission environment, and adjusting video codec and transmission parameters in combination with strategy optimization model, the problems of video transmission stability and fluency in dynamic network environments in the existing technology are solved, achieving high Quality video transmission.
Patent Information
- Application Number
- CN202411237931.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-05
AI Technical Summary
Existing video encoding, codec and transmission technologies are difficult to ensure the stability and fluency of video transmission in dynamic network environments, and fail to fully consider the impact of changes in network status on video encoding, codec and transmission.
By generating comprehensive feature vectors, predicting network status, using generative adversarial network to simulate video transmission environment, combining strategy optimization model to determine optimization strategies, and adjusting video codec and transmission parameters to adapt to network changes.
In a dynamic network environment, the quality and stability of video transmission are significantly improved and the high-quality video transmission is ensured.
Smart Images

Figure CN118764627B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a video encoding and decoding and transmission method, device, equipment and storage medium. Background Art
[0002] In today's rapidly developing digital age, video content consumption has become an indispensable part of daily life. With the popularization of high-definition video formats such as 4K and 8K and the development of emerging technologies such as virtual reality (VR) and augmented reality (AR), higher requirements are placed on video encoding, decoding and transmission technologies.
[0003] A current video coding and transmission technology is: adjusting the video coding settings according to the complexity of the video content, the video coding settings include setting the resolution, frame rate and encoding quality, etc.; encoding the video according to the adjusted video coding settings to obtain the video data to be transmitted, and then transmitting the video data to be transmitted to the target device; in the process of transmitting the video to be transmitted, the adaptive bitstream technology (ABR) is used to adjust the transmission bit rate according to the user's network status to ensure the stability of the video transmission process.
[0004] However, current video coding and transmission technologies do not take into account the dynamic changes in network status (network congestion and bandwidth fluctuations, etc.) when performing video coding and transmission. When the network status is in a dynamic change, current video coding and transmission technologies are difficult to ensure the stability and smoothness of video transmission. It can be seen that the level of existing technologies in ensuring high-quality video transmission in a dynamic network environment needs to be improved. Summary of the invention
[0005] In order to improve the level of ensuring high-quality video transmission in a dynamic network environment, the embodiments of the present application provide a video encoding and decoding and transmission method, device, equipment and storage medium.
[0006] In a first aspect, an embodiment of the present application provides a video encoding, decoding and transmission method, including:
[0007] Generate a comprehensive feature vector based on the acquired video and the current state of the network;
[0008] Processing the comprehensive feature vector based on a preset network status prediction model to obtain a network prediction status;
[0009] Processing the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality;
[0010] The network simulation environment information is processed based on a policy optimization model to obtain an optimization strategy, and the video is encoded, decoded and transmitted based on the optimization strategy.
[0011] In a second aspect, an embodiment of the present application provides a video encoding, decoding and transmission device, including:
[0012] A vector calculation module, used to generate a comprehensive feature vector based on the acquired video and the current state of the network;
[0013] A state prediction module, used for processing the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state;
[0014] An information generation module, configured to obtain network simulation environment information by processing the current network state and the predicted network state based on a preset generative adversarial network, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality;
[0015] The strategy optimization module is used to process the network simulation environment information based on the strategy optimization model to obtain the optimization strategy, and encode, decode and transmit the video based on the optimization strategy.
[0016] In a third aspect, an embodiment of the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.
[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned method when executed by a processor.
[0018] In a fifth aspect, the embodiments of the present application further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0019] The embodiments of the above-mentioned video encoding and decoding and transmission methods, devices, equipment and storage media generate a comprehensive feature vector based on the acquired video and the current state of the network; process the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state; process the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, and the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency and video quality; process the network simulation environment information based on a strategy optimization model to obtain an optimization strategy, and encode, decode and transmit the video based on the optimization strategy. Through the implementation of the above-mentioned embodiments, the network state is predicted and simulated, and then the optimization strategy is determined in the simulated network simulation environment, and the video is encoded, decoded and transmitted according to the optimization strategy; in this way, before performing video encoding, decoding and transmission, the influence of the network state on video encoding, decoding and transmission is fully considered, and the optimal video encoding, decoding and transmission strategy is calculated, so that the level of ensuring high-quality video transmission in a dynamic network environment can be improved.
[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0022] Figure 1 This is an application environment diagram of a video encoding, decoding and transmission method in one embodiment of the present application;
[0023] Figure 2 A flow chart of a video encoding, decoding and transmission method provided in Embodiment 1 of the present application;
[0024] Figure 3 A flow chart of a video encoding, decoding and transmission method provided in Embodiment 2 of the present application;
[0025] Figure 4 A flow chart of a video encoding, decoding and transmission method provided in Embodiment 3 of the present application;
[0026] Figure 5 A flow chart of a video encoding, decoding and transmission method provided in Embodiment 4 of the present application;
[0027] Figure 6A flow chart of a video encoding, decoding and transmission method provided in Embodiment 5 of the present application;
[0028] Figure 7 A schematic diagram of the structure of a video encoding, decoding and transmission device provided in one embodiment of the present application;
[0029] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application;
[0030] Fig. 9 An internal structure diagram of a computer-readable storage medium provided in one embodiment of the present application. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0033] In this article, the term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.
[0034] To solve the above problems, the present disclosure provides a video encoding and decoding and transmission method, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.
[0035] Embodiment 1
[0036] Figure 2 A video encoding and decoding and transmission method flow chart provided in Example 1 of the present application, refer to Figure 2 The method may be performed by a device for performing the method, and the device may be implemented by software and / or hardware. The method includes:
[0037] S110, generating a comprehensive feature vector based on the acquired video and the current state of the network.
[0038] Among them, the video is a video stream that needs to be encoded, decoded and transmitted, and the video stream includes multiple continuous video frames. The network is a network used for video transmission, and the current state of the network at least includes network bandwidth, network delay, packet loss rate and video buffer information; the comprehensive feature vector is a combination of the spatial features of the video content and the time series features of the current state of the network. The comprehensive feature vector is used to understand the relationship between the video content and the current state of the network, and to provide decision support for the subsequent encoding, decoding and transmission of the video; in addition, the acquired video and the current state of the network can be processed by a feature extraction network in a preset video processing system to obtain a comprehensive feature vector, and the feature extraction network includes a first feature extraction network for processing the video and a second feature extraction network for processing the current state of the network; it should be noted that the above-mentioned video processing system is used to encode, decode and transmit the acquired video; in this embodiment, the acquired video is recorded as V, the current state of the network is recorded as N, and the comprehensive feature vector is recorded as F.
[0039] Specifically, the video V is processed by a first feature extraction network corresponding to the video V to obtain a first processing result The current state of the network is processed by a second feature extraction network corresponding to the current state of the network to obtain a second processing result , and then the video processing system combines the first processing result With the second processing result The comprehensive feature vector is recorded as F.
[0040] S120: Processing the comprehensive feature vector based on a preset network status prediction model to obtain a predicted network status.
[0041] The video processing system is also provided with a network state prediction model that communicates with the feature extraction network. The network state prediction model is used to predict the network state corresponding to a specific time t. The network prediction state is the predicted network state corresponding to a specific time t. The network state at least includes the network bandwidth, network delay, packet loss rate and video buffer information corresponding to the specific time t. The video is composed of video frames corresponding to different times. The comprehensive feature vector F specifically includes the network state corresponding to the video frames at different times. In this embodiment, the network state prediction model calculates the network state corresponding to a specific time t The calculation formula is as follows:
[0042]
[0043] It should be noted that is the network status corresponding to a specific time t, k and m are preset upper limits, is the coefficient of the autoregressive model in the network status prediction model, is the network status corresponding to the video frame at time (ti), is the coefficient of the moving average model in the network status prediction model, is the change in the network state corresponding to the video frame at time (tj), , γ is the constant term or intercept of the network status prediction model point, which is used to provide the baseline value of the network status prediction model. It is an error item, which is used to characterize the difference between the predicted network status and the actual network status.
[0044] Specifically, the network states corresponding to the video frames at different times in the comprehensive feature vector F are The network state prediction model is used to predict the network state at a specific time t, which is the network prediction state. .
[0045] S130. Processing the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality.
[0046] Among them, the generative adversarial network (GAN) is set in the video processing system and communicates with the network state prediction model and the feature extraction network; the generative adversarial network (GAN) is used to process the current state N of the network and the predicted state of the network , thereby obtaining the network simulation environment information, which is used to simulate the network environment corresponding to the video during encoding, decoding and transmission; the video transmission delay time D(t) is the delay when the video processing system transmits the video V, the video encoding and decoding efficiency E(t) is the efficiency of the video processing system when encoding and decoding the video, and the video quality Q(t) is the quality level of the video transmitted from the video processing system to the user end.
[0047] Specifically, the current network state N obtained in step S110 is compared with the network prediction state generated in step S120. The data is input into a preset generative adversarial network (GAN) for processing to obtain network simulation environment information. In this embodiment, the network simulation environment information at least includes video transmission delay time D(t), video encoding and decoding efficiency E(t) and video quality Q(t).
[0048] S140, processing the network simulation environment information based on a policy optimization model to obtain an optimization strategy, and encoding, decoding and transmitting the video based on the optimization strategy.
[0049] Among them, the policy optimization model is set in the video processing system and communicates with the generative adversarial network (GAN). The policy optimization model is used to process the network simulation environment information output by the generative adversarial network (GAN) and generate an optimization strategy. The optimization strategy includes the best encoding and decoding scheme and the best video transmission scheme adapted to the current state of the network. The encoding and decoding scheme includes at least H.264 encoding and decoding technology, H.265 encoding and decoding technology, and AV1 encoding and decoding technology, etc. The video transmission scheme includes transmission control methods with different bit rates; the optimization strategy is used to control the video processing system to encode, decode and transmit the acquired video V.
[0050] Specifically, the network simulation environment information is input into the policy optimization model for processing, thereby obtaining an optimization strategy. Further, the video processing system controls the process of encoding, decoding and transmitting the acquired video V according to the optimization strategy.
[0051] It should be noted that this embodiment generates a comprehensive feature vector based on the acquired video and the current state of the network; processes the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state; processes the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, and the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality; processes the network simulation environment information based on a strategy optimization model to obtain an optimization strategy, and encodes, decodes, and transmits the video based on the optimization strategy. Through the implementation of the above embodiments, the network state is predicted and simulated, and then the optimization strategy is determined in the simulated network simulation environment, and the video is encoded, decoded, and transmitted according to the optimization strategy; in this way, before encoding, decoding, and transmitting the video, the influence of the network state on the video encoding, decoding, and transmission is fully considered, and the optimal video encoding, decoding, and transmission strategy is calculated, so that the level of ensuring high-quality video transmission in a dynamic network environment can be improved.
[0052] Embodiment 2
[0053] Figure 3 A video encoding and decoding and transmission method flow chart provided in Example 2 of the present application, refer to Figure 3 , the method optimizes the "generating a comprehensive feature vector based on the acquired video and the current state of the network" in the first embodiment; it should be noted that, for the part not described in detail in this embodiment, reference can be made to the description of other embodiments, and the method includes:
[0054] S211, extracting features from the acquired video to obtain video features.
[0055] The feature extraction network in the first embodiment includes a video feature extraction network, in which a two-dimensional convolution layer Conv2D is provided, and the two-dimensional convolution layer Conv2D is used to perform two-dimensional convolution on the video V to obtain primary video features; and the convolution parameter of the two-dimensional convolution layer Conv2D is recorded as ; The video feature extraction network also has a nonlinear activation function Relu, which is used to further process the primary video features to obtain the target video features , the nonlinear activation function Relu is specifically used to increase the expression ability of the video feature extraction network. The nonlinear activation function Relu can set all negative values in the primary video features to 0, thereby introducing nonlinear video features; the calculation formula corresponding to the video feature extraction network is:
[0056]
[0057] Specifically, the acquired video V is input into the video feature extraction network in the feature extraction network for processing. The video feature extraction network first performs two-dimensional convolution on the video frame of the video V through the two-dimensional convolution layer Conv2D to obtain the primary video feature Furthermore, the video feature extraction network uses the nonlinear activation function Relu to extract the primary video features. Processing to obtain the target video features .
[0058] S212: Extract features from the acquired current network state to obtain network state features.
[0059] Among them, the feature extraction network in the first embodiment also includes a network state feature extraction network, and the network state feature extraction network is specifically a long short-term memory network (LSTM). The long short-term memory network (LSTM) is a special recurrent neural network (RNN) that can process and memorize long-term information; in this embodiment, the long short-term memory network (LSTM) is specifically used to extract network state features from the current state N of the network , and the network parameters of the long short-term memory network (LSTM) are recorded as , It is used to determine how LSTM extracts time series features from the sequence of the current state of the network; the calculation formula corresponding to the network state feature extraction network is:
[0060]
[0061] Specifically, the current state of the network is input into the network state feature extraction network for processing, thereby obtaining the network state feature .
[0062] S213: Perform feature connection on the video feature and the network status feature to obtain a comprehensive feature vector.
[0063] Among them, the feature extraction network in the first embodiment also includes a feature connection function Concat, which is used to extract the video features. With the network status characteristics Perform feature connection to obtain a comprehensive feature vector F; the calculation expression of the comprehensive feature vector F is:
[0064]
[0065] Specifically, the video features generated in step S211 are The network status characteristics generated in step S212 Input it into the feature connection function Concat for processing to obtain the comprehensive feature vector F.
[0066] S220: Process the comprehensive feature vector based on a preset network status prediction model to obtain a predicted network status.
[0067] S230. Processing the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality.
[0068] S240: Process the network simulation environment information based on a policy optimization model to obtain an optimization strategy, and encode, decode and transmit the video based on the optimization strategy.
[0069] In one embodiment, the preset steps of the generative adversarial network in the first embodiment are optimized, and the steps of constructing the generative adversarial network include:
[0070] S1. Build a generator based on the preset generator network parameters.
[0071] Among them, the generative adversarial network (GAN) includes two core components: the generator G and the discriminator D. The generator G is used to generate network state data that is as close to the real one as possible, and the discriminator D is used to distinguish the generated network state data from the real network state data. Through the generator G and the discriminator D, the generative adversarial network (GAN) can generate more realistic network simulation environment information; the calculation formula corresponding to the generator G is as follows:
[0072]
[0073] It should be noted that is the network state data calculated by generator G, z is the random noise input into generator G, are the generator network parameters.
[0074] Specifically, according to the preset generator network parameters Construct the generator G(z, ).
[0075] S2. Construct a discriminator based on the discriminator network parameters, the preset fully connected layer network and the activation function.
[0076] Among them, the calculation formula corresponding to the discriminator D is as follows:
[0077]
[0078] It should be noted that is the network status data calculated by the generator G, is the discriminator network parameter, FC is the preset fully connected layer, which is used to input the network status data in the discriminator D Converted into a probability value, Sigmoid is the preset activation function, which is used to convert the probability value output by the fully connected layer FC into a probability value between 0 and 1.
[0079] Specifically, according to the discriminator network parameters , the preset fully connected layer network FC and activation function Sigmoid construct the discriminator D according to the calculation formula of the discriminator D.
[0080] S3. Construct a generative adversarial network based on the generator and the discriminator.
[0081] Specifically, the generator G constructed in step S1 establishes a communication relationship with the discriminator constructed in step S2, that is, the output of the generator G As the input of the discriminator D, a generative adversarial network (GAN) is then constructed based on the two core components of the generator G and the discriminator D.
[0082] Embodiment 3
[0083] Figure 4 A video encoding and decoding and transmission method flow chart provided in Example 3 of the present application, refer to Figure 4 The method optimizes the "processing the network simulation environment information based on the policy optimization model to obtain the optimization strategy" in the first embodiment; it should be noted that, for the part not described in detail in this embodiment, reference can be made to the description of other embodiments, and the method includes:
[0084] S310: Generate a comprehensive feature vector based on the acquired video and the current state of the network.
[0085] S320: Process the comprehensive feature vector based on a preset network status prediction model to obtain a predicted network status.
[0086] S330. Process the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality.
[0087] S341. Process the video transmission delay time, the video encoding and decoding efficiency, and the video quality based on the reward function in the strategy optimization model to obtain a reward result.
[0088] Among them, a reward function R(t) is provided in the strategy optimization model, and the reward function R(t) is used to calculate the reward obtained by a strategy adopted in a current state of a certain network. The strategy includes a coding scheme and a video transmission scheme. In implementation, in order to meet the needs of different current states of the network, a plurality of strategies are provided in the video processing system. In this embodiment, the network simulation environment information specifically includes the video transmission delay time D(t), the video coding efficiency E(t) and the video quality Q(t). The strategy optimization model is preset with a first weight parameter for the video transmission delay time D(t). , a second weight parameter is preset for the video encoding and decoding efficiency E(t) , a third weight parameter is preset for the video quality Q(t) ; The reward result is the output result of the reward function R(t), and the calculation formula corresponding to the reward function R(t) is as follows:
[0089]
[0090] in, It is used to smooth the impact of video transmission delay time D(t) on the reward calculation results. The lower the video transmission delay time D(t), the higher the reward result corresponding to the video transmission delay time D(t). log(E(t)) is used to smooth the impact of video encoding and decoding efficiency E(t) on the reward calculation results. The higher the video encoding and decoding efficiency E(t), the higher the reward result corresponding to the video encoding and decoding efficiency E(t). Video quality Q(t) is an important user experience indicator for the video. The higher the video quality Q(t), the higher the reward result corresponding to the video quality Q(t).
[0091] Specifically, the video transmission delay time D(t), video encoding and decoding efficiency E(t), and video quality Q(t) in the network simulation environment information are input into the reward function R(t) for calculation to obtain the reward result.
[0092] S342: Determine an optimization strategy based on the reward result and a strategy selection function in the strategy optimization model.
[0093] Among them, the reward result corresponding to each strategy can be calculated through the strategy optimization model. The higher the reward result, the more likely it is that the corresponding strategy can transmit the best quality video in a dynamic network environment. In the strategy selection function setting and the strategy optimization model, the strategy selection function is a function that determines the corresponding strategy when the reward result is maximized.
[0094] Specifically, the strategy optimization model calculates the reward results corresponding to various strategies under the current state of the network corresponding to a certain time t through the reward function R(t), and then selects the strategy corresponding to the maximum reward result as the optimization strategy through the strategy selection function.
[0095] S343: Encode, decode and transmit the video based on the optimization strategy.
[0096] The optimization strategy includes a specific video encoding and decoding solution and a video transmission solution.
[0097] Specifically, the video V is encoded, decoded and transmitted according to a specific video encoding and decoding scheme and a video transmission scheme corresponding to the optimization strategy.
[0098] Embodiment 4
[0099] Figure 5 A video encoding and decoding and transmission method flow chart provided in Example 4 of the present application, refer to Figure 5 , the method optimizes the "determining the optimization strategy based on the reward result and the strategy selection function in the strategy optimization model" in the third embodiment; it should be noted that, for the part not described in detail in this embodiment, please refer to the description of other embodiments, the method includes:
[0100] S410: Generate a comprehensive feature vector based on the acquired video and the current state of the network.
[0101] S420: Process the comprehensive feature vector based on a preset network status prediction model to obtain a predicted network status.
[0102] S430. Process the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, where the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality.
[0103] S441. Process the video transmission delay time, the video encoding and decoding efficiency, and the video quality based on the reward function in the strategy optimization model to obtain a reward result.
[0104] S4421. Update the function parameters of the strategy selection function based on the reward result to obtain a target strategy selection function.
[0105] Among them, the strategy selection function has corresponding function parameters , the calculated reward result can be used to select the function parameters of the strategy Update to get the updated function parameters , and then update the function parameters Updating the policy selection function can obtain an updated policy selection function, and the target policy selection function is also the updated policy selection function.
[0106] Specifically, after the reward result is calculated through step S441, the current function parameters of the strategy selection function are further updated through the reward result to obtain updated function parameters, and then the updated function parameters are updated to the strategy selection function to obtain the target strategy selection function.
[0107] S4422. Determine an optimization strategy based on the target strategy selection function and the reward result.
[0108] Specifically, the strategy corresponding to the reward result with the largest value is determined by the target strategy selection function as the optimization strategy.
[0109] S443: Encode, decode and transmit the video based on the optimization strategy.
[0110] Embodiment 5
[0111] Figure 6 A video encoding and decoding and transmission method flow chart provided in Example 1 of the present application, refer to Figure 6 The method may be performed by a device for performing the method, and the device may be implemented by software and / or hardware. The method includes:
[0112] S510: Generate a comprehensive feature vector based on the acquired video and the current state of the network.
[0113] S520: Process the comprehensive feature vector based on a preset network status prediction model to obtain a predicted network status.
[0114] S530. Process the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, where the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality.
[0115] S540: Process the network simulation environment information based on a policy optimization model to obtain an optimization strategy, and encode, decode and transmit the video based on the optimization strategy.
[0116] S550: Process the current state of the network and the acquired user feedback information based on a preset loss function to obtain a loss result.
[0117] Among them, the video is encoded and decoded through the optimization strategy and transmitted to the user end, and the user end feeds back the video quality evaluation to the video processing system. The user feedback information is also the video quality evaluation fed back by the user end; the loss function is set in the video processing system, and the loss function is used to calculate the loss result according to the current state N of the network and the obtained user feedback information. The loss result is used to characterize the quality of video encoding, decoding and transmission through the current optimization strategy.
[0118] Specifically, the current state N of the network and the acquired user feedback information are input into a preset loss function for processing to obtain a loss result.
[0119] S560, calculating updated model parameters based on the loss result, a preset learning rate, and current model parameters of the strategy optimization model.
[0120] Among them, the updated model parameters are recorded as , updated model parameters The calculation formula is as follows:
[0121]
[0122] It should be noted that is the current model parameter of the strategy optimization model, eta is the preset learning rate, J( ) is the gradient of the loss result.
[0123] Specifically, calculate the gradient of the loss result J( ), then calculate J( ) multiplied by the preset learning rate , further, calculate the current model parameters of the strategy optimization model Multiplying the above The difference between the two gets the updated model parameters .
[0124] S570: Update the policy optimization model based on the updated model parameters to obtain a target policy optimization model, and use the target policy optimization model as a new policy optimization model.
[0125] Among them, the updated model parameters are used to replace the original current model parameters in the policy optimization model, so as to achieve the update of the policy optimization model, and the target policy optimization model is also the updated policy optimization model.
[0126] Specifically, the model parameters will be updated Update to the strategy optimization model to replace the original current model parameters , thereby obtaining a target policy optimization model. Further, the target policy optimization model is used as a new policy optimization model in step S540, and a video encoding, decoding and transmission method shown in this embodiment is re-executed.
[0127] It should be noted that the policy optimization model can be updated in real time by obtaining the current network status and user feedback information, thereby improving the adaptability of the policy optimization model, so that the truly optimal strategy can be determined through the updated policy optimization model, thereby further improving the level of ensuring high-quality video transmission in a dynamic network environment.
[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0129] Embodiment 6
[0130] Based on the same inventive concept, the embodiment of the present disclosure also provides a video coding and transmission device for implementing the above-mentioned video coding and transmission method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video coding and transmission device embodiments provided below can refer to the above-mentioned limitations on the video coding and transmission method, and will not be repeated here.
[0131] In this embodiment, Figure 7 As shown, a video encoding, decoding and transmission device is provided, comprising:
[0132] A vector calculation module, used to generate a comprehensive feature vector based on the acquired video and the current state of the network;
[0133] A state prediction module, used for processing the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state;
[0134] An information generation module, configured to obtain network simulation environment information by processing the current network state and the predicted network state based on a preset generative adversarial network, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality;
[0135] The strategy optimization module is used to process the network simulation environment information based on the strategy optimization model to obtain the optimization strategy, and encode, decode and transmit the video based on the optimization strategy.
[0136] Each module in the above video encoding, decoding and transmission device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0137] It should be noted that this embodiment generates a comprehensive feature vector based on the acquired video and the current state of the network; processes the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state; processes the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, and the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality; processes the network simulation environment information based on a strategy optimization model to obtain an optimization strategy, and encodes, decodes, and transmits the video based on the optimization strategy. Through the implementation of the above embodiments, the network state is predicted and simulated, and then the optimization strategy is determined in the simulated network simulation environment, and the video is encoded, decoded, and transmitted according to the optimization strategy; in this way, before encoding, decoding, and transmitting the video, the influence of the network state on the video encoding, decoding, and transmission is fully considered, and the optimal video encoding, decoding, and transmission strategy is calculated, so that the level of ensuring high-quality video transmission in a dynamic network environment can be improved.
[0138] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video encoding, decoding and transmission method is implemented.
[0139] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the computer device to which the scheme of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0140] In one embodiment, a computer readable storage medium is provided. Fig. 9As shown, a computer program is stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0141] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0143] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (DCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present disclosure may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present disclosure may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to this.
[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0145] The above-described embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the attached claims.
Claims
1. A video encoding, decoding and transmission method, characterized in that: include: Generate a comprehensive feature vector based on the acquired video and the current state of the network; Processing the comprehensive feature vector based on a preset network status prediction model to obtain a network prediction status; Processing the current state of the network and the predicted state of the network based on a preset generative adversarial network to obtain network simulation environment information, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality; Processing the network simulation environment information based on a policy optimization model to obtain an optimization strategy, and encoding, decoding and transmitting the video based on the optimization strategy; The step of generating a comprehensive feature vector based on the acquired video and the current state of the network includes: Perform feature extraction on the acquired video to obtain video features; Perform feature extraction on the acquired current network state to obtain network state features; Performing feature connection on the video feature and the network status feature to obtain a comprehensive feature vector; The step of constructing the generative adversarial network includes: Build a generator based on the preset generator network parameters; Construct a discriminator based on the discriminator network parameters, the preset fully connected layer network and the activation function; Constructing a generative adversarial network based on the generator and the discriminator; Wherein, the process of processing the network simulation environment information based on the policy optimization model to obtain the optimization strategy includes: Processing the video transmission delay time, the video encoding and decoding efficiency, and the video quality based on the reward function in the strategy optimization model to obtain a reward result; An optimization strategy is determined based on the reward result and a strategy selection function in the strategy optimization model.
2. The method according to claim 1, characterized in that: The determining of the optimization strategy based on the reward result and the strategy selection function in the strategy optimization model includes: Based on the reward result, the function parameters of the strategy selection function are updated to obtain a target strategy selection function; An optimization strategy is determined based on the target strategy selection function and the reward result.
3. The method according to claim 1, characterized in that Also includes: Based on the preset loss function, the current state of the network and the obtained user feedback information are processed to obtain the loss result; Calculating updated model parameters based on the loss result, a preset learning rate, and current model parameters of the strategy optimization model; The policy optimization model is updated based on the updated model parameters to obtain a target policy optimization model, and the target policy optimization model is used as a new policy optimization model.
4. A video encoding, decoding and transmission device, characterized in that: The device comprises: A vector calculation module, used to generate a comprehensive feature vector based on the acquired video and the current state of the network; A state prediction module, used for processing the comprehensive feature vector based on a preset network state prediction model to obtain a network prediction state; An information generation module, configured to obtain network simulation environment information by processing the current network state and the predicted network state based on a preset generative adversarial network, wherein the network simulation environment information at least includes video transmission delay time, video encoding and decoding efficiency, and video quality; A policy optimization module, used for processing the network simulation environment information based on a policy optimization model to obtain an optimization policy, and encoding, decoding and transmitting the video based on the optimization policy; The step of generating a comprehensive feature vector based on the acquired video and the current state of the network includes: Perform feature extraction on the acquired video to obtain video features; Perform feature extraction on the acquired current network state to obtain network state features; Performing feature connection on the video feature and the network status feature to obtain a comprehensive feature vector; The step of constructing the generative adversarial network includes: Build a generator based on the preset generator network parameters; Construct a discriminator based on the discriminator network parameters, the preset fully connected layer network and the activation function; Constructing a generative adversarial network based on the generator and the discriminator; Wherein, the process of processing the network simulation environment information based on the policy optimization model to obtain the optimization strategy includes: Processing the video transmission delay time, the video encoding and decoding efficiency, and the video quality based on the reward function in the strategy optimization model to obtain a reward result; An optimization strategy is determined based on the reward result and a strategy selection function in the strategy optimization model.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.
7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.
Citation Information
Patent Citations
Method for dynamically adapting video image parameters for facilitating subsequent applications
CN103518210A
Real-time video code rate self-adaptive regulation and control method and system based on reinforcement learning
CN111901642A
A method and system for resisting weak network for audio and video transmission
CN118101941A
Deep-reinforcement-learning-based adaptive efficient resource allocation method for cloud data center
WO2023184939A1
Video transmission method, apparatus and device, and storage medium
WO2024120134A1
