Adaptive adjustment video stream encoding optimization method, device, equipment and product

Abstract

The application discloses a self-adaptive video stream coding optimization method and device, equipment and product, and relates to the technical field of multimedia communication. The method comprises the following steps: inputting image basic data of a video stream into a video feature extraction model; calculating image feature differences between video frames by the video feature extraction model to obtain target image features; determining user viewing preferences by using a pre-trained binary classifier according to the image basic data and the target image features; and performing coding optimization on a current video by a video stream coding model based on a current network state and the user viewing preferences. The pre-trained binary classifier is used to determine the user viewing preferences, the viewing preferences of different users for different video contents are determined, the viewing preferences of the users and real-time network conditions are combined, the optimization points are considered from multiple angles, the best coding configuration is selected for real-time interactive video streams, and the self-adaptability of the interactive video streams in different scenes is improved, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia communication, in particular to a video stream coding optimization method and device with adaptive adjustment, equipment and product. BACKGROUND

[0002] Video live broadcast and remote conference are important services on the Internet today, and these services all rely on the transmission of real-time interactive video streams. In immersive experience applications such as Virtual Reality (VR) and Augmented Reality (AR), interactive video streams also have an important influence on user experience. Through experiments, it is found that different video contents can lead users to have different viewing preferences. For example, for sports competition videos, due to the rapid changes in the picture and the high-intensity sports scenes, users have very high requirements for the frame rate of the video. High frame rate can ensure the smoothness and continuity of the picture, avoid missing key moments, and thus guarantee the viewing experience of users. For natural scenery videos with colorful colors, users pay more attention to the delicacy and clarity of the picture. Such videos are rich in details and color levels, and users have more stringent requirements for resolution. High resolution can better display the delicate texture and color changes of natural scenery, and bring users a more realistic visual enjoyment.

[0003] Traditional coding and decoding methods have great difficulty in real-time and accurate perception of network conditions, and do not consider video scene changes, so the optimization consideration factors are relatively single. SUMMARY

[0004] The main purpose of the present application is to provide a video stream coding optimization method and device with adaptive adjustment, equipment and product, aiming to solve the technical problem of single optimization consideration factor.

[0005] To achieve the above purpose, the present application provides a video stream coding optimization method with adaptive adjustment, which comprises:

[0006] inputting image basic data of a video stream into a video feature extraction model, calculating the image feature difference between video frames through the video feature extraction model, and obtaining target image features;

[0007] determining user viewing preferences by using a pre-trained binary classifier according to the image basic data and the target image features;

[0008] based on the current network state and the user viewing preferences, performing coding optimization on the current video by using a video stream coding model.

[0009] In an embodiment, the video stream encoding model comprises a state space, an action space and a reward function, and the step of encoding the current video based on the current network state and the user viewing preference through the video stream encoding model comprises:

[0010] The current network state and the user viewing preference are defined as the state space, and a plurality of gear change demand values are defined as the action space;

[0011] Based on the state space, an initial optimized encoding configuration is determined according to the action space, and the current encoding configuration is adjusted according to the initial optimized encoding configuration;

[0012] A reward function at the current time is calculated based on the current encoding configuration, the encoding configuration is adjusted according to the reward function at the current time, and a target optimized encoding configuration is determined;

[0013] The current video is encoded and optimized according to the target optimized encoding configuration.

[0014] In an embodiment, the step of calculating a reward function at the current time based on the current encoding configuration, adjusting the encoding configuration according to the reward function at the current time, and determining a target optimized encoding configuration comprises:

[0015] A reward function at the current time is calculated based on the current encoding configuration;

[0016] If the reward function at the current time is greater than a preset threshold, the current encoding configuration is output as the target encoding configuration.

[0017] In an embodiment, the step of calculating a reward function at the current time based on the current encoding configuration comprises:

[0018] If the reward function at the current time is less than or equal to the preset threshold, the step of determining an initial optimized encoding configuration according to the current encoding configuration and the action space is returned until the reward function at the current time is greater than the preset threshold.

[0019] In an embodiment, the step of determining an initial optimized encoding configuration according to the action space comprises:

[0020] An encoding configuration decision set is determined according to the action space;

[0021] A decision probability corresponding to each encoding configuration decision in the encoding configuration set is determined according to the user viewing preference;

[0022] According to the decision probability, one configuration decision is selected from the encoding configuration set as the initial optimized encoding configuration.

[0023] In an embodiment, the user viewing preference comprises a resolution preference or a frame rate preference, and the step of determining the user viewing preference according to the image base data and the target image feature by using a pre-trained binary classifier comprises:

[0024] inputting the image base data of the current video and the target image feature into the binary classifier to obtain an output value output by the binary classifier;

[0025] when the output value is greater than a preset preference threshold, taking the resolution preference of the user as the user viewing preference;

[0026] when the output value is less than or equal to the preset preference threshold, taking the frame rate preference of the user as the user viewing preference.

[0027] In addition, to achieve the above-mentioned purpose, the present application further provides a video stream encoding optimization device for adaptive adjustment, which comprises:

[0028] a target image feature acquisition module, configured to input image base data of a video stream into a video feature extraction model, calculate image feature differences between video frames by using the video feature extraction model, and obtain target image features;

[0029] a user preference confirmation module, configured to determine a user viewing preference by using a pre-trained binary classifier according to the image base data and the target image features;

[0030] an encoding optimization module, configured to perform encoding optimization on a current video by using a video stream encoding model based on a current network state and the user viewing preference.

[0031] In addition, to achieve the above-mentioned purpose, the present application further provides a video stream encoding optimization device for adaptive adjustment, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the above-mentioned video stream encoding optimization method for adaptive adjustment.

[0032] In addition, to achieve the above-mentioned purpose, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned video stream encoding optimization method for adaptive adjustment.

[0033] In addition, to achieve the above-mentioned purpose, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned video stream encoding optimization method for adaptive adjustment.

[0034] The one or more technical solutions provided in the application have at least the following technical effects:

[0035] The image basis data of the video stream is input into a video feature extraction model, the image feature difference between video frames is calculated through the video feature extraction model, and target image features are obtained; the pre-trained binary classifier is used to determine the user viewing preference according to the image basis data and the target image features; and the current video is encoded and optimized through a video stream encoding model based on the current network state and the user viewing preference. The pre-trained binary classifier is used to determine the user viewing preference, the viewing preference of different users for different video contents is determined, the viewing preference of the user and the real-time network condition are combined, the optimization point is considered from multiple angles, the best encoding configuration is selected for the real-time interactive video stream, the self-adaptability of the interactive video stream in different scenes is improved, and the user experience is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0038] Figure 1 The system device architecture of the embodiments of the application is shown in the figure;

[0039] Figure 2 The flowchart of the first embodiment of the adaptive video stream encoding optimization method of the application is shown in the figure;

[0040] Figure 3 The flowchart of the second embodiment of the adaptive video stream encoding optimization method of the application is shown in the figure;

[0041] Figure 4 The user preference perception method of the embodiments of the application is shown in the figure;

[0042] Figure 5 The flowchart of the third embodiment of the adaptive video stream encoding optimization method of the application is shown in the figure;

[0043] Figure 6 The user viewing preference action design flowchart of the embodiments of the application is shown in the figure;

[0044] Figure 7 The encoding configuration decision method of the embodiments of the application is shown in the figure;

[0045] Figure 8 A module structure diagram of a video stream encoding optimization device for adaptive adjustment of an embodiment of the present application;

[0046] Figure 9 A device structure diagram of a hardware operating environment involved in a video stream encoding optimization method for adaptive adjustment in an embodiment of the present application.

[0047] The purposes, functional features and advantages of the present application will be further described with reference to the accompanying drawings in conjunction with embodiments. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0049] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0050] Since the traditional encoding and decoding method is based on the network transmission situation, such as the current network perceived bandwidth, delay, jitter, etc., to determine the highest resolution and frame rate of the video that can be transmitted, and then to encode the video in real time based on this resolution. However, due to the jitter of the cellular network, WiFi and other wireless networks, it is difficult to accurately perceive the network situation in real time. Moreover, in a weak network (small transmission bandwidth, large delay) environment, if the video resolution during video encoding and decoding is completely determined by the network state, the transmittable video resolution is often poor, and the user experience is not high. In addition, different video content will lead users to have different viewing preferences. For example, for sports videos, due to the rapid change of the picture and the high-intensity sports scene, users have very high requirements for the video frame rate. High frame rate can ensure the smoothness and continuity of the picture, avoid missing key moments, and thus guarantee the user's viewing experience. For natural scenery videos with colorful colors, users pay more attention to the delicacy and clarity of the picture. Such videos are rich in details and color levels, and users have more stringent requirements for resolution. High resolution can better display the delicate texture and color change of natural scenery, and bring users a more realistic visual enjoyment. If the video quality decreases, the key details may become blurred, which seriously affects the user's viewing experience.

[0051] To address the need to combine the aforementioned factors requiring optimization, this application provides a solution. The basic image data of the video stream is input into a video feature extraction model. This model calculates the differences in image features between video frames to obtain target image features. Based on the basic image data and target image features, a pre-trained binary classifier is used to determine user viewing preferences. Finally, based on the current network state and user viewing preferences, the video stream encoding model optimizes the encoding of the current video. By using a pre-trained binary classifier to determine user viewing preferences, different users' viewing preferences for different video content are identified. By combining user viewing preferences with real-time network conditions, optimization points are considered from multiple perspectives, selecting the optimal encoding configuration for the real-time interactive video stream. This improves the adaptability of the interactive video stream in different scenarios while enhancing the user experience.

[0052] It should be noted that, in one embodiment of this application, the system device architecture diagram can be as follows: Figure 1 As shown, the system consists of a control center and two core methods: user preference awareness and encoding configuration decision-making (shown in the dashed box diagram). The workflow is as follows: When the sender of the real-time interactive video stream needs to change its encoding configuration, the control center is activated via control flow information, and the collected statistical information from the transport layer and video encoder is transmitted to the control center. Then, the control center activates the user preference awareness method, transmitting the current frame number, video resolution, and frame rate information to this method. The user preference awareness module extracts low-level image features of the currently playing video based on the current information, determines the user's viewing preference for the current video, and sends the result to the encoding configuration decision-making method. Upon receiving the user viewing preference transmitted by the user preference awareness module and the transport layer and video encoder statistical information transmitted by the control center, the encoding configuration decision-making module uses a reinforcement learning algorithm to determine the latest encoding configuration strategy in real time.

[0053] Based on this, embodiments of this application provide an adaptive adjustment video stream coding optimization method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the adaptive adjustment video stream coding optimization method of this application.

[0054] In this embodiment, the adaptive adjustment video stream coding optimization method includes steps S10 to S30:

[0055] Step S10: Input the basic image data of the video stream into the video feature extraction model, calculate the image feature differences between video frames through the video feature extraction model, and obtain the target image features.

[0056] It should be noted that the image basis data of the video stream can be pixel information in the video frame and related metadata. For example, the image basis data of the video stream can include a frame number being currently encoded, configuration information corresponding to the frame number, and pixel information of each frame image, etc. The pixel information of the image can be understood as the original pixel data of the current video frame. For example, the pixel information can include color information (such as RGB value), brightness, contrast, etc. The video feature extraction model can quickly locate and extract the required video frame, and calculate the image feature difference between the video frames according to the image basis data of each video frame. The target image feature can be a high-level feature vector representing the content of the video frame after being processed by the feature extraction model.

[0057] For example, the frame number being currently encoded and the current configuration information can be obtained from the control center. The frame number helps the video feature extraction module to accurately understand which part of the video content is currently being processed by the video encoder and the configuration information corresponding to different content. With this information, the corresponding video frame image is further retrieved from the local file. Since the transmitted video content is determined, the video to be transmitted is encoded into different resolution versions in advance by using a codec tool, and stored locally. The video feature extraction module can quickly locate and extract the required video frame, and it can calculate the image feature difference between the video frames. In order to ensure the real-time decision, it is necessary to reduce the computational complexity as much as possible while ensuring the effectiveness of the features. Five feature differences are extracted, which are pixel difference, region difference, edge difference, gray histogram difference and speeded up robust features (SURF) difference. These image feature differences are used as video features. Among them, the pixel difference can be obtained by calculating the pixel value difference between adjacent video frames. The region difference can be obtained by first performing Gaussian filtering smoothing on the image, and then calculating the contour information of the image. The ratio of the contour information to the total pixels is used as the region difference. The edge difference can be obtained by first extracting the edge information of the image, and then calculating the ratio of the edge information difference between the frames to the total pixels as the edge difference. The gray histogram difference can be obtained by calculating the chi-square distance of the gray histogram of the two video frames. The speeded up robust features difference can be obtained by calculating the Euclidean distance between the speeded up robust features descriptors of the two video frames.

[0058] In step S20, a pre-trained binary classifier is used to determine the user viewing preference according to the image basis data and the target image feature.

[0059] It should be noted that the pre-trained binary classifier is used to classify the input data into two categories, such as video content that the user likes or dislikes. The user viewing preference can be the preference of the user for the video content, which is usually obtained by analyzing the viewing behavior or feedback data of the user.

[0060] For example, according to the image basic data and the target image feature, the pre-trained binary classifier can identify that the user preference is high requirement for the picture quality of movie content and low requirement for the picture quality of TV series content.

[0061] Step S30, encoding optimization of the current video based on the current network state and the user viewing preference through the video stream encoding model.

[0062] It should be noted that the current network state can include parameters such as bandwidth, delay and jitter, and these information can be obtained through network monitoring tools or protocols. Video encoding can be re-encoding of the current video stream to optimize video transmission performance while meeting user viewing preferences. It can be understood that the encoded video stream will be sent to the user end to ensure the best viewing experience under different network conditions.

[0063] In this embodiment, the image basic data of the video stream is input into the video feature extraction model, the image feature difference between the video frames is calculated through the video feature extraction model, and the target image feature is obtained; according to the image basic data and the target image feature, the pre-trained binary classifier is used to determine the user viewing preference; based on the current network state and the user viewing preference, the current video is encoded and optimized through the video stream encoding model. The pre-trained binary classifier is used to determine the user viewing preference, which determines the viewing preference of different users for different video content. By combining the user's viewing preference and real-time network condition, the optimization point is considered from multiple angles, the best encoding configuration is selected for real-time interactive video stream, and the self-adaptability of the interactive video stream in different scenes is improved, and the user experience is enhanced.

[0064] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the adaptive video stream encoding optimization method of the present application is shown in the above Figure 1 Based on the first embodiment shown in the above

[0065] In the second embodiment, the step S20 comprises:

[0066] Step S201, input the image basic data and the target image feature of the current video into the binary classifier to obtain the output value output by the binary classifier.

[0067] It should be noted that the image base data of the current video can include current video resolution and frame rate information, wherein the resolution is an important indicator for measuring how much data in a video frame, the higher the video resolution, the more data in a frame of video, the clearer the video, the higher the video quality, and the unit is pixel value (pixel). Common resolutions are 1080P (1920x1080), 720P (1280x720), 480P (854x480) and 360P (640x360). Frame rate refers to the frame rate of the player playing the video. The higher the frame rate of the player playing the video, the smoother the video plays, and the unit is frame per second (fps). Common frame rates are 30fps, 25fps, 20fps, 15fps and 10fps. The output value can be understood as the output result of the binary classifier, which is used for subsequent confirmation of user viewing preferences.

[0068] As shown in Figure 4 The whole user preference perception method mainly consists of two parts of video feature extraction and binary classifier. First, the video image features are extracted, and then the user preference is determined in the binary classifier. The model of the binary classifier can be a simple fully connected neural network, and the number of hidden layer neurons is 32. The binary classifier can be trained based on the Adam optimizer and the BCE loss function, and the training data set can use the public video data set. The public data set should include the image features of the video and the record of the user's preference. The binary classifier receives a 7-dimensional vector, which includes 5 levels of image features extracted by the video feature extraction module and the current video resolution and frame rate information provided by the current video encoder. Such design aims to make full use of the information of video content and encoding configuration to more accurately judge the user's viewing preference. In terms of output, the dimension of the binary classifier is 1, and we use the Sigmoid function to normalize the output value to the range of [0, 1].

[0069] Step S202, when the output value is greater than the preset preference threshold, the resolution preference of the user is taken as the user viewing preference.

[0070] It should be noted that the preset preference threshold can take the middle value of the output range, such as 0.5. For example, when the output value is greater than 0.5, we determine that the user's viewing preference is inclined to resolution.

[0071] Step S203, when the output value is less than or equal to the preset preference threshold, the frame rate preference of the user is taken as the user viewing preference.

[0072] It can be understood that when the output value is less than or equal to 0.5, we determine that the user's viewing preference is inclined to resolution.

[0073] In this embodiment, by inputting the image base data of the current video and the target image features into the binary classifier, the output value output by the binary classifier is obtained, so that the information of the video content and the encoding configuration can be fully utilized to more accurately judge the viewing preference of the user. Further, when the output value is greater than the preset preference threshold, the resolution preference of the user is taken as the viewing preference of the user, and when the output value is less than or equal to the preset preference threshold, the frame rate preference of the user is taken as the viewing preference of the user.

[0074] With reference to Figure 5 , Figure 5 FIG. 1 is a flowchart of a third embodiment of the adaptive video stream encoding optimization method of the present application, based on the second embodiment shown in FIG. 2. Figure 3 The third embodiment of the adaptive video stream encoding optimization method of the present application is proposed based on the second embodiment shown in FIG. 2.

[0075] In the third embodiment, the step S30 comprises:

[0076] In step S301, the current network state and the user viewing preference are defined as a state space, and a plurality of gear change demand values are defined as an action space.

[0077] It should be noted that the state space can be a set of all possible states, which describes all possible situations of the environment. For example, the state space can include combinations of the current network state (such as bandwidth, delay, and jitter) and the user viewing preference (such as resolution preference and frame rate preference). The action space is a set of all possible actions, which describes all possible behaviors performed in each state. For example, the action space can include different video encoding configurations, such as combinations of various resolutions, frame rates, and bit rates.

[0078] It can be understood that, considering the limited computing power of mobile device video encoding and the convenience of real-time adjustment of the encoding configuration by the video encoder, a plurality of encoding configuration gears can be set. For example, three resolution gears and three frame rate gears are set, which are Res={540P, 360P, 180P} and FR={30fps, 25fps, 20fps}, respectively. Each time the encoding configuration of the real-time interactive video application needs to be modified, the video encoder of Hughes selects an appropriate encoding configuration from the three resolution gears and the three frame rate gears for video encoding.

[0079] For example, the method of reinforcement learning can be used. The state obtained by the reinforcement learning from the environment at time t can be represented as s Each vector in the state s t contains statistical information of the past k time intervals. In s t , represents the historical bandwidth estimation value information of the transmission layer statistics; history network RTT information representing transmission layer statistics; is history packet loss rate information of transmission layer statistics; finally, are history video Quantization Parameter (QP), resolution, frame rate information of video encoder statistics. It can be understood that after the reinforcement learning agent captures the state from the environment, it will make its own action, and the action of the reinforcement learning agent is defined as the gear that needs to be downgraded / upgraded. For example, the action space can be represented as A = {-2, -1, 0, 1, 2}, and we use a t to represent the action taken by the reinforcement learning at time t, and a t ∈ A. It can be understood that the action space is adjusted and optimized on the basis of multiple encoding configuration gears.

[0080] Step S302, based on the state space, determining an initial optimized encoding configuration according to the action space, and adjusting the current encoding configuration according to the initial optimized encoding configuration.

[0081] For example, as shown in FIG. 3, based on the action space, the current encoding configuration is adjusted to an initial optimized encoding configuration. Figure 6 For example, as shown in FIG. 3, based on the action space, the current encoding configuration is adjusted to an initial optimized encoding configuration. t to represent the action taken by the reinforcement learning at time t, and based on the action space, the encoding configuration is adjusted. The adjusted encoding configuration can be understood as the initial optimized encoding configuration. For example, when a t <0, it means that the current encoding configuration needs to be downgraded, and the degree of downgrade is the absolute value of a t > 0, it means that the current encoding configuration can be upgraded, and the degree of upgrade is the absolute value of a t = 0, it means that the current encoding configuration does not need to be changed. Considering that the network bandwidth of the cellular network may suddenly decrease / increase, we set the maximum degree of encoding configuration downgrade / upgrade to 2, which means that the encoding configuration can be downgraded / upgraded by at most two gears.

[0082] It should be noted that the step of determining the initial optimized encoding configuration according to the action space comprises: determining an encoding configuration decision set according to the action space; determining a decision probability corresponding to each encoding configuration decision in the encoding configuration set according to the user's viewing preference; and selecting an encoding configuration decision from the encoding configuration set as the initial optimized encoding configuration according to the decision probability.

[0083] The set of encoding configuration decisions can include multiple encoding configuration decisions, and each policy defines a rule or probability distribution for selecting an action in each state. The policy can be deterministic (selecting a fixed action in each state) or stochastic (selecting an action according to a probability in each state). For example, when the reinforcement learning agent decides that the current encoding configuration needs to be downgraded / upgraded, the encoding configuration decision method combines the user viewing preference delivered by the user preference perception method at the current time to reasonably regulate the downgrade / upgrade allocation of the resolution and frame rate levels. We take the current encoding configuration needing to be downgraded and the user viewing preference as the video resolution as an example to introduce the adjustment and decision of the encoding configuration as follows:

[0084] When the encoding configuration needs to be downgraded by 2 levels, i.e., a t = -2, assuming that the current resolution level is r and the frame rate level is f. We have three encoding configuration decisions, which are r-2, f-2 and r-1 & f-1. When there is no user viewing preference, the probabilities of the three encoding decisions are 1 / 3, i.e., P(r-2) = 1 / 3, P(f-2) = 1 / 3 and P((r-1) & (f-1)) = 1 / 3. If the current user viewing preference is resolution, the encoding decision probabilities are adjusted, and the adjustment methods are shown in formulas (1) to (3):

[0085] P(r-2) -= 0.5 * P(r-2) (1)

[0086] P((r-1) & (f-1)) -= 0.25 * P((r-1) & (f-1)) (2)

[0087] P(f-2) += 0.5 * P(r-2) + 0.25 * P(r-1) & (f-1)) (3)

[0088] The three formulas represent the selection of the protection resolution. Formula (1) indicates that the probability of reducing the resolution by two levels becomes 1 / 2 of itself, formula (2) indicates that the probability of reducing the resolution by one level and reducing the frame rate by one level becomes 3 / 4 of itself, and formula (3) indicates that the reduced part is all added to the probability of reducing the frame rate by two levels in order to maintain the probability of 1.

[0089] In step S303, the reward function at the current time is calculated based on the current encoding configuration, and the encoding configuration is adjusted according to the feedback of the reward function at the current time to determine the target optimized encoding configuration.

[0090] It should be noted that the reward function defines the immediate feedback obtained after performing a certain action, which measures the goodness of the action. For example, the design of the reward function mainly considers three factors, which are the video bitrate (unit: kbps) of the receiver feedback, the video frame rate (unit: fps) and the RTT (unit: ms) of the network link. The selection of these three factors takes into account both objective QoE indicators (video bitrate, video frame rate) and QoS indicators (packet loss rate, RTT). Here, the video bitrate is used instead of the video quality of the objective QoE indicator because the video bitrate is the most easily obtained at the sender to meet the requirements of online training and inference. The reward function at time t can be expressed as formula (4) as follows:

[0091] reward t =α*b t +β*f t +δ*RTT t (4)

[0092] Wherein, b t is the video bitrate at time t, f t is the frame rate at time t, RTT t is the round-trip delay at time t. α, β, δ represent the correlation coefficients.

[0093] It should be noted that step S303 includes: calculating the reward function at the current time based on the current encoding configuration; if the reward function at the current time is greater than a preset threshold, output the current encoding configuration, and the current encoding configuration is the target encoding configuration. After the step of calculating the reward function at the current time based on the current encoding configuration, it includes: if the reward function at the current time is less than or equal to the preset threshold, returning to the step of determining the initial optimized encoding configuration according to the current encoding configuration and the action space until the reward function at the current time is greater than the preset threshold.

[0094] Wherein, the preset threshold is used to judge whether the current reward is good enough, for example, a predefined reward function is used, and the reward value is calculated by the reward function. For example: high video clarity and smooth playback: +10 points, video with slight lag: +5 points, video with serious lag: -10 points, set the preset threshold to 15 points, and compare the size of the reward value and the preset threshold. When the reward value is greater than the preset threshold, output the current encoding configuration as the target encoding configuration. When the reward value is less than or equal to 15, return to the step of determining the initial optimized encoding configuration according to the current encoding configuration and the action space until the reward function at the current time is greater than the preset threshold. It can be understood that the reward value can be made as large as possible, and the reward value is continuously calculated by substituting the current encoding configuration into the reward function to determine the encoding configuration corresponding to the maximum reward value as the target encoding configuration.

[0095] Step S304, encoding optimization is performed on the current video according to the target optimization encoding configuration.

[0096] For example, based on the above state space, encoding decision, user viewing preference and reward function design, the basic deep reinforcement learning algorithm such as PPO (Proximal Policy Optimization), TD3 (Twin Delayed DDPG) can be used to optimize the encoding of the current video. Steps S304 to S304 can be represented as Figure 7 , Figure 7 An encoding configuration decision method in the embodiment is shown, which is a reinforcement learning-based decision method. The observed network state information and user preference are used as reinforcement learning inputs, and the optimal encoding configuration is adjusted through feedback. Finally, the encoding configuration that maximizes the user QoE under the current network state is output.

[0097] In the embodiment, the current network state and user viewing preference are defined as the state space, which comprehensively describes the current system state and ensures that all influencing factors are considered. Multiple gear change demand values are defined as the action space, which provides the possibility of flexible adjustment of the encoding configuration. Based on the state space, the initial optimization encoding configuration is determined according to the action space, and the current encoding configuration is adjusted according to the initial optimization encoding configuration. By combining the current state and the available action, an initial optimization encoding configuration is quickly determined, which provides a basis for further optimization. The reward function at the current time is calculated based on the current encoding configuration, and the encoding configuration is adjusted according to the reward function at the current time. The target optimization encoding configuration is determined, and the current video is encoded and optimized according to the target optimization encoding configuration. By calculating the reward function, the effect of the current encoding configuration is quantified, and a clear evaluation standard is provided. The target optimization encoding configuration is applied to optimize the encoding of the video, and the efficiency and quality of the video transmission are ensured based on the user viewing preference, which improves the user experience.

[0098] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the adaptive adjustment of the video stream encoding optimization method. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0099] The present application also provides an adaptive adjustment of a video stream encoding optimization device, please refer to Figure 8 , the adaptive adjustment of the video stream encoding optimization device comprises:

[0100] The target image feature acquisition module 10 is configured to input the image basic data of the video stream into a video feature extraction model, calculate the image feature difference between the video frames through the video feature extraction model, and obtain the target image feature.

[0101] The user preference confirmation module 20 is configured to determine the user viewing preference by using a pre-trained binary classifier according to the image base data and the target image feature.

[0102] The encoding optimization module 30 is configured to perform encoding optimization on the current video by using a video stream encoding model based on the current network status and the user viewing preference.

[0103] The adaptive video stream encoding optimization apparatus provided in the present application adopts the adaptive video stream encoding optimization method in the above embodiments, and can solve the technical problem of single optimization consideration factor. Compared with the prior art, the adaptive video stream encoding optimization apparatus provided in the present application has the same beneficial effects as the adaptive video stream encoding optimization method provided in the above embodiments, and other technical features in the adaptive video stream encoding optimization apparatus are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0104] The present application provides an adaptive video stream encoding optimization device, which comprises at least one processor and a memory connected with the at least one processor in communication. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the adaptive video stream encoding optimization method in Embodiment I.

[0105] Reference will now be made to the drawings, and specific language will be used to describe the same. Figure 9 FIG. 1 shows a structural diagram of an adaptive video stream encoding optimization device suitable for implementing the embodiments of the present application. The adaptive video stream encoding optimization device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 9 The adaptive video stream encoding optimization device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0106] As shown in FIG. 1, the adaptive video stream encoding optimization device can include a communication interface 101, a user interface 102, a storage 103, a processor 104, and the like. Figure 9As shown, the adaptive adjusted video stream encoding optimization device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for the adaptive adjusted video stream encoding optimization device to operate are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the adaptive adjusted video stream encoding optimization device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 9 The adaptive adjusted video stream encoding optimization device with various systems is shown in the foregoing embodiments, but it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0107] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0108] The adaptive adjusted video stream encoding optimization device provided in the present application adopts the adaptive adjusted video stream encoding optimization method in the above-mentioned embodiments, and can solve the technical problem of single optimization consideration factor. Compared with the prior art, the adaptive adjusted video stream encoding optimization device provided in the present application has the same beneficial effects as the adaptive adjusted video stream encoding optimization method provided in the above-mentioned embodiments, and other technical features in the adaptive adjusted video stream encoding optimization device are the same as the features disclosed in the above-mentioned method, which will not be described here.

[0109] It is to be understood that the various parts of the disclosure can be implemented in hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0110] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by any person skilled in the art within the spirit and scope of the application are intended to be encompassed by the claims.

[0111] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the adaptive video stream encoding optimization method in the above embodiments.

[0112] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0113] The above computer readable storage medium can be included in the adaptive video stream encoding optimization device; or can exist separately and not be assembled into the adaptive video stream encoding optimization device.

[0114] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the adaptive video stream coding optimization device, cause the adaptive video stream coding optimization device to: input image basic data of a video stream to a video feature extraction model, calculate image feature differences between video frames by the video feature extraction model, and obtain target image features; determine a user viewing preference by using a pre-trained binary classifier according to the image basic data and the target image features; and perform coding optimization on a current video by a video stream coding model based on a current network state and the user viewing preference.

[0115] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0116] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0117] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0118] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the adaptive video stream encoding optimization method described above, and can solve the technical problem of single optimization consideration factor. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the adaptive video stream encoding optimization method provided by the above embodiments, and will not be described here.

[0119] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the adaptive video stream encoding optimization method as described above.

[0120] The computer program product provided by the present application can solve the technical problem of single optimization consideration factor. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the adaptive video stream encoding optimization method provided by the above embodiments, and will not be described here.

[0121] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the contents of the specification and drawings are included in the patent protection scope of the present application.

Claims

1. An adaptive adjustment method for real-time interactive video stream coding optimization, characterized in that, The method comprises: inputting image basic data of a video stream to a video feature extraction model, calculating image feature differences between video frames through the video feature extraction model, and obtaining target image features; determining a user viewing preference by using a pre-trained binary classifier according to the image basic data and the target image features; optimizing encoding of a current video based on a current network state and the user viewing preference through a video stream encoding model; The video stream encoding model comprises a state space, an action space, and a reward function. The step of optimizing encoding of the current video based on the current network state and the user viewing preference through the video stream encoding model comprises: defining the current network state and the user viewing preference as the state space, and defining a plurality of gear change requirement values as the action space, wherein the gear is an encoding configuration gear; determining an initial optimized encoding configuration according to the action space based on the state space, and adjusting a current encoding configuration according to the initial optimized encoding configuration; calculating a reward function at a current time based on the current encoding configuration, adjusting the encoding configuration according to the reward function at the current time, and determining a target optimized encoding configuration; optimizing encoding of the current video according to the target optimized encoding configuration.

2. The method of claim 1, wherein, The step of calculating the reward function at the current time based on the current encoding configuration, adjusting the encoding configuration according to the reward function at the current time, and determining the target optimized encoding configuration comprises: calculating the reward function at the current time based on the current encoding configuration; if the reward function at the current time is greater than a preset threshold, outputting the current encoding configuration, wherein the current encoding configuration is the target encoding configuration.

3. The method of claim 2, wherein, After the step of calculating the reward function at the current time based on the current encoding configuration, the method comprises: if the reward function at the current time is less than or equal to the preset threshold, returning to the step of determining the initial optimized encoding configuration according to the action space based on the state space, until the reward function at the current time is greater than the preset threshold.

4. The method of claim 1, wherein, The step of determining the initial optimized encoding configuration according to the action space comprises: determining a set of encoding configuration decisions according to the action space; determining a decision probability corresponding to each encoding configuration decision in the set of encoding configuration decisions according to the user viewing preference; selecting one configuration decision from the set of encoding configuration decisions as the initial optimized encoding configuration according to the decision probability.

5. The method of any one of claims 1 to 4, wherein, The user viewing preference comprises a resolution preference or a frame rate preference. The step of determining the user viewing preference by using the pre-trained binary classifier according to the image basic data and the target image features comprises: inputting the image basic data of the current video and the target image features to the binary classifier to obtain an output value output by the binary classifier; when the output value is greater than a preset preference threshold, taking the resolution preference of the user as the user viewing preference; when the output value is less than or equal to the preset preference threshold, taking the frame rate preference of the user as the user viewing preference.

6. An apparatus for adaptive adjustment of real-time interactive video streaming encoding optimization, comprising: The device comprises: The target image feature acquisition module is configured to input image basic data of the video stream to a video feature extraction model, calculate image feature differences between video frames by the video feature extraction model, and obtain target image features; The user preference confirmation module is configured to determine a user viewing preference by using a pre-trained binary classifier based on the image basic data and the target image features; The encoding optimization module is configured to perform encoding optimization on a current video by a video stream encoding model based on a current network state and the user viewing preference, the video stream encoding model including a state space, an action space, and a reward function. The encoding optimization module is further configured to define the current network state and the user viewing preference as the state space, define a plurality of gear change requirement values as the action space, and define a gear as an encoding configuration gear; determine an initial optimized encoding configuration according to the action space based on the state space, and adjust a current encoding configuration according to the initial optimized encoding configuration; calculate a reward function at a current time based on the current encoding configuration, feed back and adjust the encoding configuration according to the reward function at the current time, and determine a target optimized encoding configuration; and perform encoding optimization on the current video according to the target optimized encoding configuration.

7. An adaptive, real-time, interactive video stream encoding optimization device, comprising: The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the self-adaptive real-time interactive video stream encoding optimization method of any one of claims 1 to 5.

8. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the self-adaptive real-time interactive video stream encoding optimization method of any one of claims 1 to 5.

9. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to implement the steps of the self-adaptive real-time interactive video stream encoding optimization method of any one of claims 1 to 5.

Images