Video stream frame rate adjustment method and its device, equipment, medium and product

Abstract

The present application relates to a method for adjusting the frame rate of a video stream and its apparatus, device, medium, and product, the method comprising: obtaining the current image frame required to generate the video stream; determining the subjective evaluation parameters of the current image frame, the subjective evaluation parameters comprising a reference frame rate suitable for encoding the current image frame, and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate; inputting the output bandwidth of the encoder into a parameter optimization model to obtain an approximate function that characterizes the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth; applying the approximate function to obtain the numerical approximation point corresponding to the subjective evaluation parameter, and obtaining the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate for encoding the current image frame. The present application can adaptively control the encoder to encode the video stream at the optimal frame rate, thereby improving the playback experience of the video stream of the live broadcast network.

Description

Technical Field

[0001] The present application relates to the field of video transmission technology, and in particular to a method for adjusting the frame rate of a video stream and its device, equipment, medium, and product. Background Art

[0002] The transmission control of video streams, when the bandwidth and resolution are fixed, is mainly achieved by adjusting the frame rate to change the transmission bit rate. The starting point is usually based on the network conditions or the hardware conditions of the computer equipment. The goal is to meet the actual conditions of the network or hardware, without considering the actual viewing experience brought to the audience by the frame rate changes in the time dimension.

[0003] Specifically, according to traditional coding principles, the quality, frame rate, and resolution of a video vary with changes in bandwidth. Assuming the resolution remains unchanged, if the frame rate varies too much, the range of video quality will be smaller, and image compression will be more difficult, unless the resolution is reduced. By reducing the resolution, the range of frame rate and video quality changes is relatively more flexible, and the coding process can gain greater flexibility. If the problem is simply solved by reducing the frame rate without considering the user's actual viewing experience, it is not user-friendly.

[0004] This shows that there is still much room for improvement in the transmission control technology for video streams. Summary of the invention

[0005] The purpose of the present application is to solve the above-mentioned problem and to provide a video stream frame rate adjustment method and its corresponding device, equipment, non-volatile readable storage medium, and computer program product.

[0006] According to one aspect of the present application, a method for adjusting a video stream frame rate is provided, comprising the following steps:

[0007] Get the current image frame required to generate the video stream;

[0008] Determine a reference frame rate and subjective quality index that matches the current image frame;

[0009] Inputting the output bandwidth of the encoder into the parameter optimization model to obtain an approximate function representing the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth;

[0010] The approximate function is applied to obtain a numerical approximation point corresponding to the subjective evaluation parameter, and a coding frame rate corresponding to the numerical approximation point is obtained as an optimal frame rate for coding the current image frame.

[0011] According to another aspect of the present application, a video stream frame rate adjustment device is provided, comprising:

[0012] An image acquisition module, used to acquire the current image frame required to generate a video stream;

[0013] A subjective evaluation module, used to determine subjective evaluation parameters of the current image frame, wherein the subjective evaluation parameters include a reference frame rate suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate;

[0014] A function modulation module, used for inputting the output bandwidth of the encoder into a parameter optimization model to obtain an approximate function representing the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth;

[0015] The frame rate determination module is used to apply the approximate function to obtain the numerical approximation point corresponding to the subjective evaluation parameter, and obtain the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate for encoding the current image frame.

[0016] According to another aspect of the present application, a video stream frame rate adjustment device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the video stream frame rate adjustment method described in the present application.

[0017] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the video stream frame rate adjustment method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0018] According to another aspect of the present application, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the method described in any one of the embodiments of the present application are implemented.

[0019] Compared with the prior art, the present application, under the constraint of the output bandwidth of the encoder, on the one hand, determines the subjective evaluation parameters corresponding to the current image frame in the video stream, wherein the subjective evaluation parameters include a reference frame rate suitable for encoding the current image frame, and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate, and evaluates the correspondence between the frame rate change and the image quality change; on the other hand, the parameter optimization model predicts an approximate function that characterizes the mapping relationship between the encoding frame rate and the objective quality index, and on this basis, obtains the numerical approximation point corresponding to the subjective evaluation parameter from the approximate function, and then determines the optimal frame rate required for encoding based on the numerical approximation point to control the encoder to encode and output the current image frame, so that the objective quality index of the video stream generated by the encoding obeys the tuning of the subjective quality index, thereby improving the overall quality perception of the video image while ensuring the stable and smooth output of the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of the network architecture corresponding to the live broadcast environment used in this application;

[0022] Figure 2 A schematic diagram of the principle on which the video stream frame rate adjustment method of the present application is based;

[0023] Figure 3 A schematic diagram of a flow chart of an embodiment of a method for adjusting a video stream frame rate of the present application;

[0024] Figure 4 A schematic diagram of a process of obtaining a current image frame in an embodiment of the present application;

[0025] Figure 5 A flowchart of a process for deciding whether to recalculate subjective evaluation parameters in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a process of determining a subjective evaluation parameter for a current image frame in an embodiment of the present application;

[0027] Figure 7 A schematic flow chart of a process of iterating a parameter optimization model in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of a process of calculating an objective quality indicator according to a video stream in an embodiment of the present application;

[0029] Fig. 9 This is a principle block diagram of the video stream frame rate adjustment device of the present application;

[0030] Fig.10 A structural diagram of a video stream frame rate adjustment device used in this application. DETAILED DESCRIPTION

[0031] The hardware referred to by the names such as "server" and "client" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0032] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of service clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0033] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0034] The models referenced or may be referenced in this application, including traditional machine learning models or deep learning models, can be deployed on a remote server and remotely called on the client, or deployed on a client with sufficient device capabilities and directly called, unless explicitly specified. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0035] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0036] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0037] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0038] See also Figure 1 The network architecture adopted in the exemplary application scenario of the present application can be used to deploy a live broadcast service. The encoding process of the video stream of the live broadcast service can be implemented by running a computer program product obtained by any embodiment of the present application. Figure 1 The application server 81 shown can be used to support the operation of the live broadcast room instance, and the media server 82 can be used to process the decoding process of the video stream pushed by each live broadcast user to achieve relay, and the terminal devices such as the computer 83 and the mobile phone 84 are generally provided to the terminal users as clients. In addition, when it is necessary to encode the video stream on the terminal device, the computer program product obtained by each embodiment of the present application can also be deployed in the terminal device, so that when encoding the video image captured by the camera, the method of any embodiment of the present application is applicable. In other words, the application scenarios disclosed above are for illustrative purposes only, and the video stream frame rate adjustment method of the present application is applicable to all scenarios where the video stream needs to be encoded.

[0039] Figure 2 The principle block diagram shown reveals the implementation principle on which the various embodiments of the present application are based. It can be seen from the figure that in order to provide the encoder with the best frame rate parameters, the encoder is adaptively controlled to encode the image frames of the video stream, and a parameter optimization model simulates an approximate function that characterizes the mapping relationship between the encoding frame rate and the objective quality index according to the output bandwidth of the encoder. An image detection module determines the subjective evaluation parameters determined by the subjective quality evaluation method of the reference image of the current image frame according to the video stream, and the subjective evaluation parameters include a reference frame rate and a subjective quality index.

[0040] Obviously, both the approximate function and the subjective evaluation parameters contain the corresponding relationship between the frame rate and the quality index. The approximate function is inferred by the parameter optimization model and optimized by the objective quality index of the encoded image frame. Therefore, the quality index inferred by it can be quantified in the form of an objective quality index. The quality index in the subjective evaluation parameter is determined by referring to the data obtained by quantifying subjective feelings according to the subjective quality evaluation method of the image. Therefore, its quality index can be quantified in the form of a subjective quality index.

[0041] Furthermore, a frame rate determination module obtains the numerical approximation point between the approximate function and the subjective evaluation parameter, obtains the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate, and controls the encoder to encode the image frames with the optimal frame rate. The encoded video stream can be remotely pushed for transmission.

[0042] In addition, through a quality evaluation module, the original current image frame is referred to to calculate the corresponding current image frame in the video stream decoded by the decoder to determine the objective quality index actually obtained by the latter, and then the weight parameters of the parameter model are corrected by backpropagation using the objective quality index and the optimal frame rate during encoding, so that the parameter optimization model is iterated cyclically and its simulation approximation function is continuously improved to predict the optimal frame rate, thereby continuously optimizing the encoding efficiency during the video stream encoding process and obtaining high-quality encoding effects.

[0043] See also Figure 3 According to one aspect of the present application, a video stream frame rate adjustment method is provided, in one embodiment of which, the method comprises the following steps:

[0044] Step S1100, obtaining a current image frame required for generating a video stream;

[0045] The video stream may be a video stream in a live network transmission process, and may be provided by a live network user. The video stream includes multiple image frames, each of which may be processed as a current image frame by the method of the present application, so as to control the encoder to encode each current image frame in the video stream.

[0046] The current image frame may be obtained from an image space before encoding, and is generally stored in a specific format, such as a YUV format or an RGB format, which may be flexibly determined by those skilled in the art.

[0047] Step S1200, determining a reference frame rate and a subjective quality index that match the current image frame, wherein the subjective evaluation parameter includes a reference frame rate suitable for encoding the current image frame, and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate;

[0048] For each current image frame in the video stream, its corresponding subjective quality index may be determined by referring to the image subjective quality evaluation method.

[0049] In one embodiment, a subjective image quality evaluation method is pre-applied to provide multiple video contents with different frame rates and play them for viewers to watch. Then, scores of each viewer are obtained from multiple preset dimensions, and weighted and quantized into a subjective quality index according to the scores given by the viewer. A mapping relationship table is established for the subjective quality indexes corresponding to the multiple frame rates, and then used as a reference for the application to determine the reference frame rate and subjective quality index that match the current image frame. Accordingly, when it is necessary to determine the reference frame rate that matches the current image frame and its corresponding subjective quality index, according to the change in the image complexity of the current image frame relative to its previous image frame, corresponding adjustments are made on the basis of the reference frame rate corresponding to its previous image frame, so that the reference frame rate corresponding to the current image frame can be evaluated. According to the evaluated reference frame rate, the corresponding subjective quality index can be determined using the mapping relationship data, thereby obtaining a reference frame rate and a corresponding subjective quality index, which constitute the subjective evaluation parameters corresponding to the current image frame.

[0050] It can be seen from this that the reference frame rate is intermediate data used as a reference, which constitutes an indication of the frame rate encoding condition of the current image frame. Correspondingly, the subjective quality index refers to the subjective evaluation result of the image quality that can be predicted when the reference frame rate is used as the encoding condition.

[0051] In one embodiment, the mapping relationship table determined according to the image subjective quality evaluation method is fitted to obtain the corresponding subjective curve function, and then the corresponding subjective quality index can be quickly obtained according to the reference frame rate determined corresponding to each current image frame to obtain its subjective evaluation parameters. In this case, since the subjective curve function has a linear characteristic after fitting, the subjective evaluation parameters determined thereby can be smooth.

[0052] The current image frame may be a transition image indicating that a video scene has switched relative to a previous image frame, and of course, it may also be a non-transition image. In this regard, in some embodiments, it may be determined whether to recalculate its subjective quality parameters based on whether the current image frame is a transition image. If it is not necessary to recalculate the subjective quality parameters, the subjective quality parameters that have been generated in the current scene where the current image frame is located may be obtained and used directly.

[0053] Step S1300: input the output bandwidth of the encoder into the parameter optimization model to obtain an approximate function that characterizes the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth;

[0054] The output bandwidth of the encoder is an objective condition, which is limited by network conditions and / or hardware conditions, and therefore has relative certainty. In this embodiment, the output bandwidth can be input into the parameter optimization model, and the control parameter optimization model simulates an approximate function for solving the optimal frame rate of the encoder.

[0055] The parameter optimization model is implemented by pre-modeling. For example, considering that the resolution of the same video stream is required to be relatively stable during transmission, and the encoding frame rate and objective quality index have greater flexibility with the change of the output bandwidth of the encoder, the objective quality index and encoding frame rate of different video streams under a specific output bandwidth are fitted. From the fitting results, it can be seen that the encoding frame rate and objective quality index of the video stream follow the natural logarithmic distribution. According to this principle, the parameter optimization model can be modeled and implemented.

[0056] An exemplary parameter optimization model is constructed according to the following formula:

[0057] y psnr =A*ln(fps)+B

[0058] Among them, y psnr is an exemplary objective quality indicator, fps is the frame rate, A is a learnable weight parameter, and B is a learnable bias parameter, which together constitute the learnable parameters of the model. According to the formula, the parameter optimization model can modify its weight parameters and bias parameters based on the objective quality indicator of the image frame output by the encoder, implement iterative training, and make it continuously approach convergence.

[0059] According to the exemplary modeling principle of the parameter optimization model, the core of the parameter optimization model is to construct a data curve corresponding to the mapping relationship between the export bandwidth, the coding frame rate, and the objective quality index, so that the mapping relationship between the coding frame rate and the objective quality index can be predicted according to the export bandwidth, that is, an approximate function is determined. Further determining a corresponding point on the determined approximate function can determine the specific coding frame rate and objective quality index. It can be seen that the data curve corresponding to a specific export bandwidth essentially reflects the mapping relationship data set between the coding frame rate and the objective quality index under the specific export bandwidth. The mapping relationship data set is represented by the approximate function obtained by the parameter optimization model, and accordingly, this approximate function can be obtained according to an export bandwidth.

[0060] It should be noted that step S1200 and step S1300 can be executed concurrently, and the execution order can be swapped, and are not limited by the order listed in this application.

[0061] In addition, the parameter optimization model can be modeled as a machine learning model with low computing resource utilization, or it can be modeled as a more intelligent deep learning model, as long as the functions disclosed in this application can be implemented according to the above principles.

[0062] Step S1400: Apply the approximate function to obtain a numerical approximation point corresponding to the subjective evaluation parameter, and obtain a coding frame rate corresponding to the numerical approximation point as an optimal frame rate for coding the current image frame.

[0063] Figuratively speaking, the approximate function represents the mapping relationship data of the linear relationship between the encoding frame rate and the objective quality index under a specific export bandwidth, which is reflected as a data curve on the rectangular coordinate system, and the subjective evaluation parameters corresponding to the current image frame, that is, its reference frame rate and the subjective quality index corresponding to the reference frame rate, can be reflected as a point on the rectangular coordinate system.

[0064] Accordingly, in one embodiment, by calculating the shortest distance between the point corresponding to the subjective parameter model on the rectangular coordinate system and the data curve of the approximate function, the point with the shortest distance on the data curve is determined to be the numerical approximation point on the data curve. The numerical approximation point corresponds to the mapping relationship data between a coding frame rate and an objective quality indicator, and the coding frame rate can be determined as the optimal frame rate.

[0065] In another embodiment, according to the mapping relationship data between the reference frame rate and the subjective quality index determined by the subjective quality evaluation method of the image, the subjective curve function is obtained in advance by data fitting. In this case, the intersection of the subjective curve function and the approximate function can be calculated, and the intersection is the numerical approximation point, thereby the corresponding encoding frame rate can also be obtained as the optimal frame rate.

[0066] The optimal frame rate determined in the present application is input into the encoder to control the encoder to encode the current image frame at the optimal frame rate so as to encode the current image frame in the video stream.

[0067] According to the above embodiments, it can be seen that, compared with the prior art, the present application, under the constraint of the output bandwidth of the encoder, on the one hand, determines the subjective evaluation parameters corresponding to the current image frame in the video stream, wherein the subjective evaluation parameters include a reference frame rate suitable for encoding the current image frame, and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate, and evaluates the correspondence between the frame rate change and the image quality change; on the other hand, the parameter optimization model predicts an approximate function that characterizes the mapping relationship between the encoding frame rate and the objective quality index, and on this basis, obtains the numerical approximation point corresponding to the subjective evaluation parameter from the approximate function, and then determines the optimal frame rate required for encoding based on the numerical approximation point to control the encoder to encode and output the current image frame, so that the objective quality index of the video stream generated by the encoding obeys the tuning of the subjective quality index, thereby improving the overall quality perception of the video image while ensuring the stable and smooth output of the video stream.

[0068] See also Figure 4 Based on the above embodiment, the step S1100, obtaining the current image frame required to generate the video stream, includes the following steps:

[0069] Step S1111, rendering the data frame collected by the camera unit into a texture image;

[0070] The computer program product of the present application can be deployed in a terminal device, such as a personal computer or mobile terminal used by a live broadcast user of a live network broadcast to generate a live video stream. The camera unit installed on the terminal device is responsible for capturing real-scene images to generate the video stream.

[0071] When the camera unit is started, it starts to collect images and obtain image data in the form of data frames. Then, a preset texture is applied to the image to generate texture data, which is rendered and displayed in the graphical user interface of the terminal device.

[0072] Step S1112, converting the texture image into an image frame in a specific format and storing it in an image space;

[0073] After conventional image preprocessing, the texture image is further converted into an image frame of a specific format and stored in an image space. The image space is a color space corresponding to a color encoding method of a corresponding specific format. The specific format can be a YUV format or an RGB format, and its format type does not affect the embodiment of the creative spirit of the present application.

[0074] Step S1113: Acquire each image frame from the image space according to the timestamp as the current image frame.

[0075] When the encoder is encoding, it continuously calls each image frame from the image space, and the called image frame is encoded by the encoder as the current image frame. Since the encoding process requires the image frames in the video stream to be organized in order, they can be acquired frame by frame in order according to the timestamp of the image frame, and each acquired image frame is the current image frame.

[0076] According to the above embodiments, the present application is suitable for deployment at a terminal device, and the terminal device can encode the image frames captured by the camera unit according to the optimal frame rate obtained by the present application, thereby obtaining high-quality image transmission quality while ensuring adaptive and stable transmission of the video stream.

[0077] Based on the above embodiment, in another embodiment different from the previous embodiment, the step S1100, obtaining the current image frame required to generate the video stream, includes the following steps:

[0078] Step S1121: Decode the original video stream submitted by the terminal device, obtain image frames in a specific format, and store them in the image space;

[0079] The computer program product of the present application can be deployed in a server, for example, in a media server of a live broadcast service on the Internet. The media server is responsible for receiving the video stream uploaded by the live broadcast user of the live broadcast room of the live broadcast service on the Internet, decoding the video stream, obtaining the image frames encoded in the default format and storing them in the image space so as to mix the streams or convert the formats thereof, and finally obtaining the image frames in a specific format stored in the image space so that these image frames can be subsequently encoded into a video stream output according to the network conditions and / or hardware conditions of the receiving user.

[0080] Step S1122: Obtain each image frame from the image space according to the timestamp as the current image frame:

[0081] Similarly, when the encoder is encoding, it continuously calls each image frame from the image space, and the called image frame is encoded by the encoder as the current image frame. Since the encoding process requires the image frames in the video stream to be organized in order, they can be acquired frame by frame in order according to the timestamp of the image frame, and each acquired image frame is the current image frame.

[0082] According to the above embodiments, the present application is suitable for deployment on a server. The server can decode and then encode the image frames of the video stream uploaded by the terminal device according to the optimal frame rate obtained by the present application, thereby ensuring adaptive and stable transmission of the video stream while obtaining high-quality image transmission quality.

[0083] See also Figure 5 Based on the above embodiment, the step S1200, determining the subjective evaluation parameters of the current image frame, includes the following steps:

[0084] Step S1210: using a preset transition recognition model to determine whether the current image frame is a transition image;

[0085] There is a phenomenon of scene switching in the video stream. By identifying the current image frame, it can be determined whether the current image is a transition image, thereby determining whether the current image frame has a scene switch relative to its previous image frame. The occurrence of scene switching usually means that the scale of the image data has changed significantly. Therefore, identifying whether the current image frame is a transition image can be used as a basis for determining whether the subjective evaluation parameters used in the current image frame need to be re-determined.

[0086] When identifying whether the current image frame is a transition image, any feasible transition identification method can be used to make a judgment. The principle is to compare the image change information between the current image frame and its adjacent previous image frame in time sequence. When the image change information changes significantly, the current image frame is determined to be a transition image.

[0087] One of the exemplary recognition methods may calculate the information difference between the current image frame and its preceding image frame in time sequence. When the information difference is greater than a preset threshold, it may be determined that the current image frame is a transition image.

[0088] In another exemplary recognition method, the transition recognition model can be a model based on deep learning. In the exemplary model structure, Resnet can be used as the backbone model to extract the image feature information of the current image frame and its previous image frame, and then feature splicing is performed to obtain comprehensive feature information. The comprehensive feature information is connected to the classifier through the fully connected layer, and the classifier outputs the classification probability corresponding to whether the current image frame is a transition image, and determines whether it is a transition image based on the classification probability. Of course, the transition recognition model should first be trained to a convergence state using a sufficient amount of training samples, which can be specifically implemented by those skilled in the art.

[0089] Step S1220: When the image is a transition image, image analysis is performed on the current image frame to determine the difference between the image size of the current image frame and the previous image frame;

[0090] In order to more conveniently determine whether it is necessary to recalculate the subjective evaluation parameters corresponding to the current image frame according to the current scene, after the current image frame is determined to be a transition image, image analysis can be performed on the transition image. By comparing the difference between the image sizes of the transition image and the previous image frame adjacent to it in time sequence, it can be determined whether the encoder's code stream will undergo a significant change, so as to timely adjust the subjective evaluation parameters corresponding to the current image frame according to the change in the code stream, and at the same time use it as the subjective evaluation parameters corresponding to each image frame in the current scene to guide the encoding of subsequent image frames in the current scene.

[0091] Step S1230: when the difference is greater than a preset threshold, recalculate the subjective evaluation parameter of the current image frame;

[0092] The analysis of the difference between the image sizes can provide a preset threshold for comparison, and the preset threshold can be an empirical threshold or a measured threshold. When the difference between the image sizes is less than the preset threshold, it indicates that the code stream changes slightly, and there is no need to redetermine the subjective evaluation parameters, and the subjective evaluation parameters of the previous scene can be used. When the difference between the image sizes is greater than the preset threshold, it indicates that the code stream changes significantly, and accordingly, the range of variation of the encoding frame rate of the encoder is also larger. At this time, the subjective evaluation parameters can be redetermined based on the image complexity of the current image frame, or based on the image complexity of the current image frame and multiple image frames thereafter, with the goal of redetermining the subjective evaluation parameters of each image frame that adapts to the current scene.

[0093] Step S1240: When the image is not in transition, the subjective evaluation parameter corresponding to the previous image frame of the current image frame is called as the subjective evaluation parameter of the current image frame.

[0094] When the current image frame is determined to be a non-transition image frame, it means that the scene has not switched. In this case, since the corresponding subjective evaluation parameters of the scene have been calculated and determined at the moment corresponding to the first image frame identified as a transition image, there is no need to recalculate the subjective evaluation parameters for the current image frame, but directly call the subjective evaluation parameters that have been determined in advance in the current scene.

[0095] It should be pointed out that when the computer program product of the present application starts working, the first image frame will be identified as the current image frame and the corresponding subjective evaluation parameters will be determined for it. In this case, there may be a lack of reference basis when calculating the subjective evaluation parameters. Therefore, it can be adaptively initialized in advance, for example, providing default initialized subjective evaluation parameters for the scene corresponding to the first image frame to call.

[0096] According to the embodiments disclosed above, in the process of encoding each image frame of the video stream, transition recognition can be performed for each current image frame, and a decision is made whether to redetermine the subjective evaluation parameters in the current scene based on the potential code stream changes. That is, a decision is made whether to redetermine the subjective evaluation parameters based on whether the difference between the transition recognition and the image size is greater than a preset threshold. This can minimize the computing pressure of the computer device, avoid frequent redetermination of the subjective evaluation parameters, and maintain the stability of the optimal frame rate within a certain range of changes, thereby improving the encoding efficiency as a whole.

[0097] See also Figure 6 On the basis of the above embodiment, the step S1200, determining the subjective evaluation parameter of the current image frame, or the step S1230, when the difference is greater than a preset threshold, recalculating the subjective evaluation parameter of the current image frame, comprises the following steps:

[0098] Step S1231, calculating the image complexity of the current image frame and several subsequent image frames;

[0099] Video coding and the image complexity of the image frame sequence are closely related. The image complexity of the sequence is divided into temporal complexity and spatial complexity. The more details in the video picture, the greater the spatial complexity; the more intense the motion of the video content, the greater the temporal complexity. The greater the video complexity, the more data is required for encoding under the premise of the same image quality. Therefore, the image complexity can be obtained by evaluating the temporal complexity and spatial complexity of the sequence. The image frame sequence can be composed of a plurality of consecutive subsequent image frames, starting with the current image frame as the starting image frame according to a preset number.

[0100] For example, TIandSI can be used to calculate the temporal complexity and spatial complexity. ITU-RBT.1788 recommends using temporal information (TI, Temporal perceptual Information, also known as temporal complexity) and spatial information (SI, Spatial perceptual Information, also known as spatial complexity) to measure the characteristics of the video. The TIandSI tool is a tool commonly used in the field for temporal complexity and spatial complexity of images. It provides a command line version TISIcmd, which was later upgraded to TIandSI. Both provide corresponding calling interfaces for calculating the corresponding temporal complexity and spatial complexity of image frames to obtain the image complexity of the image frame sequence.

[0101] Step S1232, calculating the difference information between the image complexity of the current image frame and the image complexity of the previous image frame;

[0102] Since the process of encoding the video stream is performed continuously for each image frame, for each current image frame that is not the first image frame of the video stream, at least one image frame that precedes it in time sequence has determined the corresponding image complexity according to the principles disclosed in the present application. In this case, by subtracting the image complexity corresponding to the current image frame from the image complexity corresponding to the previous image frame, the difference information between the image complexity of the current image frame and that of the previous image frame can be determined, and the difference information can be quantified into a numerical value for calculation.

[0103] For the current image frame that is the first image frame, since it has no previous image frame, its image complexity that can be referenced can be regarded as zero value. Therefore, the difference information of the image complexity corresponding to the current image frame can also be determined by the same logic.

[0104] Step S1233, according to the difference information, adjusting the reference frame rate of the current image frame based on the reference frame rate of the previous image frame;

[0105] The mapping relationship data between the difference information of image complexity and the frame rate change is pre-quantified, so that the coding frame rate change range caused by the image complexity change can be quantified. Based on this, according to the change of the difference information, the change value used to adjust the theoretically applicable frame rate of the current image frame relative to its previous image frame can be determined. That is, when the previous image frame has determined its reference frame rate for obtaining the coding frame rate with the approximate function obtained by the parameter optimization model, the change value that needs to be adjusted based on the reference frame rate of the previous image frame can be determined based on the difference information, and then the change value is superimposed on the reference frame rate of the previous image frame for corresponding adjustment, so as to obtain the reference frame rate corresponding to the current image frame. The reference frame rate of the current image frame is determined by the relative difference method, which can maintain the smoothness of the data and avoid the sharp jitter of the image quality of the encoded video stream caused by different data reference benchmarks.

[0106] In addition, it should be understood that since the image complexity in this embodiment is determined based on the current image frame and several subsequent image frames, when the current image frame is an image frame of the second scene, the reference frame rate corresponding to the previous image frame is usually also the reference frame rate of the previous first scene.

[0107] Step S1234: applying a preset subjective curve function representing a mapping relationship between a reference frame rate and a subjective quality index, and determining a corresponding subjective quality index according to the reference frame rate.

[0108] It is not difficult to understand that when the encoder output bandwidth is determined, once the time complexity and spatial complexity of the image frames in the image frame sequence change, it will also affect the change of its encoding frame rate. This change relationship can be determined in advance by technical personnel in this field.

[0109] For example, a mapping relationship table between a reference frame rate and a subjective quality index can be constructed according to any feasible image subjective quality evaluation method, such as DSIS, DSCQS, SSM, SSCQE, etc., so that the corresponding subjective quality index under a certain reference frame rate can be queried through the mapping relationship table. The frame rate in the mapping relationship table determined according to the image subjective quality evaluation method is mainly used to indicate intermediate data and is for reference only, so it is named the reference frame rate.

[0110] In one embodiment, when determining the mapping relationship table, multiple continuous reference frame rates can be integrated into the same frame rate interval to establish a mapping relationship between the frame rate interval and the subjective quality indicator, for example, the frame rate interval 21fps-18fps is mapped correspondingly to a subjective quality indicator. The mapping relationship between the reference frame rate and the subjective quality indicator is established in the form of a frame rate interval, mainly considering that the audience's subjective experience is not sensitive to the frame rate difference within a certain frame rate variation range. Therefore, by establishing a mapping relationship between the frame rate interval and the subjective quality indicator, the efficiency of establishing the mapping relationship table is improved and the computational complexity can be further reduced.

[0111] After determining the mapping relationship table, data fitting can be further performed according to the mapping relationship table to fit it into a data curve to obtain a corresponding subjective curve function, so as to characterize the mapping relationship between the reference frame rate and the subjective quality index.

[0112] When it is necessary to determine the subjective quality index according to the reference frame rate corresponding to the current image frame, the subjective curve function is applied and substituted into the reference frame rate of the current image frame to calculate the corresponding subjective quality index. The reference frame rate of the current image frame and the subjective quality index form a data pair, which is the subjective evaluation parameter of the current image frame.

[0113] It should be noted that the above steps S1231 to S1234 can be performed when the subjective evaluation parameters of the current image frame need to be recalculated after scene transition recognition is performed on the current image frame, or can be performed independently without relying on the scene transition recognition process.

[0114] According to the above embodiments, when determining the subjective evaluation parameters of the current image frame for finding the numerical approach point with the approximate function of the parameter optimization model, the difference information is evaluated based on the image complexity of the current image frame relative to the previous image frame, and then based on the correspondence between the difference information and the frame rate change, the reference frame rate corresponding to the current image frame is corrected on the basis of the reference frame rate of the previous image frame, and then the subjective curve function determined according to the image subjective quality evaluation method is applied to determine the subjective quality index of the current image frame below the reference frame rate, forming a subjective evaluation parameter, which can be used to determine the optimal frame rate required for encoding for the current image frame. In the whole process, the amount of calculation is small and the operation efficiency is high. The obtained subjective evaluation parameter has a reference role in indicating the user's subjective quality perception. The corresponding optimal frame rate is determined by finding the numerical approach point with the approximate function of the parameter optimization model and then encoding is performed, so that the encoded video stream can obtain excellent subjective quality indicators on the basis of taking into account the transmission efficiency.

[0115] See also Figure 7In another embodiment expanded on the basis of any one of the embodiments of the present application, after determining the optimal frame rate, the following steps are further included:

[0116] Step S1500, calling the encoder to encode the current image frame at the optimal frame rate and the export bandwidth, and encoding the current image frame into the video stream;

[0117] After determining the optimal frame rate, the encoder uses the optimal frame rate as the encoding frame rate for encoding the current image frame. The encoder encodes the current image frame of a given resolution under the constraint of a given export bandwidth, thereby encoding the current image frame in the video stream. It can be understood that for image frames subsequent to the current image frame, when the scene does not switch, it is highly likely that the subsequent image frames will continue to be encoded at the same optimal frame rate.

[0118] Step S1600: Decode the video stream to obtain the current image frame, evaluate its objective quality index, and iteratively update the parameter optimization model with the objective quality index and the optimal frame rate:

[0119] The video stream encoded by the encoder is continuously output. For example, the video stream encoded in the media server is broadcast and pushed to the terminal users in the live broadcast room, or the video stream encoded and generated in the terminal device of the live broadcast user in the live broadcast room is pushed to the media server.

[0120] For the encoded output video stream, the local device can decode it to obtain the current image frame, and further obtain the current image frame before encoding according to the timestamp of the decoded current image frame, call the algorithm corresponding to the objective quality indicator, and calculate the objective quality indicator corresponding to the current image frame in the video stream with reference to the current image frame before encoding. Then, the objective quality indicator and the best frame rate corresponding to the current image frame are back-propagated to correct the weight parameters and bias parameters of the parameter optimization model, so that the parameter optimization model continuously approaches convergence and gradually improves its ability to predict the best frame rate that meets the image frame encoding requirements.

[0121] In a recommended embodiment, when correcting the weights of the parameter optimization model, the least squares method can be used to calculate the model loss value based on the optimal frame rate and the objective quality index, and then the weights are corrected based on the model loss value.

[0122] According to the above embodiments, it can be seen that the objective quality index corresponding to each image frame of the encoded output video stream is calculated, and the learnable parameters of the parameter optimization model are reversely corrected according to the objective quality index and the optimal frame rate, so that the parameter optimization model is continuously iterated and upgraded, and its ability to determine the optimal frame rate of the encoder is cyclically improved, thereby realizing dynamic and flexible control of the video stream encoding process, so that the parameter optimization model can adaptively control the encoder to adjust its encoding frame rate according to the subjective quality perception under a given export bandwidth, while ensuring the stable and smooth output of the video stream, and also improving the overall quality perception of the video image.

[0123] See also Figure 8 Based on the previous embodiment, the step S1600, decoding the video stream to obtain the current image frame and evaluating its objective quality index, includes the following steps:

[0124] Step S1610: Decode the video stream to obtain the current image frame as the first image frame:

[0125] As mentioned above, each current image frame is encoded and output by the encoder, included in the video stream, and transmitted to a media server or terminal device. Therefore, the image frames included in the video stream theoretically represent the image quality that can be obtained by the receiving device. Based on this, the video stream output by the encoder can be decoded to obtain each image frame therein, so as to be used to calculate the corresponding objective quality index. For the specified current image frame, it can be called the first image frame.

[0126] Step S1620, obtaining the current image frame before encoding as the second image frame;

[0127] In order to calculate the objective quality index of the first image frame, the current image frame before being encoded by the encoder may be obtained as the second image frame according to the corresponding relationship on the timestamp. The second image frame may be obtained by extracting it from the image space before the encoder.

[0128] Step S1630: using the second image frame as a reference, applying a preset formula to calculate and obtain an objective quality index of the first image frame;

[0129] The type of objective quality indicator used is not limited to the examples in the foregoing. Specifically, when the computer program product obtained by any embodiment of the present application is deployed on a terminal device, the objective quality indicator can be represented by mean square error (MSE) or peak signal-to-noise ratio (PSNR), because its calculation amount is small, the computing power of the terminal device can be saved and the operation efficiency can be improved; when the computer program product obtained by any embodiment of the present application is deployed on a media server, the objective quality indicator can be represented by peak signal-to-noise ratio (PSNR), structural similarity (SSIM) or mean square error (MSE). Of course, if the computing power of the terminal device is sufficient, SSIM can also be used as a quality indicator at the terminal device. In this regard, those skilled in the art can flexibly select and implement according to the principles disclosed above.

[0130] On the basis of determining the specific type of the objective quality indicator, a calculation formula corresponding to the objective quality indicator may be applied to calculate the objective quality indicator of the first image frame with reference to the second image frame.

[0131] Taking the objective quality indicator PSNR as an example, the PSNR image quality indicator is the peak signal-to-noise ratio indicator, and the corresponding formula of its algorithm is as follows:

[0132]

[0133] Among them, bits refers to the number of bits occupied by each pixel.

[0134] PSNR (Peak Signal to Noise Ratio) is the ratio of the peak signal energy to the average energy of the noise. It is usually expressed in logarithm and converted to decibels (dB). Since the mean square error (MSE) is the energy mean of the difference between the real image (second image frame) and the noisy image (first image frame), and the difference between the two is the noise, PSNR is the ratio of the peak signal energy to MSE.

[0135] It is not difficult to understand that due to the low computational complexity of PSNR, it is more suitable for deployment and implementation on the terminal device side. Of course, it can also be deployed on the server side, which can be flexibly determined by technical personnel in this field.

[0136] The objective quality index of the MSE image is the mean square error index, and the corresponding formula of its algorithm is as follows:

[0137]

[0138] Among them, f ij , f′ ijRepresent the second image frame and the first image frame respectively, M and N represent the height and width of the image frame respectively. Since the second image frame and the first image frame are of the same scale, their height and width are the same.

[0139] SSIM (Structural Similarity) is an indicator used to measure the similarity of images. SSIM consists of three parts: brightness contrast, contrast contrast, and structure contrast. The formula corresponding to its algorithm can be flexibly implemented by technicians in this field, so it is omitted here.

[0140] According to the embodiments disclosed above, it can be known that the objective quality index of the encoded image frame can be evaluated based on the correspondence between the image frame in the video stream encoded and output by the encoder and the image frame before encoding, and the objective quality index actually obtained by the current image frame output by the encoder is affected by the subjective quality index. Therefore, it is used with the corresponding optimal frame rate for encoding the current image frame to correct the weight parameters and bias parameters of the parameter optimization model, so as to improve the prediction ability of the optimal frame rate of the parameter optimization model, so that the video image encoded according to the optimal frame rate predicted by the parameter optimization model can better match the user's perception.

[0141] According to another embodiment of the present application, after step S1500, calling the encoder to encode the current image frame at the optimal frame rate and the export bandwidth, and encoding the current image frame in the video stream, the following steps are included:

[0142] Step S1700: Push the video stream to a terminal device or a media server participating in a live network broadcast service.

[0143] When the encoder is run in a terminal device placed in a live broadcast room of a live broadcast service, the computer program product implemented by the present application can be run and implemented in the terminal device. As a result, the video stream generated by it is naturally pushed to the media server participating in the live broadcast service, so that the media server can further push it to the terminal users of the corresponding live broadcast room.

[0144] On the contrary, when the encoder is run in a media server placed in a live broadcast room of a live network broadcast service, the computer program product implemented by the present application can be run and implemented in the media server, whereby the video stream generated by it is naturally pushed to the terminal devices where the terminal users of the live broadcast room participating in the live network broadcast service are located.

[0145] According to the above embodiments, it can be known that the deployment scenario of the present application is broad. It can be deployed in a server or in a terminal device. When it serves the network live broadcast service, it will definitely enhance the actual viewing experience of the live video stream and improve the user experience of the live broadcast room.

[0146] See also Fig. 9 According to one aspect of the present application, a video stream frame rate adjustment device is provided, comprising an image acquisition module 1100, a subjective evaluation module 1200, a function modulation module 1300, and a frame rate determination module 1400, wherein the image acquisition module is used to acquire a current image frame required for generating a video stream; the subjective evaluation module 1200 is used to determine a subjective evaluation parameter of the current image frame, the subjective evaluation parameter comprising a reference frame rate suitable for encoding the current image frame, and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate; the function modulation module 1300 is used to input an output bandwidth of the encoder into a parameter optimization model to obtain an approximate function that characterizes a mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth; the frame rate determination module 1400 is used to apply the approximate function to obtain a numerical approximation point corresponding to the subjective evaluation parameter, and obtain the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate for encoding the current image frame.

[0147] Based on the above embodiments, in one embodiment, the image acquisition module includes: an image rendering inverse unit, used to render the data frame collected by the camera unit into a texture image; a format conversion unit, used to convert the texture image into an image frame of a specific format and store it in an image space; an image calling unit, used to obtain each image frame from the image space according to a timestamp as a current image frame.

[0148] Based on the above embodiments, in another embodiment, the image acquisition module includes: a decoding processing unit, used to decode the original video stream submitted by the terminal device, obtain image frames of a specific format therein, and store them in an image space; an image calling unit, used to obtain each image frame from the image space according to a timestamp as a current image frame.

[0149] Based on any embodiment of the present application, in one embodiment, the subjective evaluation module 1200 includes: a transition recognition unit, which is used to use a preset transition recognition model to determine whether the current image frame belongs to a transition image; an image analysis unit, which is used to perform image analysis on the current image frame when it is a transition image, and determine the difference between the image size between it and the previous image frame; a parameter recalculation unit, which is used to recalculate the subjective evaluation parameters of the current image frame when the difference is greater than a preset threshold; and a parameter calling unit, which is used to call the subjective evaluation parameters corresponding to the previous image frame of the current image frame as the subjective evaluation parameters of the current image frame when it is a non-transition image.

[0150] Based on any embodiment of the present application, in one embodiment, the subjective evaluation module 1200 includes: a complexity calculation unit, used to calculate the image complexity of a current image frame and several subsequent image frames; a difference calculation unit, used to calculate the difference information between the image complexity of the current image frame and the image complexity of its previous image frame; a frame rate adjustment unit, used to obtain a reference frame rate of the current image frame based on the reference frame rate of the previous image frame according to the difference information; and a parameter calculation unit, used to apply a preset subjective curve function that characterizes the mapping relationship between the reference frame rate and the subjective quality indicator, and determine the corresponding subjective quality indicator according to the reference frame rate.

[0151] Based on any embodiment of the present application, in one embodiment, after the frame rate determination module 1400, it includes: an encoding execution module, used to call the encoder to encode the current image frame with the optimal frame rate and the export bandwidth, and encode the current image frame in the video stream; an iterative correction module, used to decode the video stream to obtain the current image frame, evaluate its objective quality index, and iteratively update the parameter optimization model with the objective quality index and the optimal frame rate.

[0152] Based on the previous embodiment, in one embodiment, the iterative correction module includes: a decoding processing unit, used to decode the video stream to obtain the current image frame as the first image frame; a pre-calling unit, used to obtain the current image frame before encoding as the second image frame; an indicator calculation unit, used to use the second image frame as a reference and apply a preset formula to calculate and obtain the objective quality indicator of the first image frame.

[0153] Based on any embodiment of the present application, in one embodiment, the objective quality indicator is a peak signal-to-noise ratio.

[0154] Another embodiment of the present application also provides a video stream frame rate adjustment device. Fig.10 As shown, a schematic diagram of the internal structure of a video stream frame rate adjustment device. The video stream frame rate adjustment device includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable non-volatile readable storage medium of the video stream frame rate adjustment device stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a video stream frame rate adjustment method.

[0155] The processor of the video stream frame rate adjustment device is used to provide computing and control capabilities to support the operation of the entire video stream frame rate adjustment device. The memory of the video stream frame rate adjustment device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the video stream frame rate adjustment method of the present application. The network interface of the video stream frame rate adjustment device is used to connect and communicate with a terminal.

[0156] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the video stream frame rate adjustment device to which the scheme of the present application is applied. The specific video stream frame rate adjustment device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In this embodiment, the processor is used to execute Fig. 9 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the video stream frame rate adjustment device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0158] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the video stream frame rate adjustment method of any embodiment of the present application.

[0159] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0161] In summary, the present application can adaptively control the encoder to encode the video stream at the optimal frame rate, thereby improving the playback experience of the live video stream.

Claims

1. A method for adjusting the frame rate of a video stream, characterized in that: The steps include: Get the current image frame required to generate the video stream; Determining a subjective evaluation parameter of the current image frame, wherein the subjective evaluation parameter includes a reference frame rate suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate; Inputting the output bandwidth of the encoder into the parameter optimization model to obtain an approximate function representing the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth; The approximate function is applied to obtain a numerical approximation point corresponding to the subjective evaluation parameter, and a coding frame rate corresponding to the numerical approximation point is obtained as an optimal frame rate for coding the current image frame.

2. The video stream frame rate adjustment method according to claim 1, characterized in that: Obtaining the current image frame required to generate a video stream includes the following steps: Rendering the data frames collected by the camera unit into texture images; Converting the texture image into an image frame in a specific format and storing the frame in an image space; Each image frame is acquired from the image space according to the time stamp as the current image frame.

3. The video stream frame rate adjustment method according to claim 1, characterized in that: Obtaining the current image frame required to generate a video stream includes the following steps: Decode the original video stream submitted by the terminal device, obtain the image frame in a specific format, and store it in the image space; Each image frame is acquired from the image space according to the time stamp as the current image frame.

4. The video stream frame rate adjustment method according to claim 1, characterized in that: Determining the subjective evaluation parameters of the current image frame includes the following steps: Using a preset transition recognition model to determine whether the current image frame belongs to a transition image; When it is a transition image, image analysis is performed on the current image frame to determine the difference between the image size of the current image frame and the previous image frame; When the difference is greater than a preset threshold, recalculating the subjective evaluation parameter of the current image frame; When it is a non-transition image, the subjective evaluation parameter corresponding to the previous image frame of the current image frame is called as the subjective evaluation parameter of the current image frame.

5. The video stream frame rate adjustment method according to claim 1, characterized in that: Determining the subjective evaluation parameters of the current image frame includes the following steps: Calculating the image complexity of a current image frame and several subsequent image frames; Calculating difference information between the image complexity of the current image frame and the image complexity of the previous image frame; According to the difference information, adjusting the reference frame rate of the current image frame based on the reference frame rate of the previous image frame; A preset subjective curve function representing a mapping relationship between a reference frame rate and a subjective quality index is applied, and a corresponding subjective quality index is determined according to the reference frame rate.

6. The video stream frame rate adjustment method according to claim 1, characterized in that: After obtaining the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate for encoding the current image frame, the following steps are included: Invoke an encoder to encode the current image frame at the optimal frame rate and export bandwidth, and encode the current image frame into the video stream; The video stream is decoded to obtain the current image frame, an objective quality index thereof is evaluated, and the parameter optimization model is iteratively updated with the objective quality index and the optimal frame rate.

7. The video stream frame rate adjustment method according to claim 6, characterized in that: Decoding the video stream to obtain the current image frame and evaluating its objective quality index comprises the following steps: Decoding the video stream to obtain the current image frame as the first image frame; Acquire the current image frame before encoding as the second image frame; Taking the second image frame as a reference, a preset formula is applied to calculate and obtain an objective quality index of the first image frame.

8. The video stream frame rate adjustment method according to any one of claims 1 to 7, characterized in that: The objective quality indicator is peak signal-to-noise ratio.

9. A video stream frame rate adjustment device, characterized in that: include: An image acquisition module, used to acquire the current image frame required to generate a video stream; A subjective evaluation module, used to determine subjective evaluation parameters of the current image frame, wherein the subjective evaluation parameters include a reference frame rate suitable for encoding the current image frame and a subjective quality index that can be expected to be obtained by encoding according to the reference frame rate; A function modulation module, used for inputting the output bandwidth of the encoder into a parameter optimization model to obtain an approximate function representing the mapping relationship between the encoding frame rate and the objective quality index under the output bandwidth; The frame rate determination module is used to apply the approximate function to obtain the numerical approximation point corresponding to the subjective evaluation parameter, and obtain the encoding frame rate corresponding to the numerical approximation point as the optimal frame rate for encoding the current image frame.

10. A video stream frame rate adjustment device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 8.

11. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 8 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

12. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 8 when executed by a processor.

Images