Super-resolution control method and apparatus for video stream, and electronic device

Abstract

The present application discloses a super-resolution control method and apparatus for a video stream, and an electronic device. The method comprises: obtaining a current frame in a video stream; obtaining predicted super-resolution time consumption of the current frame; when the predicted super-resolution time consumption meets a super-resolution condition, executing super-resolution processing on the current frame; and when the predicted super-resolution time consumption does not meet the super-resolution condition, executing frame dropping processing on the current frame. Hence, in the present application, by predicting super-resolution time consumption of each frame in a video stream, and then realizing super-resolution control or frame dropping on the basis of the predicted super-resolution time consumption, the stuttering phenomenon caused by excessive super-resolution time consumption is avoided, and the viewing experience of users on the video stream is improved.

Description

Video stream super-resolution control method, device and electronic equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 27, 2023, with application number 202311263451.X and invention name “Super-resolution control method, device and electronic device for video stream”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a method, device and electronic device for super-resolution control of a video stream. Background Art

[0003] When the player plays a video, if the video source is not clear, the player can perform super-resolution when decoding the video image to enhance the image quality, so that you can experience the viewing effect of ultra-high-definition video with a low-definition video stream.

[0004] However, the amount of calculation required for super-resolution on the player is relatively large. In some cases, the performance of the device where the player is located cannot meet the requirements for super-resolution processing, resulting in lag on the player.

[0005] Therefore, there is an urgent need for a technical solution that can effectively avoid playback jams caused by video over-resolution.

[0006] Summary of the Invention

[0007] In view of this, the present application provides a method, device and electronic device for super-resolution control of video streams, which are used to solve the playback jamming problem caused by video super-resolution and improve the user's viewing experience.

[0008] A super-resolution control method for a video stream, comprising:

[0009] Get the current frame in the video stream;

[0010] Obtaining the predicted super-resolution time consumption of the current frame;

[0011] When the predicted super-resolution time meets the super-resolution condition, performing super-resolution processing on the current frame;

[0012] When the predicted super-resolution time does not satisfy the super-resolution condition, a frame drop process is performed on the current frame.

[0013] In the above method, preferably, obtaining the predicted super-resolution time consumption of the current frame includes:

[0014] Obtaining a first region and a second region in the current frame; wherein the first region is an image region in the current frame that needs to be super-resolved, and the second region is other regions in the current frame except the first region;

[0015] Obtaining an estimated regional super-resolution time corresponding to the first region according to an average super-resolution rate corresponding to the first region, wherein the average super-resolution rate represents the super-resolution time per unit area in the region where super-resolution processing is performed;

[0016] Obtaining an estimated regional fusion time corresponding to the second region according to an average fusion rate corresponding to the second region; the average fusion rate represents the fusion time per unit area in the region where the fusion process is performed;

[0017] The estimated region super-resolution time corresponding to the first region and the estimated region fusion time corresponding to the second region are added together to obtain the predicted super-resolution time of the current frame.

[0018] In the above method, preferably, the average super-resolution rate corresponding to the first region is the average super-resolution rate corresponding to the image type to which the first region belongs;

[0019] The average super-resolution rate corresponding to each image type in the current frame is obtained by:

[0020] Obtaining an actual super-resolution rate of a historical frame in the video stream on which super-resolution processing is performed; wherein the actual super-resolution rate of the historical frame represents: a super-resolution time per unit area in an image region corresponding to the image type in the historical frame;

[0021] averaging the actual super-resolution rate according to the number of historical frames to obtain an average super-resolution rate corresponding to the image type;

[0022] Alternatively, the average super-resolution rate corresponding to each image type in the current frame is obtained by:

[0023] multiplying the difference between the number of historical frames on which super-resolution processing is performed in the video stream and one by an average super-resolution rate corresponding to the image type in a frame previous to the current frame to obtain a product;

[0024] Adding the product to the actual super-resolution rate corresponding to the image type in the previous frame to obtain a sum value;

[0025] The sum is divided by the number of historical frames in the video stream on which super-resolution processing is performed, so as to obtain an average super-resolution rate corresponding to the image type.

[0026] Preferably, the above method further comprises, after performing super-resolution processing on the current frame:

[0027] Obtaining an actual super-resolution rate of the first region, where the actual super-resolution rate of the first region represents a super-resolution time consumption per unit area of ​​the first region;

[0028] An average super-resolution rate corresponding to the image type to which the first region belongs is updated according to the actual super-resolution rate of the first region.

[0029] In the above method, preferably, different image types correspond to different average super-resolution rates.

[0030] In the above method, preferably, the average fusion rate corresponding to the second region is obtained by:

[0031] Obtaining an actual fusion rate of historical frames on which fusion processing is performed in the video stream, where the actual fusion rate of the historical frames represents a fusion time per unit area within an area on which fusion processing is performed in the historical frames;

[0032] averaging the actual fusion rates according to the number of historical frames to obtain an average fusion rate corresponding to the second region;

[0033] Alternatively, the average fusion rate corresponding to the second region is obtained by:

[0034] multiplying the difference between the number of historical frames on which fusion processing is performed in the video stream minus one by an average super-resolution rate of the image region on which fusion processing is performed in a frame previous to the current frame to obtain a product;

[0035] Adding the product to the actual super-resolution rate of the image region on which the fusion process is performed in the previous frame to obtain a sum value;

[0036] The sum is divided by the number of historical frames in the video stream on which fusion processing is performed to obtain an average super-resolution rate corresponding to the second area.

[0037] Preferably, the above method further comprises, after performing super-resolution processing on the current frame:

[0038] Obtaining an actual fusion rate of the second region, where the actual fusion rate of the second region represents a fusion time per unit area within the second region;

[0039] The average fusion rate corresponding to the second area is updated according to the actual fusion rate of the second area.

[0040] The above method preferably further includes, before obtaining the predicted super-resolution time consumption of the current frame:

[0041] Obtaining a current state of a super-resolution control; wherein, when a frame loss rate corresponding to the video stream is greater than or equal to a frame loss threshold, the super-resolution control is in an off state, and when a frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in an on state; the frame loss rate represents the number of lost frames of the video stream per unit time;

[0042] When the current state is the on state, the step of obtaining the predicted super-resolution time consumption of the current frame is performed.

[0043] A super-resolution control device for a video stream, comprising:

[0044] A frame obtaining unit, used to obtain a current frame in a video stream;

[0045] A prediction obtaining unit, configured to obtain a predicted super-resolution time consumption of the current frame;

[0046] A frame control unit is configured to perform super-resolution processing on the current frame if the predicted super-resolution time meets the super-resolution condition; and to perform frame drop processing on the current frame if the predicted super-resolution time does not meet the super-resolution condition.

[0047] An electronic device, comprising:

[0048] A memory for storing computer programs and data generated by the execution of the computer programs;

[0049] A processor is used to execute the computer program to achieve: obtaining a current frame in a video stream; obtaining a predicted super-resolution time of the current frame; performing super-resolution processing on the current frame when the predicted super-resolution time meets a super-resolution condition; and performing frame drop processing on the current frame when the predicted super-resolution time does not meet the super-resolution condition.

[0050] From the above technical solution, it can be seen that in the method, device and electronic device for super-resolution control of a video stream disclosed in this application, the super-resolution time consumption is predicted for each frame of the video stream that currently needs to be played, and thus the super-resolution processing or frame drop processing for the current frame is determined based on the obtained predicted super-resolution time consumption. It can be seen that in this application, by predicting the super-resolution time consumption of each frame in the video stream, super-resolution control or frame drop processing is achieved based on the predicted super-resolution time consumption, thereby avoiding the jamming phenomenon caused by excessive super-resolution time consumption and improving the user's viewing experience of the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] FIG1 is a flow chart of a method for super-resolution control of a video stream provided in Example 1 of the present application;

[0053] FIG2 is an example diagram of a video stream requiring super-resolution control being decoded by a player in accordance with an embodiment of the present application;

[0054] FIG3 is a flowchart of obtaining the predicted super-resolution time consumption of the current frame in a super-resolution control method for a video stream provided in Example 1 of the present application;

[0055] FIG4 is an example diagram of the current frame in an embodiment of the present application;

[0056] FIG5 is a flowchart of obtaining an average super-resolution rate corresponding to each image type in a current frame in a super-resolution control method for a video stream provided in Example 1 of the present application;

[0057] FIG6 is a flowchart of obtaining an average super-resolution rate corresponding to each image type in a current frame in a super-resolution control method for a video stream provided in Example 1 of the present application;

[0058] FIG7 is a flowchart of updating an average super-resolution rate corresponding to an image type to which a first region belongs in a super-resolution control method for a video stream provided in Example 1 of the present application;

[0059] FIG8 is a flowchart of obtaining an average fusion rate corresponding to a second region in a method for super-resolution control of a video stream provided in Example 1 of the present application;

[0060] FIG9 is a flowchart of obtaining an average fusion rate corresponding to a second region in a method for super-resolution control of a video stream provided in Example 1 of the present application;

[0061] FIG10 is a flowchart of updating the average fusion rate corresponding to the second region in a super-resolution control method for a video stream provided in Example 1 of the present application;

[0062] FIG11 is another flow chart of a method for super-resolution control of a video stream provided in Example 1 of the present application;

[0063] FIG12 is a schematic structural diagram of a video stream super-resolution control device provided in Example 2 of the present application;

[0064] 13 and 14 are schematic structural diagrams of a video stream super-resolution control device provided in Example 2 of the present application;

[0065] FIG15 is a schematic structural diagram of an electronic device provided in Example 3 of the present application;

[0066] FIG16 is an overall flow chart of the present application for implementing super-resolution judgment in a variety show playback scenario;

[0067] FIG17 is an example diagram of a current frame of a variety show video stream in a variety show playback scenario applicable to the present application;

[0068] FIG18 is a schematic diagram of the super-resolution time estimation process for the current frame in the variety show playback scenario according to the present application;

[0069] FIG19 is a flow chart of the present application for calculating the portrait explicit model statistic g1 based on the portrait explicit region set and the total super-resolution execution time of the portrait explicit regions in a variety show playback scenario;

[0070] FIG20 is a flowchart illustrating the calculation of the subtitle model statistic g2 based on the subtitle region set and the total super-resolution execution time of the subtitle region in the present application in a variety show playback scenario;

[0071] FIG21 is a flow chart of the present application for calculating the logo model statistic g3 based on the logo area set and the total super-resolution execution time of the logo area in a variety show playback scenario;

[0072] FIG22 is a flowchart of calculating the fusion calculation statistic g4 based on the remaining area and the total fusion execution time of the remaining areas in the variety show playback scenario of the present application;

[0073] FIG23 is a flowchart of calculating the total super-resolution real-time statistical value m based on the current frame and the total super-resolution execution time of the current frame in the variety show playback scenario of the present application;

[0074] Figure 24 is a flow chart of the present application for calculating the real-time statistical value d of frame loss based on the total number of frame losses in a single playback and the playback duration in a variety show playback scenario. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] Example 1

[0077] Referring to Figure 1, there is shown a flowchart illustrating a method for super-resolution control of a video stream provided in Example 1 of the present application. This method can be applied to electronic devices capable of image processing, such as computers or servers. The technical solution in this embodiment is primarily used to implement super-resolution control and improve the user's viewing experience of the video stream.

[0078] Specifically, the method in this embodiment may include the following steps:

[0079] Step 101: Get the current frame in the video stream.

[0080] In this embodiment, during the playback of the video stream, each image frame to be output is acquired, and each image frame is used as the current frame to execute the control scheme of this embodiment.

[0081] For example, as shown in FIG2 , the video stream is decoded by the player, and each decoded image frame is sequentially used as the current frame to execute the control scheme of this embodiment.

[0082] Step 102: Obtain the predicted super-resolution time consumption of the current frame.

[0083] Specifically, in this embodiment, the super-resolution time consumption of the current frame can be predicted based on the super-resolution time consumption of historical frames in the video stream that have been super-resolved, so as to obtain the predicted super-resolution time consumption of the current frame.

[0084] Step 103: Determine whether the predicted super-resolution time meets the super-resolution condition. If the predicted super-resolution time meets the super-resolution condition, execute step 104; if the predicted super-resolution time does not meet the super-resolution condition, execute step 105.

[0085] Step 104: Perform super-resolution processing on the current frame.

[0086] Among them, after the current frame is super-resolved, the current frame after super-resolved is put into the rendering queue and output after being rendered and other processes.

[0087] It should be noted that the super-resolution processing performed on the current frame in step 104 includes: super-resolution processing on the first region in the current frame, and fusion processing of the super-resolution first region and the second region.

[0088] In the case that there is only one first region, super-resolution is performed on the first region to obtain a super-resolution region; then, the super-resolution region and the second region are fused to obtain a current frame that has been super-resolution processed.

[0089] In the case that there are multiple first regions, each first region is super-resolved in turn to obtain a super-resolved region corresponding to each first region, and then all the super-resolved regions are fused with the second region to obtain a current frame that has been super-resolved.

[0090] Step 105: Perform frame drop processing on the current frame.

[0091] It should be noted that the frame dropping process here can be understood as: directly discarding the frames without performing super-resolution or rendering.

[0092] In one implementation, the super-resolution condition may be: the predicted super-resolution time is less than or equal to a super-resolution threshold. Based on this, if the predicted super-resolution time is less than or equal to the super-resolution threshold, super-resolution processing is performed on the current frame; if the predicted super-resolution time is greater than the super-resolution threshold, frame drop processing is performed on the current frame.

[0093] In another implementation, the super-resolution condition may be that the statistical super-resolution time corresponding to the predicted super-resolution time is less than or equal to the super-resolution threshold. The statistical super-resolution time is the average total super-resolution time of historical frames in the video stream that were super-resolved before the current frame and the current frame.

[0094] For example, in this embodiment, the actual total super-resolution time of all historical frames that have been super-resolved in the video stream and the predicted super-resolution time of the current frame can be added together, and then divided by the sum of the number of historical frames that have been super-resolved plus 1. The resulting quotient is the average total super-resolution time of the historical frames and the current frame, that is, the statistical super-resolution time.

[0095] For example, in this embodiment, the average total super-resolution time corresponding to the current frame in the video stream can be multiplied by the number of historical frames, the product can be added to the predicted super-resolution time of the current frame, and then the obtained sum can be divided by the sum of the number of historical frames plus 1. The obtained quotient is the average total super-resolution time of the historical frames and the current frame, that is, the statistical super-resolution time.

[0096] The average super-resolution time corresponding to the current frame is: the average super-resolution time of the historical frames that were super-resolved before the current frame, specifically: the average value obtained by dividing the sum of the actual super-resolution times of the historical frames that were super-resolved by the number of historical frames.

[0097] It should be noted that the super-resolution threshold is related to the playback speed of the video stream. The higher the playback speed, the smaller the super-resolution threshold. For example, if the video stream is played at 1x speed and the super-resolution threshold is 30ms, if the predicted super-resolution time of the current frame or the statistical super-resolution time corresponding to the predicted super-resolution time is greater than 30ms, then the current frame will be dropped and neither output nor super-resolution processing will be performed; if the predicted super-resolution time of the current frame is less than or equal to 30ms, then super-resolution processing will be performed on the current frame, and the super-resolution current frame will be placed in the rendering queue and output after rendering and other processing.

[0098] For example, if the video stream is played at 2x speed, the over-resolution threshold is 15ms; if the video stream is played at 3x speed, the over-resolution threshold is 10ms.

[0099] From the above technical solution, it can be seen that in the super-resolution control method of a video stream provided in Example 1 of the present application, the super-resolution time consumption is predicted for each frame of the video stream that currently needs to be played, and thus the super-resolution processing or frame drop processing for the current frame is determined based on the obtained predicted super-resolution time consumption. It can be seen that in this application, by predicting the super-resolution time consumption of each frame in the video stream, super-resolution control or frame drop is implemented based on the predicted super-resolution time consumption, thereby avoiding the jamming phenomenon caused by excessive super-resolution time consumption and improving the user's viewing experience of the video stream.

[0100] In one implementation, in step 102, the predicted super-resolution time consumption of the current frame may be obtained in the following manner, as shown in FIG3 :

[0101] Step 301: obtaining a first region and a second region in a current frame; the first region is an image region in the current frame that needs to be super-resolution processed, and the second region is other regions in the current frame except the first region.

[0102] The current frame may contain one or more first regions, each of which corresponds to a corresponding image type based on the image content contained therein, such as a subtitle type, a portrait type, an icon type, etc. The second region is the region other than these first regions. For example, as shown in FIG4 , the first region of the current frame includes: the image region containing subtitles, the image region containing portraits, the image region containing station logos, and the image region containing program logos, and the second region of the current frame is the region other than these first regions.

[0103] In one implementation, in this embodiment, during the decoding of the video stream, the image content in the current frame may be identified by an image recognition algorithm to obtain the first region and the second region.

[0104] Step 302: Obtain an estimated regional super-resolution time consumption corresponding to the first region according to an average super-resolution rate corresponding to the first region; the average super-resolution rate represents the super-resolution time consumption per unit area in the region where super-resolution processing is performed.

[0105] In one embodiment, the average super-resolution time consumption corresponding to the first region is multiplied by the area of ​​the first region to obtain the estimated regional super-resolution time consumption corresponding to the first region.

[0106] It should be noted that different image types have different average super-resolution rates. For example, at least two of the average super-resolution times corresponding to the subtitle type, the average super-resolution time corresponding to the portrait type, and the average super-resolution time corresponding to the icon type are different, or even each of them is different. Based on this, in this embodiment, when obtaining the estimated regional super-resolution time corresponding to the first region, the average super-resolution time corresponding to the image type to which the first region belongs can be selected according to different image types and multiplied by the area of ​​the first region to obtain the estimated regional super-resolution time corresponding to the first region.

[0107] For example, taking Figure 4 as an example, in this embodiment, the average super-resolution time g1 corresponding to the portrait type is multiplied by the area s1 of the image area where the portrait is located to obtain the estimated area super-resolution time g1*s1 of the portrait area; the average super-resolution time g2 corresponding to the subtitle type is multiplied by the area s2 of the image area where the subtitle is located to obtain the estimated area super-resolution time g2*s2 of the subtitle area; the average super-resolution time g3 corresponding to the station logo and program logo is multiplied by the area s3 of the image area where the station logo and program logo are located to obtain the estimated area super-resolution time g3*s3 of the station logo and program logo area.

[0108] Step 303: Obtain an estimated regional fusion time corresponding to the second region according to the average fusion rate corresponding to the second region; the average fusion rate represents the fusion time per unit area in the region where the fusion process is performed.

[0109] In one embodiment, the average fusion time corresponding to the second region is multiplied by the area of ​​the second region to obtain the estimated regional fusion time corresponding to the second region.

[0110] The fusion processing refers to fusing the second region with the first region that has undergone super-resolution processing (ie, the super-resolution region).

[0111] For example, taking FIG. 4 as an example, in this embodiment, the average fusion time g4 corresponding to the remaining region is multiplied by the area s4 of the remaining region to obtain the estimated region fusion time g4*s4 of the remaining region.

[0112] It should be noted that the execution order between step 302 and step 303 is not limited to the execution order shown in the accompanying drawings. For example, in other embodiments, step 303 may be executed first and then step 302, or step 302 and step 303 may be executed simultaneously. The execution order of step 302 and step 303 is different, and the resulting different technical solutions are all within the scope of protection of this application.

[0113] Step 304: Add the estimated regional super-resolution time corresponding to the first region and the estimated regional fusion time corresponding to the second region to obtain the predicted super-resolution time of the current frame.

[0114] For example, taking Figure 4 as an example, g1*s1, g2*s2, g3*s3, and g4*s4 are added together to obtain the predicted super-resolution time of the current frame.

[0115] In one implementation, the average super-resolution rate corresponding to each image type in the current frame is obtained by the following method, as shown in FIG5 :

[0116] Step 501: obtaining an actual super-resolution rate of a historical frame in a video stream that has been subjected to super-resolution processing; the actual super-resolution rate of a historical frame represents: the super-resolution time per unit area in an image region corresponding to an image type in the historical frame.

[0117] The actual super-resolution rate of the historical frames in the video stream that have been super-resolved refers to the actual super-resolution rate of the image region corresponding to the image type in the historical frames in the video stream that have been super-resolved. The actual super-resolution rate of the historical frames can be obtained by dividing the actual super-resolution time of the image region corresponding to the image type in the historical frames by the area of ​​the image region corresponding to the image type in the historical frames.

[0118] For example, there are two historical frames in the video stream in which subtitle areas are super-resolved. The actual super-resolution rate of the image area corresponding to the subtitle type in each historical frame is: the actual super-resolution time of the image area corresponding to the subtitle type in the historical frame, divided by the area of ​​the image area corresponding to the subtitle type in the historical frame.

[0119] For another example, there are three historical frames in the video stream in which portrait areas are super-resolved. The actual super-resolution rate of the image area corresponding to the portrait type in each historical frame is: the actual super-resolution time of the image area corresponding to the portrait type in the historical frame, divided by the area of ​​the image area corresponding to the portrait type in the historical frame.

[0120] Step 502: According to the number of historical frames, the actual super-resolution rates are averaged to obtain an average super-resolution rate corresponding to the image type.

[0121] For example, if there are two historical frames in the video stream in which the subtitle area is super-resolved, then the average super-resolution rate corresponding to the subtitle area is: the average value obtained by dividing the actual super-resolution rate of the image area corresponding to the subtitle type in all historical frames by 2.

[0122] For example, if there are three historical frames in the video stream in which portrait areas are super-resolved, then the average super-resolution rate corresponding to the portrait areas is: the average value obtained by dividing the actual super-resolution rate of the image areas corresponding to the portrait type in all historical frames by 3.

[0123] In another implementation, the average super-resolution rate corresponding to the image type in the current frame is also obtained by the following method, as shown in FIG6 :

[0124] Step 601: multiply the difference between the number of historical frames in the video stream that have been super-resolved by one by the average super-resolution rate corresponding to the image type in the frame before the current frame to obtain a product.

[0125] Step 602: Add the product to the actual super-resolution rate corresponding to the image type in the previous frame to obtain a sum value.

[0126] Step 603: Divide the sum by the number of historical frames in the video stream that have been super-resolved to obtain an average super-resolution rate corresponding to the image type.

[0127] That is to say, in this embodiment, the difference between the number of historical frames in the video stream that have been super-resolved minus one is used and multiplied by the average super-resolution rate corresponding to the image type in the frame before the current frame, the obtained product is added to the actual super-resolution rate corresponding to the image type in the previous frame, and the obtained sum is divided by the number of historical frames in the video stream that have been super-resolved to obtain the average super-resolution rate corresponding to the image type in the current frame.

[0128] The method for obtaining the average super-resolution rate corresponding to the image type in the frame before the current frame may refer to the method for obtaining the average super-resolution rate corresponding to the image type in the current frame.

[0129] For example, taking the portrait type as an example, starting from the first frame in the video stream that is super-resolved, the actual super-resolution rate corresponding to the image area corresponding to each image type in the first frame is used as the average super-resolution rate corresponding to the corresponding image type of the next frame; after the second frame is super-resolved, the average super-resolution rate corresponding to each image type is multiplied by 1 and then added to the actual super-resolution rate of the image area corresponding to the corresponding image type on the second frame, and the sum is divided by 2, and the quotient is used as the average super-resolution rate corresponding to the corresponding image type of the next frame; after the third frame is super-resolved, the average super-resolution rate corresponding to each image type is multiplied by 1 and then added to the actual super-resolution rate of the image area corresponding to the corresponding image type on the second frame, and the sum is divided by 2, and the quotient is used as the average super-resolution rate corresponding to the corresponding image type of the next frame; The average super-resolution rate corresponding to the type is multiplied by 2 and then added to the actual super-resolution rate of the image area corresponding to the corresponding image type on the third frame. The resulting sum is divided by 3, and the quotient is used as the average super-resolution rate corresponding to the corresponding image type of the next frame. This process is repeated until the previous frame of the current frame is super-resolved. The average super-resolution rate corresponding to each image type is multiplied by the number of historical frames minus one, and then the actual super-resolution rate of the image area corresponding to the corresponding image type on the previous frame is added. The resulting sum is divided by the number of historical frames, and the quotient is used as the average super-resolution rate corresponding to the corresponding image type of the current frame.

[0130] Furthermore, after step 103, in this embodiment, the average super-resolution rate corresponding to the image type to which the first region belongs may be updated, as shown in FIG7 .

[0131] Step 701: Obtain an actual super-resolution rate of a first region, where the actual super-resolution rate of the first region represents the super-resolution time consumption per unit area in the first region.

[0132] The actual super-resolution rate of the first region can be obtained by dividing the actual super-resolution time consumed by the first region by the area of ​​the first region.

[0133] Step 702: Update the average super-resolution rate corresponding to the image type to which the first region belongs according to the actual super-resolution rate of the first region.

[0134] In one implementation, step 702 may multiply the average super-resolution rate corresponding to the image type to which the first region belongs by the number of historical frames, add the actual super-resolution rate of the first region, and divide the resulting sum by the sum of the number of historical frames plus 1. The quotient obtained is the average super-resolution rate corresponding to the image type to which the first region belongs after the update.

[0135] In another implementation, in step 702, the actual super-resolution rate of the image area of ​​the image type to which the first area in each historical frame belongs and the actual super-resolution rate of the first area can be added together, and then the obtained sum is divided by the sum of the number of historical frames plus 1, and the quotient obtained is the average super-resolution rate corresponding to the image type to which the first area belongs after the update.

[0136] In one implementation, the average fusion rate corresponding to the second region is obtained by the following method, as shown in FIG8 :

[0137] Step 801: obtaining an actual fusion rate of historical frames on which fusion processing is performed in a video stream, where the actual fusion rate of historical frames represents the fusion time per unit area in the region on which fusion processing is performed in the historical frames.

[0138] The actual fusion rate of historical frames in a video stream that are fused refers to the actual fusion rate of image regions in the historical frames that are super-resolved in the video stream. The actual fusion rate of historical frames can be obtained by dividing the actual fusion time of the image regions in the historical frames that are fused by the area of ​​the image regions in the historical frames that are fused.

[0139] Step 802: average the actual fusion rates according to the number of historical frames to obtain an average fusion rate corresponding to the second region.

[0140] For example, if there are two historical frames in the video stream in which the remaining areas except subtitles, portraits and icons are fused, then the average fusion rate corresponding to the remaining areas is: the average value of the actual fusion rate of the image areas fused in all historical frames divided by 2.

[0141] In another implementation, the average fusion rate corresponding to the second region is also obtained by the following method, as shown in FIG9 :

[0142] Step 901: multiply the difference between the number of historical frames subjected to fusion processing in the video stream and the number of frames subjected to fusion processing by one by the average fusion rate of the image region subjected to fusion processing in the frame before the current frame to obtain a product.

[0143] Step 902: Add the product to the actual fusion rate of the image region on which the fusion process is performed in the previous frame to obtain a sum value.

[0144] Step 903: Divide the sum by the number of historical frames in the video stream that have been fused, to obtain an average fusion rate corresponding to the second region.

[0145] That is to say, in this embodiment, the difference between the number of historical frames on which fusion processing is performed in the video stream minus one is used and multiplied by the average fusion rate of the image area on which fusion processing is performed in the frame before the current frame, the obtained product is added to the actual fusion rate of the image area on which the fusion processor is executed in the previous frame, and the obtained sum is divided by the number of historical frames on which fusion processing is performed in the video stream to obtain the average fusion rate corresponding to the second area in the current frame.

[0146] The method for obtaining the average fusion rate of the image region subjected to fusion processing in the frame before the current frame may refer to the method for obtaining the average fusion rate of the second region, ie, the image region subjected to fusion processing, in the current frame.

[0147] For example, starting from the first image frame in the video stream that is super-resolved, the actual fusion rate of the image area that is fused in the first frame is used as the average fusion rate of the image area that is fused in the next frame; after the second frame is super-resolved, the average fusion rate of the image area that is fused is multiplied by 1 and then added to the actual fusion rate of the image area that is fused on the second frame, the sum is divided by 2, and the quotient is used as the average fusion rate of the image area that is fused in the next frame; after the third frame is super-resolved, the image area that is fused is multiplied by 1 and then added to the actual fusion rate of the image area that is fused on the second frame, and the sum is divided by 2, and the quotient is used as the average fusion rate of the image area that is fused in the next frame; The average fusion rate of the image area is multiplied by 2 and added to the actual fusion rate of the image area on which the fusion processing is performed on the third frame. The obtained sum is divided by 3, and the quotient is used as the average fusion rate of the image area on which the fusion processing is performed on the next frame. This process is repeated until the super-resolution processing is performed on the frame before the current frame. The average fusion rate of the image area on which the fusion processing is performed is multiplied by the number of historical frames minus one, and then the actual fusion rate of the image area on which the fusion processing is performed on the previous frame is added. The obtained sum is divided by the number of historical frames, and the quotient is used as the average fusion rate corresponding to the second area in the current frame.

[0148] Furthermore, after step 103, in this embodiment, the average fusion rate corresponding to the second region may be updated, as shown in FIG10 :

[0149] Step 1001: Obtain an actual fusion rate of the second region, where the actual fusion rate of the second region represents the fusion time per unit area in the second region.

[0150] The actual fusion rate of the second region can be obtained by dividing the actual fusion time of the second region by the area of ​​the second region.

[0151] Step 1002: Update the average fusion rate corresponding to the second area according to the actual fusion rate of the second area.

[0152] In one implementation, step 1002 may multiply the average fusion rate corresponding to the second region by the number of historical frames, add the actual fusion rate of the second region, and divide the resulting sum by the sum of the number of historical frames plus 1. The quotient thus obtained is the updated average fusion rate corresponding to the second region.

[0153] In another implementation, in step 1002, the actual fusion rate of the image area on which the fusion processing is performed in each historical frame and the actual fusion rate of the second area can be added together, and then the obtained sum can be divided by the sum of the number of historical frames plus 1, and the quotient obtained is the average fusion rate corresponding to the updated second area.

[0154] It should be noted that, when the super-resolution condition is that the statistical super-resolution time corresponding to the predicted super-resolution time is less than or equal to the super-resolution threshold, in this embodiment, after step 103, the average super-resolution total time corresponding to the current frame may also be updated. For example, the statistical super-resolution time corresponding to the predicted super-resolution time of the current frame is directly used as the new average super-resolution total time.

[0155] Therefore, in this embodiment, after each super-resolution processing is performed, the average super-resolution rate, average fusion rate and average total super-resolution time are updated once, so that when super-resolution control is performed on the next frame, these updated data can be directly read without performing complex calculations again, thereby improving the efficiency of super-resolution control.

[0156] In one implementation, before obtaining the predicted super-resolution time consumption of the current frame in step 102, the present embodiment may further include the following processing, as shown in FIG11 :

[0157] Step 101-1: Obtain the current state of the super-resolution control;

[0158] Among them, when the frame loss rate corresponding to the video stream is greater than or equal to the frame loss threshold, the super-resolution control is in the off state; when the frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in the on state; the frame loss rate represents the number of lost frames in the video stream per unit time.

[0159] It should be noted that the frame loss rate in this embodiment is obtained by real-time calculation. Specifically, in this embodiment, the frame loss rate is updated once after each image frame in the video stream is processed.

[0160] For example, after step 104 or step 105, the current playback duration of the video stream and the number of lost frames of the image frames in the video stream that are subjected to frame loss processing are obtained in this embodiment, and then the number of lost frames is divided by the current playback duration to obtain the frame loss rate.

[0161] The frame loss threshold varies depending on the playback speed of the video stream. For example, at 1x speed, the frame loss threshold is 5 frames per second; at 2x speed, the frame loss threshold is 30 frames per second; and at 3x speed, the frame loss threshold is 40 frames per second.

[0162] Step 101-2: Determine whether the current state is on. If the current state is on, execute step 102 to obtain the predicted super-resolution time of the current frame, and then perform super-resolution processing on the current frame if the predicted super-resolution time meets the super-resolution condition; if the predicted super-resolution time does not meet the super-resolution condition, perform frame drop processing on the current frame.

[0163] It should be noted that, when the current state is closed, step 102 is not executed. At this time, the current frame can be directly placed in the rendering queue and output after rendering and other processing. That is, no super-resolution judgment and super-resolution processing are performed, but direct low-definition output is performed.

[0164] As can be seen, in this embodiment, before super-resolution is determined for the current frame, the frame loss rate is first determined. If the frame loss rate is too high, the super-resolution function is disabled for the video stream to avoid excessive frame loss and poor viewing quality caused by super-resolution processing. Super-resolution is determined only when the frame loss rate is low, and super-resolution processing is performed only when the super-resolution conditions are met to achieve high-definition output.

[0165] It should be noted that the super-resolution control is turned on by default before the video stream is output by the player. As the video stream is super-resolved, the super-resolution control is turned off due to the high frame loss rate.

[0166] After the super-resolution control is turned off, it can no longer be turned on, and the frame loss rate is no longer updated. That is, the super-resolution control remains off until the video stream ends. After the video stream ends and before the next video stream is output, the super-resolution control is turned on.

[0167] Alternatively, after the super-resolution control is turned off, the frame loss rate may continue to be updated, and the super-resolution control may be turned on again based on the possibly reduced frame loss rate, until the frame loss rate rises again to the frame loss threshold, at which point the super-resolution control is turned off again.

[0168] Example 2

[0169] Referring to Figure 12 , there is shown a schematic diagram of the structure of a video stream super-resolution control device provided in Example 2 of this application. This device can be configured in an electronic device capable of image processing, such as a computer or server. The technical solution in this embodiment is primarily used to implement super-resolution control and improve the user's viewing experience of the video stream.

[0170] Specifically, the device in this embodiment may include the following units:

[0171] The frame obtaining unit 1201 is used to obtain the current frame in the video stream;

[0172] The prediction obtaining unit 1202 is configured to obtain the predicted super-resolution time consumption of the current frame;

[0173] The frame control unit 1203 is configured to perform super-resolution processing on the current frame if the predicted super-resolution time meets the super-resolution condition; and perform frame drop processing on the current frame if the predicted super-resolution time does not meet the super-resolution condition.

[0174] As can be seen from the above technical solution, in the super-resolution control device for a video stream provided in the second embodiment of the present application, a prediction of the super-resolution time consumption is performed for each frame of the video stream that is currently required to be played, thereby determining whether to perform super-resolution processing or frame drop processing on the current frame based on the obtained predicted super-resolution time consumption. It can be seen that in this embodiment, by predicting the super-resolution time consumption of each frame in the video stream, super-resolution control or frame drop processing is implemented based on the predicted super-resolution time consumption, thereby avoiding the jamming phenomenon caused by excessive super-resolution time consumption and improving the user's viewing experience of the video stream.

[0175] In one implementation, the prediction obtaining unit 1202 is specifically used to: obtain a first area and a second area in the current frame; the first area is the image area in the current frame that needs to be super-resolution processed, and the second area is other areas in the current frame except the first area; according to the average super-resolution rate corresponding to the first area, obtain the estimated area super-resolution time corresponding to the first area; the average super-resolution rate represents the super-resolution time per unit area in the area where super-resolution processing is performed; according to the average fusion rate corresponding to the second area, obtain the estimated area fusion time corresponding to the second area; the average fusion rate represents the fusion time per unit area in the area where fusion processing is performed; the estimated area super-resolution time corresponding to the first area and the estimated area fusion time corresponding to the second area are added to obtain the predicted super-resolution time of the current frame.

[0176] In one implementation, the average super-resolution rate corresponding to the first region is an average super-resolution rate corresponding to the image type to which the first region belongs;

[0177] The device in this embodiment further includes an average rate processing unit 1204, as shown in FIG13 :

[0178] The average rate processing unit 1204 obtains an average super-resolution rate corresponding to each image type in the current frame by: obtaining an actual super-resolution rate of a historical frame in the video stream on which super-resolution processing is performed; the actual super-resolution rate of the historical frame represents: a super-resolution time per unit area within an image region corresponding to the image type in the historical frame; and averaging the actual super-resolution rates according to the number of historical frames to obtain an average super-resolution rate corresponding to the image type.

[0179] Alternatively, the average rate processing unit 1204 obtains the average super-resolution rate corresponding to each image type in the current frame in the following manner: multiplying the difference between the number of historical frames in the video stream that have been super-resolved by one by the average super-resolution rate corresponding to the image type in the previous frame of the current frame to obtain a product; adding the product to the actual super-resolution rate corresponding to the image type in the previous frame to obtain a sum; and dividing the sum by the number of historical frames in the video stream that have been super-resolved to obtain the average super-resolution rate corresponding to the image type.

[0180] In one implementation, the average rate processing unit 1204 is used to: obtain the actual super-resolution rate of the first area after the frame control unit 1203 performs super-resolution processing on the current frame, the actual super-resolution rate of the first area representing the super-resolution time per unit area in the first area; and update the average super-resolution rate corresponding to the image type to which the first area belongs according to the actual super-resolution rate of the first area.

[0181] Among them, different image types correspond to different average super-resolution rates.

[0182] In one implementation, the average rate processing unit 1204 obtains the average fusion rate corresponding to the second area by: obtaining an actual fusion rate of the historical frames on which the fusion process is performed in the video stream, where the actual fusion rate of the historical frames represents the fusion time per unit area within the area on which the fusion process is performed in the historical frames; averaging the actual fusion rates according to the number of the historical frames to obtain the average fusion rate corresponding to the second area;

[0183] Alternatively, the average rate processing unit 1204 obtains the average fusion rate corresponding to the second area in the following manner: using the difference between the number of historical frames on which fusion processing is performed in the video stream minus one, multiplying it by the average super-resolution rate of the image area on which fusion processing is performed in the frame before the current frame to obtain a product; adding the product to the actual super-resolution rate of the image area on which fusion processing is performed in the previous frame to obtain a sum; dividing the sum by the number of historical frames on which fusion processing is performed in the video stream to obtain the average super-resolution rate corresponding to the second area.

[0184] In one implementation, the average rate processing unit 1204 is used to: obtain the actual fusion rate of the second area after the frame control unit 1203 performs super-resolution processing on the current frame, the actual fusion rate of the second area representing the fusion time per unit area in the second area; and update the average fusion rate corresponding to the second area according to the actual fusion rate of the second area.

[0185] In one implementation, the apparatus in this embodiment further includes a frame loss determination unit 1205, as shown in FIG14 :

[0186] The frame loss judgment unit 1205 is used to: obtain the current state of the super-resolution control before the prediction acquisition unit 1202 obtains the predicted super-resolution time of the current frame; wherein, when the frame loss rate corresponding to the video stream is greater than or equal to the frame loss threshold, the super-resolution control is in the closed state, and when the frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in the open state; the frame loss rate represents the number of lost frames of the video stream per unit time; when the current state is in the open state, the prediction acquisition unit 1202 is triggered to obtain the predicted super-resolution time of the current frame.

[0187] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0188] Example 3

[0189] 15 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present application. The electronic device may include the following structure:

[0190] Memory 1501, used to store computer programs and data generated by the execution of the computer programs;

[0191] Processor 1502 is used to execute the computer program to achieve: obtaining a current frame in a video stream; obtaining a predicted super-resolution time of the current frame; performing super-resolution processing on the current frame when the predicted super-resolution time meets a super-resolution condition; and performing frame drop processing on the current frame when the predicted super-resolution time does not meet the super-resolution condition.

[0192] As can be seen from the above technical solution, in an electronic device provided in Example 3 of the present application, a prediction of the super-resolution time consumption is performed for each frame of the image currently required to be played in the video stream, thereby determining whether to perform super-resolution processing or frame drop processing on the current frame based on the obtained predicted super-resolution time consumption. It can be seen that in this embodiment, by predicting the super-resolution time consumption of each frame in the video stream, and then implementing super-resolution control or frame drop processing based on the predicted super-resolution time consumption, the freeze phenomenon caused by excessive super-resolution time consumption is avoided, thereby improving the user's viewing experience of the video stream.

[0193] Taking the video stream playback scenario of a TV variety show as an example, the technical solution of this application is illustrated below:

[0194] First, when the player plays a video, if the user's local network is poor, or if they choose to play the video in both standard definition and ultra-high definition to save data, or if the video source is not clear to begin with, the player will perform AI super-resolution to enhance the image quality when the video is decoded. This allows users to experience the ultra-high-definition viewing effect with a low-definition video stream. However, AI super-resolution is computationally intensive, and even when super-resolution is performed on feature areas (such as portrait visibility, subtitles, and station logos), performance will still not be achieved on low-end devices or at high magnifications, resulting in lag and audio-visual asynchrony.

[0195] In order to solve this problem, this application proposes a dynamic frame drop mechanism. By real-time statistics of the time consumed for super-resolution and fusion of each area, the super-resolution execution time of the current frame is estimated (i.e., the predicted super-resolution time consumed in the previous article). Then, based on the estimated super-resolution execution time consumed by the current frame, a judgment is made to drop the frame or perform super-resolution rendering. Then, based on the actual super-resolution data, the time consumption statistics of each time consumption are fed back (i.e., the average super-resolution rate and the average fusion rate are updated). At the same time, the frame rate in the case of frame loss can be judged based on the entire super-resolution time consumption statistics. If it does not reach the set value (i.e., the frame loss rate is greater than or equal to the frame loss threshold), the AI ​​super-resolution function is terminated.

[0196] As shown in Figure 16, this is the overall flow chart for implementing super-resolution judgment in this application. After the video frame is decoded and the current video frame is obtained, it is divided into: determining whether to enable super-resolution, estimating the super-resolution time, determining whether to super-resolution based on the estimated time, executing super-resolution and feeding back the updated estimated value, and entering the rendering queue.

[0197] It should be noted that the final region set of the current frame is generated by the server during video encoding and obtained through a fusion algorithm. It is mainly divided into three image types: portrait explicit feature area, subtitle area, and station logo area. The details are as follows:

[0198] 1. Video frame decoding to obtain the current video frame: The player decodes the video stream sent by the server to obtain continuous video image (i.e., image frame) data and a set of feature interval sequences for the image. A fusion algorithm is then used to obtain the final set of regions to be super-resolution calculated (i.e., the first and second regions mentioned above).

[0199] 2. Determine whether to enable super-resolution: Super-resolution calculation is enabled by default each time playback is started. As the video stream plays, if it is determined that the super-resolution conditions are not met (mainly due to insufficient performance), that is, the frame loss per second exceeds the frame loss threshold, the super-resolution calculation function will be turned off.

[0200] 3. Super-resolution time estimation: Based on the final feature set and the statistical parameters of each region, estimate the time required to perform super-resolution calculations on the current frame (i.e., the predicted super-resolution time mentioned above).

[0201] 4. Determine whether to perform super-resolution based on the estimated time consumption: The time consumption calculated based on the super-resolution time consumption is the predicted super-resolution time consumption mentioned above, and determine whether to perform super-resolution. If the predicted super-resolution time consumption exceeds the super-resolution threshold, discard the current frame. If it does not exceed the super-resolution threshold, determine to perform super-resolution calculation.

[0202] 5. Perform super-resolution calculations: Perform super-resolution calculations on the final set of regions to obtain the super-resolution image, as well as the actual and total time consumption of the fusion calculations of each region. Feedback is provided to update the time consumption statistics of each type (i.e., the average fusion rate and the average super-resolution rate) and the total time consumption statistics (i.e., the average total super-resolution time).

[0203] 6. Enter the rendering queue: replace the current frame with the image frame calculated by super-resolution and enter the rendering queue for image rendering.

[0204] The specific plan is as follows:

[0205] First, the values ​​that need to be estimated in real time in this embodiment are: portrait explicit model statistic g1, subtitle model statistic g2, station logo model statistic g3, fusion calculation statistic g4, total super-resolution real-time statistic m, and frame loss real-time statistic d.

[0206] g1, g2, g3, and g4 are the total computation time of each model or fusion divided by the corresponding area, expressed in milliseconds per second. The default values ​​of g1, g2, g3, and g4 are 0 before the video stream is played. m is the total statistical computation time (i.e., the average total time it takes to super-resolve a single frame), expressed in milliseconds. d is the number of frame drops per second, also known as the frame loss rate.

[0207] Taking the current frame shown in Figure 17 as an example, Figure 18 shows a schematic diagram of the super-resolution time estimation process for the current frame, which is mainly divided into: portrait explicit area model estimation, subtitle model estimation, station logo model estimation, and fusion calculation estimation.

[0208] 1.1 Based on the total area of ​​the portrait explicit region set as s1, we multiply the area s1 by the portrait explicit model statistic g1 to obtain the estimated value t1 for the portrait explicit region model calculation (i.e., the portrait region super-resolution calculation), which is the estimated region super-resolution time mentioned above. The formula is t1 = s1 * g1.

[0209] 1.2 Based on the total area of ​​the subtitle region set being s2, the subtitle model statistic g2 is multiplied by the area s2 to obtain the estimated value t2 for the subtitle region model calculation (i.e., the subtitle region super-resolution calculation). This is the formula t2 = s2 * g2.

[0210] 1.3 Based on the total area of ​​the logo region set being s3, multiplying the logo model statistic g3 by the logo region model calculation estimate t3 (i.e., logo region super-resolution calculation) can be obtained using the formula t3 = s3 * g3.

[0211] 1.4 The remaining area is fused and calculated. The remaining area s4 = s - s1 - s2 - s3 (s is the total area of ​​the image. There may be overlap among s1, s2, and s3, but this overlap is negligible due to the small percentage of overlap in actual video frames). The area s4 is then multiplied by the fusion calculation statistic g4 to obtain the estimated value t4 for the remaining area fusion calculation (i.e., the remaining area super-resolution calculation). The formula is t4 = s4 * g4.

[0212] 1.5 The total time consumed for super-resolution calculation of the current frame is t=t1+t2+t3+t4.

[0213] Taking the current frame shown in FIG17 as an example, FIG19 , FIG20 , FIG21 , FIG22 , FIG23 and FIG24 are schematic diagrams of a calculation process for feedback updating estimated values ​​after super-resolution processing, as follows:

[0214] 2.1 Update of the portrait explicitness model statistic g1, calculated using the formula g1 = (g1*(n-1) + tn / s1) / n, as shown in Figure 19. At the start of playback, g1 is initialized to 0. n is the order of the current frame in the video stream where super-resolution processing is performed. tn, when super-resolution calculations are being performed, is the total super-resolution time (t) for the portrait explicitness region of the current frame. g1 is not updated when frame drop logic is executed.

[0215] 2.2 Update the subtitle model statistical value g2, calculated using the formula g2 = (g2 * (n - 1) + tn / s2) / n, as shown in Figure 20. At the start of playback, g2 is initially 0. When performing super-resolution calculations, tn is the total super-resolution execution time (t) for the subtitle region of the current frame. When executing the frame drop logic, g2 is not updated.

[0216] 2.3 Update the logo model statistical value g3, calculated using the formula g3 = (g3*(n-1)+tn / s3) / n, as shown in Figure 21. At the start of playback, g3 is initialized to 0. When performing super-resolution calculations, tn is the total super-resolution execution time (t) for the logo area in the current frame. g3 is not updated when the frame drop logic is executed.

[0217] 2.4 Update of the fusion calculation statistic g4, calculated using the formula g4 = (g4 * (n - 1) + tn / s4) / n, as shown in Figure 22. At the start of playback, g4 is initialized to 0. When performing super-resolution calculations, tn is the total fusion execution time (t) for the fusion region of the current frame. g4 is not updated when executing the frame drop logic.

[0218] 2.5 Update the total super-resolution real-time statistical value m, calculated as m = (m*(n-1) + tn) / n, as shown in Figure 23. This is the average value of the current real-time super-resolution time. tn has two values: when performing super-resolution calculations, it is the total super-resolution execution time t for the current frame; when executing frame drop logic, it is 0.

[0219] 2.6 Update of the real-time frame loss statistics value d. The calculation formula is d = count / time, as shown in Figure 24, where count is the total number of lost frames in a single playback and time is the playback duration.

[0220] The following is the process of super-resolution judgment:

[0221] (1) Logic judgment on whether to perform super-resolution. The initial value of starting playback is true, which represents the logic of super-resolution judgment. When the real-time frame loss statistics value d is greater than the threshold (usually 5, 30 in the 2x speed playback scenario, and 40 in the 3x speed playback scenario), it is set to false to disable the super-resolution logic and jump to (2).

[0222] (2) Whether super-resolution calculation is performed on the current frame. The value C is calculated based on the total super-resolution calculation time t estimated for the current frame and the total super-resolution real-time statistical value m. The formula is C = (m*(n-1)+t) / n, where n is the total number of executions currently. Determine C and the threshold S (S is 30ms when playing at 1x speed, 15ms when playing at 2x speed, and 10ms when playing at 3x speed). If C is less than or equal to S, super-resolution calculation is performed and the process goes to (3). If it is greater than the threshold, the frame loss logic is bypassed and the process goes to (4).

[0223] (3) Perform super-resolution calculations based on the final region set of the current frame to obtain the super-resolution image, the actual total time consumed, and the super-resolution effect of each region. The super-resolution image is then used to replace the original image and placed into the rendering queue. At the same time, the statistical values ​​of each module are updated based on the actual time consumed.

[0224] (4) First, update the real-time frame loss statistics d, and then determine whether d is greater than the threshold (the threshold is 5 in the case of 1x speed, the threshold is 30 in the case of 2x speed, and the threshold is 55 in the case of 3x speed). If d is greater than the threshold, the judgment on whether to perform super-resolution is set to false, that is, the super-resolution control is in the closed state.

[0225] It can be seen that the technical solution of this application makes improvements in the following aspects:

[0226] First, by calculating the computational time and fusion time for each classification (i.e., image type) in the super-resolution region, the super-resolution computation time of the current frame is estimated. A decision is made to discard the frame or perform super-resolution rendering, and the updated statistical value is fed back. In the event of super-resolution and rendering timeouts, dynamic frame dropping reduces video freezes and audio-visual asynchrony during real-time super-resolution on low-end devices.

[0227] Second, the super-resolution time consumption is calculated by statistics. When the maximum threshold is exceeded (the video frame rate does not reach 20 frames in the case of frame loss, and the normal video frame rate is 25 frames), the super-resolution function is automatically disabled.

[0228] 3. In the case of multiple playback, different thresholds are set according to different playback speeds, and appropriate frame drops are dynamically selected to ensure that there are no freezes or audio and video out of sync caused by insufficient performance during multiple playback.

[0229] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0230] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0231] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0232] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for super-resolution control of a video stream, characterized in that: include: Get the current frame in the video stream; Obtaining the predicted super-resolution time consumption of the current frame; When the predicted super-resolution time consumption satisfies the super-resolution condition, performing super-resolution processing on the current frame; When the predicted super-resolution time consumption does not satisfy the super-resolution condition, performing frame drop processing on the current frame; The time consumption of obtaining the predicted super-resolution of the current frame includes: Obtaining a first region and a second region in the current frame; the first region is an image region in the current frame that needs to be super-resolution processed, and the second region is other regions in the current frame except the first region; the first region corresponds to a corresponding image type based on the image content contained therein; Obtaining an estimated regional super-resolution time consumption corresponding to the first region according to an average super-resolution rate corresponding to the first region; the average super-resolution rate represents the super-resolution time consumption per unit area in the region where super-resolution processing is performed; Obtaining an estimated regional fusion time corresponding to the second region according to an average fusion rate corresponding to the second region; the average fusion rate represents the fusion time per unit area in the region where the fusion process is performed; Adding the estimated region super-resolution time corresponding to the first region and the estimated region fusion time corresponding to the second region to obtain the predicted super-resolution time of the current frame; Before obtaining the predicted super-resolution time consumption of the current frame, the method further includes: Obtaining the current state of the super-resolution control; wherein, when the frame loss rate corresponding to the video stream is greater than or equal to the frame loss threshold, the super-resolution control is in a closed state, and when the frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in an open state; the frame loss rate represents the number of frame losses of the video stream per unit time; When the current state is the on state, performing the steps of: obtaining the predicted super-resolution time consumption of the current frame; The playback speed of the video stream is different, and the corresponding frame loss threshold is different.

2. The method according to claim 1, characterized in that The average super-resolution rate corresponding to the first region is the average super-resolution rate corresponding to the image type to which the first region belongs; The average super-resolution rate corresponding to each image type in the current frame is obtained by: Obtaining an actual super-resolution rate of a historical frame on which super-resolution processing is performed in the video stream; the actual super-resolution rate of the historical frame represents: a super-resolution time per unit area in an image region corresponding to the image type in the historical frame; According to the number of the historical frames, the actual super-resolution rate is averaged to obtain an average super-resolution rate corresponding to the image type; Alternatively, the average super-resolution rate corresponding to each image type in the current frame is obtained by: Multiplying the difference between the number of historical frames on which super-resolution processing is performed in the video stream and the average super-resolution rate corresponding to the image type in a frame before the current frame by one, to obtain a product; Adding the product to the actual super-resolution rate corresponding to the image type in the previous frame to obtain a sum value; The sum is divided by the number of historical frames on which super-resolution processing is performed in the video stream to obtain an average super-resolution rate corresponding to the image type.

3. The method according to claim 2, characterized in that After performing super-resolution processing on the current frame, the method further includes: Obtaining an actual super-resolution rate of the first region, where the actual super-resolution rate of the first region represents a super-resolution time consumption per unit area in the first region; According to the actual super-resolution rate of the first region, an average super-resolution rate corresponding to the image type to which the first region belongs is updated.

4. The method according to claim 2, characterized in that: Different image types correspond to different average super-resolution rates.

5. The method according to claim 1, characterized in that: The average fusion rate corresponding to the second region is obtained by: Obtain an actual fusion rate of the historical frames in the video stream that are subjected to fusion processing, wherein the actual fusion rate of the historical frames represents the fusion rate per unit area in the region in which the fusion processing is performed in the historical frames. time consuming; According to the number of the historical frames, the actual fusion rate is averaged to obtain an average fusion rate corresponding to the second area; Alternatively, the average fusion rate corresponding to the second region is obtained by: The difference between the number of historical frames subjected to fusion processing in the video stream and one is multiplied by the average super-resolution rate of the image region subjected to fusion processing in a frame before the current frame to obtain a product; Adding the product to the actual super-resolution rate of the image region on which the fusion process is performed in the previous frame to obtain a sum value; The sum is divided by the number of historical frames on which fusion processing is performed in the video stream to obtain an average super-resolution rate corresponding to the second area.

6. The method according to claim 1 or 5, characterized in that: After performing super-resolution processing on the current frame, the method further includes: Obtaining an actual fusion rate of the second region, where the actual fusion rate of the second region represents a fusion time per unit area in the second region; According to the actual fusion rate of the second area, the average fusion rate corresponding to the second area is updated.

7. A super-resolution control device for a video stream, characterized in that: include: A frame obtaining unit, used for obtaining a current frame in a video stream; A prediction obtaining unit, used for obtaining the predicted super-resolution time consumption of the current frame; A frame control unit, configured to perform super-resolution processing on the current frame if the predicted super-resolution time meets the super-resolution condition; and perform frame drop processing on the current frame if the predicted super-resolution time does not meet the super-resolution condition; The prediction obtaining unit is used to obtain a first region and a second region in the current frame; the first region is an image region in the current frame that needs to be super-resolution processed, and the second region is other regions in the current frame except the first region; the first region corresponds to a corresponding image type based on the image content contained therein; according to an average super-resolution rate corresponding to the first region, an estimated region super-resolution time corresponding to the first region is obtained; the average super-resolution rate represents the super-resolution time per unit area in the region where super-resolution processing is performed; according to an average fusion rate corresponding to the second region, an estimated region super-resolution time corresponding to the first region is obtained. The estimated regional fusion time corresponding to the second region is obtained; the average fusion rate represents the fusion time per unit area in the region where the fusion process is performed; the estimated regional super-resolution time corresponding to the first region and the estimated regional fusion time corresponding to the second region are added together to obtain the predicted super-resolution time of the current frame; A frame loss judgment unit, used for obtaining the current state of the super-resolution control before the prediction obtaining unit obtains the predicted super-resolution time consumption of the current frame; wherein, when the frame loss rate corresponding to the video stream is greater than or equal to the frame loss threshold, the super-resolution control is in a closed state, and when the frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in an open state; the frame loss rate represents the number of lost frames of the video stream per unit time; when the current state is in an open state, the prediction obtaining unit is triggered; The playback speed of the video stream is different, and the corresponding frame loss threshold is different.

8. An electronic device, characterized in that: include: A memory, used to store a computer program and data generated by the execution of the computer program; A processor, configured to execute the computer program to implement: obtaining a current frame in a video stream; obtaining a predicted super-resolution time of the current frame; performing super-resolution processing on the current frame if the predicted super-resolution time satisfies a super-resolution condition; performing frame drop processing on the current frame if the predicted super-resolution time does not satisfy the super-resolution condition; The time consumption of obtaining the predicted super-resolution of the current frame includes: Obtaining a first region and a second region in the current frame; the first region is an image region in the current frame that needs to be super-resolution processed, and the second region is other regions in the current frame except the first region; the first region corresponds to a corresponding image type based on the image content contained therein; Obtaining an estimated regional super-resolution time consumption corresponding to the first region according to an average super-resolution rate corresponding to the first region; the average super-resolution rate represents the super-resolution time consumption per unit area in the region where super-resolution processing is performed; Obtaining an estimated regional fusion time corresponding to the second region according to an average fusion rate corresponding to the second region; the average fusion rate represents the fusion time per unit area in the region where the fusion process is performed; The estimated area corresponding to the first area is super-resolved and the estimated area corresponding to the second area is super-resolved. The domain fusion time is added to obtain the predicted super-resolution time of the current frame; Before obtaining the predicted super-resolution time consumption of the current frame, the method further includes: Obtaining the current state of the super-resolution control; wherein, when the frame loss rate corresponding to the video stream is greater than or equal to the frame loss threshold, the super-resolution control is in a closed state, and when the frame loss rate corresponding to the video stream is less than the frame loss threshold, the super-resolution control is in an open state; the frame loss rate represents the number of frame losses of the video stream per unit time; When the current state is the on state, performing the steps of: obtaining the predicted super-resolution time consumption of the current frame; The playback speed of the video stream is different, and the corresponding frame loss threshold is different.

Images