A method, apparatus and storage medium for region-based video super-resolution processing

Abstract

The embodiment of the present application discloses a method, device and storage medium for super-resolution processing of video by region. The method includes: decoding a video stream to obtain a plurality of frames of video images and a set of feature region sequences corresponding to each frame of video images; intercepting corresponding feature images from the current frame of video images according to the set of feature region sequences corresponding to the current frame of video images to obtain a set of feature images; performing super-resolution processing on the images in the set of feature images to obtain a set of feature images after super-resolution processing; performing fusion calculation based on the set of feature images after super-resolution processing and the current frame of video images to obtain the current frame of video images after super-resolution processing. It can be seen that in the embodiment of the present application, super-resolution processing is performed on the feature images in the original video images according to the set of feature region sequences, and there is no need to perform super-resolution processing on all the original video images, which greatly reduces the amount of client calculation while enhancing the video quality.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and more specifically, to a method, device and storage medium for processing video super-resolution by region. Background Art

[0002] When the player plays a video, if the user's local network is poor, or if the user chooses to play the video in standard definition or ultra-fast definition to save traffic, or if the video source is not clear, the player will perform super-resolution processing when the video is decoded. In this way, by performing super-resolution processing on the decoded video, the video quality can be enhanced, and the viewing effect of ultra-high-definition video can be experienced with low-definition video streams, which can not only reduce user traffic consumption, but also reduce server export bandwidth and reduce operating costs.

[0003] However, currently the player performs super-resolution processing on every frame of the video, which results in a very large amount of computation for super-resolution processing on the client side and places high performance requirements on the client side. Summary of the invention

[0004] In view of this, the embodiments of the present application disclose a method, device and storage medium for region-based video super-resolution processing, so as to reduce the computational complexity of client super-resolution processing while enhancing video quality.

[0005] The technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for processing a video super-resolution by region, the method comprising:

[0007] Decode the video stream to obtain multiple video frames and a set of feature region sequences corresponding to each video frame;

[0008] According to the feature region sequence set corresponding to the current frame video picture, a corresponding feature picture is intercepted from the current frame video picture to obtain a feature picture set;

[0009] Performing super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing;

[0010] A fusion calculation is performed based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing.

[0011] In a possible implementation, the extracting corresponding feature pictures from the current frame video picture according to the feature region sequence set corresponding to the current frame video picture to obtain the feature picture set includes:

[0012] Splitting the feature region sequence set corresponding to the current frame video picture to obtain a portrait explicit region set, a subtitle region set, and a logo region set corresponding to the current frame video picture;

[0013] According to the portrait explicit area set, the subtitle area set and the identification area set, corresponding feature pictures are intercepted from the current frame video picture to obtain a feature picture set.

[0014] In a possible implementation, the extracting corresponding feature pictures from the current frame video picture according to the portrait explicit area set, the subtitle area set, and the identification area set to obtain the feature picture set includes:

[0015] Processing the overlapping areas in the portrait explicit area set to obtain a final portrait explicit area set;

[0016] Processing the areas in the subtitle area set that overlap with the areas in the final portrait explicit area set to obtain a final subtitle area set;

[0017] Processing the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain a final marked area set;

[0018] According to the final portrait explicit region set, corresponding feature images are intercepted from the current frame video image to obtain a portrait explicit image set;

[0019] According to the final subtitle area set, corresponding feature pictures are intercepted from the current frame video picture to obtain a subtitle picture set;

[0020] According to the final identification area set, corresponding feature pictures are intercepted from the current frame video picture to obtain an identification picture set.

[0021] In a possible implementation, the processing of the overlapping regions in the set of portrait dominant regions to obtain a final set of portrait dominant regions includes:

[0022] Arrange the regions in the portrait explicit region set from large to small according to the region area to obtain a first region set;

[0023] Taking out a first region from the first region set and putting it into the second region set; the first region is the region with the largest area in the first region set;

[0024] Determine whether the current first region set is empty;

[0025] If the current first region set is not empty, taking a second region from the current first region set; the second region is the region with the largest area in the current first region set;

[0026] Traversing the regions in the second region set, and determining whether the second region overlaps with the currently traversed third region;

[0027] If the second area overlaps with the third area, a first overlapping index is calculated;

[0028] Determining whether the first overlap index is greater than a first preset threshold;

[0029] If the first overlap index is greater than a first preset threshold, merging the second region with the third region to obtain a fourth region;

[0030] The third area in the second area set is replaced by the fourth area, and the following steps are returned to be executed: determining whether the current first area set is empty;

[0031] If the current first region set is empty, the current second region set is used as the final portrait explicit region set.

[0032] In a possible implementation, the processing of the areas in the subtitle area set that overlap with the areas in the final portrait explicit area set to obtain the final subtitle area set includes:

[0033] Determine whether the current subtitle area set is empty;

[0034] If the current subtitle area set is not empty, taking a fifth area from the current subtitle area set; the fifth area is any area in the current subtitle area set;

[0035] Traversing the final set of portrait explicit regions, and determining whether the fifth region overlaps with the currently traversed sixth region;

[0036] If the fifth area overlaps with the sixth area, a second overlapping index is calculated;

[0037] Determining whether the second overlap index is greater than a second preset threshold;

[0038] If the second overlapping index is greater than a second preset threshold, filtering out the area in the fifth area that overlaps with the sixth area to obtain a filtered fifth area;

[0039] Put the filtered fifth area into the third area set, and return to perform the following steps: determine whether the current subtitle area set is empty;

[0040] If the current subtitle area set is empty, the current third area set is used as the final subtitle area set.

[0041] In a possible implementation, the processing of the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain the final marked area set includes:

[0042] Determine whether the current identification area set is empty;

[0043] If the current identification area set is not empty, taking out a seventh area from the current identification area set; the seventh area is any area in the current identification area set;

[0044] Traversing the final set of portrait explicit regions, and determining whether the seventh region overlaps with the currently traversed eighth region;

[0045] If the seventh area overlaps with the eighth area, a third overlapping index is calculated;

[0046] Determining whether the third overlap index is greater than a third preset threshold;

[0047] If the third overlapping index is greater than a third preset threshold, filtering out the area in the seventh area that overlaps with the eighth area to obtain a filtered seventh area;

[0048] Put the filtered seventh area into the fourth area set, and return to execute the following steps: determine whether the current marked area set is empty;

[0049] If the current marked area set is empty, the current fourth area set is used as the final marked area set.

[0050] In a possible implementation, performing super-resolution processing on the feature picture set to obtain the feature picture set after super-resolution processing includes:

[0051] Performing super-resolution processing on the portrait explicit picture set by using a portrait explicit super-resolution model to obtain a super-resolution processed portrait explicit picture set;

[0052] Performing super-resolution processing on the subtitle picture set by using a subtitle super-resolution model to obtain a subtitle picture set after super-resolution processing;

[0053] The logo image set is subjected to super-resolution processing using a logo super-resolution model to obtain a logo image set after super-resolution processing.

[0054] In a second aspect, an embodiment of the present application provides a regional video super-resolution processing device, the device comprising:

[0055] A decoding unit, used to decode the video stream to obtain multiple video frames and a set of feature region sequences corresponding to each video frame;

[0056] A capture unit, configured to capture a corresponding feature picture from the current frame video picture according to a feature region sequence set corresponding to the current frame video picture, to obtain a feature picture set;

[0057] A super-resolution processing unit, used to perform super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing;

[0058] The fusion unit is used to perform fusion calculation based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing.

[0059] In a third aspect, an embodiment of the present application provides a device for processing video super-resolution by region, the device comprising: a processor, a memory, and a system bus;

[0060] The processor and the memory are connected via the system bus;

[0061] The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the regional video super-resolution processing method described in any one of the first aspects above.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the regional video super-resolution processing method described in any one of the first aspects above.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the regional video super-resolution processing method described in any one of the first aspects above.

[0064] Based on the above technical solution, this application has the following beneficial effects:

[0065] The embodiment of the present application discloses a method, device and storage medium for super-resolution processing of video by region. The method includes: decoding a video stream to obtain a plurality of video frames and a set of feature region sequences corresponding to each video frame; intercepting corresponding feature images from the current video frame according to the feature region sequence set corresponding to the current video frame to obtain a feature image set; performing super-resolution processing on the images in the feature image set to obtain a set of feature images after super-resolution processing; performing fusion calculation based on the feature image set after super-resolution processing and the current video frame to obtain the current video frame after super-resolution processing. It can be seen that in the embodiment of the present application, a feature region sequence set is obtained by decoding the video stream, so that the feature images in the original video image can be super-resolution processed according to the feature region sequence set, without the need to perform super-resolution processing on all the original video images, thereby greatly reducing the computational complexity of the client super-resolution processing while enhancing the video quality. Moreover, each frame of video image corresponds to a set of feature area sequences. As the video changes, the feature areas corresponding to the video images will dynamically increase or decrease, and the feature images will be dynamically captured for super-resolution processing, which increases the flexibility of super-resolution processing and avoids wasting client resources by performing super-resolution processing on video images with few or no feature areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying any creative work.

[0067] Figure 1 A flowchart of a method for processing video super-resolution by region disclosed in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a current frame video picture disclosed in an embodiment of the present application;

[0069] Figure 3 A flowchart of a method for obtaining a portrait explicit picture set, a subtitle picture set, and a logo picture set disclosed in an embodiment of the present application;

[0070] Figure 4 A flowchart of a method for obtaining a final set of portrait explicit regions disclosed in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of overlapping regions in a set of portrait explicit regions disclosed in an embodiment of the present application;

[0072] Figure 6 A flowchart of a method for obtaining a final subtitle area set disclosed in an embodiment of the present application;

[0073] Figure 7 A flowchart of a method for obtaining a final identification area set disclosed in an embodiment of the present application;

[0074] Figure 8 A flowchart of super-resolution processing of a feature image disclosed in an embodiment of the present application;

[0075] Fig. 9 A flowchart of fusing a feature image after super-resolution processing with an original video image disclosed in an embodiment of the present application;

[0076] Fig.10 A schematic diagram of the structure of a regional video super-resolution processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] The embodiment of the present application discloses a method, device and storage medium for super-resolution processing of video by region. The method includes: decoding a video stream to obtain a plurality of video frames and a set of feature region sequences corresponding to each video frame; intercepting corresponding feature images from the current video frame according to the feature region sequence set corresponding to the current video frame to obtain a feature image set; performing super-resolution processing on the images in the feature image set to obtain a set of feature images after super-resolution processing; performing fusion calculation based on the feature image set after super-resolution processing and the current video frame to obtain the current video frame after super-resolution processing. It can be seen that in the embodiment of the present application, a feature region sequence set is obtained by decoding the video stream, so that the feature images in the original video image can be super-resolution processed according to the feature region sequence set, without the need to perform super-resolution processing on all the original video images, thereby greatly reducing the computational complexity of the client super-resolution processing while enhancing the video quality. Moreover, each frame of video image corresponds to a set of feature area sequences. As the video changes, the feature areas corresponding to the video images will dynamically increase or decrease, and the feature images will be dynamically captured for super-resolution processing, which increases the flexibility of super-resolution processing and avoids wasting client resources by performing super-resolution processing on video images with few or no feature areas.

[0079] See also Figure 1 , a flowchart of a method for processing a video super-resolution by region disclosed in an embodiment of the present application, the method comprising:

[0080] S101, decoding a video stream to obtain multiple video frames and a feature region sequence set corresponding to each video frame;

[0081] In the embodiment of the present application, the feature region sequence set is generated when the server-side video is encoded and encoded into the video stream. The client player will receive the video stream sent by the server and then decode the video stream. Among them, the multi-frame video picture refers to continuous video picture data. The feature region sequence set may include a portrait explicit region set, a subtitle region set, a logo region set, etc. The logo region set may include a station logo region and a program logo logo region. There is no specific limitation and it can be set according to actual needs.

[0082] In the embodiment of the present application, the region can be expressed as (x, y, w, h), where x and y represent the coordinates of the upper left corner of the region in the original video image, w represents the width of the region, and h represents the height of the region. In this way, the unique region position can be determined in the original video image.

[0083] S102, intercepting corresponding feature pictures from the current frame video picture according to the feature region sequence set corresponding to the current frame video picture to obtain a feature picture set;

[0084] See also Figure 2 , is a schematic diagram of a current frame video image disclosed in an embodiment of the present application, wherein the current frame video image includes a portrait dominant area (30, 32, 22, 31), a subtitle area (12, 5, 28, 3), a station logo area (2, 36, 14, 4) and a program logo area (46, 7, 11, 4). In a possible implementation, the embodiment of the present application will intercept corresponding feature images from the current frame video image based on the portrait dominant area, the subtitle area and the two logo areas to obtain a feature image set including 4 feature images. It can be understood that the above is only an exemplary description and should not be construed as a limitation to the present application.

[0085] S103, performing super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing;

[0086] It should be noted that super-resolution processing is to reconstruct low-resolution images into high-resolution images, which can greatly improve the quality of the image.

[0087] S104, performing fusion calculation based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing.

[0088] It is understandable that after obtaining the current frame video picture after super-resolution processing, the next frame video picture will be processed. In the embodiment of the present application, after obtaining the current frame video picture after super-resolution processing, the current frame video picture after super-resolution processing can also replace the original current frame video picture and add it to the rendering queue for image rendering.

[0089] It can be seen that in the embodiment of the present application, a feature region sequence set is obtained by decoding a video stream, so that the feature image in the original video image can be super-resolution processed according to the feature region sequence set, without having to perform super-resolution processing on all the original video images, thus greatly reducing the computational complexity of the client super-resolution processing while enhancing the video quality. Moreover, each frame of the video image corresponds to a feature region sequence set, so that as the video changes, the feature region corresponding to the video image will dynamically increase or decrease, and the feature image will be dynamically intercepted for super-resolution processing, which increases the flexibility of super-resolution processing and avoids the waste of client resources caused by super-resolution processing of video images with few or no feature regions.

[0090] See also Figure 3 , a flowchart of a method for obtaining a portrait explicit picture set, a subtitle picture set, and an identification picture set disclosed in an embodiment of the present application. S102 in the region-based video super-resolution processing method provided in an embodiment of the present application may specifically include:

[0091] S1021, splitting a feature region sequence set corresponding to a current frame video picture to obtain a portrait explicit region set, a subtitle region set, and a logo region set corresponding to the current frame video picture;

[0092] The logo area set includes a station logo area and a program logo area.

[0093] S1022, processing mutually overlapping regions in the portrait dominant region set to obtain a final portrait dominant region set;

[0094] In the embodiment of the present application, considering that multiple portrait display areas in the same frame may overlap with each other, the overlapping areas only need to be processed once for super-resolution, so the overlapping areas will be processed. Specifically, the areas in the portrait display area set can be traversed to merge the two overlapping portrait display areas into one portrait display area.

[0095] S1023, processing the areas in the subtitle area set that overlap with the areas in the final portrait explicit area set to obtain a final subtitle area set;

[0096] In the embodiment of the present application, it is considered that the subtitle areas in the subtitle area set will not overlap with each other, but may overlap with the portrait explicit area, so it is necessary to process the area where the subtitle area overlaps with the final portrait explicit area. Specifically, a filtering algorithm can be used to filter out the area in the subtitle area that overlaps with the final portrait explicit area.

[0097] S1024, processing the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain a final marked area set;

[0098] In the embodiment of the present application, it is considered that the identification areas in the identification area set will not overlap with each other, but may overlap with the portrait explicit area, so it is necessary to process the area where the subtitle area overlaps with the final portrait explicit area. Specifically, a filtering algorithm can be used to filter out the area in the identification area that overlaps with the final portrait explicit area.

[0099] S1025, intercepting corresponding feature pictures from the current frame video picture according to the final portrait explicit region set to obtain a portrait explicit picture set;

[0100] S1026, extracting corresponding feature pictures from the current frame video picture according to the final subtitle area set to obtain a subtitle picture set;

[0101] S1027. According to the final identification area set, corresponding feature images are intercepted from the current frame video image to obtain an identification image set.

[0102] It can be seen that in the embodiment of the present application, different feature areas in the video image are distinguished, and different feature areas are processed separately, so that the final portrait explicit area set, the final subtitle area set and the final logo area set obtained are more accurate, so that the corresponding feature images can be accurately captured, avoiding super-resolution processing of video images with few or no feature areas and causing waste of client resources.

[0103] See also Figure 4 , a flow chart of a method for obtaining a final set of portrait explicit regions disclosed in an embodiment of the present application. S1022 in the region-based video super-resolution processing method provided in the embodiment of the application may specifically include:

[0104] S401, arranging the regions in the portrait dominant region set from large to small according to the region area, to obtain a first region set;

[0105] In the embodiment of the present application, before executing step S401, it can also be determined whether the portrait explicit region set is empty; when the portrait explicit region set is not empty, the regions in the portrait explicit region set are arranged from large to small according to the region area. When the portrait explicit region set is empty, for example, when the lens switching screen is all black, there is no need to process the portrait explicit region set of the current frame video picture.

[0106] S402, taking out a first region from the first region set, and putting it into a second region set; the first region is the region with the largest area in the first region set;

[0107] It should be noted that the second region set is an initially empty set.

[0108] S403, determining whether the current first region set is empty;

[0109] It should be noted that the current first region set refers to the first region set after the first region is taken out.

[0110] S404: If the current first region set is not empty, taking a second region from the current first region set; the second region is the region with the largest area in the current first region set;

[0111] S405, traversing the regions in the second region set, and determining whether the second region overlaps with the currently traversed third region;

[0112] It should be noted that in the embodiment of the present application, the regions in the second region set are traversed sequentially. For example: the second region set includes region 1 and region 2, then when traversing the regions in the second region set, region 1 will be traversed first to determine whether the second region overlaps with region 1 (i.e., the third region currently traversed), and then if S405 is returned to be executed, the regions in the second region set will be traversed to region 2 to determine whether the second region overlaps with region 2 (i.e., the third region currently traversed). It can be understood that the above is only an exemplary description and should not be construed as a limitation on the present application.

[0113] S406: If the second area overlaps with the third area, calculate a first overlap index;

[0114] If the second area does not overlap with the third area, determine whether the third area is the last area of ​​the second area set; if the third area is not the last area of ​​the second area set, return to step S405; if the third area is the last area of ​​the second area set, place the second area in the second area set, and return to step S403.

[0115] See also Figure 5 , which is a schematic diagram of overlapping areas in a portrait explicit area set disclosed in an embodiment of the present application. Figure 5 S1 and S2 are the dominant areas of the portrait, S1 and S2 overlap, the overlapping area is S3, and the calculation method of the first overlapping index is S3 / (S1+S2), S3 refers to the area of ​​the overlapping area S3, S1 refers to the area of ​​the dominant area S1, and S2 refers to the area of ​​the dominant area S2. That is, in a possible implementation, the first overlapping index = the area of ​​the overlapping area between the second area and the third area / (the area of ​​the second area+the area of ​​the third area).

[0116] S407, determining whether the first overlap index is greater than a first preset threshold;

[0117] It should be noted that the first preset threshold can be set according to actual conditions and is not specifically limited.

[0118] S408: If the first overlap index is greater than a first preset threshold, merge the second region with the third region to obtain a fourth region;

[0119] If the first overlap index is not greater than the first preset threshold, determine whether the third area is the last area of ​​the second area set; if the third area is not the last area of ​​the second area set, return to execute step S405; if the third area is the last area of ​​the second area set, place the second area in the second area set, and return to execute step S403.

[0120] See also Figure 5 , assuming Figure 5 If the overlap index of S1 and S2 is greater than the first preset threshold, the fourth region after fusion is Figure 5 S4 in.

[0121] S409, replacing the third area in the second area set with the fourth area, and returning to step S403;

[0122] S410: If the current first region set is empty, use the current second region set as a final portrait explicit region set.

[0123] For example: arrange the areas in the portrait explicit area set from large to small according to the area area to obtain the first area set {area 1, area 2, area 3}; take area 1 from the first area set and put it into the second area set which is initially empty to obtain the second area set {area 1}; determine whether the current first area set {area 2, area 3} is empty; determine that the current first area set {area 2, area 3} is not empty, then take area 2 from the current first area set; traverse the second area set {area 1} to determine whether area 2 overlaps with the currently traversed area 1; determine that area 2 overlaps with area 1, and calculate the first overlapping index between area 2 and area 1; determine whether the first overlapping index between area 2 and area 1 is greater than the first set threshold; determine that the first overlapping index between area 2 and area 1 is not greater than the first preset threshold, then determine whether area 1 is the last area of ​​the second area set {area 1}; determine that area 1 is the last area of ​​the second area set {area 1}, put area 2 into the second area set, and obtain the second area set {area 1, area 1}. 2}, return to execute to determine whether the current first area set {area 3} is empty; determine that the current first area set {area 3} is not empty, then take area 3 from the current first area set {area 3}; traverse the second area set {area 1, area 2}, and determine whether area 3 overlaps with the currently traversed area 1; determine that area 3 does not overlap with area 1, then return to execute traversal of the second area set {area 1, area 2}, and determine whether area 3 overlaps with the currently traversed area 2; determine that area 3 overlaps with area 2, and calculate the first overlapping index of area 3 and area 2; determine whether the overlapping index of area 3 and area 2 is greater than the first set threshold; determine that the first overlapping index of area 3 and area 2 is greater than the first set threshold, merge area 3 and area 2 to obtain area 4, use area 4 to replace area 2 in the second area set, obtain the second area set {area 1, area 4}, and return to execute to determine whether the first area set is empty; determine that the first area set is empty, and use the second area set {area 1, area 4} as the final portrait explicit area set. It should be understood that the above description is merely exemplary and should not be construed as limiting the present application.

[0124] It can be seen that in the embodiment of the present application, the regions in the portrait explicit region set are first arranged from large to small according to the region area, and then the portrait explicit regions are processed in sequence from large to small according to the region area size, which can reduce the complexity of calculation and ensure the effectiveness of subsequent fusion. Moreover, by calculating the overlap index and comparing the overlap index with the preset threshold, it can be accurately determined whether two overlapping portrait explicit regions need to be fused, so that an accurate final portrait explicit region set can be obtained, avoiding secondary super-resolution processing of larger overlapping regions, and further reducing the amount of calculation of the client.

[0125] See also Figure 6 , a flow chart of a method for obtaining a final subtitle area set disclosed in an embodiment of the present application. S1023 in the embodiment of the present application may specifically include:

[0126] S601, determining whether the current subtitle region set is empty;

[0127] In the embodiment of the present application, before executing step S601, the third region set is initialized to obtain an initially empty third region set. It is understandable that if after the first execution of determining whether the current subtitle region set is empty, it is determined that the current subtitle region set is empty, it means that there is no subtitle region in the current frame video picture, and the current third region set is empty at this time, then the empty current third region set is used as the final subtitle region set.

[0128] S602: If the current subtitle region set is not empty, taking a fifth region from the current subtitle region set; the fifth region is any region in the current subtitle region set;

[0129] S603, traversing the final set of portrait explicit regions, and determining whether the fifth region overlaps with the currently traversed sixth region;

[0130] S604: If the fifth area overlaps with the sixth area, calculate a second overlap index;

[0131] If the fifth area does not overlap with the sixth area, determine whether the sixth area is the last area of ​​the final portrait explicit area set; if the sixth area is the last area of ​​the final portrait explicit area set, put the fifth area into the third area set and return to execute S601; if the sixth area is not the last area of ​​the final portrait explicit area set, return to execute S603.

[0132] The second overlap index refers to the overlap index between the fifth region and the sixth region, and the calculation method of the second overlap index is similar to the calculation method of the first overlap index. In a possible implementation, the second overlap index = the area of ​​the overlap between the fifth region and the sixth region / (the area of ​​the fifth region + the area of ​​the sixth region).

[0133] S605: Determine whether the second overlap index is greater than a second preset threshold;

[0134] It should be noted that the second preset threshold can be set according to actual conditions and is not specifically limited.

[0135] S606: If the second overlapping index is greater than a second preset threshold, filter out the area in the fifth area that overlaps with the sixth area to obtain a filtered fifth area;

[0136] If the second overlapping index is not greater than the second preset threshold, determine whether the sixth area is the last area of ​​the final portrait explicit area set; if the sixth area is the last area of ​​the final portrait explicit area set, put the fifth area into the third area set, and return to execute S601; if the sixth area is not the last area of ​​the final portrait explicit area set, return to execute S603.

[0137] S607, putting the filtered fifth region into the third region set, and returning to execute S601;

[0138] S608: If the current subtitle area set is empty, use the current third area set as the final subtitle area set.

[0139] It can be seen that in the embodiment of the present application, by calculating the overlap index and comparing the overlap index with the preset threshold, it is possible to accurately determine whether the overlapping areas in the subtitle area set and the areas in the final portrait explicit area set need to be filtered. In this way, an accurate final subtitle area set can be obtained, avoiding secondary super-resolution processing of larger overlapping areas, and further reducing the amount of calculation on the client.

[0140] See also Figure 7 , a flowchart of a method for obtaining a final identification area set disclosed in an embodiment of the present application. S1024 in the embodiment of the present application may specifically include:

[0141] S701, determining whether the current identification area set is empty;

[0142] In the embodiment of the present application, before executing step S701, the fourth region set is initialized to obtain an initially empty fourth region set. It is understandable that if after the first execution of determining whether the current subtitle region set is empty, it is determined that the current subtitle region set is empty, it means that there is no identification region in the current frame video picture, and the current fourth region set is empty at this time, then the empty current fourth region set is taken as the final subtitle region set.

[0143] S702: If the current identification area set is not empty, taking out a seventh area from the current identification area set; the seventh area is any area in the current identification area set;

[0144] S703, traversing the final set of portrait explicit regions, and determining whether the seventh region overlaps with the currently traversed eighth region;

[0145] S704: If the seventh area overlaps with the eighth area, a third overlapping index is calculated;

[0146] If the seventh area does not overlap with the eighth area, determine whether the eighth area is the last area of ​​the final portrait explicit area set; if the eighth area is the last area of ​​the final portrait explicit area set, put the seventh area into the fourth area set and return to execute S701; if the eighth area is not the last area of ​​the final portrait explicit area set, return to execute S703.

[0147] The third overlap index refers to the overlap index between the seventh region and the eighth region. The calculation method of the third overlap index is similar to the calculation method of the first overlap index and the second overlap index. In a possible implementation, the third overlap index = the area of ​​the overlap between the seventh region and the eighth region / (the area of ​​the seventh region + the area of ​​the eighth region)

[0148] S705, determining whether the third overlap index is greater than a third preset threshold;

[0149] It should be noted that the third preset threshold can be set according to actual conditions and is not specifically limited.

[0150] S706: If the third overlapping index is greater than a third preset threshold, filter out the area in the seventh area that overlaps with the eighth area to obtain a filtered seventh area;

[0151] If the third overlapping index is not greater than the third preset threshold, determine whether the eighth area is the last area of ​​the final portrait explicit area set; if the eighth area is the last area of ​​the final portrait explicit area set, put the seventh area into the fourth area set, and return to execute S701; if the eighth area is not the last area of ​​the final portrait explicit area set, return to execute S703.

[0152] S707, putting the filtered seventh region into the fourth region set, and returning to execute S701;

[0153] S708: If the current marked area set is empty, use the current fourth area set as the final marked area set.

[0154] It can be seen that in the embodiment of the present application, by calculating the overlap index and comparing the overlap index with the preset threshold, it is possible to accurately determine whether the overlapping areas in the identification area set and the areas in the final portrait explicit area set need to be filtered. In this way, an accurate final subtitle area set can be obtained, avoiding secondary super-resolution processing of larger overlapping areas, and further reducing the amount of calculation on the client.

[0155] See also Figure 8 , a flowchart of super-resolution processing of a feature image disclosed in an embodiment of the present application. S103 in the embodiment of the present application specifically includes:

[0156] S801, performing super-resolution processing on the portrait explicit picture set by using the portrait explicit super-resolution model to obtain a super-resolution processed portrait explicit picture set;

[0157] S802, performing super-resolution processing on the subtitle picture set by using the subtitle super-resolution model to obtain a subtitle picture set after super-resolution processing;

[0158] S803: Perform super-resolution processing on the logo image set by using the logo super-resolution model to obtain a logo image set after super-resolution processing.

[0159] In the embodiment of the present application, the explicit portrait training data, subtitle training data and logo training data are collected and screened respectively, such as 300,000 explicit portrait pictures, 100,000 subtitle pictures, 50,000 station logo pictures, and 50,000 program logo pictures. There is no specific limitation and the setting can be selected according to actual needs. Then three versions of models are designed for the three categories respectively, among which the parameters of the explicit portrait super-resolution model are slightly larger, and the subtitle and station logo use the same model with fewer parameters; then the collected training is used to train the corresponding models respectively, and finally three super-resolution models are obtained.

[0160] It can be seen that in the embodiment of the present application, the super-resolution model is designed according to the type of feature area, and the super-resolution model is trained specifically for different types of feature areas, thereby improving the overall super-resolution effect of the video. For simple scenes such as subtitles and labels, the complexity of the model algorithm can be reduced. While ensuring the super-resolution effect, the super-resolution calculation time can be greatly reduced, and the cost and consumption of the calculation can also be reduced.

[0161] Fig. 9 This is a flow chart of fusing a feature image after super-resolution processing with an original video image disclosed in an embodiment of the present application. S104 in the embodiment of the present application specifically includes:

[0162] S901, define and initialize a blank target picture; wherein the target picture is N times the current frame picture, and N is greater than 1;

[0163] Wherein, N can be 1.5, 2, etc., and is not specifically limited and can be set according to actual needs. In a possible implementation, a specific value of N can be obtained according to the selected super-resolution model.

[0164] S902, calculating a super-resolutioned portrait explicit region set according to the final portrait explicit region set, traversing the super-resolutioned portrait explicit picture set, and filling pictures in the super-resolutioned portrait explicit picture set into the target picture according to the super-resolutioned portrait explicit region set;

[0165] It should be noted that the super-resolution portrait explicit region set and the target image have the same scaling factor. For example, the target image is twice the current frame image. Assuming that region 1 in the final portrait explicit region set is (2,2,1,1), then based on region 1 (2,2,1,1), the super-resolution region 1 in the super-resolution portrait explicit region set is calculated to be (4,4,2,2). It will be understood that the above is merely an exemplary description and should not be construed as a limitation on the present application.

[0166] S903, calculating a super-resolution subtitle region set according to the final subtitle region set, traversing the subtitle picture set after super-resolution processing, and filling pictures in the subtitle picture set after super-resolution processing into the target picture according to the super-resolution subtitle region set;

[0167] S904, calculating a super-resolution marked area set according to the final marked area set, traversing the marked picture set after super-resolution processing, and filling pictures in the marked picture set after super-resolution processing into the target picture according to the super-resolution marked area set;

[0168] S905 , traverse the unfilled areas in the target image, obtain the corresponding image to be filled from the current frame image, enlarge the image to be filled and fill it into the target image, and obtain the current frame video image after super-resolution processing.

[0169] It should be noted that bilinear interpolation can be used to enlarge the image to be filled in the implementation of this application. Bilinear interpolation is used to enlarge the image. Bilinear interpolation uses existing pixels to calculate other pixels to achieve pixel expansion, that is, bilinear interpolation is a calculation at the pixel level.

[0170] It can be seen that in the embodiment of the present application, the super-resolution subtitle area set is calculated based on the final subtitle area set, and the feature image after super-resolution processing can be accurately filled into the target image based on the super-resolution subtitle area set to accurately obtain the super-resolution processed video image.

[0171] Fig.10 This is a schematic diagram of the structure of a regional video super-resolution processing device disclosed in an embodiment of the present application, the device comprising:

[0172] A decoding unit 101 is used to decode the video stream to obtain multiple video frames and a set of feature region sequences corresponding to each video frame;

[0173] The interception unit 102 is used to intercept the corresponding feature picture from the current frame video picture according to the feature region sequence set corresponding to the current frame video picture to obtain a feature picture set;

[0174] A super-resolution processing unit 103 is used to perform super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing;

[0175] The fusion unit 104 is used to perform fusion calculation based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing.

[0176] The embodiment of the present application discloses a device for processing video super-resolution by region, wherein a decoding unit is used to decode a video stream to obtain a plurality of video frames and a set of feature region sequences corresponding to each video frame; a capture unit is used to capture corresponding feature images from the current video frame according to the feature region sequence set corresponding to the current video frame to obtain a feature image set; a super-resolution processing unit is used to perform super-resolution processing on images in the feature image set to obtain a set of feature images after super-resolution processing; a fusion unit is used to perform fusion calculation based on the feature image set after super-resolution processing and the current video frame to obtain the current video frame after super-resolution processing. It can be seen that in the embodiment of the present application, a feature region sequence set is obtained by decoding the video stream, so that super-resolution processing can be performed on the feature regions in the original video image according to the feature region sequence set, without the need to perform super-resolution processing on all the original video images, thereby greatly reducing the amount of computation for super-resolution processing on the client while enhancing the video quality. Moreover, each frame of video image corresponds to a set of feature area sequences. As the video changes, the feature areas corresponding to the video images will dynamically increase or decrease, and the feature images will be dynamically captured for super-resolution processing, which increases the flexibility of super-resolution processing and avoids wasting client resources by performing super-resolution processing on video images with few or no feature areas.

[0177] In a possible implementation, the regional video super-resolution processing device provided in the embodiment of the present application further includes:

[0178] The splitting unit 105 is used to split the feature region sequence set corresponding to the current frame video picture to obtain a portrait explicit region set, a subtitle region set and a logo region set corresponding to the current frame video picture;

[0179] The interception unit 102 is specifically configured to intercept corresponding feature images from the current frame video image according to the portrait explicit area set, the subtitle area set and the identification area set to obtain a feature image set.

[0180] In a possible implementation, the regional video super-resolution processing device provided in the embodiment of the present application further includes:

[0181] A first processing unit 106 is used to process the overlapping areas in the portrait dominant area set to obtain a final portrait dominant area set;

[0182] A second processing unit 107 is used to process the area in the subtitle area set that overlaps with the area in the final portrait explicit area set to obtain a final subtitle area set;

[0183] A third processing unit 108 is used to process the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain a final marked area set;

[0184] The interception unit 102 is specifically used for:

[0185] According to the final portrait explicit region set, corresponding feature images are intercepted from the current frame video image to obtain a portrait explicit image set;

[0186] According to the final subtitle area set, corresponding feature pictures are intercepted from the current frame video picture to obtain a subtitle picture set;

[0187] According to the final identification area set, corresponding feature pictures are intercepted from the current frame video picture to obtain an identification picture set.

[0188] In a possible implementation, the first processing unit 106 in the regional video super-resolution processing device provided in the embodiment of the present application is specifically configured to:

[0189] Arrange the regions in the portrait explicit region set from large to small according to the region area to obtain a first region set;

[0190] Taking out a first region from the first region set and putting it into the second region set; the first region is the region with the largest area in the first region set;

[0191] Determine whether the current first region set is empty;

[0192] If the current first region set is not empty, taking a second region from the current first region set; the second region is the region with the largest area in the current first region set;

[0193] Traversing the regions in the second region set, and determining whether the second region overlaps with the currently traversed third region;

[0194] If the second area overlaps with the third area, a first overlapping index is calculated;

[0195] Determining whether the first overlap index is greater than a first preset threshold;

[0196] If the first overlap index is greater than a first preset threshold, merging the second region with the third region to obtain a fourth region;

[0197] The third area in the second area set is replaced by the fourth area, and the following steps are returned to be executed: determining whether the current first area set is empty;

[0198] If the current first region set is empty, the current second region set is used as the final portrait explicit region set.

[0199] In a possible implementation, the first processing unit 106 in the regional video super-resolution processing device provided in the embodiment of the present application is further configured to:

[0200] If the second area does not overlap with the third area, determining whether the third area is the last area of ​​the second area set;

[0201] If the third area is not the last area of ​​the second area set, return to perform the following steps: traverse the areas in the second area set, and determine whether the second area overlaps with the currently traversed third area;

[0202] If the third area is the last area of ​​the second area set, the second area is placed in the second area set, and the following steps are returned to: determining whether the current first area set is empty;

[0203] If the first overlap index is not greater than a first preset threshold, determining whether the third area is the last area of ​​the second area set;

[0204] If the third area is not the last area of ​​the second area set, return to perform the following steps: traverse the areas in the second area set, and determine whether the second area overlaps with the currently traversed third area;

[0205] If the third area is the last area of ​​the second area set, the second area is placed in the second area set, and the process returns to execute the following steps: determining whether the current first area set is empty.

[0206] In a possible implementation, the second processing unit 107 in the regional video super-resolution processing device provided in the embodiment of the present application is specifically used for:

[0207] Determine whether the current subtitle area set is empty;

[0208] If the current subtitle area set is not empty, taking a fifth area from the current subtitle area set; the fifth area is any area in the current subtitle area set;

[0209] Traversing the final set of portrait explicit regions, and determining whether the fifth region overlaps with the currently traversed sixth region;

[0210] If the fifth area overlaps with the sixth area, a second overlapping index is calculated;

[0211] Determining whether the second overlap index is greater than a second preset threshold;

[0212] If the second overlapping index is greater than a second preset threshold, filtering out the area in the fifth area that overlaps with the sixth area to obtain a filtered fifth area;

[0213] Put the filtered fifth area into the third area set, and return to perform the following steps: determine whether the current subtitle area set is empty;

[0214] If the current subtitle area set is empty, the current third area set is used as the final subtitle area set.

[0215] In a possible implementation, the second processing unit 107 in the regional video super-resolution processing device provided in the embodiment of the present application is further configured to:

[0216] If the fifth area does not overlap with the sixth area, determining whether the sixth area is the last area of ​​the final portrait explicit area set;

[0217] If the sixth region is the last region of the final portrait explicit region set, the fifth region is placed in the third region set, and the following steps are returned to: determining whether the current subtitle region set is empty;

[0218] If the sixth region is not the last region of the final set of portrait explicit regions, return to perform the following steps: traverse the final set of portrait explicit regions to determine whether the fifth region overlaps with the currently traversed sixth region;

[0219] If the second overlapping index is not greater than a second preset threshold, determining whether the sixth region is the last region of the final portrait dominant region set;

[0220] If the sixth region is the last region of the final portrait explicit region set, the fifth region is placed in the third region set, and the following steps are returned to: determining whether the current subtitle region set is empty;

[0221] If the sixth region is not the last region of the final portrait explicit region set, the process returns to execute the following steps: traverse the final portrait explicit region set to determine whether the fifth region overlaps with the currently traversed sixth region.

[0222] In a possible implementation, the third processing unit 108 in the regional video super-resolution processing device provided in the embodiment of the present application is specifically used for:

[0223] Determine whether the current identification area set is empty;

[0224] If the current identification area set is not empty, taking out a seventh area from the current identification area set; the seventh area is any area in the current identification area set;

[0225] Traversing the final set of portrait explicit regions, and determining whether the seventh region overlaps with the currently traversed eighth region;

[0226] If the seventh area overlaps with the eighth area, a third overlapping index is calculated;

[0227] Determining whether the third overlap index is greater than a third preset threshold;

[0228] If the third overlapping index is greater than a third preset threshold, filtering out the area in the seventh area that overlaps with the eighth area to obtain a filtered seventh area;

[0229] Put the filtered seventh area into the fourth area set, and return to execute the following steps: determine whether the current marked area set is empty;

[0230] If the current marked area set is empty, the current fourth area set is used as the final marked area set.

[0231] In a possible implementation, the third processing unit 108 in the regional video super-resolution processing device provided in the embodiment of the present application is further configured to:

[0232] If the seventh area does not overlap with the eighth area, determining whether the eighth area is the last area of ​​the final portrait explicit area set;

[0233] If the eighth region is the last region of the final portrait explicit region set, the seventh region is placed in the fourth region set, and the following steps are returned to: determining whether the current marked region set is empty;

[0234] If the eighth region is not the last region of the final set of portrait explicit regions, the following steps are returned to be executed: the final set of portrait explicit regions is traversed to determine whether the seventh region overlaps with the currently traversed eighth region;

[0235] If the third overlap index is not greater than a third preset threshold, determining whether the eighth region is the last region of the final portrait dominant region set;

[0236] If the eighth region is the last region of the final portrait explicit region set, the seventh region is placed in the fourth region set, and the following steps are returned to: determining whether the current marked region set is empty;

[0237] If the eighth region is not the last region of the final set of portrait explicit regions, the process returns to the following steps: traverse the final set of portrait explicit regions to determine whether the seventh region overlaps with the currently traversed eighth region.

[0238] In a possible implementation, the super-resolution processing unit 103 in the region-based video super-resolution processing device provided in the embodiment of the present application is specifically used for:

[0239] Performing super-resolution processing on the portrait explicit picture set by using the portrait explicit super-resolution model to obtain a super-resolution processed portrait explicit picture set;

[0240] Using the subtitle super-resolution model to perform super-resolution processing on the subtitle picture set, to obtain a subtitle picture set after super-resolution processing;

[0241] The logo super-resolution model is used to perform super-resolution processing on the logo picture set to obtain a logo picture set after super-resolution processing.

[0242] In a possible implementation, the fusion unit 104 in the regional video super-resolution processing device provided in the embodiment of the present application specifically includes:

[0243] A definition unit, used to define and initialize a blank target picture; wherein the target picture is N times the current frame picture, and N is greater than 1;

[0244] A filling unit, configured to calculate a super-resolution explicit portrait region set according to the final portrait explicit region set, traverse the super-resolution explicit portrait picture set, and fill pictures in the super-resolution explicit portrait picture set into the target picture according to the super-resolution explicit portrait region set;

[0245] The filling unit is further used to calculate a super-resolution subtitle region set according to the final subtitle region set, traverse the subtitle picture set after super-resolution processing, and fill pictures in the subtitle picture set after super-resolution processing into the target picture according to the super-resolution subtitle region set;

[0246] The filling unit is further used to calculate a super-resolution marked area set according to the final marked area set, traverse the marked picture set after super-resolution processing, and fill the pictures in the marked picture set after super-resolution processing into the target picture according to the super-resolution marked area set;

[0247] The filling unit is also used to traverse the unfilled area in the target image, obtain the corresponding image to be filled from the current frame image, enlarge the image to be filled and fill it into the target image, so as to obtain the current frame video image after super-resolution processing.

[0248] Furthermore, the embodiment of the present application also provides a regional video super-resolution processing device, including: a processor, a memory, and a system bus;

[0249] The processor and the memory are connected via the system bus;

[0250] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned sub-region video super-resolution processing method.

[0251] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned regional video super-resolution processing method.

[0252] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned region-based video super-resolution processing method.

[0253] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0254] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0255] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0256] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing video super-resolution by region, characterized in that: The method comprises: Decode the video stream to obtain multiple video frames and a set of feature region sequences corresponding to each video frame; According to the feature region sequence set corresponding to the current frame video picture, a corresponding feature picture is intercepted from the current frame video picture to obtain a feature picture set; Performing super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing; Perform fusion calculation based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing; The step of extracting corresponding feature images from the current frame video picture according to the feature region sequence set corresponding to the current frame video picture to obtain the feature image set includes: Splitting the feature region sequence set corresponding to the current frame video picture to obtain a portrait explicit region set, a subtitle region set, and a logo region set corresponding to the current frame video picture; Processing the portrait explicit region set, the subtitle region set and the logo region set respectively to obtain a final portrait explicit region set, a final subtitle region set and a final logo region set; According to the final portrait explicit area set, the final subtitle area set and the final logo area set, corresponding feature images are intercepted from the current frame video image to obtain a feature image set; The final set of portrait explicit regions is obtained, including: Arrange the regions in the portrait explicit region set from large to small according to the region area to obtain a first region set; Taking out a first region from the first region set and putting it into the second region set; the first region is the region with the largest area in the first region set; Determine whether the current first region set is empty; If the current first region set is not empty, taking a second region from the current first region set; the second region is the region with the largest area in the current first region set; Traversing the regions in the second region set, and determining whether the second region overlaps with the currently traversed third region; If the second area overlaps with the third area, a first overlapping index is calculated; Determining whether the first overlap index is greater than a first preset threshold; If the first overlap index is greater than a first preset threshold, merging the second region with the third region to obtain a fourth region; The third area in the second area set is replaced by the fourth area, and the following steps are returned to be executed: determining whether the current first area set is empty; If the current first region set is empty, the current second region set is used as the final portrait explicit region set.

2. The method according to claim 1, characterized in that The portrait explicit region set, the subtitle region set and the logo region set are processed respectively to obtain a final portrait explicit region set, a final subtitle region set and a final logo region set, including: Processing the overlapping areas in the portrait explicit area set to obtain a final portrait explicit area set; Processing the areas in the subtitle area set that overlap with the areas in the final portrait explicit area set to obtain a final subtitle area set; Processing the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain a final marked area set; Then, according to the final portrait explicit area set, the final subtitle area set and the final logo area set, corresponding feature pictures are intercepted from the current frame video picture to obtain a feature picture set, including: According to the final portrait explicit region set, corresponding feature images are intercepted from the current frame video image to obtain a portrait explicit image set; According to the final subtitle region set, corresponding feature images are intercepted from the current frame video image to obtain a subtitle image set; According to the final identification area set, corresponding feature pictures are intercepted from the current frame video picture to obtain an identification picture set.

3. The method according to claim 2, characterized in that The step of processing the overlapping areas in the subtitle area set and the areas in the final portrait explicit area set to obtain the final subtitle area set includes: Determine whether the current subtitle area set is empty; If the current subtitle area set is not empty, taking a fifth area from the current subtitle area set; the fifth area is any area in the current subtitle area set; Traversing the final set of portrait explicit regions, and determining whether the fifth region overlaps with the currently traversed sixth region; If the fifth area overlaps with the sixth area, a second overlapping index is calculated; Determining whether the second overlap index is greater than a second preset threshold; If the second overlapping index is greater than a second preset threshold, filtering out the area in the fifth area that overlaps with the sixth area to obtain a filtered fifth area; Put the filtered fifth area into the third area set, and return to perform the following steps: determine whether the current subtitle area set is empty; If the current subtitle area set is empty, the current third area set is used as the final subtitle area set.

4. The method according to claim 2, characterized in that: The step of processing the areas in the marked area set that overlap with the areas in the final portrait explicit area set to obtain the final marked area set includes: Determine whether the current identification area set is empty; If the current identification area set is not empty, taking out a seventh area from the current identification area set; the seventh area is any area in the current identification area set; Traversing the final set of portrait explicit regions, and determining whether the seventh region overlaps with the currently traversed eighth region; If the seventh area overlaps with the eighth area, a third overlapping index is calculated; Determining whether the third overlap index is greater than a third preset threshold; If the third overlapping index is greater than a third preset threshold, filtering out the area in the seventh area that overlaps with the eighth area to obtain a filtered seventh area; Put the filtered seventh area into the fourth area set, and return to execute the following steps: determine whether the current marked area set is empty; If the current marked area set is empty, the current fourth area set is used as the final marked area set.

5. The method according to claim 2, characterized in that: The step of performing super-resolution processing on the feature picture set to obtain a feature picture set after super-resolution processing includes: Performing super-resolution processing on the portrait explicit picture set by using a portrait explicit super-resolution model to obtain a super-resolution processed portrait explicit picture set; Performing super-resolution processing on the subtitle picture set by using a subtitle super-resolution model to obtain a subtitle picture set after super-resolution processing; The logo image set is subjected to super-resolution processing using a logo super-resolution model to obtain a logo image set after super-resolution processing.

6. A regional video super-resolution processing device, characterized in that: The device comprises: A decoding unit, used to decode the video stream to obtain multiple video frames and a set of feature region sequences corresponding to each video frame; A capture unit, configured to capture a corresponding feature picture from the current frame video picture according to a feature region sequence set corresponding to the current frame video picture, to obtain a feature picture set; A super-resolution processing unit, used to perform super-resolution processing on the pictures in the feature picture set to obtain a feature picture set after super-resolution processing; A fusion unit, configured to perform fusion calculation based on the feature picture set after super-resolution processing and the current frame video picture to obtain the current frame video picture after super-resolution processing; The interception unit intercepts a corresponding feature picture from the current frame video picture according to a feature region sequence set corresponding to the current frame video picture to obtain a feature picture set, including: Splitting the feature region sequence set corresponding to the current frame video picture to obtain a portrait explicit region set, a subtitle region set, and a logo region set corresponding to the current frame video picture; Processing the portrait explicit region set, the subtitle region set and the logo region set respectively to obtain a final portrait explicit region set, a final subtitle region set and a final logo region set; According to the final portrait explicit area set, the final subtitle area set and the final logo area set, corresponding feature images are intercepted from the current frame video image to obtain a feature image set; Wherein, obtaining the final portrait explicit region set includes: arranging the regions in the portrait explicit region set from large to small according to the region area to obtain a first region set; taking out a first region from the first region set and putting it into a second region set; the first region is the region with the largest area in the first region set; judging whether the current first region set is empty; if the current first region set is not empty, taking out a second region from the current first region set; the second region is the region with the largest area in the current first region set; traversing the regions in the second region set to judge whether the second region overlaps with the currently traversed third region; if the second region overlaps with the third region, calculating a first overlap index; judging whether the first overlap index is greater than a first preset threshold; if the first overlap index is greater than the first preset threshold, merging the second region with the third region to obtain a fourth region; replacing the third region in the second region set with the fourth region, and returning to perform the following steps: judging whether the current first region set is empty; if the current first region set is empty, taking the current second region set as the final portrait explicit region set.

7. A regional video super-resolution processing device, characterized in that: The device comprises: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the regional video super-resolution processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method for processing sub-regional video super-resolution according to any one of claims 1 to 5.

Images