Image quality enhancement model optimization method, video quality enhancement method, device, equipment and medium

Abstract

The present invention relates to the field of image processing technology, and provides a method for optimizing a picture quality enhancement model, a method for enhancing video quality, and related devices. The method comprises: first, based on each source stream video, obtaining an unenhanced transcoded video corresponding to the source stream video and a plurality of video pairs with different enhancement degrees, wherein each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video; then, based on the source stream video and its corresponding unenhanced transcoded video, obtaining an unenhanced image sample group, and based on the plurality of video pairs corresponding to the source stream video, obtaining a plurality of enhanced image sample groups with different enhancement degrees; finally, based on the unenhanced image sample group and all enhanced image sample groups of each source stream video, optimizing a pre-trained end-side picture quality enhancement model to obtain an optimized end-side picture quality enhancement model. This improves the picture quality enhancement capability of the end-side picture quality enhancement model and enhances the video quality enhancement effect.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for optimizing an image quality enhancement model, a method for enhancing video quality, and related devices. Background Art

[0002] In live broadcast scenarios, the source stream video of the anchor's live broadcast is usually first enhanced by the cloud-based image quality enhancement model on the server, and then the enhanced video is transcoded into videos of different clarity using different bit rates. Then, based on the clarity selected by the viewer, the corresponding video is sent to the viewer's client, and the end-side image quality enhancement model on the client is used to enhance the image quality of the received video. That is, the live video seen by the audience has actually undergone two image quality enhancements.

[0003] For major streamers with high-quality source video, using two-step image enhancement can ensure the quality of their live broadcasts. However, for smaller streamers, due to the poor quality of their source video, two-step enhancement often results in distortion in the live broadcast, such as excessive smoothing or sharpening of faces. Because the performance of the on-device image enhancement model depends on the client's hardware performance, it can be difficult for the on-device image enhancement model to distinguish between enhanced and unenhanced images. This can affect its effectiveness on enhanced but low-quality images output by the cloud-based image enhancement model. Therefore, optimizing the on-device image enhancement model is crucial. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a method for optimizing an image quality enhancement model, a method for enhancing video quality, and related devices.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for optimizing an image quality enhancement model, the method comprising:

[0007] For each source stream video, obtaining a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video;

[0008] For each source stream video, obtaining an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtaining multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video;

[0009] Based on the unenhanced image sample group and all enhanced image sample groups of each source stream video, the pre-trained end-side image quality enhancement model is optimized to obtain an optimized end-side image quality enhancement model.

[0010] In an optional embodiment, each enhancement level has a corresponding enhancement coefficient; the step of obtaining a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video includes:

[0011] Transcoding the source stream video according to a preset bit rate to obtain an unenhanced transcoded video corresponding to the source stream video;

[0012] Using a preset cloud-based image quality enhancement model, the source stream video is enhanced according to an enhancement coefficient corresponding to a maximum enhancement degree, to obtain an enhanced, non-transcoded video corresponding to the maximum enhancement degree;

[0013] Obtaining enhanced non-transcoded videos corresponding to each other enhancement level based on the enhancement coefficient corresponding to each other enhancement level, the source stream video, and the enhanced non-transcoded video of the maximum enhancement level;

[0014] Each enhanced non-transcoded video is transcoded according to a preset bit rate to obtain each enhanced transcoded video, and the enhanced non-transcoded video and the enhanced transcoded video with the same enhancement degree are combined into a video pair to obtain multiple video pairs with different enhancement degrees corresponding to the source stream video.

[0015] In an optional embodiment, the step of obtaining an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtaining a plurality of enhanced image sample groups with different enhancement degrees based on a plurality of video pairs corresponding to the source stream video, includes:

[0016] Randomly obtain a plurality of original images from the source stream video, and obtain, based on each of the original images, each corresponding unenhanced image from the unenhanced transcoded video corresponding to the source stream video;

[0017] Pairing each of the original images with a corresponding unenhanced image to obtain an unenhanced image sample group including a plurality of unenhanced image sample pairs, thereby obtaining an unenhanced image sample group of the source stream video;

[0018] For each video pair corresponding to the source stream video, obtaining, according to each original image, each corresponding first enhanced image from the enhanced non-transcoded video of the video pair, and each corresponding second enhanced image from the enhanced transcoded video of the video pair;

[0019] A first enhanced image and a second enhanced image corresponding to the same original image are paired to obtain an enhanced image sample group including multiple enhanced image sample pairs, and an enhanced image sample group for each video pair corresponding to the source stream video is obtained, where the multiple enhanced image sample groups of the source stream video have different degrees of enhancement.

[0020] In an optional embodiment, the step of optimizing the pre-trained end-side image quality enhancement model based on the unenhanced image sample group and all enhanced image sample groups of each source stream video to obtain the optimized end-side image quality enhancement model includes:

[0021] Obtain all unenhanced image sample pairs in the unenhanced image sample group of each source stream video to obtain an unenhanced image sample set;

[0022] Based on all enhanced image sample pairs in all enhanced image sample groups of each source stream video, a plurality of enhanced image sample sets are obtained; wherein the enhanced image sample pairs in each enhanced image sample set have the same enhancement degree;

[0023] According to the enhancement degree from small to large, all enhanced image sample sets are sorted to obtain the target sequence;

[0024] Optimizing a pre-trained end-side image quality enhancement model using the unenhanced image sample set to obtain an initial end-side image quality enhancement model;

[0025] According to the target sequence, the initial end-side image quality enhancement model is optimized using each enhanced image sample set in turn to obtain an optimized end-side image quality enhancement model.

[0026] In an optional embodiment, the step of optimizing the initial end-side image quality enhancement model using each enhanced image sample set in sequence according to the target sequence to obtain an optimized end-side image quality enhancement model includes:

[0027] Using the unenhanced image sample set as an old image sample set, and using the first enhanced image sample set in the target sequence as a new image sample set;

[0028] The first ratio value in the preset ratio sequence is used as the current ratio value; the ratio values ​​in the preset ratio sequence are arranged in ascending order;

[0029] Mixing the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set;

[0030] If the current scale value is not the last scale value in the preset scale sequence, then after taking the scale value following the current scale value as the next current scale value, re-performing the steps of mixing the old image sample set and the new image sample set according to the current scale value to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set;

[0031] If the current scale value is the last scale value in the preset scale sequence and the new image sample set is not the last enhanced image sample set in the target sequence, then after using the new image sample set as the next old image sample set and using the next image sample set after the new image sample set as the next new image sample set, re-performing the step of using the first scale value in the preset scale sequence as the current scale value;

[0032] If the current ratio value is the last ratio value in the preset ratio sequence and the new image sample set is the last enhanced image sample set in the target sequence, an optimized end-side image quality enhancement model is obtained; wherein, when the current ratio value is the last ratio value in the preset ratio sequence, the mixed image sample set is the new image sample set.

[0033] In an optional embodiment, the old image sample set includes a plurality of old image sample pairs, and the new image sample set includes a plurality of new image sample pairs; and the step of mixing the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set includes:

[0034] Replacing a plurality of old image sample pairs in the old image sample set with new image sample pairs in the new image sample set to obtain a replaced image sample set, wherein the proportion of the new image sample pairs in the replaced image sample set is the current proportion value;

[0035] According to the current ratio value, a plurality of old image sample pairs are obtained from the replaced image sample set to obtain each target old image sample pair;

[0036] For each of the target old image sample pairs, a target new image sample pair that matches the target old image sample pair is obtained in the new image sample set, and image regions of the target old image sample pair are replaced according to the target new image sample pair to obtain each replaced image sample pair, thereby obtaining the mixed image sample set; wherein the proportion of the replaced region in each image sample of the replaced image sample pair is the current ratio value.

[0037] In a second aspect, the present invention provides a method for enhancing video quality, which is applied to a system including a server and a client, and the method for enhancing video quality comprises:

[0038] The server uses the pre-stored cloud-based image quality enhancement model to enhance the image quality of the target source stream video according to the current enhancement level to obtain the enhanced video;

[0039] The server transcodes the enhanced video based on the bitrate corresponding to the target definition selected by the user, obtains the transcoded video, and sends it to the client;

[0040] The client obtains a target end-side image quality enhancement model that matches the current enhancement level and target clarity from multiple pre-stored end-side image quality enhancement models, and uses the target end-side image quality enhancement model to enhance the image quality of the transcoded video to obtain the target live video; wherein, each end-side image quality enhancement model is obtained according to the image quality enhancement model optimization method described in any one of the aforementioned embodiments.

[0041] In a third aspect, the present invention provides an optimization device for an image quality enhancement model, the device comprising:

[0042] A video acquisition module is configured to obtain, for each source stream video, a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video;

[0043] a sample acquisition module configured to obtain, for each source stream video, an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtain multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video;

[0044] The model optimization module is used to optimize the pre-trained end-side image quality enhancement model based on the unenhanced image sample group and all enhanced image sample groups of each source stream video to obtain an optimized end-side image quality enhancement model.

[0045] In a fourth aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the image quality enhancement model optimization method described in any one of the aforementioned embodiments is implemented.

[0046] In a fifth aspect, the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image quality enhancement model optimization method described in any one of the aforementioned embodiments is implemented.

[0047] The present invention provides a method for optimizing a picture quality enhancement model, a method for enhancing video picture quality, and related devices. The method comprises: first, based on each source stream video, obtaining an unenhanced transcoded video corresponding to the source stream video and a plurality of video pairs with different enhancement degrees, each video pair including an enhanced unenhanced video and an enhanced transcoded video; then, based on the source stream video and its corresponding unenhanced transcoded video, obtaining an unenhanced image sample group, and based on the plurality of video pairs corresponding to the source stream video, obtaining a plurality of enhanced image sample groups with different enhancement degrees; finally, based on the unenhanced image sample group and all enhanced image sample groups of each source stream video, optimizing a pre-trained end-side picture quality enhancement model to obtain an optimized end-side picture quality enhancement model. By using unenhanced image samples and image samples with different enhancement degrees to optimize the end-side picture quality enhancement model, the end-side picture quality enhancement model has the ability to enhance images that have been enhanced but have low picture quality, thereby improving the picture quality enhancement capability of the end-side picture quality enhancement model, avoiding over-enhancement or under-enhancement of video picture quality, and improving the video quality enhancement effect and the audience's viewing experience.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A schematic diagram of a system provided by an embodiment of the present invention is shown;

[0051] Figure 2 A block diagram of an electronic device provided by an embodiment of the present invention is shown;

[0052] Figure 3 A schematic diagram showing the flow of an image quality enhancement model optimization method provided by an embodiment of the present invention is shown;

[0053] Figure 4 A schematic diagram showing a flow chart of a method for enhancing video quality provided by an embodiment of the present invention is shown;

[0054] Figure 5 The figure shows the functional modules of the image quality enhancement model optimization device provided by the embodiment of the present invention.

[0055] Icon: 100 - electronic device; 110 - processor; 120 - memory; 130 - communication module; 300 - image quality enhancement model optimization device; 310 - video acquisition module; 330 - sample acquisition module; 350 - model optimization module. DETAILED DESCRIPTION

[0056] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0058] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0059] Please refer to Figure 1 , is a schematic diagram of a system provided by an embodiment of the present invention. The system includes a server and a client in communication connection, and there may be multiple clients. The server may be a standalone server or a server cluster consisting of multiple servers. The client may be a smartphone, a personal computer, a tablet computer, a netbook, a laptop computer, an ultra-mobile personal computer (UMPC), a personal digital assistant (PDA), etc., which are not limited in the embodiment of the present invention.

[0060] Please refer to Figure 2, is a block diagram of an electronic device provided by an embodiment of the present invention. The structure of the electronic device 100 can be used to implement the above Figure 1 The electronic device 100 includes a processor 110, a memory 120, and a communication module 130. Each component is electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0061] The processor 110 is used to read / write data or programs stored in the memory 120 and execute corresponding functions. It can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP digital signal processor, an ASIC application-specific integrated circuit, an FPGA off-the-shelf programmable gate array or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0062] The memory 120 is used to store programs or data. The memory 120 can be RAM (Random Access Memory), ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electric Erasable Programmable Read-Only Memory), etc.

[0063] The communication module 130 is used to communicate signaling or data with other devices.

[0064] It is understandable that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown. Figure 2 The components shown in the figure may be implemented using hardware, software, or a combination thereof. For example, to implement the corresponding functions of the client, the electronic device 100 may further include other modules, such as a radio frequency circuit, an I / O interface, a battery, a touch screen, a microphone / speaker, a camera module, etc., but this is not limited in the embodiment of the present invention.

[0065] Currently, the source stream video of a live stream is enhanced twice: by a cloud-based image quality enhancement model on the server and by an end-to-end image quality enhancement model on the client. Due to the poor image quality enhancement capabilities of the end-to-end image quality enhancement model, the video quality enhancement effect is affected, resulting in poor video quality and a reduced viewing experience. Therefore, embodiments of the present invention provide a method for optimizing an end-to-end image quality enhancement model to improve its image quality enhancement capabilities and enhance the video quality enhancement effect.

[0066] See also Figure 3 , is a flow chart of the image quality enhancement model optimization method provided by an embodiment of the present invention.

[0067] Step S202 : For each source stream video, obtain a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video.

[0068] Step S204 : For each source stream video, obtain an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtain multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video.

[0069] In this embodiment, based on the business data of the live broadcast platform, such as anchor ratings, PCU (Peak concurrent users, peak number of online users in the live broadcast room), image quality evaluation and other dimensions, source stream videos with lower image quality can be screened out from the live broadcast platform, and these source stream videos are pre-processed, such as deduplication of content and elimination of irrelevant video data, to obtain multiple source stream videos for optimizing the model, and these multiple source stream videos are all unenhanced and untranscoded videos.

[0070] It is understandable that each source stream video is processed in a similar manner. For ease of understanding, the following description will be made using the source stream video V as an example.

[0071] By performing transcoding operations and / or enhancement operations of varying degrees based on the source video stream V, an unenhanced transcoded video LV corresponding to the source video stream V and multiple video pairs with varying degrees of enhancement are obtained. For example, assuming there are five enhancement levels, namely, enhancement levels 1 to 5, then based on the source video stream V, video pairs corresponding to each of the five enhancement levels can be obtained, such as video pairs 1 to 5, and each video pair includes an enhanced unenhanced video and an enhanced transcoded video.

[0072] Then, given that both the source video stream V and the unenhanced transcoded video LV have not undergone image quality enhancement, an unenhanced image sample group can be obtained based on these two videos. Furthermore, given that both the enhanced unenhanced video and the enhanced transcoded video in a video pair have undergone image quality enhancement, an enhanced image sample group with the same degree of enhancement can be obtained based on this video pair. That is, based on video pairs 1 through 5, enhanced sample groups corresponding to five different degrees of enhancement can be obtained, such as enhanced sample groups 1 through 5.

[0073] Step S206 : Optimize the pre-trained end-side image quality enhancement model based on the unenhanced image sample group and all enhanced image sample groups of each source stream video to obtain an optimized end-side image quality enhancement model.

[0074] In this embodiment, similar processing is performed on each source stream video used for model optimization, following the same processing method as the source stream video V. This yields a set of unenhanced image samples and a set of all enhanced image samples for each source stream video. The pre-trained on-device image quality enhancement model is then optimized based on these unenhanced image samples and image samples with varying degrees of enhancement, yielding an optimized on-device image quality enhancement model.

[0075] It can be understood that the present invention optimizes the end-side image quality enhancement model by using unenhanced image samples and image samples with different enhancement levels, enabling it to enhance enhanced but low-quality images, thereby improving the image quality enhancement capability of the end-side image quality enhancement model, avoiding over-enhancement or under-enhancement of video quality, and improving the video quality enhancement effect and the audience's viewing experience. At the same time, optimizing the end-side image quality enhancement model only optimizes the model parameters and changes the model structure. The computing resources required by the optimized end-side image quality enhancement model are the same as those before optimization. That is, the enhancement capability of the end-side image quality enhancement model can be improved without improving the client's hardware performance.

[0076] It can be seen that based on the above steps, first, based on each source stream video, the corresponding unenhanced transcoded video of the source stream video and multiple video pairs with different enhancement degrees are obtained, and each video pair includes an enhanced unenhanced video and an enhanced transcoded video; then, based on the source stream video and its corresponding unenhanced transcoded video, an unenhanced image sample group is obtained, and based on the multiple video pairs corresponding to the source stream video, multiple enhanced image sample groups with different enhancement degrees are obtained; finally, based on the unenhanced image sample group of each source stream video and all enhanced image sample groups, the pre-trained end-side image quality enhancement model is optimized to obtain an optimized end-side image quality enhancement model. By using unenhanced image samples and image samples with different enhancement degrees to optimize the end-side image quality enhancement model, it has the ability to enhance enhanced but low-quality images, thereby improving the image quality enhancement capability of the end-side image quality enhancement model, avoiding over-enhancement or under-enhancement of video quality, and improving the video quality enhancement effect and the audience's viewing experience.

[0077] Optionally, for the process of obtaining the corresponding non-enhanced transcoded video and multiple video pairs with different enhancement degrees based on the source stream video in step S202, an embodiment of the present invention provides a possible implementation method.

[0078] Step S202-1: transcode the source stream video according to a preset bit rate to obtain an unenhanced transcoded video corresponding to the source stream video.

[0079] In step S202-3, the preset cloud-based image quality enhancement model is used to enhance the image quality of the source stream video according to the enhancement coefficient corresponding to the maximum enhancement degree, thereby obtaining an enhanced non-transcoded video corresponding to the maximum enhancement degree.

[0080] Step S202-5: Obtain the enhanced non-transcoded video corresponding to each other enhancement level based on the enhancement coefficient corresponding to each other enhancement level, the source stream video, and the enhanced non-transcoded video of the maximum enhancement level.

[0081] In step S202-7, each enhanced non-transcoded video is transcoded according to a preset bit rate to obtain each enhanced transcoded video, and the enhanced non-transcoded video and the enhanced transcoded video with the same enhancement degree are combined into a video pair to obtain multiple video pairs with different enhancement degrees corresponding to the source stream video.

[0082] In this embodiment, each enhancement level has a corresponding enhancement coefficient. For ease of understanding, the five enhancement levels described above are used as examples. It is assumed that the enhancement coefficients corresponding to enhancement levels 1 to 5 are 0.2, 0.4, 0.6, 0.8, and 1, respectively. The greater the enhancement level, the greater the enhancement coefficient, i.e., enhancement level 5 is the maximum enhancement level. It should be understood that the type of enhancement level and the enhancement coefficient corresponding to each enhancement level can be set according to actual conditions and are not limited in this embodiment of the present invention.

[0083] First, the source stream video V is transcoded according to the preset bit rate, and the unenhanced transcoded video LV corresponding to the source stream video is obtained. Among them, the preset bit rate needs to be the same as the bit rate of the corresponding clarity of the pre-trained end-side image quality enhancement model. It can be understood that for videos with different clarity after transcoding, different end-side image quality enhancement models are needed to enhance their image quality. That is, the pre-trained end-side image quality enhancement model has the corresponding clarity. Then, when optimizing the end-side image quality enhancement model, it is also necessary to use transcoded videos with the same clarity to perform model optimization to ensure the enhancement effect after model optimization.

[0084] The source video V is then enhanced using a preset cloud-based image quality enhancement model, using an enhancement coefficient of 1 corresponding to the maximum enhancement level, i.e., enhancement level 5, to obtain an enhanced, untranscoded video V5 corresponding to enhancement level 5. The source video V and the enhanced, untranscoded video V5 are then interpolated based on the enhancement coefficients corresponding to enhancement levels 1, 2, 3, and 4, respectively, to obtain enhanced, untranscoded videos corresponding to enhancement levels 1 to 4, such as enhanced, untranscoded videos V1 to V4.

[0085] For example, the process of obtaining the enhanced non-transcoded video V1 is as follows: according to the preset interpolation formula, based on the enhancement coefficient 0.2 corresponding to the enhancement level 1, each frame image of the source video V and the enhanced non-transcoded video V5 is interpolated, and multiple interpolated images are obtained, that is, the enhanced non-transcoded video V1 composed of these multiple interpolated images is obtained. The preset interpolation formula is: HR s =(HR max -HR)×s+HR, where HR represents a frame of image obtained from the source video stream, HRmax represents the corresponding frame of image obtained from the enhanced, non-transcoded video corresponding to the maximum enhancement level, s represents the enhancement coefficient, and HRs represents the interpolated image. For example, if HR is the first frame of image in the source video stream, then HRmax is the first frame of image in the enhanced, non-transcoded video corresponding to the maximum enhancement level.

[0086] Finally, enhanced untranscoded videos V1 through V5 are transcoded at a preset bitrate, resulting in five enhanced transcoded videos, namely, enhanced transcoded videos LV1 through LV5. Furthermore, enhanced untranscoded videos and enhanced transcoded videos with the same degree of enhancement are paired together to form video pairs corresponding to the five enhancement levels.

[0087] That is, video pair 1 corresponding to enhancement level 1 includes enhanced non-transcoded video V1 and enhanced transcoded video LV1, video pair 2 corresponding to enhancement level 2 includes enhanced non-transcoded video V2 and enhanced transcoded video LV2, video pair 3 corresponding to enhancement level 3 includes enhanced non-transcoded video V3 and enhanced transcoded video LV3, video pair 4 corresponding to enhancement level 4 includes enhanced non-transcoded video V4 and enhanced transcoded video LV4, and video pair 5 corresponding to enhancement level 5 includes enhanced non-transcoded video V5 and enhanced transcoded video LV5.

[0088] It can be understood that since transcoding has a nonlinear effect on the image quality of the enhanced image, that is, it will have different degrees of impact on different textures in the video image, the present invention adopts the method of first enhancing and then transcoding to obtain the enhanced and transcoded video, which can avoid the poor training samples affecting the model optimization, thereby ensuring the optimization effect of the model.

[0089] Optionally, for the process in step S204 of obtaining an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtaining multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video, an embodiment of the present invention provides a possible implementation method.

[0090] Step S204 - 1 : randomly obtain a plurality of original images from the source stream video, and obtain, based on each original image, each corresponding unenhanced image from the unenhanced transcoded video corresponding to the source stream video.

[0091] Step S204 - 3 : Pair each original image with the corresponding unenhanced image to obtain an unenhanced image sample group including a plurality of unenhanced image sample pairs, thereby obtaining an unenhanced image sample group of the source stream video.

[0092] Step S204-5: For each video pair corresponding to the source stream video, according to each original image, obtain each corresponding first enhanced image from the enhanced non-transcoded video of the video pair, and obtain each corresponding second enhanced image from the enhanced transcoded video of the video pair.

[0093] Step S204-7, pairing the first enhanced image and the second enhanced image corresponding to the same original image to obtain an enhanced image sample group including multiple enhanced image sample pairs, and obtaining an enhanced image sample group for each video pair corresponding to the source stream video, wherein the multiple enhanced image sample groups of the source stream video have different enhancement degrees.

[0094] It is understandable that the image quality of the transcoded video is significantly lower than that of the untranscoded video. In this case, the images in the transcoded video can be used as supervisory images, and the images in the untranscoded video can be used as supervised images. The image samples composed of the supervisory images and the corresponding supervised images can be used to optimize the end-side image quality enhancement model.

[0095] For ease of understanding, the above example is continued for explanation. Multiple frames of images are randomly obtained from the source stream video V to obtain multiple original images. And based on each original image, the image of the corresponding frame is obtained from the unenhanced transcoded video LV, and the unenhanced image corresponding to each original image is obtained. Then each original image is paired with its corresponding unenhanced image to obtain multiple unenhanced image sample pairs, and the original image in each unenhanced image sample pair is the supervisory image and the unenhanced image is the supervised image. The multiple unenhanced image sample pairs obtained constitute the unenhanced image sample group of the source stream video V.

[0096] It is understood that each video pair corresponding to the source stream video V is processed in a similar manner. For ease of understanding, the following description uses video pair 1 as an example. For each original image, the corresponding frame image is obtained from the enhanced, untranscoded video V1 of video pair 1 to obtain a first enhanced image corresponding to each original image. Furthermore, for each original image, the corresponding frame image is obtained from the enhanced, transcoded video LV1 of video pair 1 to obtain a second enhanced image corresponding to each original image.

[0097] By pairing the first enhanced image and the second enhanced image corresponding to the same original image, multiple enhanced image sample pairs are obtained, where the first enhanced image in each enhanced image sample pair is the supervisory image and the second enhanced image is the supervised image. These multiple enhanced image sample pairs constitute the enhanced image sample group for video pair 1, namely, enhanced image sample group 1. Similarly, video pairs 2 through 5 are processed separately to obtain enhanced image sample groups 2 through 5. This results in an unenhanced image sample group for source video stream V and five enhanced image sample groups with varying degrees of enhancement.

[0098] Optionally, for step S206, an embodiment of the present invention provides a possible implementation method.

[0099] Step S206 - 1 , obtaining all pairs of unenhanced image samples in the unenhanced image sample group of each source stream video to obtain an unenhanced image sample set.

[0100] Step S206 - 3 : obtaining multiple enhanced image sample sets based on all enhanced image sample pairs in all enhanced image sample groups of each source stream video; the enhanced image sample pairs in each enhanced image sample set have the same enhancement degree.

[0101] Step S206-5: sort all enhanced image sample sets in ascending order of enhancement degree to obtain a target sequence.

[0102] Step S206 - 7 : Optimize the pre-trained end-side image quality enhancement model using the unenhanced image sample set to obtain an initial end-side image quality enhancement model.

[0103] Step S206 - 9 : According to the target sequence, each enhanced image sample set is used in turn to optimize the initial end-side image quality enhancement model to obtain an optimized end-side image quality enhancement model.

[0104] For ease of understanding, we continue with the above example. Obtaining the unenhanced image sample group for each source stream video results in an unenhanced image sample set P0 containing multiple unenhanced image sample pairs, where each unenhanced image sample pair includes a supervisory image and a supervised image. Furthermore, all enhanced image sample groups for each source stream video are divided according to the degree of enhancement, resulting in five enhanced image sample sets. The enhanced image sample pairs in each enhanced image sample set have the same degree of enhancement, and each enhanced image sample pair includes a supervisory image and a supervised image.

[0105] The five enhanced image sample sets obtained are: enhanced image sample set P1 corresponding to enhancement level 1, enhanced image sample set P2 corresponding to enhancement level 2, enhanced image sample set P3 corresponding to enhancement level 3, enhanced image sample set P4 corresponding to enhancement level 4, and enhanced image sample set P5 corresponding to enhancement level 5. These five enhanced image sample sets are then sorted in ascending order of enhancement level to obtain the target sequence {P1, P2, P3, P4, P5}.

[0106] The pre-trained end-side image quality enhancement model is then optimized using the unenhanced image sample set P0. Specifically, the pre-trained end-side image quality enhancement model is used to enhance the supervised image in each unenhanced image sample pair to obtain each enhanced image. The pre-trained end-side image quality enhancement model is then optimized and trained based on the loss calculated for each enhanced image and the supervised image in each unenhanced image sample pair to obtain an initial end-side image quality enhancement model. For example, the loss can be calculated using a preset loss function, such as a pixel error loss function, a perceptual loss function, or a GAN adversarial loss function.

[0107] Finally, according to the target sequence, each enhanced image sample set is used in turn to optimize the initial end-side image quality enhancement model, and the optimized end-side image quality enhancement model is obtained.

[0108] It can be understood that the embodiment of the present invention optimizes the end-side image quality enhancement model by using image samples with corresponding enhancement levels in order from small to large, so as to guide the end-side image quality enhancement model to learn image quality enhancement from easy to difficult, thereby improving the image quality enhancement capability of the end-side image quality enhancement model and ensuring the optimization effect of the model.

[0109] Optionally, for step S206-9, an embodiment of the present invention provides a possible implementation method.

[0110] Step S206-9-1: Use the unenhanced image sample set as the old image sample set, and use the first enhanced image sample set in the target sequence as the new image sample set.

[0111] Step S206-9-3: The first ratio value in the preset ratio sequence is used as the current ratio value; the ratio values ​​in the preset ratio sequence are arranged in ascending order.

[0112] Step S206-9-5: Mix the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set, and use the mixed image sample set to optimize the initial end-side image quality enhancement model.

[0113] Step S206-9-7A: If the current ratio value is not the last ratio value in the preset ratio sequence, the next ratio value after the current ratio value is used as the next current ratio value, and then step S206-9-5 is executed again.

[0114] In step S206-9-7B, if the current ratio value is the last ratio value in the preset ratio sequence and the new image sample set is not the last enhanced image sample set in the target sequence, then step S206-9-3 is re-executed after the new image sample set is used as the next old image sample set and the next image sample set after the new image sample set is used as the next new image sample set.

[0115] Step S206-9-7C, if the current ratio value is the last ratio value in the preset ratio sequence and the new image sample set is the last enhanced image sample set in the target sequence, then an optimized end-side image quality enhancement model is obtained; wherein, when the current ratio value is the last ratio value in the preset ratio sequence, the mixed image sample set is the new image sample set.

[0116] It is understandable that, to ensure the optimization effect of the model, the present invention uses a mixed sample approach to gradually update the image sample set, so that the model gradually learns image quality enhancement. For ease of understanding, the following continues with the above example. The unenhanced image sample set P0 is used as the old image sample set, and the first enhanced image sample set P1 in the target sequence is used as the new image sample set.

[0117] The first ratio value, 25%, in a preset ratio sequence, such as {25%, 50%, 75%, 100%}, is used as the current ratio value. It should be noted that the ratio values ​​in the preset ratio sequence must be arranged in ascending order, and the specific values ​​and total number of ratio values ​​can be set according to actual circumstances and are not limited in this embodiment of the present invention.

[0118] At the current ratio of 25%, the old image sample set P0 and the new image sample set P1 are mixed to obtain a mixed image sample set. The mixed image sample set is then used to optimize the initial end-side image quality enhancement model. A determination is then made as to whether the current ratio is the last ratio in the preset ratio sequence. If the current ratio is the last ratio, it indicates that the old image sample set has been completely replaced by the new image sample set, meaning that the mixed image sample set is the new image sample set.

[0119] If it is determined that the current scale value is not the last scale value, it means that the old image sample set has not been completely replaced by the new image sample set. In this case, the scale value after the current scale value is used as the next current scale value, and S206-9-5 is re-executed to remix the image samples and perform model optimization. If the current scale value is the last scale value, it means that the old image sample set has been completely replaced by the new image sample set.

[0120] For example, if the current ratio value is 25% and it is not the last ratio value in the preset ratio sequence, then the next ratio value after the current ratio value of 25%, that is, 50%, is used as the next current ratio value, and the image samples are remixed and the model is optimized. This process is repeated in sequence until the current ratio value is the last ratio value, that is, the unenhanced image sample set P0 is completely replaced by the enhanced image sample set P1. It can be understood that the present invention gradually replaces the old image sample set (i.e., the image sample set with a lower degree of enhancement) with the new image sample set (i.e., the image sample set with a higher degree of enhancement) by gradually increasing the ratio value, so that the model gradually adapts to the replacement of image samples, thereby avoiding the direct replacement of the image sample set and affecting the model optimization, and ensuring the optimization effect of the model.

[0121] After the current ratio value is the last ratio value in the preset ratio sequence, that is, after the old image sample set has been completely replaced by the new image sample set, it is necessary to determine whether the new image sample set is the last enhanced image sample set. If the new image sample set is not the last enhanced image sample set, it indicates that there are still unused enhanced image sample sets. Then, after using the new image sample set as the next old image sample set and the image sample set after the new image sample set as the next new image sample set, step S206-9-3 is re-executed, that is, the image samples are mixed and the model optimization is performed again according to the ratio values ​​in the preset ratio sequence. If the new image sample set is the last enhanced image sample set, it indicates that all enhanced image sample sets have been used for model optimization, and an optimized end-side image quality enhancement model is obtained.

[0122] For example, if the new image sample set P1 is not the last enhanced image sample set in the target sequence, then the new image sample set P1 is used as the next old image sample set, and after the next enhanced image sample set P2 after the new image sample set P1 is used as the next new image sample set, the image sample sets are mixed and the model is optimized again according to the ratio values ​​in the preset ratio sequence. This process is repeated until the new image sample set is the last enhanced image sample set, that is, all enhanced image sample sets are used for model optimization. It can be understood that the present invention optimizes the model by using image sample sets with gradually increasing enhancement levels to guide the end-side image quality enhancement model to learn image quality enhancement from easy to difficult, thereby gradually improving the image quality enhancement capability of the model and ensuring the effect of model optimization.

[0123] Optionally, for the process of mixing the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set in step S206-9-5, an embodiment of the present invention provides a possible implementation method.

[0124] Step S206-9-51, replacing multiple old image sample pairs in the old image sample set with new image sample pairs in the new image sample set to obtain a replaced image sample set, wherein the proportion of the new image sample pairs in the replaced image sample set is the current proportion value.

[0125] Step S206-9-52, according to the current ratio value, obtain multiple old image sample pairs from the replaced image sample set to obtain each target old image sample pair.

[0126] Step S206-9-53, for each target old image sample pair, obtain the target new image sample pair that matches the target old image sample pair in the new image sample set, and replace the image area of ​​the target old image sample pair according to the target new image sample pair to obtain each replaced image sample pair, and obtain a mixed image sample set; wherein the proportion of the replaced area in each image sample of the replaced image sample pair is the current ratio value.

[0127] In this embodiment, the old image sample set includes multiple old image sample pairs, and the new image sample set includes multiple new image sample pairs. For ease of understanding, the example in which the old image sample set is the unenhanced image sample set P0 and the new image sample set is the enhanced image sample set P1 is continued. In this case, the unenhanced image sample pairs in the unenhanced image sample set P0 are the old image sample pairs, and the enhanced image sample pairs in the enhanced image sample set P1 are the new image sample pairs.

[0128] For the convenience of description, the unenhanced image sample pairs in the unenhanced image sample set P0 are represented as LR0-HR0, where LR0 represents the supervised image in the image sample pair, and HR0 is the supervisory image in the image sample pair; the enhanced image sample pairs in the enhanced image sample set P1 are represented as LR1-HR1, where LR1 represents the supervised image in the image sample pair, and HR1 is the supervisory image in the image sample pair.

[0129] Based on the current ratio of 25%, 25% of the old image sample pairs are obtained from the old image sample set P0 to obtain multiple old image sample pairs to be replaced. Then, in the new image sample set P1, the new image sample pairs that match each old image sample pair to be replaced are determined to obtain multiple new image sample pairs to be replaced. These multiple old image sample pairs to be replaced are deleted from the old image sample set P0 and added to the multiple new image sample pairs to be replaced, thereby obtaining a replaced image sample set. In the replaced image sample set, 75% of the old image sample pairs, namely LR0-HR0, and 25% of the new image sample pairs, namely LR1-HR1, are old image sample pairs.

[0130] Based on the current ratio of 25%, 25% of the old image sample pairs are obtained from 75% of the old image sample pairs in the replaced image sample set to obtain each target old image sample pair, and image region replacement is performed on each target old image sample pair. It is understood that the processing method for each target old image sample pair is similar, and the following uses one target old image sample pair as an example for explanation.

[0131] Based on the target old image sample pair LR0-HR0, obtain the target new image sample pair LR1-HR1 that matches it in the new image sample set P1. Then, determine the target image region in the supervised image LR0 of the target old image sample pair LR0-HR0, and determine the image region at the same position in the supervised image HR0 as the target image region. It should be noted that the position of the target image region can be randomly selected, but the ratio of the area of ​​the target image region to the total area of ​​the supervised image in the target old image sample pair must be consistent with the current ratio value.

[0132] Next, the target new image sample is used to replace the target image region in the supervised image LR0 with the image region at the same position in the supervised image LR1 of LR1-HR1, thereby obtaining the replaced supervised image LR01. Furthermore, the target new image sample is used to replace the target image region in the supervisory image HR0 of LR1-HR1 with the image region at the same position in the supervisory image HR1 of LR1-HR1, thereby obtaining the replaced supervisory image HR01. The replaced supervised image and the replaced supervisory image are obtained, i.e., the replaced image sample pair LR01-HR01.

[0133] Each old image sample pair is processed in a similar manner to obtain each replaced image sample pair, that is, the mixed image sample set. In the mixed image sample set, 25% are new image sample pairs, namely LR1-HR1, 56.25% are old image sample pairs, namely LR0-HR0, and 18.75% are replaced image sample pairs, namely LR01-HR01.

[0134] It can be understood that the present invention achieves mixing of image sample sets by replacing old and new image sample pairs, and uses this mixed image sample set to optimize the model, enabling the model to adaptively enhance image quality for images with varying degrees of enhancement. Furthermore, by replacing image regions within new and old image sample pairs, image sample mixing is achieved, and the model is optimized using this mixed image sample set, enabling the model to adaptively enhance image quality for regions with varying degrees of enhancement within the same image. This improves the image quality enhancement capabilities of the on-device image quality enhancement model.

[0135] Optionally, based on the optimized end-side image quality enhancement model obtained according to the above method, an embodiment of the present invention further provides a video quality enhancement method, which is applied to a system including a server and a client. Figure 4 , is a flow chart of a video quality enhancement method provided by an embodiment of the present invention.

[0136] In step S212, the server uses a pre-stored cloud-based image quality enhancement model to enhance the image quality of the target source stream video according to the current enhancement level to obtain an enhanced video.

[0137] In step S214, the server transcodes the enhanced video based on the bit rate corresponding to the target definition selected by the user, obtains the transcoded video, and sends it to the client.

[0138] In step S216, the client obtains a target end-side image quality enhancement model that matches the current enhancement level and target clarity from multiple pre-stored end-side image quality enhancement models, and uses the target end-side image quality enhancement model to enhance the image quality of the transcoded video to obtain the target live video.

[0139] It can be understood that based on the characteristics of the end-side image quality enhancement model being small in size and occupying little space, multiple end-side image quality enhancement models can be set based on the multiple enhancement levels and multiple clarity settings of the cloud-based image quality enhancement model, so that the end-side image quality enhancement model with corresponding clarity and enhancement level can be used to perform image quality enhancement, thereby improving the video quality enhancement effect.

[0140] For example, assuming the cloud-based image quality enhancement model has two enhancement levels and three definition settings, six end-side image quality enhancement models are set. Furthermore, the above-mentioned image quality enhancement model optimization method can be used to optimize these six end-side image quality enhancement models and then install them on the client.

[0141] In this embodiment, the server obtains the current enhancement level based on the current enhancement level of a pre-stored cloud-based image quality enhancement model. Using this pre-stored cloud-based image quality enhancement model, the server enhances the target source stream video according to the current enhancement level, generating an enhanced video. The enhanced video is then transcoded based on the bitrate corresponding to the target definition selected by the user, resulting in the transcoded video and sent to the client.

[0142] The client obtains a target end-side image quality enhancement model that matches the current enhancement level and target clarity from multiple pre-stored end-side image quality enhancement models, and uses the target end-side image quality enhancement model to enhance the image quality of the transcoded video, thereby obtaining the target live video and displaying the target live video.

[0143] It can be understood that since each end-side image quality enhancement model in the client is a model optimized according to the above-mentioned image quality enhancement model optimization method, then using the end-side image quality enhancement model in the client to enhance the image quality of the video can avoid excessive or insufficient enhancement of the video quality, thereby improving the video quality enhancement effect and the audience's viewing experience.

[0144] In order to execute the corresponding steps in the above embodiments and various possible methods, a method for implementing an image quality enhancement model optimization device is given below. Figure 5 , Figure 5 This is a functional module diagram of a device 300 for optimizing an image quality enhancement model provided by an embodiment of the present invention. It should be noted that the basic principles and technical effects of the device 300 provided by this embodiment are the same as those of the above-mentioned embodiments. For the sake of simplicity, any details not mentioned in this embodiment can be referred to the corresponding contents of the above-mentioned embodiments. The device 300 for optimizing an image quality enhancement model includes:

[0145] The video acquisition module 310 is configured to obtain, for each source stream video, a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video;

[0146] The sample acquisition module 330 is configured to obtain, for each source stream video, an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtain multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video;

[0147] The model optimization module 350 is used to optimize the pre-trained end-side image quality enhancement model based on the unenhanced image sample group and all enhanced image sample groups of each source stream video to obtain an optimized end-side image quality enhancement model.

[0148] Optionally, the video acquisition module 310 is also used to: transcode the source stream video according to a preset bit rate to obtain an unenhanced transcoded video corresponding to the source stream video; use a preset cloud-based image quality enhancement model to enhance the image quality of the source stream video according to the enhancement coefficient corresponding to the maximum enhancement degree to obtain an enhanced unenhanced video corresponding to the maximum enhancement degree; based on the enhancement coefficient corresponding to each other enhancement degree, the source stream video and the enhanced unencoded video with the maximum enhancement degree, obtain an enhanced unencoded video corresponding to each other enhancement degree; transcode each enhanced unencoded video according to the preset bit rate to obtain each enhanced transcoded video, and form an enhanced unencoded video and an enhanced transcoded video with the same enhancement degree into a video pair to obtain multiple video pairs with different enhancement degrees corresponding to the source stream video.

[0149] Optionally, the sample acquisition module 330 is also used to: randomly acquire multiple original images from the source stream video, and according to each original image, acquire each corresponding unenhanced image from the unenhanced transcoded video corresponding to the source stream video; pair each original image with the corresponding unenhanced image to obtain an unenhanced image sample group including multiple unenhanced image sample pairs, and obtain an unenhanced image sample group of the source stream video; for each video pair corresponding to the source stream video, according to each original image, acquire each corresponding first enhanced image from the enhanced unenhanced video of the video pair, and acquire each corresponding second enhanced image from the enhanced transcoded video of the video pair; pair the first enhanced image and the second enhanced image corresponding to the same original image to obtain an enhanced image sample group including multiple enhanced image sample pairs, and obtain an enhanced image sample group for each video pair corresponding to the source stream video, and the multiple enhanced image sample groups of the source stream video have different degrees of enhancement.

[0150] Optionally, the model optimization module 350 is also used to: obtain all unenhanced image sample pairs in the unenhanced image sample group of each source stream video to obtain an unenhanced image sample set; obtain multiple enhanced image sample sets based on all enhanced image sample pairs in all enhanced image sample groups of each source stream video; the enhanced image sample pairs in each enhanced image sample set have the same degree of enhancement; sort all enhanced image sample sets in order of enhancement degree from small to large to obtain a target sequence; use the unenhanced image sample set to optimize the pre-trained end-side image quality enhancement model to obtain an initial end-side image quality enhancement model; according to the target sequence, use each enhanced image sample set in turn to optimize the initial end-side image quality enhancement model to obtain an optimized end-side image quality enhancement model.

[0151] Optionally, the model optimization module 350 is further configured to: use the unenhanced image sample set as the old image sample set, and use the first enhanced image sample set in the target sequence as the new image sample set; use the first ratio value in the preset ratio sequence as the current ratio value; arrange the ratio values ​​in the preset ratio sequence in ascending order; mix the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set, and use the mixed image sample set to optimize the initial end-side image quality enhancement model; if the current ratio value is not the last ratio value in the preset ratio sequence, use the next ratio value after the current ratio value as the next current ratio value, and then re-execute the mixing of the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set, and The step of optimizing the initial end-side image quality enhancement model using the mixed image sample set; if the current ratio value is the last ratio value in the preset ratio sequence and the new image sample set is not the last enhanced image sample set in the target sequence, then using the new image sample set as the next old image sample set and the next image sample set after the new image sample set as the next new image sample set, and then re-performing the step of using the first ratio value in the preset ratio sequence as the current ratio value; if the current ratio value is the last ratio value in the preset ratio sequence and the new image sample set is the last enhanced image sample set in the target sequence, then obtaining the optimized end-side image quality enhancement model; wherein, when the current ratio value is the last ratio value in the preset ratio sequence, the mixed image sample set is the new image sample set.

[0152] Optionally, the model optimization module 350 is also used to: replace multiple old image sample pairs in the old image sample set with new image sample pairs in the new image sample set to obtain a replaced image sample set, and the proportion of new image sample pairs in the replaced image sample set is the current proportion value; according to the current proportion value, obtain multiple old image sample pairs from the replaced image sample set to obtain each target old image sample pair; for each target old image sample pair, obtain a target new image sample pair that matches the target old image sample pair in the new image sample set, and replace the image area of ​​the target old image sample pair according to the target new image sample pair to obtain each replaced image sample pair to obtain a mixed image sample set; wherein, the proportion of the replaced area in each image sample of the replaced image sample pair is the current proportion value.

[0153] An embodiment of the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the image quality enhancement model optimization method disclosed in the embodiment of the present invention is implemented.

[0154] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the image quality enhancement model optimization method disclosed in the embodiment of the present invention is implemented.

[0155] The embodiment of the present invention further provides a system, including the server and client provided by the embodiment of the present invention, to implement the video quality enhancement method disclosed in the embodiment of the present invention.

[0156] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0157] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0158] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0159] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for optimizing an image quality enhancement model, characterized in that: The image quality enhancement model optimization method includes: For each source stream video, obtaining a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video; For each source stream video, obtaining an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtaining multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video; Based on the unenhanced sample group and all enhanced sample groups of each source video, an unenhanced image sample set and multiple enhanced image sample sets with different enhancement degrees are obtained, and the pre-trained end-side image quality enhancement model is optimized using the unenhanced image sample set to obtain an initial end-side image quality enhancement model; In order of enhancement degree from small to large, obtaining the first enhanced image sample set as the new image sample set, and taking the unenhanced image sample set as the old image sample set; sequentially mixing the old image sample set and the new image sample set according to each ratio value arranged from small to large in the preset ratio sequence to obtain a mixed image sample set, optimizing the initial end-side image quality enhancement model using the mixed image sample set, using the first enhanced image sample set as the next old image sample set and using the next enhanced image sample set after the first enhanced image sample set as the next new image sample set; The old image sample set and the new image sample set are mixed in sequence according to each ratio value arranged from small to large in the preset ratio sequence to obtain a mixed image sample set, and the initial end-side image quality enhancement model is optimized using the mixed image sample set; the cycle is repeated until the last enhanced image sample set is reached to obtain the optimized end-side image quality enhancement model.

2. The image quality enhancement model optimization method according to claim 1, characterized in that: Each degree of enhancement has a corresponding enhancement coefficient; The step of obtaining a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video includes: Transcoding the source stream video according to a preset bit rate to obtain an unenhanced transcoded video corresponding to the source stream video; Using a preset cloud-based image quality enhancement model, the source stream video is enhanced according to an enhancement coefficient corresponding to a maximum enhancement degree, to obtain an enhanced, non-transcoded video corresponding to the maximum enhancement degree; Obtaining enhanced non-transcoded videos corresponding to each other enhancement level based on the enhancement coefficient corresponding to each other enhancement level, the source stream video, and the enhanced non-transcoded video of the maximum enhancement level; Each enhanced non-transcoded video is transcoded according to a preset bit rate to obtain each enhanced transcoded video, and the enhanced non-transcoded video and the enhanced transcoded video with the same enhancement degree are combined into a video pair to obtain multiple video pairs with different enhancement degrees corresponding to the source stream video.

3. The image quality enhancement model optimization method according to claim 1, characterized in that: The step of obtaining an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtaining a plurality of enhanced image sample groups with different enhancement degrees based on a plurality of video pairs corresponding to the source stream video, comprises: Randomly obtain a plurality of original images from the source stream video, and obtain, based on each of the original images, each corresponding unenhanced image from the unenhanced transcoded video corresponding to the source stream video; Pairing each of the original images with a corresponding unenhanced image to obtain an unenhanced image sample group including a plurality of unenhanced image sample pairs, thereby obtaining an unenhanced image sample group of the source stream video; For each video pair corresponding to the source stream video, obtaining, according to each original image, each corresponding first enhanced image from the enhanced non-transcoded video of the video pair, and each corresponding second enhanced image from the enhanced transcoded video of the video pair; A first enhanced image and a second enhanced image corresponding to the same original image are paired to obtain an enhanced image sample group including multiple enhanced image sample pairs, and an enhanced image sample group for each video pair corresponding to the source stream video is obtained, where the multiple enhanced image sample groups of the source stream video have different degrees of enhancement.

4. The image quality enhancement model optimization method according to claim 1, characterized in that: After the step of optimizing the pre-trained end-side image quality enhancement model using the unenhanced image sample set to obtain an initial end-side image quality enhancement model, the method further includes: According to the enhancement degree from small to large, all enhanced image sample sets are sorted to obtain the target sequence.

5. The image quality enhancement model optimization method according to claim 4, characterized in that: The step of sequentially mixing the old image sample set and the new image sample set according to each ratio value arranged from small to large in the preset ratio sequence to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set includes: Using the unenhanced image sample set as an old image sample set, and using the first enhanced image sample set in the target sequence as a new image sample set; The first ratio value in the preset ratio sequence is used as the current ratio value; the ratio values ​​in the preset ratio sequence are arranged in ascending order; Mixing the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set; If the current scale value is not the last scale value in the preset scale sequence, after taking the scale value after the current scale value as the next current scale value, re-performing the steps of mixing the old image sample set and the new image sample set according to the current scale value to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set; until the current scale value is the last scale value in the preset scale sequence.

6. The image quality enhancement model optimization method according to claim 5, characterized in that: The old image sample set includes a plurality of old image sample pairs, and the new image sample set includes a plurality of new image sample pairs; and the step of mixing the old image sample set and the new image sample set according to the current ratio value to obtain a mixed image sample set includes: Replacing a plurality of old image sample pairs in the old image sample set with new image sample pairs in the new image sample set to obtain a replaced image sample set, wherein the proportion of the new image sample pairs in the replaced image sample set is the current proportion value; According to the current ratio value, a plurality of old image sample pairs are obtained from the replaced image sample set to obtain each target old image sample pair; For each of the target old image sample pairs, a target new image sample pair that matches the target old image sample pair is obtained in the new image sample set, and image regions of the target old image sample pair are replaced according to the target new image sample pair to obtain each replaced image sample pair, thereby obtaining the mixed image sample set; wherein the proportion of the replaced region in each image sample of the replaced image sample pair is the current ratio value.

7. A method for enhancing video quality, characterized in that: Applied to a system including a server and a client, the video quality enhancement method includes: The server uses the pre-stored cloud-based image quality enhancement model to enhance the image quality of the target source stream video according to the current enhancement level to obtain the enhanced video; The server transcodes the enhanced video based on the bitrate corresponding to the target definition selected by the user, obtains the transcoded video, and sends it to the client; The client obtains a target end-side image quality enhancement model that matches the current enhancement level and target clarity from multiple pre-stored end-side image quality enhancement models, and uses the target end-side image quality enhancement model to enhance the image quality of the transcoded video to obtain the target live video; wherein each end-side image quality enhancement model is obtained according to the image quality enhancement model optimization method described in any one of claims 1 to 6.

8. An optimization device for an image quality enhancement model, characterized in that: The device comprises: A video acquisition module is configured to obtain, for each source stream video, a corresponding unenhanced transcoded video and a plurality of video pairs with different enhancement levels based on the source stream video; each video pair includes an enhanced unenhanced transcoded video and an enhanced transcoded video; a sample acquisition module configured to obtain, for each source stream video, an unenhanced image sample group based on the source stream video and its corresponding unenhanced transcoded video, and obtain multiple enhanced image sample groups with different enhancement degrees based on multiple video pairs corresponding to the source stream video; a model optimization module, configured to obtain, based on the unenhanced sample set and all enhanced sample sets of each source video, an unenhanced image sample set and multiple enhanced image sample sets with different enhancement levels, and optimize a pre-trained on-device image quality enhancement model using the unenhanced image sample set to obtain an initial on-device image quality enhancement model; In order of enhancement degree from small to large, obtaining the first enhanced image sample set as the new image sample set, and taking the unenhanced image sample set as the old image sample set; sequentially mixing the old image sample set and the new image sample set according to each ratio value arranged from small to large in the preset ratio sequence to obtain a mixed image sample set, and optimizing the initial end-side image quality enhancement model using the mixed image sample set; The first enhanced image sample set is used as the next old image sample set and the next enhanced image sample set after the first enhanced image sample set is used as the next new image sample set; the old image sample set and the new image sample set are mixed in sequence according to each ratio value arranged from small to large in the preset ratio sequence to obtain a mixed image sample set, and the initial end-side image quality enhancement model is optimized using the mixed image sample set; the cycle is repeated until the last enhanced image sample set is set to obtain the optimized end-side image quality enhancement model.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method for optimizing the image quality enhancement model according to any one of claims 1 to 6 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the image quality enhancement model optimization method according to any one of claims 1 to 6.

Images