Image quality enhancement method, device, terminal and readable storage medium

Abstract

The image quality enhancement method, device, terminal and readable storage medium provided by the present invention are applied to the terminal, and the method includes: obtaining the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information of the terminal; determining the target enhancement model according to the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information; when it is determined that there is a need for image quality enhancement, enhancing the image to be enhanced according to the target enhancement model. It can be seen that the entire process of the image quality enhancement method provided by the embodiment of the present invention is completed in the terminal, which effectively alleviates the resource pressure on the server side. At the same time, the method provided by the present invention can determine the adaptive enhancement model according to the type of terminal and the type of live broadcast room, distinguish the enhancement model of the platform hardware, so that the enhancement scheme is more compatible and meets people's demand for high image quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image quality enhancement method, device, terminal and readable storage medium. Background Art

[0002] With the development of the Internet era, graphics card capabilities have gradually increased, and users' demand for high-definition and high-smoothness live broadcast experience on the terminal side has become increasingly intense.

[0003] In the traditional live broadcast process, image quality and bit rate are negatively correlated. The higher the image quality, the higher the bit rate required, the greater the transmission pressure, and the bottleneck of background transcoding enhancement is becoming more and more obvious. The existing image quality enhancement is performed on the server side. However, the model used for image quality enhancement on the server side has been replaced for several generations, resulting in the inability to improve the image quality enhancement effect. In addition, the existing GPU image source on the server side is gradually consumed, causing the image quality enhancement effect to encounter a bottleneck, unable to achieve better results, and unable to meet user needs. Summary of the invention

[0004] One of the purposes of the present invention is to provide an image quality enhancement method, device, terminal and readable storage medium to improve the image quality enhancement effect and efficiency and meet the user's demand for high image quality. The embodiments of the present invention can be implemented as follows:

[0005] In a first aspect, the present invention provides an image quality enhancement method, applied to a terminal, the method comprising:

[0006] Obtain the terminal's device type, live broadcast room type, live broadcast bitrate, and enhanced model configuration information;

[0007] Determine a target enhancement model according to the device type, live broadcast room type, live broadcast bit rate, and enhancement model configuration information;

[0008] When it is determined that there is a need for image quality enhancement, the image to be enhanced is enhanced according to the target enhancement model.

[0009] In a second aspect, the present invention provides an image quality enhancement device, comprising:

[0010] The acquisition module is used to obtain the terminal's device type, live broadcast room type, live broadcast bit rate, and enhanced model configuration information;

[0011] A determination module, used to determine a target enhancement model according to the device type, the live broadcast room type, the live broadcast bit rate and the enhancement model configuration information;

[0012] The enhancement module is used to enhance the image quality of the image to be enhanced according to the target enhancement model when it is determined that there is a need for image quality enhancement.

[0013] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the method described in the first aspect.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0015] The image quality enhancement method, device, terminal and readable storage medium provided by the present invention are applied to the terminal, and the method includes: obtaining the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information of the terminal; determining the target enhancement model according to the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information; when it is determined that there is a need for image quality enhancement, enhancing the image to be enhanced according to the target enhancement model. It can be seen that the entire process of the image quality enhancement method provided by the embodiment of the present invention is completed in the terminal, which effectively alleviates the resource pressure on the server side. At the same time, the method provided by the embodiment of the present invention can determine the adaptive enhancement model according to the type of terminal and the type of live broadcast room, distinguish the enhancement model of the platform hardware, so that the enhancement scheme is more compatible and meets people's demand for high image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of an existing application scenario of image quality enhancement;

[0018] Figure 2 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention;

[0019] Figure 3 A schematic flow chart of an image quality enhancement method provided by an embodiment of the present invention;

[0020] Figure 4 A schematic flow chart of step S303 provided in an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of data resources before and after processing of different types of terminals provided in an embodiment of the present invention;

[0022] Figure 6A schematic flow chart of another implementation of step S303 provided in an embodiment of the present invention;

[0023] Figure 7 A schematic flow chart of another method for enhancing image quality provided by an embodiment of the invention;

[0024] Figure 8 This is a functional module diagram of an image quality enhancement device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0028] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0029] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0030] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0031] With the rapid development of live broadcast technology, users have an increasing demand for high-quality and high-smooth live broadcasts on the terminal side. Image quality enhancement processing is to analyze the image, determine the color information, object edges and other data in the image, and enhance these data to a certain extent through enhancement algorithms, such as sharpening the edge of the object, improving the contrast and saturation, etc., so that the image quality is improved. This is very important for improving the audience's experience.

[0032] See also Figure 1 , Figure 1 This is a schematic diagram of an existing application scenario of image quality enhancement. In this application scenario, a terminal 110 and a server 120 are included, wherein the terminal 110 and the server 120 can be connected via a network communication. The terminal 110 may include but is not limited to: a tablet computer, a smart phone, a PC, a smart wearable device, etc., and the server 120 may include but is not limited to: a live broadcast server, a game server, a streaming media server, etc.

[0033] Assuming that in a live broadcast scenario, the terminal 110 sends the captured image to the server 120. The image can be considered as the image before enhancement. The image may have image quality problems such as blur and noise. After the server 120 obtains the image, it can use the enhancement model to enhance the image, and then feed the enhanced image back to the terminal 110, and the terminal 110 can display the enhanced image.

[0034] From the above application scenarios, it can be seen that the existing image quality enhancement is performed on the server side. However, the model used for image quality enhancement on the server side has been continuously optimized, resulting in the inability to improve the image quality enhancement effect. In addition, the existing GPU image source on the server side is gradually consumed, causing the image quality enhancement effect to encounter a bottleneck, unable to achieve better results, and unable to meet user needs.

[0035] It should be noted that Figure 1 The image quality issues shown are examples only and do not represent actual image quality defects.

[0036] In order to solve the above problems, an embodiment of the present invention provides an image quality enhancement method, which can be applied to Figure 1 On the terminal 110 in the image quality enhancement process, in order to achieve the effect of image quality enhancement, the embodiment of the present invention can match the enhancement type suitable for the terminal for the terminal of different device types, thereby improving the compatibility of the image quality enhancement scheme in the embodiment of the present invention. At the same time, in the process of enhancing the image, the enhancement speed can be optimized, the enhancement efficiency can be improved, and the user's demand for real-time enhancement can be met.

[0037] The following first introduces a terminal for executing an image quality enhancement method in an embodiment of the present invention.

[0038] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. The terminal 110 may include an image quality enhancement device 400 , a memory 111 , a processor 112 and a communication interface 113 .

[0039] Among them, the memory 111, the processor 112 and the communication interface 113 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The image quality enhancement device 400 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the terminal 110. The processor 112 is used to execute the executable module stored in the memory 111, such as the software function module or computer program included in the human image quality enhancement device 400.

[0040] The memory 111 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 112 executes the programs after receiving the execution instruction.

[0041] The processor 112 may be an integrated circuit chip with signal processing capabilities. The processor 112 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a voice processor, and a video processor, etc.; it may also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. A general-purpose processor may be a microprocessor or the processor 112 may also be any conventional processor, etc.

[0042] The communication interface 113 is used to couple various input / output devices to the processor 112 and the memory 111 .

[0043] The following describes the implementation process of the image quality enhancement method provided by the embodiment of the present invention. The image quality enhancement method can be implemented by Figure 2 Execute it at terminal 110 in the Figure 3 , Figure 3 A schematic flow chart of an image quality enhancement method provided by an embodiment of the present invention, the method may include:

[0044] S301, obtaining the terminal device type, live broadcast room type, live broadcast bit rate and enhanced model configuration information.

[0045] S302, determining a target enhancement model according to the device type, the live broadcast room type, the live broadcast bit rate, and the enhancement model configuration information.

[0046] S303: When it is determined that there is a need for image quality enhancement, the image to be enhanced is enhanced according to a target enhancement model.

[0047] According to the image quality enhancement method provided by the embodiment of the present invention, the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information of the terminal are first obtained, and then the target enhancement model can be determined according to the device type, live broadcast room type, live broadcast bit rate and enhancement model configuration information. Finally, when there is an enhancement requirement, the image to be enhanced is enhanced based on the enhancement model. It can be seen that the entire process of the image quality enhancement method provided by the embodiment of the present invention is completed in the terminal, which effectively alleviates the resource pressure on the server side. At the same time, the method provided by the embodiment of the present invention can determine the adapted enhancement model according to the type of terminal and the type of live broadcast room, distinguish the enhancement model of the platform hardware, so that the enhancement solution is more compatible and meets the user's demand for high image quality.

[0048] The following is an introduction to each exemplary step S301 to step S304 in the above embodiment.

[0049] In step S301, the device type of the terminal, the type of the live broadcast room, the live broadcast bit rate and the enhanced model configuration information are obtained.

[0050] In an embodiment of the present invention, the device type of the terminal can be but is not limited to a PC or a mobile device for a smart phone. It can also be distinguished according to the mobile operating system installed inside the mobile device, for example, an iOS terminal, an Android terminal, etc., to obtain an enhanced model adapted to different mobile operating systems; the live broadcast room type refers to the type of live broadcast room currently entered by the user or the type of live broadcast room opened by the user.

[0051] In an embodiment of the present invention, the above-mentioned enhanced model configuration information may be, but is not limited to, sent from a server to a terminal. For each enhanced model, the configuration information may include a model name. For each model name, the configuration information may include, but is not limited to, the following information: framework name, live broadcast room type, live broadcast bit rate, enhanced type, version, related framework suffix, etc. The device terminal may select the enhanced framework to be matched through the "framework name". The live broadcast room type and live broadcast bit rate may be used to verify whether the current live broadcast room can use the model; the enhanced type is the optimal model type determined by the terminal performance. Different enhanced types may be distinguished by, but are not limited to, different numbers. For example, "0" indicates any model type, "1" indicates an enhanced model, and "2" indicates a super-resolution model. It should be noted that the enhanced type variable is mainly used on terminals using the Android operating system, and different chips select different models.

[0052] In step S302, the target enhancement model is determined according to the device type, the live broadcast room type, the live broadcast bit rate and the enhancement model configuration information.

[0053] At present, the more commonly used enhancement frameworks include MNN, TNN and NCNN. However, these enhancement frameworks will inevitably encounter the same problem during use. The image source output by terminal decoding does not match the unified input of the enhancement framework. This means that before enhancement, the decoded output image data needs to be converted into the unified input of the enhancement framework. This process will inevitably result in a lot of performance consumption, which greatly increases the difficulty of product launch.

[0054] In order to solve the above problems, the embodiments of the present invention design different enhancement models for terminals with different operating systems in combination with the decoding outputs of the different terminals. The following embodiments will be described by taking PC, iOS and Android terminals as examples.

[0055] On the PC side, the data format of the decoded output image source is D3D11Texture. Therefore, the embodiment of the present invention preferably uses D3D11 computer shader (a renderer) to implement the general convolution and transposed convolution required for the enhancement process, so as to meet the implementation of the enhancement process. It can be considered that after the data decoded by the PC side is subjected to the enhancement process, an image data similar to D3D11Texture will be obtained, which is convenient for subsequent direct use in layered rendering. It can be understood that the input and output of the enhancement model for the PC side are both D3D11Texture, and no additional processing is required, which avoids unnecessary conversion and copying, and can improve the enhancement efficiency.

[0056] In this embodiment, for the PC side, since its equipment is very similar to the traditional background server enhancement, the model used in the previous background enhancement can be used to perform on-side enhancement.

[0057] On the iOS terminal, the data format of the decoded output image source is CVPixelBufferRef. Therefore, the present embodiment preferably uses the existing MLCore framework to perform related enhancement processing operations. However, since the MLCore model cannot exist in the static library, in order to overcome this problem, the functions provided by the MLCore framework can be used, which can be imported into the app through online download and local compilation. The whole process can adopt the loading synchronous and compiling asynchronous method.

[0058] In this embodiment, for iOS terminals, the enhanced model can be designed using a model with seven layers of ordinary convolution and one inverted convolution.

[0059] On the Android terminal, the data format of the decoded image source is GL.Texture. In the existing enhancement frameworks on various terminals, the input is set as the Buffer interface related to each platform, which has the defects of long enhancement time and high copying time. Therefore, the implementation of the present invention first encapsulates the decoded image data into the corresponding Handler, and then draws the enhanced encapsulated data onto GL.Texture using OpenGL technology.

[0060] In this embodiment, for Android terminals, a three-layer UNet structure can be used to enhance the model design with a model that has two downsamplings and three upsamplings.

[0061] In step S303, when it is determined that there is a need for image quality enhancement, the image to be enhanced is enhanced according to the target enhancement model.

[0062] In the embodiment of the present invention, in order to determine whether the user currently has a need for image quality enhancement, the embodiment of the present invention also provides a possible implementation method:

[0063] a1: Output the image quality enhancement mark according to the determined target enhancement model.

[0064] a2: When an enabling operation for the image quality enhancement flag is received, it is determined that there is a need for image quality enhancement.

[0065] Based on the above configuration information, the terminal can be matched according to the configuration information. If there is information matching the terminal and the live broadcast room type, the image quality enhancement mark can be output, that is, the user-operable mark is displayed in the live broadcast room. For example, the operable mark is displayed in the live broadcast room in the form of a button, so that the user can choose to turn on or off the button according to their needs. If the user turns on the button, it can be considered that the user has a demand for image quality enhancement. If the user turns off the button, it can be determined that there is no demand for image quality enhancement.

[0066] It is understandable that if the terminal detects that there is no matching enhancement model configuration information, a picture quality enhancement logo will appear in the live broadcast room, but prompt information such as not supporting picture quality enhancement can be output to the user to facilitate the user to make timely adjustments.

[0067] In some embodiments, the image to be enhanced may be, for example, one of a plurality of continuous frames of a video. By performing more targeted image quality enhancement processing on each frame in the video, the overall subjective quality of the video is enhanced, and the user's experience of watching the video is improved.

[0068] Optionally, in the embodiment of the present invention, after obtaining the target enhancement model, the image to be enhanced may be enhanced. In a possible implementation, the above step S303 may be performed as follows. Figure 4 , Figure 4 A schematic flow chart of step S303 provided in an embodiment of the present invention:

[0069] S303-3: When it is determined that there is a need for image quality enhancement, an enhancement reasoner is created according to the target enhancement model.

[0070] In the implementation of the present invention, the enhancement reasoner can be understood as an enhancement instance, which is a set of algorithms with enhancement processing functions generated based on a determined target enhancement model. The input of the enhancement reasoner can be the image data to be enhanced, and the format of the output image data matches the format of the output image data.

[0071] S303-4, decoding the image to be enhanced, and obtaining an enhanced image according to the decoded image to be enhanced and the enhancement inference device.

[0072] In order to facilitate understanding of the above step S303-4, the following will take a PC terminal, an iOS terminal and an Android terminal as examples for explanation.

[0073] The implementation of step S303 - 4 may be as follows: when the device type is a first type device, the decoded image to be enhanced is input into the enhancement reasoner for processing, and the output image is used as the enhanced image.

[0074] In an embodiment of the present invention, the PC is a first-class device, and the data format of the image to be enhanced after being decoded by the PC is D3D11Texture, which can be directly input into the enhanced reasoner. The output data format is still D3D11Texture, which can be directly used as the input for subsequent rendering without additional processing, thereby improving the enhancement efficiency.

[0075] The implementation method of the above step S303-4 can be: when the device type is a second type device, extract the brightness channel data of the decoded image to be enhanced, input the brightness channel data into the enhancement reasoner for processing, and use the output image as the enhanced image.

[0076] In an embodiment of the present invention, the iOS terminal is a second type of device. The data format of the image to be enhanced after being decoded by the iOS terminal is PixelBuffer. The input to the enhancement inferencer is the extracted brightness channel data, and the output is the enhanced brightness channel data, and the data format is also PixelBuffer. The output PixelBuffer format data can be directly used for rendering.

[0077] The implementation method of the above step S303-4 can be: when the device type is a third-category device, the decoded image to be enhanced is encapsulated into data in a preset data format, and the data in the preset format is input into the enhancement reasoner, and the output data is drawn to the decoded image to be enhanced to obtain the enhanced image.

[0078] In an embodiment of the present invention, the Android terminal is a third type of device. In a possible implementation, the image decoder of the Android terminal can decode the input image source in combination with preset parameters, and the decoded image data can be encapsulated through a converter. The encapsulated data can be directly input into the enhanced reasoner for enhancement processing, and the corresponding buffer (enhanced data) is output. The enhanced data is then drawn to GL.Texture using OpenGL technology, and the obtained GL.Texture can be directly used as input for subsequent rendering.

[0079] From the above content, it can be seen that for different types of terminals, the input data format and output data format of the enhanced model are different. Figure 5 For example, Figure 5 A schematic diagram of data resources before and after processing of different types of terminals provided in an embodiment of the present invention.

[0080] In order to reduce the amount of data copied, each platform needs to use its decoded output resources as the input of the enhancement process, and the output of the enhancement process as the input of the rendering process. In the traditional enhancement process, it needs to consume 2 CPU-to-GPU copies and 2 GPU-to-CPU copies, resulting in high CPU / GPU usage and serious power consumption. Figure 5 It can be seen that the optimization of the enhanced process in the embodiment of the present invention is mainly reflected in the reduction of the number of data copies, for example:

[0081] The output of PC terminal decoding can be directly used as the input of the enhancement model, and the output of the enhancement model can be directly used as the input of rendering, so only 0 copies are made.

[0082] iOS terminals need to extract the brightness channel data before enhancement. The output PixelBuffer after enhancement can be used directly as rendering input without copying, so only one copy from CPU to GPU is performed.

[0083] On Android terminals, an additional copy process needs to be added to the enhanced output, which is to convert the CPU data into the Texture resources required for rendering. Therefore, only one CPU-to-GPU copy is performed.

[0084] Therefore, the embodiment of the present invention adopts enhancement models for different terminals, which can greatly reduce the amount of data copying in the enhancement processing flow, and prevent the problems of excessive CPU / GPU usage and severe power consumption.

[0085] Optionally, before the above step S303-3, the method may further include: Figure 6 The steps shown, Figure 6 A schematic flow chart of another implementation of step S303 provided in an embodiment of the present invention:

[0086] S303-1, determine whether there is a pre-generated enhanced reasoner.

[0087] If it exists, step S303 - 2 is executed to obtain the pre-generated enhanced reasoner.

[0088] If not present, execute step S303-3.

[0089] That is to say, before creating an enhanced reasoner according to the target enhanced model, it is also possible to first determine whether there is an enhanced reasoner that has been created. If not, a new enhanced reasoner can be created. If it exists, it can be used directly to prevent excessive creation and waste of image resources.

[0090] Optionally, before the above step S303, the following steps may also be included:

[0091] c1, determine whether the terminal has the target enhancement model.

[0092] c2, if present, loads the target augmentation model.

[0093] c3, if it does not exist, download the target enhancement model from the server and respond to the compilation operation for the target enhancement model.

[0094] The above steps c1 to c3 of the embodiment of the present invention are mainly for IOS terminals. It can be seen from the above content that the enhanced model corresponding to the IOS terminal is the MLCore model, which cannot exist in the static library. Therefore, if the terminal type is an IOS terminal, it is possible to first determine whether there is a model file in the terminal. When the model file does not exist, the switch on the terminal does not display the ability to download and compile files in the background. When the file exists, the file is loaded synchronously. This is the concept of synchronous loading and asynchronous compilation.

[0095] Optionally, the embodiment of the present invention may also perform rendering according to the enhanced data to achieve the effect of displaying the enhanced image on the terminal. Therefore, after the above step S303, the following steps may also be included: Figure 7 The steps shown, Figure 7 A schematic flow chart of another method for enhancing image quality provided by an embodiment of the invention:

[0096] S304, obtaining brightness channel data corresponding to the enhanced image to be enhanced.

[0097] S305: Rendering is performed based on the brightness channel data, and the rendered image is displayed.

[0098] It can be seen that in the rendering process, the embodiment of the present invention selects the brightness channel data corresponding to the enhanced image to be enhanced, and uses the enhanced brightness channel data to cover the original Y channel data source of the three channels of YUV (Y represents brightness, U and V represent chrominance), and then renders it to the screen to complete the rendering process. In other words, the rendered Y channel data comes from the enhanced Y channel data, and the rendered UV channel data comes from the UV channel data before enhancement.

[0099] The reason why the embodiment of the present invention chooses the brightness channel (Y channel) data for rendering is that the data volume of the traditional RGBA four-channel enhancement is w*h*4, which is large in data volume and high in computational complexity. In addition, in the process of model verification, it is found that the main enhancement point of the YUV three-channel enhancement lies in the Y channel rather than the UV channel. Therefore, the enhancement of the UV component can be abandoned and switched to the enhancement of the Y single-channel data. The enhancement speed has been optimized to meet the requirements of real-time enhancement on the terminal. At the same time, the data volume is reduced from the original w*h*4 to w*h, which greatly optimizes the computational complexity.

[0100] In order to implement each step in the above embodiment to achieve the image quality enhancement effect, the image quality enhancement method provided in the embodiment of the present invention can be executed in a hardware device or in the form of a software module. When the training method of the feature extraction model is implemented in the form of a software module, the embodiment of the present invention also provides an image quality enhancement device, see Figure 8 , Figure 8 This is a functional module diagram of an image quality enhancement device provided by an embodiment of the present invention. The image quality enhancement device 400 may include:

[0101] The acquisition module 410 is used to obtain the device type, live broadcast room type, live broadcast bit rate and enhanced model configuration information of the terminal.

[0102] The determination module 420 is used to determine the target enhancement model according to the device type, the live broadcast room type, the live broadcast bit rate and the enhancement model configuration information.

[0103] The enhancement module 430 is used to enhance the image quality of the image to be enhanced according to the target enhancement model when it is determined that there is a need for image quality enhancement.

[0104] It is understandable that the obtaining module 410, the determining module 420 and the enhancing module 430 provided in the embodiment of the present invention can be executed in coordination. Figure 3 The various steps in the process are performed to achieve the effect of image quality enhancement.

[0105] In an optional implementation, the enhancement module 430 is specifically used to: when it is determined that there is a need for image quality enhancement, create an enhancement reasoner according to a target enhancement model; decode the image to be enhanced, and obtain an enhanced image according to the decoded image to be enhanced and the enhancement reasoner.

[0106] In an optional embodiment, the enhancement module 430 is further specifically used to: extract brightness channel data of the decoded image to be enhanced, and input the brightness channel data into the enhancement reasoner for processing, and use the output image as the enhanced image; or, input the decoded image to be enhanced into the enhancement reasoner for processing, and use the output image as the enhanced image; or, encapsulate the decoded image to be enhanced into data in a preset data format, and input the data in the preset format into the enhancement reasoner, draw the output data to the decoded image to be enhanced, and obtain the enhanced image.

[0107] In an optional embodiment, the determination module 420 is further used to: determine whether there is a pre-generated enhanced reasoner; if so, obtain the pre-generated enhanced reasoner; if not, execute the step of creating an enhanced reasoner according to the target enhanced model.

[0108] In an optional implementation, the image quality enhancement device 400 may further include a rendering module for acquiring brightness channel data corresponding to the enhanced image to be enhanced; performing rendering based on the brightness channel data, and displaying the rendered image.

[0109] In an optional implementation, the determination module 420 is further used to determine whether the terminal has a target enhancement model; if so, load the target enhancement model; if not, download the target enhancement model from the server and respond to a compilation operation for the target enhancement model.

[0110] In an optional implementation, the determination module 420 is further configured to output a picture quality enhancement flag according to the determined target enhancement model; and when an enabling operation for the picture quality enhancement flag is received, it is determined that there is a need for picture quality enhancement.

[0111] The embodiment of the present invention further provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the image quality enhancement method as described in any of the above embodiments is implemented. The computer readable storage medium can be, but is not limited to, a USB flash drive, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a magnetic disk, or an optical disk, and other media that can store program codes.

[0112] It should be understood that the apparatus and method disclosed in the present application can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0114] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes. It should be noted that in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0115] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

Claims

1. A method for enhancing image quality, characterized in that: Applied to a terminal, the method comprises: Obtaining the device type, live broadcast room type, live broadcast bit rate, and enhanced model configuration information of the terminal; the device type is differentiated according to the operating system of the terminal, including first-class devices, second-class devices, and third-class devices; The target enhancement model is determined according to the device type, the live broadcast room type, the live broadcast bit rate and the enhancement model configuration information; the target enhancement model corresponding to the terminal with different operating systems is designed according to the data format of the image source decoded and output by the mobile terminal; when it is determined that there is a need for image quality enhancement, the image to be enhanced is enhanced according to the target enhancement model; wherein, during the enhancement process, the input of the target enhancement model of the second type of device is the bright channel data decoded and output by the second type of device; the decoded output of the first type of device is directly used as the input of the target enhancement model; the input of the target enhancement model of the third type of device is the decoded output of the third type of device encapsulated by the converter, and the input and output data formats of the target enhancement model are the same.

2. The image quality enhancement method according to claim 1, characterized in that: When it is determined that there is a need for image quality enhancement, the image to be enhanced is enhanced according to the target enhancement model, including: When it is determined that there is a need for image quality enhancement, an enhancement reasoner is created according to the target enhancement model; The image to be enhanced is decoded, and an enhanced image is obtained according to the decoded image to be enhanced and the enhancement reasoner.

3. The image quality enhancement method according to claim 2, characterized in that: Decoding the image to be enhanced, and obtaining an enhanced image according to the decoded image to be enhanced and the enhancement reasoner, comprising: When the device type is the second type of device, extracting brightness channel data of the decoded image to be enhanced, inputting the brightness channel data into the enhancement reasoner for processing, and using the output image as the enhanced image; When the device type is the first type of device, the decoded image to be enhanced is input into the enhancement reasoner for processing, and the output image is used as the enhanced image; When the device type is the third type device, the decoded image to be enhanced is encapsulated as data in a preset data format, and the data in the preset data format is input into the enhanced reasoner, and the output data is drawn to obtain the enhanced image.

4. The image quality enhancement method according to claim 2, characterized in that: Before creating an enhanced reasoner according to the target enhanced model, the method further includes: determining whether a pre-generated enhanced reasoner exists; If it exists, the pre-generated enhanced reasoner will be obtained; If not, the step of creating an enhanced reasoner according to the target enhanced model is executed.

5. The image quality enhancement method according to claim 1, characterized in that: When it is determined that there is a need for image quality enhancement, after enhancing the image quality of the image to be enhanced according to the target enhancement model, the method further includes: Acquire brightness channel data corresponding to the enhanced image to be enhanced; Rendering is performed based on the brightness channel data, and the rendered image is displayed.

6. The image quality enhancement method according to claim 1, characterized in that: Before performing image quality enhancement on the image to be enhanced according to the target enhancement model, the method further includes: Determining whether the target enhancement model exists in the terminal; If it exists, then load the target enhancement model; If it does not exist, the target enhancement model is downloaded from the server, and a compilation operation for the target enhancement model is responded to.

7. The image quality enhancement method according to claim 1, characterized in that: When it is determined that there is a need for image quality enhancement, before performing image quality enhancement on the image to be enhanced according to the target enhancement model, the method further includes: Outputting the image quality enhancement mark according to the determined target enhancement model; When an enabling operation for the image quality enhancement flag is received, it is determined that the image quality enhancement requirement exists.

8. An image quality enhancement device, characterized in that: include: An acquisition module, used to obtain the device type, live broadcast room type, live broadcast bit rate and enhanced model configuration information of the terminal; the device type is distinguished according to the operating system of the terminal, including first-class devices, second-class devices and third-class devices; A determination module, used to determine a target enhancement model according to the device type, the live broadcast room type, the live broadcast bit rate and the enhancement model configuration information; wherein the target enhancement model corresponding to the terminal of different operating systems is designed according to the data format of the image source decoded and output by the mobile terminal; An enhancement module is used to enhance the image quality of the image to be enhanced according to the target enhancement model when it is determined that there is a need for image quality enhancement; wherein the input of the target enhancement model of the second type of device is the bright channel data decoded and output by the second type of device; the decoded output of the first type of device is directly used as the input of the target enhancement model; the input of the target enhancement model of the third type of device is the decoded output of the third type of device encapsulated by a converter, and the input and output data formats of the target enhancement model are the same.

9. A terminal, characterized in that , comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the method described in any one of claims 1-7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images