Image processing method and device

Abstract

The embodiment of the invention provides an image processing method. The image processing method comprises the following steps: decoding a received video file into a video frame in a first format; uploading the first format video frame to a display memory of the terminal device and binding the first format video frame as a first format texture; converting the first format texture into an RGB texture; performing image quality enhancement on the RGB texture to obtain the RGB texture after image quality enhancement; and rendering the RGB texture after image quality enhancement on a screen of the terminal equipment. According to the technical scheme provided by the embodiment of the invention, image quality enhancement can be realized at the user terminal, the image quality during video playing is improved, meanwhile, image quality enhancement at the server side is avoided, the server cost is saved, and extra code rate cost is not brought.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] The core goal of picture quality enhancement is to improve image details, color, clarity and contrast to achieve a better visual experience.

[0003] At present, the mainstream picture quality enhancement technology solution is the picture quality enhancement algorithm based on deep learning. The picture quality enhancement algorithm based on deep learning is mainly used in the cloud, which can use the cloud's more computing power to provide resource support and achieve better picture quality enhancement effects.

[0004] However, achieving image quality enhancement in the cloud will increase cloud procurement and maintenance costs, and will also significantly increase the bit rate of the original film, causing the video platform to pay additional operating costs caused by the increased bit rate.

[0005] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0006] The embodiments of the present application provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0007] One aspect of an embodiment of the present application provides an image processing method, which is applied to a terminal device, and the method includes: Decoding the received video file into a first format video frame; Uploading the first format video frame to the display memory of the terminal device and binding it as a first format texture; Convert the first format texture into an RGB texture; Performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality; The RGB texture with enhanced image quality is rendered on the screen of the terminal device.

[0008] Optionally, the performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality includes: The RGB texture is subjected to image quality enhancement by adopting a target algorithm to obtain an RGB texture after image quality enhancement, wherein the target algorithm at least includes part of an image interpolation algorithm, a sharpening algorithm, a saturation enhancement algorithm, a filtering algorithm, a super-resolution algorithm, and a deep learning algorithm.

[0009] Optionally, the performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality includes: Determining the picture quality of the video file; When the picture quality meets the first preset condition, using a first algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality does not meet the first preset condition, a second algorithm is used to enhance the image quality of the RGB texture to obtain an enhanced RGB texture, and the complexity of the second algorithm is less than that of the first algorithm.

[0010] Optionally, determining the picture quality of the video file includes: Receiving a picture quality evaluation result of the video file sent by the server; The picture quality of the video file is determined based on the picture quality evaluation result.

[0011] Optionally, the performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality includes: Receiving algorithm information sent by the server for the video file, where the algorithm information is used to describe an algorithm used for image quality enhancement; A third algorithm corresponding to the algorithm information is determined, and the third algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0012] Optionally, the performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality includes: Determining a target performance indicator of the terminal device; When the target performance indicator meets the second preset condition, using a fourth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the target performance indicator does not meet the second preset condition, a fifth algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement, and the complexity of the fifth algorithm is less than that of the fourth algorithm.

[0013] Optionally, the performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality includes: Determining the picture quality of the video file and the target performance index of the terminal device; When the picture quality and the target performance index meet the third preset condition, using a sixth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality or the target performance index does not meet the third preset condition, the seventh algorithm is used to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement, and the complexity of the seventh algorithm is less than that of the sixth algorithm.

[0014] Another aspect of an embodiment of the present application provides an image processing apparatus, applied to a terminal device, the apparatus comprising: A decoding module, used for decoding the received video file into a first format video frame; An uploading module, used for uploading the first format video frame to the display memory of the terminal device and binding it as a first format texture; A conversion module, used for converting the first format texture into an RGB texture; An enhancement module, used for enhancing the image quality of the RGB texture to obtain an RGB texture with enhanced image quality; A rendering module is used to render the RGB texture with enhanced image quality on the screen of the terminal device.

[0015] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0016] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0017] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0018] The above technical solution adopted in the embodiment of the present application may have the following advantages: By decoding the received video text into a first format video frame on a terminal device, uploading the first format video frame to a display memory of the terminal device and binding it as a first format texture, converting the first format texture into an RGB texture, enhancing the RGB texture to obtain an RGB texture with enhanced image quality, and rendering the RGB texture with enhanced image quality on a screen of the terminal device, image quality enhancement can be achieved on a user terminal, thereby improving the picture quality during video playback. At the same time, it can avoid the need for high-performance configuration for image quality enhancement on a server side, thereby saving server costs. Furthermore, image quality enhancement on a user terminal will not increase the bit rate of the video and distribution, and will not bring about additional bit rate costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0020] Figure 1 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown; Figure 2 The following is a schematic diagram showing the principle of an image processing method according to the first embodiment of the present application; Figure 3 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 4 Schematically shows Figure 3 Flow chart of sub-steps of step S200; Figure 5 Schematically shows Figure 1 Another sub-step flow chart of step S106; Figure 6 Schematically shows Figure 1 Another sub-step flow chart of step S106; Figure 7 Schematically shows Figure 1 Another sub-step flow chart of step S106; Figure 8 A block diagram schematically shows an image processing device according to the second embodiment of the present application; and Fig. 9 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0024] First, the following terms are explained: Video decoding: refers to the process of restoring compressed digital video data to its original image sequence that can be directly viewed by the human eye. In simple terms, it is to "decompress" the compressed video file and restore it to a state that can be played.

[0025] Off-Screen Rendering: refers to the process of rendering in a buffer outside the screen. The rendering result will not be displayed on the screen immediately, but saved in a texture or frame buffer in memory.

[0026] YUV: is a color encoding method, mainly used for video compression and transmission. It divides color information into brightness (Y) and chrominance (U, V). Compared with the RGB model, YUV is more in line with the visual characteristics of the human eye and can effectively reduce bandwidth requirements.

[0027] YcbCr: It is a commonly used color encoding method, especially widely used in video processing. It decomposes the image signal into two parts: brightness (Y) and chrominance (Cb, Cr). Cb represents the blue chrominance, that is, the difference between blue and yellow. Cr represents the red chrominance, that is, the difference between red and cyan.

[0028] Image quality enhancement: refers to improving the visual effect of an image through a series of technical processing, highlighting the information of interest in the image, suppressing unnecessary information, thereby improving the quality of the image and making it more suitable for human eye observation or computer analysis.

[0029] Texture: It is a very important visual feature in an image, which reflects the structural organization and arrangement properties of the surface of an object. Simply put, texture is a pattern or structure on the surface of an object.

[0030] Bit rate: Simply put, it is the amount of data transmitted per unit time. In digital signal processing, it means the number of bits transmitted per second.

[0031] Peak signal-to-noise ratio (PSNR): is a commonly used image quality evaluation indicator. It is mainly used to measure the similarity between two images and is often used to evaluate the performance of algorithms such as image compression, image restoration, and image denoising.

[0032] Structural Similarity Index (SSIM): is an indicator to measure the similarity between two images. Compared with traditional indicators such as mean square error (MSE), SSIM is more in line with the perceptual characteristics of the human visual system and can better reflect the degree of image distortion.

[0033] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: At present, the more mainstream technical solution for image quality enhancement is the image quality enhancement algorithm based on deep learning. Depending on the specific business scenario, the image quality enhancement algorithm based on deep learning may be a combination of super-resolution algorithm, general restoration algorithm, face restoration algorithm and denoising algorithm.

[0034] Image quality enhancement algorithms based on deep learning are mainly used in the cloud, because the cloud can deploy more GPU computing power to provide computing power support for advanced algorithms to achieve better image quality enhancement effects.

[0035] However, to achieve image quality enhancement in the cloud, it is necessary to purchase expensive GPUs to implement high-performance servers, thereby achieving image quality enhancement for a large number of tasks, which makes the procurement and maintenance costs of the server high. In addition, cloud-based image quality enhancement usually involves processing the original film. After enhancement, the bit rate of the original film will increase significantly, causing the distribution bit rate to increase as well. The video platform needs to pay additional operating costs for the increase in bit rate.

[0036] To this end, the embodiment of the present application provides an image processing technical solution. In this technical solution, by implementing image quality enhancement on the user terminal, the picture quality during video playback is improved. At the same time, image quality enhancement on the server side is avoided, so that the server side does not need to perform high-performance configuration, saving server costs; and image quality enhancement on the user terminal will not increase the bit rate of the video and distribution, and will not bring additional bit rate costs. See below for details.

[0037] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0038] Embodiment 1 Figure 1The flowchart of the image processing method according to the first embodiment of the present application is schematically shown.

[0039] like Figure 1 As shown, the image processing method is applied to a terminal device, and may specifically include steps S100 to S108, wherein: Step S100: decoding the received video file into a first format video frame.

[0040] Step S102: uploading the first format video frame to the display memory of the terminal device and binding it to a first format texture.

[0041] Step S104: convert the first format texture into an RGB texture.

[0042] Step S106, enhancing the image quality of the RGB texture to obtain an RGB texture with enhanced image quality.

[0043] Step S108, rendering the RGB texture with enhanced image quality on the screen of the terminal device.

[0044] The image processing method provided in this embodiment decodes the received video text into a first format video frame at the terminal device, uploads the first format video frame to the display memory of the terminal device and binds it as a first format texture, converts the first format texture into an RGB texture, performs image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality, and renders the RGB texture with enhanced image quality on the screen of the terminal device. This can achieve image quality enhancement on the user terminal and improve the picture quality during video playback. At the same time, it can avoid the need for high-performance configuration for image quality enhancement on the server side, thereby saving the cost of the server. Furthermore, performing image quality enhancement on the user terminal will not increase the bit rate of the video and distribution, and will not bring additional bit rate costs.

[0045] The following combination Figure 1 , each step in steps S100~S108 and other optional steps are explained in detail.

[0046] Step S100 , decoding the received video file into a first format video frame.

[0047] The first format may be a format suitable for video compression and transmission, such as a YUV format or a YcbCr format.

[0048] Specifically, the terminal device receives a video file sent by the server. When the video file is received, a decoder (such as FFmpeg library) may be used on the terminal device to decode the received video file to obtain a video frame in a first format.

[0049] Step S102, uploading the first format video frame to the display memory of the terminal device and binding it as a first format texture.

[0050] Taking the first format as YUV as an example, the YUV video frame may be uploaded to the display memory of the graphics card of the terminal device and bound as a YUV texture.

[0051] Specifically, you can create an OpenGL texture object, set texture parameters (such as texture format, texture filtering method, and texture wrapping method), and then upload the image data of the YUV video frame to the created texture object and bind it as a YUV texture. When uploading the texture, you can upload the Y, U, and V components as independent textures or merged texture formats through OpenGL's glTexImage2D interface. Then, through multi-texture binding, the Y, U, and V components are integrated into the same fragment shader for subsequent processing on the GPU.

[0052] Step S104 , convert the first format texture to RGB texture.

[0053] Continuing to take YUV as the first format as an example, specifically, the YUV texture can be converted into an RGB texture through a fragment shader. Since the display of the terminal device generally requires the input to be in RGB format, the video frame in the YUV format cannot be directly displayed on the screen of the terminal device. Converting the YUV format to the RGB format can meet the input requirements of the display. When converting YUV to an RGB texture through a fragment shader, the corresponding RGB value can be calculated using the YUV value of each pixel using the fragment shader.

[0054] Step S106 , performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality.

[0055] Specifically, some image quality enhancement algorithms may be used to enhance the image quality of the converted RGB texture, thereby obtaining an RGB texture with enhanced image quality.

[0056] In an optional embodiment, step S106 may include: using a target algorithm to enhance the image quality of the RGB texture to obtain an RGB texture with enhanced image quality, the target algorithm at least including some algorithms such as image interpolation algorithm, sharpening algorithm, saturation enhancement algorithm, filtering algorithm, super-resolution algorithm and deep learning algorithm.

[0057] When the computing power of the terminal device is strong, a picture quality enhancement algorithm with higher computational complexity can be used; when the computing power of the terminal device is relatively limited, a picture quality enhancement algorithm with lower computational complexity can be used. For example, for terminal devices with strong computing power such as personal computers (PCs), a picture quality enhancement algorithm with higher computational complexity, such as a super-resolution algorithm, can be used; for terminal devices with limited computing power such as mobile terminals, a picture quality enhancement algorithm with lower computational complexity, such as an image interpolation algorithm, a sharpening algorithm, or a saturation enhancement algorithm, can be used.

[0058] Step S108 , rendering the RGB texture with enhanced image quality on the screen of the terminal device.

[0059] Specifically, the enhanced RGB texture can be bound to a full-screen rectangle through the OpenGL rendering pipeline, and the entire image can be rendered to a frame buffer directly connected to the screen at one time by drawing the rectangle, and then the image can be transmitted to the display device through the frame buffer; during the refresh cycle, the content of the frame buffer is scanned and output to the screen to complete the display of one video frame.

[0060] like Figure 2 As shown, it is a principle example diagram of the image processing method of the embodiment of the present application. As shown in the figure, the terminal device decodes the received video file to obtain a YUV video frame, which is uploaded to the display content of the terminal device and bound as an OpenGL texture; then, the YUV texture is converted into an RGB texture through a fragment shader with a color space conversion function; the RGB texture is enhanced to obtain an RGB texture with enhanced image quality; finally, the RGB texture with enhanced image quality is drawn into an RGB image through the OpenGL rendering pipeline and rendered on the screen.

[0061] It is understandable that the original picture quality of each video may be different, for example, some may have better picture quality and some may have worse picture quality; and the computing performance of each terminal device may also be different, some may have better computing performance and some may have more limited computing performance. As to which image quality enhancement algorithm to adopt to obtain a better image quality enhancement effect, the original picture quality of the video and the computing performance of the terminal device are factors that need to be considered. Therefore, different image quality enhancement algorithms may be adopted according to the original picture quality and / or computing performance of each video. Some exemplary solutions are provided below.

[0062] In an optional embodiment, if Figure 3 As shown, in step S106, the image quality of the RGB texture is enhanced to obtain the RGB texture after the image quality enhancement, which may include: Step S200, determining the picture quality of the video file.

[0063] Step S202: When the picture quality meets the first preset condition, a first algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0064] Step S204, when the picture quality does not meet the first preset condition, adopt a second algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement, and the complexity of the second algorithm is less than that of the first algorithm.

[0065] When determining the picture quality of a video file, the server or the terminal device may determine the picture quality. When determining the picture quality of a video file, an index related to the picture quality in the video file may be evaluated, and the picture quality of the video file may be determined based on the evaluation result. For example, the PSNR or SSIM of the frame image of the video file may be calculated, and the picture quality of the video file may be determined based on the calculation result.

[0066] The first preset condition may be a condition indicating good picture quality, for example, a related value of the picture quality may be greater than a preset threshold. The first algorithm may be an algorithm with a higher computational complexity among image interpolation algorithms, sharpening algorithms, saturation enhancement algorithms, filtering algorithms, super-resolution algorithms, and deep learning algorithms, and the second algorithm may be an algorithm with a lower computational complexity among image interpolation algorithms, sharpening algorithms, saturation enhancement algorithms, filtering algorithms, super-resolution algorithms, and deep learning algorithms.

[0067] Specifically, the picture of the video file can be evaluated by the server or the terminal device according to some preset indicators, and the picture quality of the video file can be determined according to the evaluation result; when the picture quality meets the first preset condition, the first algorithm with higher computational complexity is used to enhance the picture quality of the RGB texture to obtain the RGB texture with enhanced picture quality; when the picture quality does not meet the first preset condition, the second algorithm with lower computational complexity is used to enhance the picture quality of the RGB texture to obtain the RGB texture with enhanced picture quality.

[0068] Since when the picture quality of the video file is good, the use of a picture quality enhancement algorithm with a lower computational complexity can no longer achieve a significant picture quality enhancement effect, therefore when the picture quality is good, using a picture quality enhancement algorithm with a higher computational complexity can improve the picture quality enhancement effect; and when the picture quality of the video file is poor, the use of a picture quality enhancement algorithm with a lower computational complexity can achieve a significant picture quality enhancement effect, therefore when the picture quality is poor, using a picture quality enhancement algorithm with a lower computational complexity can also achieve a better picture quality enhancement effect, and at the same time can also reduce the computational complexity and improve the real-time performance of image processing.

[0069] In an optional embodiment, if Figure 4 As shown, in step S200, determining the picture quality of the video file may include: Step S300: receiving a picture quality evaluation result of the video file sent by the server.

[0070] Step S302: determining the picture quality of the video file based on the picture quality evaluation result.

[0071] Specifically, before sending the video file to the terminal device, the server can use a picture quality assessment method or a multimodal large model to evaluate the picture quality of the video file to obtain a picture quality assessment result; when sending the video file to the terminal device, the picture quality assessment result is also sent to the terminal device; after receiving the picture quality assessment result of the video file sent by the server, the terminal device determines the picture quality of the currently received video file according to the picture quality assessment result.

[0072] In this embodiment, by receiving the image quality assessment result of the video file sent by the server and determining the picture quality of the video file based on the image quality assessment result, the server can share the task of image quality assessment, thereby saving more computing power for the terminal device to enhance the image quality of the video file; at the same time, the image quality assessment of the video file by the server can also obtain a more accurate image quality assessment result with the help of the server's more powerful computing power.

[0073] In an optional embodiment, if Figure 5 As shown, in step S106, the image quality of the RGB texture is enhanced to obtain the RGB texture after the image quality enhancement, which may include: Step S400: receiving algorithm information sent by the server for the video file, wherein the algorithm information is used to describe the algorithm used for image quality enhancement.

[0074] Step S402: determine a third algorithm corresponding to the algorithm information, and use the third algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0075] Among them, the third algorithm can be one or more algorithms selected from the group consisting of an image interpolation algorithm, a sharpening algorithm, a saturation enhancement algorithm, a filtering algorithm, a super-resolution algorithm, and a deep learning algorithm.

[0076] Specifically, the server can evaluate the image quality of the video file, and then determine the algorithm information corresponding to the video file based on the result of the image quality evaluation, and then the server sends the algorithm information of the video file to the terminal device; when the terminal device performs image quality enhancement on the RGB texture, it determines the third algorithm used for image quality enhancement based on the algorithm information sent by the server, and then uses the third algorithm to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement. Among them, a mapping table of the image quality evaluation results and the corresponding algorithms can be maintained in advance on the server, so that after obtaining the image quality evaluation results, the server can determine the corresponding algorithm according to the mapping table and generate algorithm information according to the corresponding algorithm. In addition, the terminal device can include multiple image quality enhancement algorithms to select some algorithms for image quality enhancement based on the algorithm information.

[0077] For example, if the server performs image quality evaluation on a video file and obtains an image quality evaluation result; then determines according to the mapping table that the image quality enhancement algorithm corresponding to the video file is an image interpolation algorithm and a sharpening algorithm, then obtains algorithm information according to the image interpolation algorithm and the sharpening algorithm, and sends the algorithm information to the terminal device; when the terminal device performs image quality enhancement on the RGB texture, it determines according to the algorithm information that the third algorithm corresponding to the image quality enhancement is an image interpolation algorithm and a sharpening algorithm, then uses the image interpolation algorithm and the sharpening algorithm to enhance the image quality of the RGB texture, and obtains the RGB texture after image quality enhancement.

[0078] In this embodiment, by receiving algorithm information sent by the server to the video file for describing the algorithm used for image quality enhancement, a third algorithm corresponding to the algorithm information is determined, and the third algorithm is used to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement. The server can determine the corresponding image quality enhancement algorithm after performing image quality evaluation on the video file, so that different image quality enhancement algorithms can be used according to the different image quality of the video file, thereby improving the flexibility and accuracy of image quality enhancement; at the same time, by providing the algorithm used for image quality enhancement through the server, more computing power can be saved for the terminal device for image quality enhancement.

[0079] In an optional embodiment, if Figure 6 As shown, in step S106, the image quality of the RGB texture is enhanced to obtain the RGB texture after the image quality enhancement, which may include: Step S500: determining a target performance indicator of the terminal device.

[0080] Step S502: When the target performance index meets the second preset condition, a fourth algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0081] Step S504, when the target performance indicator does not meet the second preset condition, adopt a fifth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement, and the complexity of the fifth algorithm is less than that of the fourth algorithm.

[0082] Among them, the target performance indicator may include performance-related data such as CPU, GPU and memory, such as the model and frequency of the CPU and GPU, memory size, etc., and may also include the utilization rate of the CPU, GPU or memory, etc. The second preset condition may be a condition indicating that the target performance indicator is good, such as the CPU utilization rate is less than a certain threshold. The fourth algorithm may be an algorithm with higher computational complexity among image interpolation algorithms, sharpening algorithms, saturation enhancement algorithms, filtering algorithms, super-resolution algorithms and deep learning algorithms, and the fifth algorithm may be an algorithm with lower computational complexity among image interpolation algorithms, sharpening algorithms, saturation enhancement algorithms, filtering algorithms, super-resolution algorithms and deep learning algorithms.

[0083] Specifically, the terminal device can obtain relevant data of the current target performance indicator and determine the target performance indicator based on these relevant data; then compare these target performance indicators with the second preset condition. If the target performance indicator meets the second preset condition, a fourth algorithm with higher computational complexity is used to enhance the image quality of the RGB texture to obtain an RGB texture with enhanced image quality; if the target performance indicator does not meet the second preset condition, a fifth algorithm with lower computational complexity is used to enhance the image quality of the RGB texture to obtain an RGB texture with enhanced image quality.

[0084] In this embodiment, by determining the target performance indicator of the terminal device, when the target performance indicator meets the second preset condition, the fourth algorithm with higher computational complexity is used to enhance the image quality of the RGB texture; when the target performance indicator does not meet the second preset condition, the fifth algorithm with lower computational complexity is used to enhance the image quality of the RGB texture. It is possible to automatically select image quality enhancement algorithms with different computational complexities according to the different computing performance of the terminal device, thereby improving the flexibility of image quality enhancement. Better image quality enhancement can be obtained when the computing performance is good, and when the computing performance is poor, it is also possible to avoid using algorithms with too high computational complexity to affect the real-time performance of the video.

[0085] In an optional embodiment, if Figure 7 As shown, in step S106, the image quality of the RGB texture is enhanced to obtain the RGB texture after the image quality enhancement, which may include: Step S600, determining the picture quality of the video file and the target performance index of the terminal device.

[0086] Step S602: When the picture quality and the target performance index meet the third preset condition, a sixth algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0087] Step S604, when the picture quality or the target performance index does not meet the third preset condition, the seventh algorithm is used to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement, and the complexity of the seventh algorithm is less than that of the sixth algorithm.

[0088] The third preset condition may be a condition indicating that the picture quality is good and the target performance is good. The sixth algorithm may be an algorithm with higher computational complexity among the image interpolation algorithm, sharpening algorithm, saturation enhancement algorithm, filtering algorithm, super-resolution algorithm and deep learning algorithm, and the seventh algorithm may be an algorithm with lower computational complexity among the image interpolation algorithm, sharpening algorithm, saturation enhancement algorithm, filtering algorithm, super-resolution algorithm and deep learning algorithm.

[0089] Specifically, the terminal device determines the picture quality of the video file and the target performance indicator of the terminal device; when the picture quality and the target performance indicator meet the third preset condition, the sixth algorithm with higher computational complexity is used to enhance the image quality of the RGB texture to obtain the RGB texture with enhanced image quality; when the picture quality does not meet the third preset condition or the target performance indicator does not meet the third preset condition, the seventh algorithm with lower computational complexity is used to enhance the image quality of the RGB texture to obtain the RGB texture with enhanced image quality.

[0090] In this embodiment, by determining the picture quality of the video file and the target performance index of the terminal device, when the picture quality and the target performance index meet the third preset condition, the sixth algorithm with higher computational complexity is used to enhance the image quality of the RGB texture, otherwise the seventh algorithm with lower computational complexity is used to enhance the image quality of the RGB texture. The picture quality of the video file and the performance index of the terminal device can be combined to select a suitable image quality enhancement algorithm to achieve image quality enhancement, so that the adopted image quality enhancement algorithm is adapted to the image quality of the video file and the performance of the terminal device, while maximizing the image quality enhancement effect and ensuring the real-time nature of the video.

[0091] Embodiment 2 Figure 8The block diagram of the image processing device according to the second embodiment of the present application is schematically shown. The device is applied to a terminal device and can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 8 As shown, the apparatus 700 may include a decoding module 710, an uploading module 720, a conversion module 730, an enhancement module 740 and a rendering module 750, wherein: The decoding module 710 is used to decode the received video file into a first format video frame; An uploading module 720, configured to upload the first format video frame to a display memory of the terminal device and bind it as a first format texture; A conversion module 730, configured to convert the first format texture into an RGB texture; An enhancement module 740 is used to enhance the image quality of the RGB texture to obtain an RGB texture with enhanced image quality; The rendering module 750 is used to render the RGB texture with enhanced image quality on the screen of the terminal device.

[0092] In an optional embodiment, the enhancement module 740 is further configured to: The RGB texture is subjected to image quality enhancement by adopting a target algorithm to obtain an RGB texture after image quality enhancement, wherein the target algorithm at least includes part of an image interpolation algorithm, a sharpening algorithm, a saturation enhancement algorithm, a filtering algorithm, a super-resolution algorithm, and a deep learning algorithm.

[0093] In an optional embodiment, the enhancement module 740 is further configured to: Determining the picture quality of the video file; When the picture quality meets the first preset condition, using a first algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality does not meet the first preset condition, a second algorithm is used to enhance the image quality of the RGB texture to obtain an enhanced RGB texture, and the complexity of the second algorithm is less than that of the first algorithm.

[0094] In an optional embodiment, the enhancement module 740 is further configured to: Receiving a picture quality evaluation result of the video file sent by the server; The picture quality of the video file is determined based on the picture quality evaluation result.

[0095] In an optional embodiment, the enhancement module 740 is further configured to: Receiving algorithm information sent by the server for the video file, where the algorithm information is used to describe an algorithm used for image quality enhancement; A third algorithm corresponding to the algorithm information is determined, and the third algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

[0096] In an optional embodiment, the enhancement module 740 is further configured to: Determining a target performance indicator of the terminal device; When the target performance indicator meets the second preset condition, using a fourth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the target performance indicator does not meet the second preset condition, a fifth algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement, and the complexity of the fifth algorithm is less than that of the fourth algorithm.

[0097] In an optional embodiment, the enhancement module 740 is further configured to: Determining the picture quality of the video file and the target performance index of the terminal device; When the picture quality and the target performance index meet the third preset condition, using a sixth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality or the target performance index does not meet the third preset condition, the seventh algorithm is used to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement, and the complexity of the seventh algorithm is less than that of the sixth algorithm.

[0098] Embodiment 3 Fig. 9 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the image processing method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of image processing methods, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0099] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0100] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0101] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0102] In this embodiment, the image processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0103] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the image processing method in the embodiment are implemented.

[0104] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the image processing method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0105] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0106] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0107] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a terminal device, the method comprises: Decoding the received video file into a first format video frame; Uploading the first format video frame to the display memory of the terminal device and binding it as a first format texture; Convert the first format texture into an RGB texture; Performing image quality enhancement on the RGB texture to obtain an RGB texture with enhanced image quality; The RGB texture with enhanced image quality is rendered on the screen of the terminal device.

2. The method according to claim 1, characterized in that The step of enhancing the image quality of the RGB texture to obtain the enhanced RGB texture includes: The RGB texture is subjected to image quality enhancement by adopting a target algorithm to obtain an RGB texture after image quality enhancement, wherein the target algorithm at least includes part of an image interpolation algorithm, a sharpening algorithm, a saturation enhancement algorithm, a filtering algorithm, a super-resolution algorithm, and a deep learning algorithm.

3. The method according to claim 1, characterized in that The step of enhancing the image quality of the RGB texture to obtain the enhanced RGB texture includes: Determining the picture quality of the video file; When the picture quality meets the first preset condition, using a first algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality does not meet the first preset condition, a second algorithm is used to enhance the image quality of the RGB texture to obtain an enhanced RGB texture, and the complexity of the second algorithm is less than that of the first algorithm.

4. The method according to claim 3, characterized in that Determining the picture quality of the video file includes: Receiving a picture quality evaluation result of the video file sent by the server; The picture quality of the video file is determined based on the picture quality evaluation result.

5. The method according to claim 1, characterized in that The step of enhancing the image quality of the RGB texture to obtain the enhanced RGB texture includes: Receiving algorithm information sent by the server for the video file, where the algorithm information is used to describe an algorithm used for image quality enhancement; A third algorithm corresponding to the algorithm information is determined, and the third algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement.

6. The method according to claim 1, characterized in that The step of enhancing the image quality of the RGB texture to obtain the enhanced RGB texture includes: Determining a target performance indicator of the terminal device; When the target performance indicator meets the second preset condition, using a fourth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the target performance indicator does not meet the second preset condition, a fifth algorithm is used to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement, and the complexity of the fifth algorithm is less than that of the fourth algorithm.

7. The method according to claim 1, characterized in that The step of enhancing the image quality of the RGB texture to obtain the enhanced RGB texture includes: Determining the picture quality of the video file and the target performance index of the terminal device; When the picture quality and the target performance index meet the third preset condition, using a sixth algorithm to enhance the image quality of the RGB texture to obtain an RGB texture after image quality enhancement; When the picture quality or the target performance index does not meet the third preset condition, the seventh algorithm is used to enhance the image quality of the RGB texture to obtain the RGB texture after image quality enhancement, and the complexity of the seventh algorithm is less than that of the sixth algorithm.

8. An image processing device, characterized in that: Applied to a terminal device, the device comprises: A decoding module, used for decoding the received video file into a first format video frame; An uploading module, used for uploading the first format video frame to the display memory of the terminal device and binding it as a first format texture; A conversion module, used for converting the first format texture into an RGB texture; An enhancement module, used for enhancing the image quality of the RGB texture to obtain an RGB texture with enhanced image quality; A rendering module is used to render the RGB texture with enhanced image quality on the screen of the terminal device.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images