Image processing model training method, image quality enhancing method and electronic equipment

Abstract

The invention provides an image processing model training method, an image quality enhancement method and electronic equipment, and relates to the technical field of image processing. After an electronic device obtains a sample image set with relatively high quality, for each sample original image in the sample image set, the quality of the sample original image is degraded to obtain an image with relatively low quality corresponding to the sample original image, that is, a sample degraded image corresponding to the sample original image is obtained; therefore, accurate determination of the training set is realized. Afterwards, the electronic equipment can utilize the training set to train a preset model, so that the preset model learns how to restore the sample degraded image with lower quality to the corresponding sample original image with higher quality, and an image processing model is obtained. The image processing model can predict the quality adjustment parameter corresponding to the image in the video stream, so that the quality of the image is enhanced by using the quality adjustment parameter corresponding to the image, the video quality is improved, and the visual experience of a user is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training method, an image quality enhancement method and an electronic device. Background Art

[0002] Video is one of the ways to record and store information, which can enable users to obtain information more intuitively and conveniently. With the development of technology, video is widely used in different fields such as life, work, education, etc. At the same time, people's requirements for video quality are getting higher and higher.

[0003] At present, in order to improve the video quality, an image processing model is generally used to predict the quality adjustment parameters corresponding to the video image, so as to utilize the quality adjustment parameters to enhance the quality of the video image, such as color, clarity, etc. Therefore, how to obtain an image processing model that can predict the quality adjustment parameters has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present application provides an image processing model training method, an image quality enhancement method and an electronic device for training an image processing model that can predict quality adjustment parameters.

[0005] In a first aspect, the present application provides an image processing model training method, which obtains a sample image set; wherein the sample image set includes a plurality of sample original images;

[0006] For each sample original image among the multiple sample original images, degrade the quality of the sample original image to obtain a sample degraded image corresponding to the sample original image; wherein the quality of the sample degraded image corresponding to the sample original image is lower than the quality of the sample original image;

[0007] Based on each sample original image and the sample degraded image corresponding to each sample original image, the preset model is trained to obtain an image processing model; the preset model is a lightweight model.

[0008] Among them, the above-mentioned image processing model is used to predict the quality adjustment parameters corresponding to the first image in the video stream, and the quality adjustment parameters are used to adjust the quality of the first image; the quality adjustment parameters include color adjustment parameters and / or texture adjustment parameters; the color adjustment parameters include at least one of contrast adjustment parameters, brightness adjustment parameters and saturation adjustment parameters, the quality includes color and / or clarity, and the color includes at least one of contrast, brightness and saturation.

[0009] In an embodiment of the present application, the electronic device can perform degradation processing on a high-quality sample original image to obtain a low-quality sample degraded image corresponding to the sample original image. Afterwards, the electronic device can use the high-quality sample original image and its corresponding low-quality sample degraded image as a training set to achieve rapid determination of the training set. Afterwards, the electronic device can use the training set to train a preset model so that the preset model learns how to restore the low-quality sample degraded image to the high-quality sample original image, ensuring the training effect of the model, so that the trained preset model, that is, the image processing model, can accurately predict the quality adjustment parameters corresponding to the image, and the quality adjustment parameters corresponding to the image represent the adjustment parameters from low to high quality of the image, so as to achieve accurate prediction of the quality adjustment parameters, thereby achieving accurate enhancement of the quality of the video image. Since the preset model is a lightweight model, that is, the image processing model is a lightweight model, it has high performance, fast image processing speed, and can reduce power consumption.

[0010] In a possible design, the quality adjustment parameter includes a color adjustment parameter, and accordingly, the quality includes color, and the plurality of sample original images include a plurality of first sample original images.

[0011] The process of determining the sample degraded image corresponding to the first sample original image may include:

[0012] For each first sample original image in the plurality of first sample original images, the electronic device may obtain a color degradation value from a preset color degradation range; wherein the values ​​in the color degradation range are all greater than 1.

[0013] Based on the color degradation value, the color of the first sample original image is degraded to obtain a sample degraded image corresponding to the first sample original image; wherein the color of the sample degraded image corresponding to the first sample original image is lower than the color of the first sample original image, thereby achieving accurate determination of the sample degraded image with poor color display effect, and since the number of the first sample original images is large enough, the number of sample degraded images with poor color display effect can be guaranteed, thereby ensuring the richness of the training set and further ensuring the training effect of the model.

[0014] In a possible design, the color adjustment parameter includes a brightness adjustment parameter, and accordingly, the color includes brightness, and the color degradation value includes a brightness degradation value. Since the brightness is related to the pixel value of the Y channel, the process of performing degradation processing on the brightness of the first sample original image may include: for each pixel point in the first sample original image, based on the brightness degradation value, reducing the pixel value of the brightness Y channel of the pixel point, obtaining the pixel value of the Y channel after brightness degradation of the pixel point, and realizing the degradation of the brightness of the first sample original image.

[0015] Specifically, for each pixel in the first sample original image, the electronic device may use the inverse of the brightness degradation value as the index of the pixel value of the Y channel of the pixel. That is, for each pixel in the first sample original image, the electronic device may use Calculate the pixel value of the Y channel after brightness degradation of the pixel point to achieve the brightness degradation of each pixel point in the first sample original image, and avoid the model being unable to recover the low-brightness sample degraded image to the corresponding first sample original image due to excessive brightness reduction, which affects the model training effect.

[0016] In a possible design, the color adjustment parameter includes a contrast adjustment parameter, and accordingly, the color includes contrast, and the color degradation value includes a contrast degradation value. Since the contrast is related to the pixel value of the Y channel, the process of degrading the contrast of the first sample original image may include: for each pixel point in the first sample original image, the electronic device may reduce the pixel value of the Y channel of the pixel point based on the brightness adjustment parameter to obtain the pixel value of the Y channel after brightness degradation of the pixel point. Afterwards, the electronic device may determine the first adjustment value of the Y channel of the pixel point based on the average value of the pixel value of the Y channel of the pixel point after brightness degradation and the pixel value of the Y channel of each pixel point corresponding to the first sample original image after brightness degradation, combined with the contrast degradation value. Afterwards, based on the first adjustment value of the Y channel of the pixel point and the average value, the pixel value of the Y channel of the pixel point after contrast degradation is obtained, so as to achieve contrast reduction, and avoid excessive contrast reduction, which may cause the model to be unable to recover well from the low-contrast sample degraded image to the corresponding first sample original image, thereby affecting the model training effect.

[0017] Specifically, for each pixel in the first sample original image, the electronic device can calculate the difference between the pixel of the Y channel after brightness degradation and the average value. After that, the electronic device calculates the ratio between the difference between the pixel of the Y channel after brightness degradation and the average value and the contrast degradation value to obtain the first adjustment value of the Y channel of the pixel. After that, the electronic device calculates the sum of the first adjustment value of the Y channel based on the pixel and the average value to obtain the pixel value of the Y channel after contrast degradation of the pixel, so as to achieve accurate reduction of contrast.

[0018] In a possible design, the color adjustment parameter includes a saturation adjustment parameter. Accordingly, the color includes saturation, and the color degradation value includes a saturation degradation value. Since saturation is related to the pixel value of the chromaticity channel (such as U and V channels), the process of degrading the saturation of the first sample original image may include: for each pixel point in the first sample original image, based on the saturation degradation value, reducing the pixel value of the chromaticity channel of the pixel point, obtaining the pixel value of the chromaticity channel after saturation degradation of the pixel point, achieving saturation reduction, and avoiding excessive saturation reduction, which causes the model to be unable to recover well from the low-saturation sample degraded image to the corresponding first sample original image, affecting the model training effect.

[0019] Specifically, for each pixel in the first sample original image, the electronic device may calculate the ratio between the pixel value of the chromaticity channel of the pixel and the saturation degradation value. That is, for each pixel in the first sample original image, the electronic device may use U s =U / s, calculate the pixel value of the U channel after saturation degradation of the pixel point. s Represents the pixel value of the U channel after saturation degradation of the pixel. U represents the original pixel value of the U channel of the pixel. s represents the saturation degradation value.

[0020] And, for each pixel point in the first sample original image, the electronic device can use V s =V / s, calculate the pixel value of the U channel after saturation degradation of the pixel point. s Represents the pixel value of the V channel after saturation degradation of the pixel. V represents the original pixel value of the U channel of the pixel.

[0021] In a possible design, the quality adjustment parameter includes a texture adjustment parameter, and correspondingly, the quality includes clarity, and the plurality of sample original images includes a plurality of second sample original images;

[0022] The process of determining the sample degraded image corresponding to the second sample original image may include:

[0023] For each second sample original image in the plurality of second sample original images, the electronic device may downsample the second sample original image, wherein the size of the downsampled second sample original image is smaller than the second sample original image.

[0024] The downsampled second sample original image is upsampled to obtain a sample degraded image corresponding to the second sample original image. The size of the sample degraded image corresponding to the second sample original image is the same as the size of the second sample original image, and the resolution of the sample degraded image corresponding to the second sample original image is lower than the resolution of the second sample original image, that is, the clarity of the sample degraded image corresponding to the second sample original image is lower than the clarity of the second sample original image, so as to achieve accurate determination of the sample degraded image with poor texture display effect, and because the number of the second sample original images is sufficient, the number of sample degraded images with poor texture display effect can be guaranteed, thereby ensuring the richness of the training set and further ensuring the training effect of the model.

[0025] In a possible design, the texture of the second sample original image (i.e., the edge and detail) can be characterized by the complexity corresponding to the second sample original image. Accordingly, when training the preset model, the complexity can be used for training. Specifically, for each second sample original image, the variance of the second sample original image is calculated to obtain the complexity corresponding to the second sample original image; based on each second sample original image, the sample degraded image corresponding to each second sample original image, and the complexity corresponding to each second sample original image, the preset model is trained so that the preset model determines the mapping relationship between the complexity and the texture adjustment parameters, so that the trained preset model can determine the texture adjustment parameters that match the complexity of the image based on the mapping relationship between the complexity and the texture adjustment parameters.

[0026] In a possible design, after the image processing model is trained, the image processing model can be used to enhance the image in the video stream in real time. Specifically, the electronic device can obtain a video stream; wherein the video stream includes at least one frame of a first image;

[0027] Afterwards, for each first image in the at least one frame of the first image, the electronic device may use the first image as an input parameter of the image processing model, run the image processing model, and obtain a quality adjustment parameter corresponding to the first image;

[0028] Afterwards, the electronic device may adjust the quality of the first image based on the quality adjustment parameter corresponding to the first image to obtain the target first image, so as to display the target first image.

[0029] In the present application, after obtaining the video stream, that is, after obtaining the current image frame (i.e., the first image), the electronic device inputs the first image into the image processing model, so as to use the image processing model to predict the quality adjustment parameters adapted to the first image, and accurately determine the quality adjustment parameters corresponding to the first image. Afterwards, the electronic device can use the quality adjustment parameters corresponding to the first image to enhance the quality of the first image, ensure the display effect of the first image, and improve the user's visual experience. And because the image processing model is a lightweight model with high performance and fast image processing speed, power consumption can be reduced.

[0030] In a possible design, before using the image processing model to predict the quality adjustment parameters corresponding to the first image, scene detection can be performed on the first image to determine whether it is necessary to use the image processing model to predict the quality adjustment parameters corresponding to the first image. Specifically, based on the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image, the scene detection result corresponding to the first image is obtained; wherein the scene detection result corresponding to the first image indicates whether the first image and the previous frame of the image belong to the same scene;

[0031] When the scene detection result indicates that the first image and the previous frame image do not belong to the same scene, the first image is used as an input parameter of the image processing model, and the image processing model is run to obtain the quality adjustment parameters corresponding to the first image.

[0032] The fact that the first image and the previous frame image belong to the same scene can be understood as the difference in contents (or objects) between the two frames of images is small, or the difference in display effects between the two frames of images is small.

[0033] In the present application, after receiving the current frame image (i.e., the first image), it is determined whether the first image and the previous frame image of the first image belong to the same scene. If they belong to the same scene, the electronic device can use the quality adjustment parameter corresponding to the previous frame image as the quality adjustment parameter corresponding to the first image, without re-predicting the quality adjustment parameter, to achieve rapid determination of the quality adjustment parameter, and improve the processing efficiency of the video image. If they do not belong to the same scene, the electronic device can re-predict the quality adjustment parameter corresponding to the first image, and obtain the quality adjustment parameter matching the scene of the first image, so that the electronic device can use the quality adjustment parameter matching the scene of the first image to adjust the quality of the first image, realize adaptive quality adjustment, realize real-time quality enhancement of the video image, improve the quality of the video image, and thus improve the display effect of the video. And because there is an error in the prediction of the quality adjustment parameter of the image processing model, that is, for the same scene, the quality adjustment parameter that may be predicted may have an error. Therefore, in the case where the current frame image and the previous frame image belong to the same scene, the quality adjustment parameter corresponding to the previous frame image is directly used to adjust the current frame image, which can avoid the display effects of two adjacent frames belonging to the same scene being different due to errors, and ensure the continuity of the display effect of the video frame.

[0034] In a possible design, the process of determining the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image may include:

[0035] For each pixel in the first image, the electronic device can calculate the difference between the pixel value of the target channel of the pixel and the pixel value of the target channel of the pixel at the corresponding position in the previous frame image, and obtain the difference of the target channel corresponding to the pixel in the first image. In other words, for each first pixel in the first image, the electronic device can calculate the difference between the pixel value of the target channel of the first pixel and the pixel value of the target channel of the corresponding second pixel in the previous frame image, and obtain the difference of the target channel corresponding to the first pixel. Among them, the position of the first pixel on the first image is the same as the position of the second pixel corresponding to the first pixel on the previous frame image.

[0036] The electronic device may obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image based on the average value of the difference of the target channel corresponding to each first pixel point in the first image.

[0037] In the present application, the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is determined based on the difference in the pixel value of the target channel of the pixel point between the current frame and the previous frame image, that is, the scene detection result is determined, and the scene detection result is determined by the overall difference value of two adjacent frames of images, thereby ensuring the accuracy of the scene detection result.

[0038] In a possible design, the number of the target channels is one or more. Accordingly, the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is obtained by averaging the difference values ​​of the target channels corresponding to the pixels in the first image, including:

[0039] For each target channel, the electronic device may calculate the average value of the difference values ​​of the target channels corresponding to each pixel point to obtain the difference value of the target channel corresponding to the first image. Afterwards, the electronic device may obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image by taking the average value of the difference values ​​of each target channel corresponding to the first image.

[0040] In the present application, the electronic device can determine the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image based on the difference between different target channels, that is, determine the scene detection result, and detect the degree of difference between the two frames of images from different dimensions to ensure the accuracy of the scene detection result.

[0041] In a possible design, the process of obtaining the scene detection result corresponding to the first image based on the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image may include:

[0042] When the difference in the target channel between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is less than or equal to the first threshold, it indicates that the difference in the pixel values ​​between the two frames of images is small, that is, the difference in the displayed objects is small. The electronic device can determine that the scene detection result corresponding to the first image indicates that the first image and the previous frame image belong to the same scene, that is, the scene of the first image has not changed.

[0043] When the difference in the target channel between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is greater than the first threshold, it indicates that the pixel value difference between the two frames of images is large, that is, the display objects are very different. The scene detection result corresponding to the first image is determined to indicate that the first image and the previous frame image do not belong to the same scene, that is, the scene of the first image has changed, thereby achieving accurate detection of the scene of the first image.

[0044] In a possible design, the target channel includes at least one of a hue H channel, a saturation S channel, and a brightness V channel.

[0045] In a second aspect, the present application provides an image quality enhancement method, where, after acquiring a video stream, the electronic device can use the first image as an input parameter of the image processing model described above for each frame of the first image in the video stream, run the image processing model, and obtain a quality adjustment parameter corresponding to the first image. The image processing model is a lightweight model.

[0046] Based on the quality adjustment parameter corresponding to the first image, the quality of the first image is adjusted to obtain a target first image. Afterwards, the target first image can be displayed electronically.

[0047] In the present application, after obtaining the video stream, that is, after obtaining the current image frame (i.e., the first image), the electronic device inputs the first image into the image processing model, so as to use the image processing model to predict the quality adjustment parameters adapted to the first image, and accurately determine the quality adjustment parameters corresponding to the first image. Afterwards, the electronic device can use the quality adjustment parameters corresponding to the first image to enhance the quality of the first image, ensure the display effect of the first image, and improve the user's visual experience. And because the image processing model is a lightweight model with high performance and fast image processing speed, power consumption can be reduced.

[0048] In a third aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described above.

[0049] In a fourth aspect, the present application provides a computer storage medium, comprising computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method described above.

[0050] In a fifth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0051] It can be understood that the beneficial effects that can be achieved by the electronic device described in the third aspect, the computer storage medium described in the fourth aspect, and the computer program product described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1A A video scene schematic diagram 1 provided in an embodiment of the present application;

[0053] Figure 1B A video scene diagram provided in an embodiment of the present application Figure 2 ;

[0054] Figure 1C A video scene diagram provided in an embodiment of the present application Figure 3 ;

[0055] Figure 1D A video scene diagram provided in an embodiment of the present application Figure 4 ;

[0056] Figure 1E A video scene diagram provided in an embodiment of the present application Figure 5 ;

[0057] Figure 1F A video scene diagram provided in an embodiment of the present application Figure 6 ;

[0058] Figure 2 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a model training process provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a model structure provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a model structure provided in an embodiment of the present application Figure 2 ;

[0062] Figure 6 A schematic diagram of a model structure provided in an embodiment of the present application Figure 3 ;

[0063] Figure 7 A schematic diagram of a model structure provided in an embodiment of the present application Figure 4 ;

[0064] Figure 8 A model training process diagram provided in the embodiment of the present application Figure 2 ;

[0065] Fig. 9 A schematic diagram of a mapping curve provided in an embodiment of the present application;

[0066] Fig.10 A schematic diagram of a quality enhancement process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0068] Video is composed of multiple frames of images. With the development of technology, video is widely used in different aspects, such as office, movies, conferences, games, collaboration, shooting, etc. Electronic devices can play videos, that is, display the images in the video. For example, when using electronic devices for work, electronic devices (such as user A's device) may share the desktop of other devices (such as user B's device). User B's device sends the desktop display data to user A's device, and user A's device displays the desktop of user B's device (such as user B's device). Figure 1A As shown), the desktop display data can be sent to the device of user A in the form of streaming media, that is, in the form of video stream. The desktop display data is equivalent to an image. It can be simply understood that the device of user B continuously sends an image including the desktop display content to the device of user A, so that the device of user A can display the image. Therefore, when the device of user B displays the content of the PPT file, the desktop display data includes the content of the displayed PPT file, that is, the image includes the content of the PPT file, and the content of the PPT file can be considered as an image. It should be understood that the PPT file can also be considered as an image in other cases, such as recording the content displayed by the electronic device, and the displayed content can include the content of the PPT file.

[0069] For example, in Figure 1B In the collaborative situation shown, the mobile phone sends screen projection data to the PC in real time. The screen projection data is sent in the form of streaming media. The screen projection data includes the screen data displayed by the mobile phone. The PC displays the screen of the mobile phone according to the received screen projection data. Here, the screen can refer to the image in the video.

[0070] For example, in Figure 1C As shown, the mobile phone displays the game screen. Since the game screen is displayed continuously, one frame of the game screen can be considered as one frame of image in the video.

[0071] For example, Figure 1D As shown, the mobile phone plays XX movie and displays the pictures in XX movie. Here, XX movie is equivalent to a video, and the pictures in the movie are equivalent to images.

[0072] For example, Figure 1EAs shown, the mobile phone is in a video recording state. When the user wants to end the recording, the user can click the stop recording button 10 as shown in 1D, and the mobile phone stops recording in response to the click operation of the stop recording button 10, and generates a corresponding video, which may include multiple frames of images.

[0073] For example, Figure 1F As shown, the mobile phone plays a short video and displays the images in the short video, where the short video can also be called a video.

[0074] Above Figures 1A to 1F The type of video shown is only an example, and other aspects may also involve videos, that is, the video may also be of other types, such as TV series type, etc., which is not limited in this application.

[0075] In order to improve the video quality, the electronic device can enhance the video image to improve the quality of the video image, thereby improving the display effect of the video image. In one implementation, for each frame of the image in the video, the electronic device can directly use the specified quality adjustment parameter to adjust the pixel value of each pixel in the image to enhance the quality of the image. The specified quality adjustment parameter may include a specified color adjustment parameter and / or a specified texture adjustment parameter. Among them, the specified color adjustment parameter is used to enhance the color of the image, and the specified texture adjustment parameter is used to enhance the texture clarity of the image. For example, the pixel value includes color channel data, and the color channel data of each pixel of the image is multiplied by the specified color adjustment parameter to enhance the color of the image. However, this implementation method will enable the images in different scenes in the video to achieve the same color enhancement effect, however, different scenes have different requirements for color enhancement effects, for example, the electronic device plays a movie, the scene of part of the picture in the movie is a darker scene, such as a night scene, the color of the object in the picture in the movie is darker, and the scene of part of the picture is a brighter scene, such as a daytime scene, the color of the object in the picture in the movie is brighter. Using this implementation method, after adjusting the picture of the movie, the color of the picture corresponding to the brighter scene will be enhanced, and the object in the picture may become brighter, and the color of the picture corresponding to the darker scene will also be enhanced, and the object in the picture will also become brighter, which reduces the display effect of the picture. It can be seen that this implementation method is not adaptive and cannot be adaptively adjusted according to the scene of the image in the video.

[0076] In another implementation, electronic devices can use deep learning algorithms to enhance video images. However, most models are highly complex and take a long time to process images, making it impossible for electronic devices with low computing power such as mobile phones, tablets, and smart screens to achieve real-time enhancement of video images.

[0077] Therefore, in response to the above problems, the present application proposes an image processing model for enhancing the video display effect. The electronic device obtains a set of sample images with higher quality. In order to ensure that the model can accurately predict the quality adjustment operation from low-quality images to high-quality images, the electronic device can reduce the quality of each sample original image in the sample image set to obtain a low-quality sample degraded image corresponding to each sample original image, so that the electronic device can input the sample original image and its corresponding sample degraded image into the basic model (such as the MobileNetV3 model), so that the basic model learns how to restore from the sample degraded image to the corresponding sample original image, thereby realizing the training of the basic model. Specifically, for each sample original image, the electronic device can reduce the color and / or clarity of the image to obtain a sample degraded image with poor color display effect and / or low clarity. Afterwards, the electronic device can input the sample original image and its corresponding sample degraded image into the basic model to train the basic model. The trained basic model and the image processing model can predict the color adjustment parameters and / or texture adjustment parameters corresponding to the image. The color adjustment parameters are used to improve the color of the image, such as improving the contrast, saturation and brightness, and the texture adjustment parameters are used to improve the clarity of the image, thereby improving the quality of the video image.

[0078] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a wearable device, a computer (such as a PC), a personal digital assistant (PDA), or other device capable of displaying images in a video. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0079] For example, Figure 2 FIG. 2 shows a schematic diagram of the structure of the electronic device 200. Figure 2 As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0080] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0082] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0083] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0084] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0085] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0086] The charging management module 211 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 211 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 211 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 211 is charging the battery 213, it may also power the electronic device through the power management module 212.

[0087] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 240, the wireless communication module 250, the modem processor and the baseband processor.

[0088] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0089] The mobile communication module 240 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 240 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 240 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 240 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 240 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 240 can be set in the same device as at least some of the modules of the processor 210.

[0090] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 240 or other functional modules.

[0091] The wireless communication module 250 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 250 can be one or more devices integrating at least one communication processing module. The wireless communication module 250 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 250 can also receive the signal to be sent from the processor 210, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0092] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0093] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0094] The electronic device 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0095] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0096] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0097] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0098] The buttons 290 include a power button, a volume button, etc. The buttons 290 may be mechanical buttons or touch buttons.

[0099] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0100] The sensor module 280 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0101] An embodiment of the present application provides an image processing model training method. An electronic device can obtain a sample image set, which can include multiple sample original images. Among them, the quality of the multiple sample original images is relatively high, such as high color display effect and / or high clarity. Afterwards, for each sample original image of the multiple sample original images, the electronic device can perform degradation processing on the quality of the sample original image to obtain a sample degraded image corresponding to the sample original image, and the quality of the sample degraded image is relatively low, such as poor color display effect and / or low clarity. Among them, the quality of the sample original image includes color and / or clarity. Afterwards, the electronic device can use the sample original image and its corresponding sample degraded image to train the MobileNetV3 model to obtain an image processing model, which can be used to predict the quality adjustment parameters corresponding to the image in the video, and the quality adjustment parameters are used to adjust the pixel values ​​of the pixels in the image, improve the image quality, enhance the display effect of the image, and thus improve the quality of the video.

[0102] Exemplarily, the above-mentioned quality adjustment parameters may include color adjustment parameters and / or texture adjustment parameters. That is, the electronic device may train the MobileNetV3 model using a sample image set to obtain a trained image processing model, and the image processing module may predict the color adjustment parameters and / or texture adjustment parameters corresponding to the image, so that the color adjustment parameters may be used to enhance the image color and / or improve the image clarity.

[0103] The process of the electronic device using the sample image set to train an image processing model capable of predicting color adjustment parameters can be introduced in the following part a. The process of the electronic device using the sample image set to train an image processing model capable of predicting texture adjustment parameters can be introduced in the following part b.

[0104] a. Color enhancement

[0105] In an embodiment of the present application, the above-mentioned sample image set may include a first sample image set, and the first sample image set may include multiple first sample original images. The electronic device may perform color degradation processing on the first sample original image to reduce the color quality of the first sample image, and obtain a sample degraded image corresponding to the first sample original image. Afterwards, the electronic device may use the first sample original image and its corresponding sample degraded image to train the MobileNetV3 model. The trained MobileNetV3 model can predict the color adjustment parameters corresponding to the image in the video, thereby using the color adjustment parameters to improve the color quality of the image, that is, to improve the color display effect of the image. Specifically, if Figure 3As shown, the process of training the MobileNetV3 model using the first sample image set may include S301-S306.

[0106] S301: The electronic device obtains a first sample image set, wherein the first sample image set includes a plurality of first sample original images.

[0107] The first sample original image has a relatively high quality, such as an image with a relatively high color quality, that is, an image with a relatively good color display effect (or referred to as a standard image).

[0108] In some embodiments, the first sample original image may be an image obtained from MIT-Adobe FiveK. MIT-Adobe FiveK includes multiple (e.g., 5,000) original images in DNG format (hereinafter referred to as original image 1), and a higher quality image corresponding to each original image 1. Accordingly, the first sample original image may be a higher quality image corresponding to the original image 1.

[0109] Among them, the higher quality image corresponding to the original image 1 may be an image manually modified by five (A, B, C, D, E) professional photo retouchers, namely professional photo retoucher A, professional photo retoucher B, professional photo retoucher C, professional photo retoucher D and professional photo retoucher E. Correspondingly, the first sample original image may include an image manually modified by a professional photo retoucher (such as professional photo retoucher C). Or the first sample original image may include images manually modified by multiple professional photo retouchers, such as images manually modified by professional photo retoucher A and professional photo retoucher C. This application is not limited thereto.

[0110] In some embodiments, the above-mentioned first sample image set can be an image set of multiple types. The types may include collaboration, movies, games, office (such as PPT), meetings, short videos, recordings and the like. Different types of image sets are produced in different ways. For example, a movie-type image set may include: multiple (such as 100) domestic and foreign 1080P / 2K high-definition movies, thereby forming a large data set that is evenly distributed in four aspects: brightness, texture complexity, motion complexity, and color complexity. In other words, it includes movie pictures with different display effects (such as color display effects), that is, images, to ensure the richness of the movie-type image set and improve the generalization of the model.

[0111] For another example, the above-mentioned office PPT type image set may include: the PPT content in the image includes common content such as text, pictures, tables, statistical charts, etc.

[0112] For another example, the image set of the above-mentioned conference type may include different facial images, such as facial images of people of different genders, facial images of people of different ages, facial images of people of different regions, etc.

[0113] For another example, the image set of the above-mentioned game type may include game screens, that is, images, in different game applications.

[0114] In some embodiments, the model training process can be simply understood as the process of how to restore the image of lower quality to the image of higher quality for two images with the same content. Therefore, after obtaining the first sample original image with higher quality, for each first sample original image, the electronic device can perform degradation processing on the color of the first sample original image to obtain an image with poor color corresponding to the first sample original image, that is, obtain a sample degraded image corresponding to the first sample original image, so that the electronic device can use each first sample original image and each sample degraded image corresponding to the first sample original image to train the model. It should be understood that the content of the first sample original image and the sample degraded image corresponding to the first sample original image, that is, the object is the same, but compared with the first sample original image, the color display effect of the object in the sample degraded image corresponding to the first sample original image is poor, such as the originally bright object becomes dark. The following will introduce a possible process of degrading the color of the first sample original image in combination with S302-S305.

[0115] S302: The electronic device obtains a brightness degradation value from a preset brightness degradation range.

[0116] S303: The electronic device obtains a contrast degradation value from a preset contrast degradation range.

[0117] S304: The electronic device obtains a saturation degradation value from a preset saturation degradation range.

[0118] S305. For each first sample original image among the multiple first sample original images, the electronic device performs degradation processing on the brightness of the first sample original image based on the brightness degradation value, performs degradation processing on the contrast of the first sample original image based on the contrast degradation value, and performs degradation processing on the saturation of the first sample original image based on the saturation degradation value, to obtain a sample degraded image corresponding to the first sample original image.

[0119] In the embodiment of the present application, since the brightness, saturation and contrast of the image are related to the color of the image, degrading the color means degrading the brightness, saturation and contrast. In other words, for each first sample original image, the electronic device needs to degrade the first sample original image from the dimensions of brightness, merging degree and contrast. Exemplarily, the electronic device can randomly obtain a brightness degradation value from a preset brightness degradation range, randomly obtain a contrast degradation value from a preset contrast degradation range, and obtain a saturation degradation value from a preset saturation degradation range.

[0120] Afterwards, since the Y (brightness, or brightness) channel data of the image (i.e., the pixel value of the Y channel) affects the brightness of the image, the electronic device can use the brightness degradation value to adjust the pixel value of the Y channel of the pixel point in the first sample original image, such as reducing the pixel value of the Y channel of the pixel point in the first sample original image, thereby achieving brightness degradation of the first sample original image.

[0121] Since the Y channel data of the image also affects the contrast of the image, the electronic device can use the contrast degradation value and the pixel value of the Y channel of the pixel after the brightness is reduced to adjust the pixel value of the Y channel of the pixel in the first sample original image to reduce the brightness of the brighter pixel and increase the brightness of the darker pixel, thereby reducing the contrast of the first sample original image, that is, achieving degradation of the contrast of the first sample original image.

[0122] Since the chrominance U and V channel data of the image (i.e., the pixel value of the U channel and the pixel value of the V channel) affect the saturation of the image, the first device can use the saturation degradation value to adjust the pixel value of the U channel and the pixel value of the V channel of the pixel point in the first sample original image, such as reducing the pixel value of the U channel of the pixel point in the first sample original image and reducing the pixel value of the V channel of the pixel point in the first sample original image, thereby achieving the degradation of the saturation of the first sample original image. Among them, U can represent the difference between the blue component and the brightness. V represents the difference between the red component and the brightness.

[0123] In the embodiment of the present application, the electronic device degrades the brightness, saturation and contrast of the first sample original image to obtain a sample degraded image corresponding to the first sample original image. Compared with the first sample original image, the overall brightness of the sample degraded image corresponding to the first sample original image is lower. And compared with the first sample original image, the contrast of the sample degraded image corresponding to the first sample original image is lower, the light and dark changes of the first sample original image are more obvious, the details of the image are clearer and more vivid, and the visibility is higher, while the light and dark changes of the sample degraded image corresponding to the first sample original image are relatively gentle.

[0124] And compared with the first sample original image, the sample degraded image corresponding to the first sample original image has a lower saturation, the color of the first sample original image is more vivid and full, while the color of the sample degraded image corresponding to the first sample original image is relatively dim.

[0125] In some embodiments, the process of performing brightness degradation processing on the first sample original image may include: for each pixel in the first sample original image, the electronic device uses the reciprocal of the brightness degradation value as the index of the pixel value of the Y channel of the pixel (or the original Y channel pixel value) to obtain the pixel value of the Y channel after brightness degradation of the pixel. Specifically, the electronic device may use the following formula 1 to calculate the pixel value of the Y channel after brightness degradation of the pixel.

[0126]

[0127] Among them, Y represents the pixel value of the original Y channel of the pixel point, Y l It represents the pixel value of the Y channel after brightness degradation of the pixel point, and l represents the brightness degradation value.

[0128] The above-mentioned process of performing degradation processing on the contrast of the first sample original image may include: for each pixel point in the first sample original image, the electronic device may calculate the difference (or first difference) between the pixel value of the Y channel of the pixel point after brightness degradation and the average value of the pixel value of the Y channel corresponding to the first sample original image after brightness degradation. Afterwards, the electronic device determines the adjustment value of the Y channel of the pixel point (or first adjustment value) based on the first difference and the contrast degradation value. Afterwards, the electronic device may determine the pixel value of the Y channel of the pixel point after contrast degradation by using the adjustment value of the Y channel of the pixel point and the average value of the pixel value of the Y channel corresponding to the first sample original image after brightness degradation, so as to reduce the brightness of the pixel point when the brightness of the pixel point exceeds the average brightness of the first sample original image after brightness degradation. When the brightness of the pixel point is lower than the average brightness of the first sample original image after brightness degradation, the brightness of the pixel point is increased to achieve brightness degradation.

[0129] It should be understood that although contrast degradation is also for the pixel value of the Y channel, for the same pixel point, the pixel value of the Y channel after contrast degradation may change compared to the pixel value of the Y channel after brightness degradation, such as being too large or too small. In other words, the pixel value of the Y channel after contrast degradation may be larger than the pixel value of the original Y channel. In other words, the brightness of the pixel point will become brighter, but it can be determined through experimental tests that, speaking of the image as a whole, the overall brightness of the image is still reduced, that is, the brightness has been degraded.

[0130] The average value of the pixel values ​​of the Y channel after brightness degradation corresponding to the first sample original image represents the average value of the pixel values ​​of the Y channel after brightness degradation of all the pixel points in the first sample original image.

[0131] Specifically, the electronic device may use the following formula 2 to calculate the pixel value of the Y channel of the pixel point after contrast degradation.

[0132]

[0133] Wherein, mean represents the average value of the pixel values ​​of the Y channel after brightness degradation corresponding to the first sample original image. c represents the contrast degradation value. c Indicates the pixel value of the Y channel after contrast degradation.

[0134] The above-mentioned process of degrading the saturation of the first sample original image may include: for each pixel point in the first sample original image, the electronic device calculates the ratio between the original U channel pixel value of the pixel point and the saturation degradation value to reduce the chromaticity of the pixel point and obtain the pixel value of the U channel of the pixel point after saturation degradation. Specifically, the electronic device may use the following formula 3 to calculate the pixel value of the U channel of the above-mentioned pixel point after saturation degradation.

[0135] U s =U / s Formula 3

[0136] Among them, U s Represents the pixel value of the U channel after saturation degradation of the pixel. U represents the original pixel value of the U channel of the pixel. s represents the saturation degradation value.

[0137] Furthermore, the electronic device calculates the ratio between the original pixel value of the V channel of the pixel and the saturation degradation value to reduce the concentration of the pixel and obtain the pixel value of the V channel of the pixel after saturation degradation. Specifically, the electronic device can use the following formula 4 to calculate the pixel value of the U channel of the above pixel after saturation degradation.

[0138] V s =V / s Formula 4

[0139] Among them, V s Represents the pixel value of the V channel after saturation degradation of the pixel. V represents the original pixel value of the V channel of the pixel.

[0140] It should be noted that before the color of the first sample original image is degraded, if the format of the first sample original image is not in YUV format, the electronic device needs to convert the format of the first sample original image into YUV format. After that, the electronic device performs the degraded processing on the color of the first sample original image in YUV format. If the format of the first sample original image is in YUV format, the electronic device can perform the degraded processing on the color of the first sample original image in YUV format.

[0141] In some embodiments, the brightness degradation value is randomly selected by the first device from a preset brightness degradation range, and the random selection process obeys a uniform distribution. Therefore, when the number of first sample original images is large enough and the number of training iterations is large enough, each value in the preset brightness degradation range can be obtained, and degraded sample images with brightness degradation to different degrees are obtained, thereby ensuring the richness of training samples, improving the accuracy of the model's prediction of brightness adjustment parameters, being able to accurately predict images of different brightnesses, and improving the generalization of the model.

[0142] Similarly, the above contrast degradation value is randomly selected by the first device from the preset contrast degradation range, and the random selection process obeys uniform distribution. Therefore, when the number of first sample original images is large enough and the number of training iterations is large enough, each value in the preset contrast degradation range can be obtained, and degraded sample images with different degrees of contrast degradation are obtained, thereby ensuring the richness of training samples, improving the model's prediction accuracy for contrast adjustment parameters, and being able to accurately predict images with different contrasts, thereby improving the generalization of the model.

[0143] The saturation degradation value is randomly selected by the first device from a preset contrast degradation range, and the random selection process obeys a uniform distribution. Therefore, when the number of first sample original images is large enough and the number of training iterations is large enough, each value in the preset saturation degradation range can be obtained, and degraded sample images with saturation degraded to different degrees are obtained, thereby ensuring the richness of training samples, improving the model's prediction accuracy for saturation adjustment parameters, being able to accurately predict images with different saturations, and improving the generalization of the model.

[0144] In some embodiments, for each first sample original image, the electronic device may utilize the target preset brightness degradation range corresponding to the type of the first sample original image to select a brightness degradation value from the target preset brightness degradation range. Furthermore, the electronic device may utilize the target preset contrast degradation range corresponding to the type of the first sample original image to select a contrast degradation value from the target preset contrast degradation range. The electronic device may utilize the target preset saturation degradation range corresponding to the type of the first sample original image to select a saturation degradation value from the target preset saturation degradation range. Exemplarily, the type may include a video recording type (or recording type, shooting type), a movie type, a conference type, a game type, a collaborative type, and the like.

[0145] However, it is found through experiments that the prediction result of the color adjustment parameters of the image by the model is independent of the type of the first sample image set. Therefore, when training the model, it is not necessary to distinguish the preset brightness degradation range. In other words, the preset brightness degradation range corresponding to all types of images is the same. Accordingly, the above-mentioned preset brightness degradation range can be [1, 1.15], the preset contrast degradation range can be [1, 1.1], and the preset saturation degradation range can be [1, 1.3].

[0146] It should be noted that the above-mentioned preset brightness degradation range, preset saturation degradation range and preset contrast degradation range are obtained through experimental testing. The electronic device performs degradation processing on the first sample original image based on the degradation values ​​in the preset brightness degradation range, preset saturation degradation range and preset contrast degradation range to obtain a sample degraded image corresponding to the first sample original image, so that the first MobileNetV3 model described below can learn how to restore the sample degraded image corresponding to the first sample original image to the first sample original image, and implement the training of the first MobileNetV3 model. If the degradation value used does not belong to the above-mentioned degradation range (such as the above-mentioned preset brightness degradation range, preset saturation degradation range, preset contrast degradation range), it may be impossible to restore the sample degraded image corresponding to the first sample original image to the first sample original image during model training, resulting in poor model training effect.

[0147] In some embodiments, in order to ensure the training effect of the model, the first sample original images may include first sample original images of various contrasts, first sample original images of various saturations, and first sample original images of various brightnesses. That is to say, the contrasts between the first sample original images are different, the brightness between the first sample original images are different, the saturation between the first sample original images is different, the contrasts between the first sample original images are the same but the brightness is different, the contrasts and saturations between the first sample original images are different, the saturations and brightness between the first sample original images are different, and the contrasts, brightnesses, and saturations between the first sample original images are different, so as to ensure the comprehensiveness of the training samples.

[0148] S306. The electronic device trains the first MobileNetV3 model based on each first sample original image and the sample degraded image corresponding to each first sample original image to obtain a trained first MobileNetV3 model. The trained first MobileNetV3 model can predict color adjustment parameters corresponding to images in videos under different scenes, and the color adjustment parameters include brightness adjustment parameters, contrast adjustment parameters, and saturation adjustment parameters.

[0149] In an embodiment of the present application, the electronic device performs degradation processing on the saturation, contrast and brightness of the first sample original image respectively to obtain a sample degraded image with reduced saturation, contrast and brightness corresponding to the first sample original image. Afterwards, the electronic device can input the first sample original image and its corresponding sample degraded image into the first MobileNetV3 model to train the first MobileNetV3 model, so that the first MobileNetV3 model learns how to restore the sample degraded image corresponding to the first sample original image to the first sample original image. In layman's terms, the first MobileNetV3 model learns how to restore an image with poor color display effect to a corresponding image with good color display effect, so that the trained first MobileNetV3 model has the ability to predict the saturation adjustment parameters, contrast adjustment parameters and brightness adjustment parameters of the image.

[0150] The brightness adjustment parameter is used to increase the brightness of the image, the contrast adjustment parameter is used to increase the contrast of the image, and the saturation adjustment parameter is used to increase the saturation of the image.

[0151] In some embodiments, the above takes the case where color includes saturation, contrast and brightness as an example, and introduces the process of degrading the saturation, contrast and brightness of the first sample original image in the first sample image set, obtaining a sample degraded image corresponding to the first sample original image, so as to train the first MobileNetV3 model using the first sample original image and its corresponding sample degraded image, and obtain the trained first MobileNetV3 model capable of predicting saturation adjustment parameters, contrast adjustment parameters and brightness adjustment parameters. Of course, the above color may also include only one of saturation, contrast and brightness, and obtain the trained first MobileNetV3 model capable of predicting the color adjustment parameters corresponding to the color type. For example, the color may include saturation, and the electronic device may degrade the saturation of the first sample original image, obtain the sample degraded image corresponding to the first sample original image, so as to determine the trained first MobileNetV3 model capable of predicting the saturation adjustment parameters using the first sample original image and its corresponding sample degraded image, where the saturation parameter refers to the color adjustment parameter. For another example, color may include contrast, and the electronic device may perform degradation processing on the contrast of the first sample original image to obtain a sample degraded image corresponding to the first sample original image, so as to use the first sample original image and its corresponding sample degraded image to determine a trained first MobileNetV3 model capable of predicting a contrast adjustment parameter. Here, the contrast adjustment parameter refers to a color adjustment parameter.

[0152] For another example, color may include brightness, and the electronic device may perform degradation processing on the brightness of the first sample original image to obtain a sample degraded image corresponding to the first sample original image, so as to use the first sample original image and its corresponding sample degraded image to determine the trained first MobileNetV3 model capable of predicting the brightness adjustment parameter. Here, the brightness adjustment parameter refers to the color adjustment parameter.

[0153] Alternatively, the above-mentioned color may also include only two of saturation, contrast and brightness. For example, the color may include saturation and contrast, and the electronic device may perform degradation processing on the saturation and contrast of the first sample original image respectively to obtain a sample degraded image corresponding to the first sample original image, so as to use the first sample original image and its corresponding sample degraded image to determine the trained first MobileNetV3 model that can predict the saturation adjustment parameter and the contrast adjustment parameter. Here, the contrast adjustment parameter and the saturation adjustment parameter refer to the color adjustment parameter.

[0154] For another example, color may include saturation and brightness, and the electronic device may perform degradation processing on the saturation and brightness of the first sample original image respectively to obtain a sample degraded image corresponding to the first sample original image, so as to use the first sample original image and its corresponding sample degraded image to determine the trained first MobileNetV3 model capable of predicting saturation adjustment parameters and brightness adjustment parameters. Here, the brightness adjustment parameter and the saturation adjustment parameter refer to the color adjustment parameter.

[0155] For another example, color may include contrast and brightness, and the electronic device may perform degradation processing on the contrast and brightness of the first sample original image respectively to obtain a sample degraded image corresponding to the first sample original image, so as to use the first sample original image and its corresponding sample degraded image to determine the trained first MobileNetV3 model capable of predicting contrast adjustment parameters and brightness adjustment parameters. Here, the brightness adjustment parameter and the contrast adjustment parameter refer to the color adjustment parameter.

[0156] In the embodiment of the present application, the color of the first sample original image can be degraded as required. For example, if the brightness, saturation and contrast of the image are to be improved, the brightness, saturation and contrast of the first sample original image can be degraded respectively to determine the relevant model that can predict the brightness adjustment parameter, saturation adjustment parameter and contrast adjustment data. For another example, if the brightness of the image is to be improved quickly, the brightness of the first sample original image can be degraded to determine the relevant model that can predict the brightness adjustment parameter, etc.

[0157] In some embodiments, the performance and speed of MobileNetV3 are greatly improved compared to MobileNetV1 and MobileNetV2, using a network architecture search (NAS) method. Figure 4 The structure of the above-mentioned first MobileNetV3 model may include a first convolutional layer, a batch normalization (BN) layer and an activation function layer, 11 coding blocks, an average pooling layer and a fully connected layer.

[0158] Correspondingly, after the electronic device inputs the image (such as the first sample original image and its corresponding sample degraded image) into the first MobileNetV3 model, the image first passes through the first convolution layer, and the electronic device performs a 3*3 convolution operation on the image to obtain a convolved image. Afterwards, the convolved image passes through a batch normalization layer, and the electronic device can perform a batch normalization operation on the convolved image, such as normalizing the pixel values ​​of the Y channel, the pixel values ​​of the U channel, and the pixel values ​​of the V channel of the pixels in the convolved image to a smaller preset range, thereby improving the generalization ability of the model.

[0159] Afterwards, the normalized image can continue to pass through the activation function layer, and the electronic device can use the nonlinear activation function (hswish) to activate the normalized image. The activation process can be understood as a gating mechanism, which avoids excessive enhancement or weakening of features by limiting the range and relative weight of the input value, so that the model can learn and express features more effectively, thereby improving the prediction accuracy of the model.

[0160] After that, the activated image passes through 11 encoding blocks in sequence. Among them, the encoding block mainly implements channel separable convolution, SE channel attention mechanism and residual connection. The structure of the encoding block can be as follows Figure 5 shown.

[0161] Among them, channel separation convolution is the main factor in achieving the lightweight of the first MobileNetV3 model. Figure 6 As shown, channel-separable convolution can include depthwise convolution, or depthwise convolution and pointwise convolution. Depthwise convolution uses a 3*3 convolution kernel, and pointwise convolution uses a 1*1 convolution kernel. In other words, channel-separable convolution is divided into two processes: 1) channel-wise channel-separable convolution; 2) normal 1X1 convolution outputs the specified number of channels.

[0162] SE can be understood as an attention module. SE is an attention mechanism in the first MobileNetV3 model, which is used to enhance the network's modeling ability for specific channels. The main function of SE is to improve the model's discriminative and expressive capabilities while maintaining a lightweight structure.

[0163] The basic structure of SE is divided into two steps: squeeze and excitation. In the squeeze step, the input feature map is globally averaged pooled to reduce its dimension to a lower dimension. Then, in the excitation step, two fully connected layers (FC1 and FC2) are used to learn the channel relationship of the features and generate a channel attention vector. Specifically, the application of SE in the first MobileNetV3 model can be as follows: Figure 7 As shown. Among them, Figure 7 The number of FC1 nodes is 1 / 4 of the number of Conv1 channels, and the activation function after FC1 uses Relu. The number of FC2 nodes is the same as the number of Conv1 channels, and the activation function after FC2 uses hardsigmoid. The 4 numbers obtained by FC2 are equivalent to the weight coefficients of each channel of Conv1.

[0164] It should be understood that the number of coding blocks (i.e., 11) is obtained through experimental testing. When the number of coding blocks in the first MobileNetV3 model is 11, the prediction accuracy of the first MobileNetV3 model is high and the prediction efficiency is high. Of course, the number of coding blocks in the first MobileNetV3 model can also be other values, and this application does not limit the number of coding blocks.

[0165] The above describes the process of an image passing through the first convolutional layer, batch normalization layer, activation function layer, and 11 coding blocks. The following will continue to introduce the process of the image (or feature map) after 11 coding blocks passing through the average pooling layer and the fully connected layer.

[0166] The feature map passes through the average pooling layer (or called the global average pooling layer), and the electronic device performs an average pooling operation on the feature map to obtain an average pooling result.

[0167] After that, the average pooling result is input to the fully connected layer. The fully connected layer is a layer with a fixed number of neurons (nodes), which takes the output of the global average pooling layer as input and maps it to the final output regression result. The regression result here can refer to color adjustment parameters, such as brightness adjustment value, contrast adjustment value and saturation adjustment value.

[0168] In some embodiments, the fully connected layer may include one or more fully connected layers, such as two fully connected layers, namely fully connected layer 1 and fully connected layer 2. Among them, the fully connected layer 1 uses the ReLU activation function to process the average pooling result, and the fully connected layer 2 uses the hard_sigmoid activation function to process the output result of the fully connected layer 1.

[0169] In some embodiments, the backbone of the MobilenetV3 model is trained using a large-scale first sample image set, and the backbone is used to extract image features. The number of iterations (epochs) of the last fully connected layer is set to 300, the learning rate of the first 30 epochs is 0.001, and the learning rate of the remaining epochs is 0.00015. And the Adam optimizer is used for training.

[0170] In some embodiments, the first MobileNetV3 model is only an example of a preset model, and other lightweight models may also be used, which are not limited in this application, such as MobileNetV1, MobileNetV2, etc.

[0171] It should be noted that the above-mentioned image processing model may include the trained first MobileNetV3 model and / or the above-mentioned second MobileNetV3 model. When the image processing model includes the trained first MobileNetV3 model, it means that the image processing model can predict color adjustment parameters, and the trained first MobileNetV3 model can be used as an image processing model. When the image processing model includes the trained second MobileNetV3 model, it means that the image processing model can predict texture adjustment parameters, and the trained second MobileNetV3 model can be used as an image processing model.

[0172] When the image processing model includes the trained first MobileNetV3 model and the above-mentioned second MobileNetV3 model, it means that the image processing model can predict both color adjustment parameters and texture adjustment parameters, and the trained first MobileNetV3 model and the second MobileNetV3 model can be used as image processing models, or the image processing model is a model that has the ability to predict both color adjustment parameters and texture adjustment parameters.

[0173] b. Texture enhancement

[0174] In an embodiment of the present application, the above-mentioned sample image set may include a second sample image set, and the second sample image set may include multiple second sample original images. The electronic device may perform texture degradation processing on the second sample original image to reduce the clarity of the second sample image, and obtain a sample degraded image corresponding to the second sample original image. Afterwards, the electronic device may use the second sample original image and its corresponding sample degraded image to train the MobileNetV3 model. The trained MobileNetV3 model can predict the texture adjustment parameters corresponding to the image in the video, thereby using the texture adjustment parameters to enhance the details of the image and improve the clarity of the image. Specifically, if Figure 8 As shown, the process of training the MobileNetV3 model using the first sample image set may include S401-S405.

[0175] S401: The electronic device obtains a second sample image set, wherein the second sample image set includes a plurality of second sample original images.

[0176] The second sample original image is of higher quality, such as an image with higher definition (or referred to as a standard image).

[0177] In some embodiments, if the format of the second sample original image is not in RGB format, the electronic device may first convert the format of the second sample original image into RGB format for training using the second sample original image in RGB format.

[0178] In some embodiments, since model training actually refers to the process of how to restore an image of high quality from an image of low quality. Therefore, after obtaining a second sample original image of high quality, for each second sample original image, the electronic device can perform degradation processing on the clarity of the second sample original image to obtain an image of poor clarity corresponding to the second sample original image, that is, obtain a sample degraded image corresponding to the second sample original image, so that the electronic device can use each second sample original image and each sample degraded image corresponding to the first sample original image to train the model. It should be understood that the content of the second sample original image and the sample degraded image corresponding to the second sample original image, that is, the object is the same, but compared with the second sample original image, the clarity of the object in the sample degraded image corresponding to the second sample original image is poor, such as the object with higher clarity originally becomes blurred. The following will introduce a possible process of degrading the clarity of the second sample original image in combination with S402-S403.

[0179] S402: For each second sample original image among the plurality of second sample original images, the electronic device downsamples the second sample original image to obtain a downsampled second sample original image.

[0180] The size corresponding to the downsampled second sample original image is smaller than the size of the second sample original image.

[0181] S403: The electronic device upsamples the downsampled second sample original image to obtain a sample degraded image corresponding to the second sample original image, wherein the size of the sample degraded image corresponding to the second sample original image is equal to the second sample original image.

[0182] In the embodiment of the present application, for each second sample original image, the electronic device may first downsample the high-quality second sample original image, and then use an image interpolation algorithm (such as a bicubic algorithm) to upsample the downsampled second sample original image to obtain a low-quality image with the same size as the second sample original image but with reduced resolution, that is, reduced clarity, that is, a sample degraded image is obtained, and the sample degraded image simulates the low-definition image corresponding to the second sample original image. In this way, the second sample original image and its corresponding sample degraded image constitute a pair of texture clarity enhanced image sets.

[0183] S404: The electronic device calculates the variance of the second sample original image to obtain the complexity corresponding to the second sample original image.

[0184] For example, the electronic device may first convert the second sample original image into a grayscale image. Afterwards, the electronic device may calculate the variance of the second sample original image using the pixel values ​​in the grayscale image. The variance may reflect the degree of discreteness of the color distribution of the second sample original image. A larger variance indicates a more drastic change in color brightness and a higher complexity of the second sample original image. A smaller variance indicates a more gradual change in color brightness and a lower complexity of the second sample original image.

[0185] S405, the electronic device trains the second MobileNetV3 model based on each second sample original image, the sample degraded image corresponding to each second sample original image, and the complexity corresponding to each second sample original image to obtain a trained second MobileNetV3 model. The trained second MobileNetV3 model can predict the texture adjustment parameters corresponding to the images in the videos under different scenes.

[0186] Among them, the texture adjustment parameter is used to enhance the texture of the image and improve the clarity of the image.

[0187] In an embodiment of the present application, the electronic device may use the variance corresponding to the second sample original image as a label of the second sample original image. Then, the electronic device may input the second sample original image with the label and its corresponding sample degraded image into the second MobileNetV3 model, so that the second MobileNetV3 model learns how to restore the sample degraded image corresponding to the second sample original image to the second sample original image, and determines the mapping relationship between texture complexity (or complexity) and texture adjustment parameters (or texture clarity strength), such as generating a mapping curve between texture complexity and texture adjustment parameters.

[0188] It should be understood that for scenes with low texture complexity, the required texture clarity strength gradually decreases. If the texture complexity of an image is low, and a larger texture clarity strength is used to adjust the texture of the image, over-sharpening may occur, resulting in unnatural and distorted images. For scenes with high texture complexity, too high texture clarity strength is also not required. The higher the texture complexity, the smaller the texture clarity strength needs to be. If the texture complexity of an image is large, and a larger texture clarity strength is used to adjust the texture of the image, over-sharpening may also occur. Therefore, the texture clarity strength corresponding to scenes with medium texture complexity should be higher. In order to achieve a mapping curve that satisfies this rule, the second MobileNetV3 model can use four tangent quadratic curves to fit a curve, as shown below. Fig. 9 The mapping curve between texture complexity and texture clarity is shown in Figure 1. Fig. 9 The range of texture complexity in is 0 to 1, which is mapped to 0 to α through a curve. The parameters at the junction between curve segments can be adjusted by adjusting Fig. 9 The parameters at the five junction points shown can change the overall shape of the curve. Among them, λ determines the maximum texture clarity strength, and α determines the texture complexity corresponding to the maximum texture clarity strength. X1 and x2 determine the junction position of the curve segment, which is used for fine-tuning the wireless details. μ determines the average texture clarity strength of the wireless as a whole. The second MobileNetV3 model is based on the overall adjustment plus the fine-tuning of the details, and finally determines a suitable mapping curve, thereby ensuring the mapping accuracy between texture complexity and texture clarity strength.

[0189] In some embodiments, the second MobileNetV3 model has a similar structure to the first MobileNetV3 model.

[0190] Optionally, the last fully connected layer of the second MobileNetV3 model may include three fully connected layers, namely fully connected layer a, fully connected layer b and fully connected layer c. Among them, the number of neurons on the relevant feature map input to the fully connected layer a is 1024, and the fully connected layer a outputs 512 neurons; the Dropout (0.5) of the fully connected layer a, that is, the dropout rate is 0.5, and the activation function used by the fully connected layer a is the ReLU activation function. The number of neurons on the relevant feature map input to the fully connected layer b is 512, and the fully connected layer b outputs 256 neurons; the Dropout (0.2) of the fully connected layer b, that is, the dropout rate is 0.5, and the activation function used by the fully connected layer b is the ReLU activation function. The number of neurons on the relevant feature map input to the fully connected layer c is 256, and the fully connected layer c outputs 1 neuron; the Dropout (0.1) of the fully connected layer c, that is, the dropout rate is 0.1, and the activation function used by the fully connected layer c is the ReLU activation function.

[0191] During the training process, the electronic device freezes all layers except the parameters of the last fully connected layer in the second MobileNetV3 model, and only trains the last fully connected layer. The electronic device can first downsample the input image (such as the second sample original image and its corresponding sample degraded image) to 256x256, set the learning rate to 0.0001, the number of iterations (epochs) to 100, and use the Adam optimizer for training. It should be understood that the electronic device can also not downsample the input image, but directly use the original image for training, but the training time is longer.

[0192] The training process of the image processing model is introduced above. After the image processing model is obtained, the target device to which the image processing model is applied can use the image processing model to predict the quality adjustment parameters of the image in the video, so as to adjust the image using the quality adjustment parameters and improve the image quality, thereby improving the video quality and ensuring the user's visual experience. The target device may be the electronic device that obtains the image processing model through training as described above, or it may not be, and this application does not limit it.

[0193] Exemplarily, the target device acquires a video stream in real time, and the video stream may be generated by the target device itself, such as when the target device is in a recording state, generating a corresponding video stream. Alternatively, the video stream may be a video stream sent by other devices, such as when the target device is in a collaborative state, displaying the display content of other devices, and the image in the video stream includes the display content of other devices.

[0194] Afterwards, the target device can input the image (or referred to as the first image) in the video stream into the image processing model, and the image processing model processes the image to obtain the quality adjustment parameter corresponding to the image and output the quality adjustment parameter. Afterwards, the target device uses the quality adjustment parameter to adjust the quality of the image, obtain the target first image, and display the target first image to ensure the user's visual experience.

[0195] In some embodiments, the above-mentioned quality adjustment parameters may include color adjustment parameters, which are used to enhance the color quality of the image. In one case, the color adjustment parameters include brightness adjustment parameters. The target device uses the brightness adjustment parameter to increase the brightness of the first image. And, the target device uses the contrast adjustment parameter to increase the contrast of the first image. And the target device uses the saturation adjustment parameter to increase the contrast of the first image. Since the brightness is related to the pixel value of the Y channel of the image, the brightness of the first image can be increased by adjusting the pixel value of the Y channel of the pixel point in the first image using the brightness adjustment parameter. Exemplarily, for each pixel point in the first image, the target device uses the brightness adjustment parameter as the index of the Y channel pixel point of the pixel point to obtain the pixel value of the Y channel of the pixel point after brightness enhancement. In an embodiment of the present application, for each pixel point in the first image, the target device can use Y L =Y0 L , calculate the pixel value of the Y channel of the pixel after brightness enhancement to enhance the brightness of the first image. L represents the pixel value of the Y channel after brightness enhancement of the pixel point, L represents the brightness adjustment parameter, and Y0 represents the original pixel value of the Y channel of the pixel point, that is, the pixel value of the Y channel of the pixel point in the first image.

[0196] In another case, the color adjustment parameter includes a contrast adjustment parameter. Since contrast refers to the degree of difference between brightness values ​​in an image, it reflects the difference in brightness between different areas in the image. It can be seen that the contrast of an image is also related to the pixel value of the Y channel of the image. Therefore, the target device can also adjust the pixel value of the Y channel of the pixel point in the first image by using the contrast adjustment parameter to improve the contrast of the first image. Exemplarily, for each pixel point in the first image, the target device can obtain the adjustment value of the Y channel of the pixel point based on the average value of the pixel value of the Y channel after brightness enhancement of the pixel point and the pixel value of the Y channel after brightness enhancement corresponding to the first image, combined with the contrast adjustment parameter. Among them, the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image represents the average value of the pixel value of the Y channel after brightness enhancement of each pixel point in the first image. Exemplarily, the target device can calculate the difference (or second difference) between the pixel value of the Y channel after brightness enhancement of the pixel point and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image. Then, the target device can determine the adjustment value of the Y channel of the pixel point based on the second difference and the contrast adjustment parameter.

[0197] Afterwards, the target device obtains the pixel value of the Y channel after contrast enhancement based on the adjusted value of the Y channel of the pixel and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image, and improves the contrast of the first image by reducing the brightness value of the darker pixel points in the first image and increasing the brightness value of the brighter pixel points in the first image, thereby enhancing the contrast of the first image.

[0198] Specifically, for each pixel in the first image, the target device calculates the second difference between the pixel value of the Y channel after brightness enhancement and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image. Then, the target device calculates the product of the second difference and the contrast adjustment parameter to obtain the adjustment value of the Y channel of the pixel. Then, the target device calculates the sum of the adjustment value of the Y channel of the pixel and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image to obtain the pixel value of the Y channel after contrast enhancement of the pixel.

[0199] In the embodiment of the present application, for each pixel in the first image, the target device may use Y C =(Y L -mean0)*C+mean0, calculate the pixel value of the Y channel of the pixel after contrast enhancement to achieve contrast enhancement of the first image. Crepresents the pixel value of the Y channel after contrast enhancement of the pixel point, C represents the contrast adjustment parameter, mean0 represents the average value of the pixel value of the Y channel after contrast enhancement corresponding to the first image, so as to achieve contrast enhancement of the first image.

[0200] It is understandable that, although both contrast and brightness enhancement are for the pixel value of the Y channel, for the same pixel point, the pixel value of the Y channel after contrast enhancement may change compared to the pixel value of the Y channel after brightness enhancement, such as being too large or too small, that is, the pixel value of the Y channel after contrast enhancement may be smaller than the pixel value of the original Y channel. In other words, the brightness of the pixel point will become darker, but it can be determined through experimental tests that, from the perspective of the first image as a whole, the overall brightness of the first image is still enhanced, that is, the brightness has been enhanced. In addition, the pixel value of the Y channel after contrast enhancement may be larger than the pixel value of the Y channel after brightness enhancement. In other words, the brightness of the pixel point will become brighter, but it can be determined through experimental tests that, from the perspective of the first image as a whole, the overall brightness of the first image will not be too bright.

[0201] In another case, the color adjustment parameter includes a saturation adjustment parameter. Since saturation indicates the purity and vividness of a color, and in the YUV color space, the UV channel indicates the chromaticity information of the image, that is, the color in the image, that is, the saturation is related to the pixel values ​​of the U and V channels of the image. Therefore, the target device can use the saturation adjustment parameter to adjust the pixel value of the U channel and the pixel value of the V channel of the pixel point in the first image respectively, thereby improving the saturation of the first image. Specifically, for each pixel point in the first image, the target device calculates the product of the pixel value of the U channel of the pixel point and the saturation adjustment parameter to obtain the pixel value of the U channel of the pixel point after saturation enhancement. In an embodiment of the present application, for each pixel point in the first image, the target device can use the U S =U0*S, calculate the pixel value of the U channel after saturation enhancement of the pixel point. S represents the pixel value of the U channel after saturation enhancement of the pixel point, and S represents the saturation adjustment parameter. U0 represents the original pixel value of the U channel of the pixel point, that is, the pixel value of the U channel of the pixel point in the first image.

[0202] And, for each pixel in the first image, the target device calculates the product of the pixel value of the V channel of the pixel and the saturation adjustment parameter to obtain the pixel value of the V channel after the saturation is enhanced. In the embodiment of the present application, for each pixel in the first image, the target device may use V S=V0*S, calculate the pixel value of the V channel after saturation enhancement. Where V S V0 represents the pixel value of the V channel after saturation enhancement of the pixel point. V1 represents the original pixel value of the V channel of the pixel point, that is, the pixel value of the V channel of the pixel point in the first image.

[0203] In another case, the color adjustment parameters include brightness adjustment parameters and contrast adjustment parameters. Specifically, for each pixel in the first image, the target device can use Y L =Y0 L , calculate the pixel value of the Y channel after contrast enhancement to achieve brightness enhancement. After that, the target device can use Y C =(Y L -mean0)*C+mean0, calculate the pixel value of the Y channel of the pixel after contrast enhancement, so as to enhance the contrast of the first image.

[0204] In another case, the color adjustment parameter includes a brightness adjustment parameter and a saturation adjustment parameter. Specifically, for each pixel in the first image, the target device can use Y L =Y0 L , calculate the pixel value of the Y channel after brightness enhancement to achieve brightness enhancement. And the target device can use U S =U0*S, calculate the pixel value of the U channel after saturation enhancement, and use V S =V0*S, calculate the pixel value of the V channel of the pixel after saturation enhancement, and realize saturation enhancement of the first image.

[0205] In another case, the color adjustment parameter includes a contrast adjustment parameter and a saturation adjustment parameter. Specifically, for each pixel in the first image, the target device can use Y C =(Y L -mean0)*C+mean0, calculate the pixel value of the Y channel of the pixel after contrast enhancement, and enhance the contrast of the first image. And the target device can use U S =U0 / S, calculate the pixel value of the U channel after saturation enhancement, and use V S =V0*S, calculate the pixel value of the V channel of the pixel after saturation enhancement, and realize saturation enhancement of the first image.

[0206] In another case, the color adjustment parameters include brightness adjustment parameters, contrast adjustment parameters and saturation adjustment parameters. Specifically, for each pixel in the first image, the target device can use Y L =Y0L , calculate the pixel value of the Y channel after brightness enhancement to achieve brightness enhancement. After that, the target device can use The pixel value of the Y channel of the pixel point after contrast enhancement is calculated to achieve contrast enhancement of the first image.

[0207] And the target device can use U S =U0*S, calculate the pixel value of the U channel after saturation enhancement, and use V S =V0*S, calculate the pixel value of the V channel after saturation enhancement of the pixel point, and realize the saturation enhancement of the first image. At this time, the color enhancement effect of the first image is optimal, that is, the color display effect of the first image is the best.

[0208] In some embodiments, the above-mentioned process of using the image processing model to perform color enhancement on the image in the video can be performed in real time. For example, in the case of a game, after the target device receives the game screen data sent by the server, the image processing model is used in real time to perform color enhancement on the game screen data, and then the enhanced game screen data is displayed. That is to say, each time the target device obtains a frame of the original image, it performs color enhancement on the original image, and then displays the color-enhanced image, so that the user perceives an image with a better color display effect, thereby ensuring the user's visual experience.

[0209] In some embodiments, before inputting the first image into the image processing model, if the format of the first image is not in YUV format, the target device may first convert the format of the first image into RGB format, and then input the first image in RGB format into the image processing model.

[0210] In another embodiment, the above-mentioned quality adjustment parameters may include texture adjustment parameters. The target device may input the first image into an image processing model, and the image processing model processes the first image to obtain the texture adjustment parameters corresponding to the first image, and the texture adjustment parameters corresponding to the first image are accurately determined. Afterwards, the target device may determine the high-frequency information of the first image. Afterwards, for each pixel point in the high-frequency information of the first image, the target device multiplies the pixel value of the Y channel of the pixel point by the texture adjustment parameter to obtain the pixel value of the Y channel of the pixel point after texture enhancement, thereby achieving accurate sharpening processing of the high-frequency area of ​​the first image and improving the clarity of the first image.

[0211] In some embodiments, the process of determining the high-frequency area of ​​the first image may include: first, the target device performs mean filtering on the pixel values ​​of the Y channel of the pixel points in the first image to obtain the blurred first image (or the first filtered image). Afterwards, the pixel value of the Y channel of each pixel point in the first image is subtracted from the pixel value of the Y channel of the pixel point at the corresponding position in the blurred first image to obtain the high-frequency information of the first image (or high-frequency edge information, edge information), so as to blur the non-high-frequency information of the first image, such as low-frequency information, that is, the part with less texture information. The blurred first image is the high-frequency information of the first image, that is, the part where the texture needs to be enhanced. For example, the pixel value of the Y channel of the first pixel point in the first image is subtracted from the pixel value of the Y channel of the corresponding position in the blurred first image, that is, the first pixel point, to obtain the pixel difference corresponding to the first pixel point in the first image. By analogy, the pixel difference corresponding to each pixel point in the first image can be obtained, and the pixel difference corresponding to each pixel point in the first image constitutes the high-frequency information of the first image.

[0212] In some embodiments, the target device calculates the product of the high-frequency information of the first image and the texture adjustment parameter λ to obtain the texture adjustment value of the Y channel of each pixel in the first image. Exemplarily, the target device may multiply the pixel value of the Y channel of each pixel in the high-frequency information of the first image by the texture adjustment parameter, that is, multiply the pixel difference corresponding to each pixel in the first image by the texture adjustment parameter to obtain the texture adjustment value of the Y channel corresponding to each pixel in the first image.

[0213] Afterwards, for each pixel in the first image, the target device adds the pixel value of the Y channel of the pixel (i.e., the original pixel value of the Y channel) to the texture adjustment value at the corresponding position to obtain the pixel value of the Y channel of the pixel after texture enhancement. For example, the pixel value of the Y channel of the first pixel in the first image is added to the texture adjustment value at the corresponding position, i.e., the texture adjustment value of the Y channel of the first pixel, to obtain the pixel value of the Y channel of the first pixel in the first image after texture enhancement.

[0214] Afterwards, the target device synthesizes the pixel values ​​of the Y channel and the UV channel of the pixel points of the first image after texture enhancement to obtain the first image after texture enhancement, that is, to obtain the target first image, thereby obtaining a texture enhanced video stream. It can be understood that the color of the target first image is also enhanced.

[0215] In some embodiments, after obtaining the high-frequency information of the first image, a threshold may be used to determine whether to retain part of the high-frequency information of the first image to suppress the noise generated by texture enhancement. Exemplarily, for the pixel value of the Y channel of each pixel point in the high-frequency information of the first image, the target device may determine whether the pixel value of the Y channel of the pixel point is greater than or equal to threshold 1. If it is greater than or equal to threshold 1, the target device may retain the pixel point, that is, the high-frequency information of the first image still includes the pixel value of the pixel point.

[0216] When the value is less than the threshold value 1, the target device may discard the pixel point, that is, the high-frequency information of the first image does not include the pixel value of the pixel point.

[0217] In the embodiment of the present application, in order to reduce the computing power and improve the speed of adjusting the texture clarity of the image, the target device can perform real-time texture clarity enhancement processing on the first image in the video stream in the YUV control. And since Y corresponds to brightness, UV corresponds to the chromaticity of color, and UV has little effect on texture clarity, the target device can only enhance the pixel value of the Y channel of the first image to ensure the texture enhancement effect of the first image and reduce the amount of data processing. In addition, the target device only performs texture enhancement on the high-frequency information of the first image, and does not perform texture enhancement on the non-high-frequency information, that is, the area with less texture information, to avoid unnecessary texture adjustment. For example, the first image includes a puppy image part and a background image part, and the background image part refers to the blue sky, which includes less texture information, so it can be blurred. The puppy image part includes the puppy's hair, which is high-frequency information, so it is necessary to enhance the texture clarity of the puppy's hair to make the detail display effect of the puppy image part clearer. It can be simply understood that determining the high-frequency information of the first image is to determine the lines of the puppy's hair, then increase the brightness of these lines, and then combine them with the pixel values ​​of the UV channel of the original first image to obtain a first image with higher definition.

[0218] In some embodiments, the high-frequency information may include noise. For example, if the video stream is a video stream generated by recording a video, the image in the captured video stream may include white spots, which are noise. Therefore, the target device may perform mean filtering on the high-frequency information of the first image again to obtain the high-frequency information of the filtered first image. Afterwards, for the pixel value of the Y channel of each pixel in the high-frequency information of the filtered first image, the target device may determine whether the pixel value of the Y channel of the pixel is less than threshold 2. If it is less than threshold 2, it indicates that the pixel is likely to be noise, and the target device may discard the pixel, that is, discard the pixel value of the Y channel of the pixel. If it is greater than or equal to threshold 2, the pixel may be retained. Afterwards, for each pixel in the high-frequency information of the filtered first image, the target device multiplies the pixel value of the Y channel of the pixel by the texture adjustment parameter to obtain the target first image.

[0219] In some embodiments, the mean filtering method used in the above blur processing is only an example, and the target device may also use other methods to blur the first image, such as Gaussian blur (or Gaussian filtering), Laplace operation, median filtering, etc. The difference in the high-frequency information of the first image determined by different blur processing methods is small, but compared with other blur processing methods, the performance of mean filtering is higher and the speed of determining the high-frequency information of the first image is faster, so that the texture enhancement rate of the first image can be effectively improved, and the real-time texture enhancement of the video stream can be ensured.

[0220] In the embodiment of the present application, after obtaining the high-frequency information of the first image, the target device uses the texture adjustment parameters that match the scene of the first image to adjust the high-frequency information of the first image, thereby achieving adaptive texture adjustment and avoiding excessive or insufficient texture enhancement of the first image. In addition, in order to reduce the influence of noise, after obtaining the high-frequency information of the first image, a mean filter is performed again, and some pixels in the high-frequency information of the first image are discarded according to the threshold value. After two noise reduction processes, the noise generated by texture enhancement is effectively suppressed.

[0221] In some embodiments, when the above-mentioned image processing model has predicted color adjustment parameters and texture adjustment parameters, the color adjustment parameters include brightness adjustment parameters and / or contrast adjustment parameters. Although the pixel values ​​of the Y channel of the first image are processed based on brightness and contrast, and the texture adjustment parameters are also processed based on the pixel values ​​of the Y channel of the first image, the texture adjustment parameters mainly adjust the pixel values ​​of the Y channel in the high-frequency area of ​​the first image, while the brightness and contrast adjustment parameters adjust the pixel values ​​of the Y channel of the entire pixel points of the first image. Therefore, after adjusting the pixel values ​​of the Y channel of the pixel points based on the color adjustment parameters and the contrast adjustment parameters, the color quality (such as brightness, contrast) and clarity of the first image as a whole can still be improved.

[0222] Exemplarily, when the pixel values ​​of the channels of the pixel points in the first image need to be processed using the color adjustment parameters and the texture adjustment parameters, for each pixel point in the image, the color adjustment parameters can be used to determine the color-enhanced pixel value of the corresponding channel of the pixel point, and the texture adjustment parameters can be used to determine the clarity-enhanced pixel value of the Y channel of the pixel point. If the channel corresponding to the pixel point also includes the Y channel, the pixel value of the target Y channel of the pixel point can be determined based on the pixel value of the Y channel after color enhancement and the pixel value of the Y channel after clarity enhancement, such as calculating the average value between the two as the pixel value of the target Y channel, or using the pixel value of the Y channel after color enhancement or the pixel value of the Y channel after clarity enhancement as the pixel value of the target Y channel.

[0223] For example, after adjusting the pixel value of the Y channel of each pixel point in the first image based on the brightness adjustment parameter, the pixel value of the Y channel of each pixel point after brightness enhancement is obtained. The pixel value of the Y channel of the pixel point in the high-frequency part of the first image is adjusted by using the texture adjustment parameter to obtain the pixel value of the Y channel of the pixel point in the high-frequency part after clarity enhancement. For the same pixel point, the average value between the pixel value of the Y channel of the pixel point after brightness enhancement and the pixel value of the Y channel of the pixel point after clarity enhancement can be calculated to obtain the pixel value of the target Y channel of the pixel point. Although the pixel value of the target Y channel of the pixel point may be smaller than the pixel value of the Y channel of the pixel point after brightness enhancement, the texture adjustment is for the high-frequency part of the first image. Overall, the brightness of the first image is still enhanced. And although the pixel value of the target Y channel of the pixel point may be smaller than the pixel value of the Y channel of the pixel point after clarity enhancement, the clarity of the first image can still be improved. In summary, although adjusting the image using both texture adjustment parameters and color adjustment parameters may change the pixel value of the Y channel of the high-frequency part of the pixels, that is, the brightness of the high-frequency part of the pixels changes, it will not affect the overall brightness display effect of the first image. Overall, the brightness of the first image is still enhanced, and the clarity is also enhanced.

[0224] In some embodiments, Fig.10 As shown, before the target device inputs the first image in the video stream into the image processing model, it can first perform scene detection to determine whether the scene has changed, that is, to determine whether the first image and the previous frame image belong to the same scene. In the case of scene change, it indicates that they do not belong to the same scene, indicating that the display content between the first image and the previous frame image is quite different, the quality difference is quite large, and the quality adjustment parameter corresponding to the previous frame image cannot be used to adjust the quality of the current image (i.e., the first image). The target device can input the first image into the target image processing model so that the image processing model processes the image, obtains the quality adjustment parameter corresponding to the image, and outputs the quality adjustment parameter. Afterwards, the target device uses the quality adjustment parameter to adjust the pixel value of the pixel point in the image.

[0225] When the scene has not changed, indicating that it belongs to the same scene, the target device can directly use the quality adjustment parameters corresponding to the previous frame image, that is, use the quality adjustment parameters corresponding to the previous frame image as the current image, that is, the quality adjustment parameters corresponding to the first image, to adjust the pixel values ​​(or original pixel values) of the pixels in the first image.

[0226] In some embodiments, the target device may use the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image to determine whether the first image and the previous frame of the image belong to the same scene, so as to obtain the scene detection result corresponding to the first image. The following describes a possible implementation method for determining the scene detection result.

[0227] In one implementation, since the HSV format corresponds to different dimensions, such as hue, saturation, and brightness, it is possible to determine whether the scene of the first image has changed in different dimensions, thereby ensuring that the obtained scene detection result is more accurate. The following will take the above-mentioned target channels including the H channel, the S channel, and the V channel as an example to describe in detail how to use HSV to obtain the scene detection result corresponding to the first image.

[0228] First, for the hue (H) dimension, the target device can calculate the difference between the pixel value of the H channel of the downsampled first image and the pixel value of the H channel of the previous frame image after downsampling of the first image, and obtain the difference of the H channel corresponding to the first image, and the difference of the H channel indicates the degree of hue difference between the first image and the previous frame image. The larger the difference of the H channel, the greater the degree of hue difference, the less the first image and the previous frame image include the same display content, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each pixel point (or first pixel point) in the downsampled first image, the target device can calculate the difference between the pixel value of the H channel of the first pixel point and the pixel value of the H channel of the pixel point (or second pixel point) at the corresponding position in the previous frame image, and obtain the difference of the H channel corresponding to the first pixel point. Among them, the position of the second pixel point at the corresponding position in the previous frame image in the previous frame image is the same as the position of the first pixel point in the first image. Afterwards, the target device may calculate the difference of the H channel corresponding to all the first pixels in the first image to obtain the difference of the H channel corresponding to the first image. Calculate the difference of the H channel corresponding to the first image. h represents the difference of the H channel corresponding to the first image, num_piexls represents the total number of pixels in the first image, current_H i represents the i-th pixel in the first image, last_H i Represents the i-th pixel in the previous frame of the first image.

[0229] For the saturation (S) dimension, the target device can calculate the difference between the pixel value of the S channel of the downsampled first image and the pixel value of the S channel of the previous frame image after downsampling of the first image, and obtain the difference of the S channel corresponding to the first image, and the difference of the S channel indicates the degree of difference in saturation between the first image and the previous frame image. The larger the difference of the S channel, the greater the degree of difference in saturation, the less the first image and the previous frame image include the same display content, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each first pixel point in the downsampled first image, the target device can calculate the difference between the pixel value of the S channel of the first pixel point and the pixel value of the H channel of the second pixel point at the corresponding position in the previous frame image, and obtain the difference of the S channel corresponding to the first pixel point. Afterwards, the target device can calculate the difference of the S channels corresponding to all the first pixels in the first image, and obtain the difference of the S channels corresponding to the first image. Specifically, the target device can use Calculate the difference of the S channel corresponding to the first image. s Indicates the difference of the S channel corresponding to the first image, current_S i Represents the pixel value of the S channel of the i-th pixel in the first image, last_S i Represents the pixel value of the S channel of the i-th pixel in the previous frame of the first image.

[0230] For the brightness (V) dimension, the target device can calculate the difference between the pixel value of the V channel of the downsampled first image and the pixel value of the V channel of the previous frame image after downsampling of the first image, and obtain the difference of the V channel corresponding to the first image, and the difference of the V channel indicates the degree of brightness difference between the first image and the previous frame image. The larger the difference of the V channel, the greater the degree of brightness difference, the less the first image and the previous frame image include the same display content, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each first pixel point in the downsampled first image, the target device can calculate the difference between the pixel value of the V channel of the first pixel point and the pixel value of the V channel of the second pixel point at the corresponding position in the previous frame image, and obtain the difference of the V channel corresponding to the first pixel point. Afterwards, the target device can calculate the difference of the V channels corresponding to all the first pixels in the first image, and obtain the difference of the V channels corresponding to the first image. Specifically, the target device can use Calculate the difference of the V channel corresponding to the first image. v represents the difference of the V channel corresponding to the first image, current_V represents the pixel value of the V channel of the i-th pixel in the first image, and last_V iIndicates the pixel value of the V channel of the i-th pixel in the previous frame of the first image.

[0231] It should be noted that the above-described process of determining the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image is only an example. The target device may not use each pixel point in the first image to determine it, but may use several pixel points in the first image to determine it. This application does not limit the method of determining the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image.

[0232] Afterwards, the target device may obtain the scene detection result corresponding to the first image according to the difference of the H channel, the S channel, and the V channel corresponding to the first image. Exemplarily, the target device may calculate the average of the difference of the H channel, the S channel, and the V channel corresponding to the first image to obtain the color difference corresponding to the first image. Specifically, the target device may use Calculate the color difference corresponding to the first image. Score represents the color difference corresponding to the first image.

[0233] When the average value, that is, the color difference corresponding to the first image is greater than the threshold value 3, it indicates that the display content of the first image and the previous frame image is quite different, that is, the quality difference between the first image and the previous frame image is quite different, the scene difference between the first image and the previous frame image is quite different, the first image and the previous frame image do not belong to the same scene, and the target device can determine that the scene detection result corresponding to the first image indicates that the scene has changed.

[0234] When the average value is less than or equal to threshold 1 (such as 27), it indicates that the display content of the first image and the previous frame image is slightly different, that is, the quality difference between the first image and the previous frame image is small, the scene difference between the first image and the previous frame image is small, and the first image and the previous frame image belong to the same scene. The target device can determine that the scene detection result corresponding to the first image indicates that the scene has not changed.

[0235] It is worth mentioning that the average value of the difference of the H channel, the difference of the S channel, and the difference of the V channel is only an example, and the scene detection result corresponding to the first image can also be determined according to other methods. For example, when the difference of the H channel corresponding to the first image is greater than a threshold value 4, the difference of the S channel is greater than a threshold value 5, and the difference of the V channel is greater than a threshold value 6, the target device determines that the scene detection result corresponding to the first image indicates that the scene has changed. When the difference of the H channel corresponding to the first image is less than or equal to a threshold value 4, the difference of the S channel is less than or equal to a threshold value 5, and the difference of the V channel is less than or equal to a threshold value 6, the target device determines that the scene detection result corresponding to the first image indicates that the scene has not changed.

[0236] In an embodiment of the present application, the target device downsamples (resizes) the first image in the video stream to increase the image processing speed and obtain the downsampled first image. Afterwards, the target device can convert the format of the downsampled first image into the HSV format to obtain the first image in the HSV format. Afterwards, the target device can respectively calculate the difference between the pixel value of the target channel of each first pixel point in the first image in the HSV format and the pixel value of the target channel of the second pixel point at the corresponding position of the previous frame image in the HSV format, and obtain the pixel difference of the target channel corresponding to each first pixel point. Afterwards, the target device can calculate the average value of the pixel difference of the target channel corresponding to each first pixel point to obtain the pixel difference of the target channel corresponding to the first image. Afterwards, the target device can calculate the average value of the pixel difference of the target channel corresponding to the first image to obtain the scene detection result corresponding to the first image, and the scene detection result indicates whether the scene has changed, so as to achieve accurate determination of the scene detection result. In addition, by downsampling the first image, the number of pixels to be calculated can be reduced, thereby reducing the amount of data processing, thereby improving the determination efficiency of the scene detection result.

[0237] The process of determining the previous frame image in the HSV format is similar to the process of determining the first image in the HSV format.

[0238] In some embodiments, generally speaking, the image format in the video stream is in YUV format, and the target device may first convert the YUV format of the downsampled first image into RGB format, and then convert the first image in RGB format into a first image in HSV format.

[0239] It should be understood that the above is an example of how to use HSV to obtain the scene detection result corresponding to the first image, taking the target channel including the H channel, the S channel and the V channel as an example. Of course, the target channel may include one or two channels of the H channel, the S channel and the V channel. The target device may calculate the difference of the target channel corresponding to the first image, so as to determine the detection result corresponding to the first image using the difference of the target channel corresponding to the first image. For example, the target channel includes the H channel and the S channel, and the target device calculates the difference of the H channel corresponding to the first image and the difference of the S channel corresponding to the first image. Afterwards, the target device may calculate the average of the difference of the H channel corresponding to the first image and the difference of the S channel corresponding to the first image. When the average is greater than a threshold value, the target device may determine that the scene detection result corresponding to the first image indicates that the scene has changed.

[0240] In an embodiment of the present application, when the target device performs quality enhancement (such as color enhancement) on the video stream, it first performs scene detection on the first image in the video stream to determine whether the current frame image (i.e., the first image) and the previous frame image of the first image belong to the same scene. If they belong to the same scene, the target device can use the color adjustment parameters corresponding to the previous frame image as the color adjustment parameters corresponding to the first image, without re-predicting the color adjustment parameters, to achieve rapid determination of the color adjustment parameters and improve the processing efficiency of the video image. If they do not belong to the same scene, the target device can re-predict the color adjustment parameters corresponding to the first image to obtain color adjustment parameters that match the color scene of the first image, so that the target device can use the color adjustment parameters that match the color scene of the first image to adjust the color of the first image and achieve adaptive color adjustment. And because there are errors in the prediction of the color adjustment parameters, that is, for the same scene, the color adjustment parameters that may be predicted may have errors. Therefore, when the current frame image and the previous frame image belong to the same scene, directly using the color adjustment parameters corresponding to the previous frame image to adjust the current frame image can avoid the different color display effects of two adjacent frames belonging to the same scene due to errors, thereby avoiding discontinuities between video frames (such as frame skipping, flickering, etc.).

[0241] In some embodiments, the target device can not only perform color enhancement on images in the video in real time, such as performing color enhancement on the first image after obtaining the first image in the video stream as mentioned above, but the target device can also perform color enhancement on images in video files that have been stored locally, as long as it can obtain pixel points in the image in the video (such as pixel values ​​of the YUV channel).

[0242] In some embodiments, the present application provides a computer storage medium including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method as described above.

[0243] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0244] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.

[0245] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0246] It should also be understood that in the present application, "when", "if" and "if" all mean that the UE or base station will take corresponding actions under certain objective circumstances. It is not a time limit, and it does not require the UE or base station to make judgments when implementing it, nor does it mean that there are other limitations.

[0247] A person skilled in the art may understand that the various numerical numbers such as first and second involved in the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application, and also indicate the order of precedence.

[0248] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one", unless otherwise specified. In this application, "at least one" is intended to mean "one or more", and "more than one" is intended to mean "two or more", unless otherwise specified.

[0249] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A can be singular or plural, and B can be singular or plural.

[0250] In this document, the term "at least one of..." or "at least one of..." means all or any combination of the listed items. For example, "at least one of A, B, and C" may mean: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time, where A may be singular or plural, B may be singular or plural, and C may be singular or plural.

[0251] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0252] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0253] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0256] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0257] The same or similar parts between the various embodiments in this application can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The above-described implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0258] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for training an image processing model, characterized in that: The image processing model is used to predict a quality adjustment parameter corresponding to a first image in a video stream, and the quality adjustment parameter is used to adjust the quality of the first image; the quality adjustment parameter includes a color adjustment parameter and / or a texture adjustment parameter; the color adjustment parameter includes at least one of a contrast adjustment parameter, a brightness adjustment parameter and a saturation adjustment parameter, the quality includes color and / or clarity, and the color includes at least one of contrast, brightness and saturation; The method comprises: Acquire a sample image set; wherein the sample image set includes a plurality of sample original images; For each sample original image among the plurality of sample original images, degrade the quality of the sample original image to obtain a sample degraded image corresponding to the sample original image; wherein the quality of the sample degraded image corresponding to the sample original image is lower than the quality of the sample original image; Based on each sample original image and the sample degraded image corresponding to each sample original image, a preset model is trained to obtain the image processing model; the preset model is a lightweight model.

2. The method according to claim 1, characterized in that The quality adjustment parameter includes a color adjustment parameter; the quality includes color; the plurality of sample original images include a plurality of first sample original images; The step of performing degradation processing on the quality of each sample original image among the plurality of sample original images to obtain a sample degraded image corresponding to the sample original image includes: For each first sample original image of the plurality of first sample original images, obtaining a color degradation value from a preset color degradation range; Based on the color degradation value, the color of the first sample original image is degraded to obtain a sample degraded image corresponding to the first sample original image; wherein the color of the sample degraded image corresponding to the first sample original image is lower than the color of the first sample original image.

3. The method according to claim 2, characterized in that The color adjustment parameter includes a brightness adjustment parameter, the color includes brightness, and the color degradation value includes a brightness degradation value; The step of performing degradation processing on the color of the first sample original image based on the color degradation value includes: For each pixel point in the first sample original image, based on the brightness degradation value, the pixel value of the brightness Y channel of the pixel point is reduced to obtain the pixel value of the brightness Y channel of the pixel point after brightness degradation.

4. The method according to claim 3, characterized in that The reducing the pixel value of the brightness Y channel of the pixel point based on the brightness degradation value includes: The reciprocal of the brightness degradation value is used as an index of the pixel value of the Y channel of the pixel point.

5. The method according to any one of claims 2 to 4, characterized in that The color adjustment parameter includes a contrast adjustment parameter, the color includes a contrast, and the color degradation value includes a contrast degradation value; The step of performing degradation processing on the color of the first sample original image based on the color degradation value includes: For each pixel point in the first sample original image, based on the brightness adjustment parameter, reduce the pixel value of the Y channel of the pixel point to obtain the pixel value of the Y channel of the pixel point after brightness degradation; Determine a first adjustment value of the Y channel of the pixel point based on an average value of the Y channel pixel values ​​of the pixel point after brightness degradation and each pixel point corresponding to the first sample original image after brightness degradation, combined with the contrast degradation value; Based on the first adjustment value of the Y channel of the pixel point and the average value, a pixel value of the Y channel of the pixel point after contrast degradation is obtained.

6. The method according to claim 5, characterized in that The step of determining the first adjustment value of the Y channel of the pixel point based on the pixel point after brightness degradation and the average value of the pixel values ​​of the Y channel of each pixel point corresponding to the first sample original image after brightness degradation, in combination with the contrast degradation value, comprises: Calculate the difference between the pixel point of the Y channel after brightness degradation of the pixel point and the average value; The difference between the pixel point of the Y channel after brightness degradation of the pixel point and the average value and the ratio between the contrast degradation values ​​are calculated to obtain a first adjustment value of the Y channel of the pixel point.

7. The method according to any one of claims 2 to 6, characterized in that The color adjustment parameter includes a saturation adjustment parameter, the color includes a saturation, and the color degradation value includes a saturation degradation value; The step of performing degradation processing on the color of the first sample original image based on the color degradation value includes: For each pixel in the first sample original image, based on the saturation degradation value, reduce the pixel value of the chroma channel of the pixel to obtain the pixel value of the chroma channel after saturation degradation of the pixel; The chrominance channels include a U channel and a V channel.

8. The method according to claim 7, characterized in that The reducing the pixel value of the chroma channel of the pixel point based on the saturation degradation value includes: The ratio between the pixel value of the chromaticity channel of the pixel point and the saturation degradation value is calculated.

9. The method according to any one of claims 1 to 8, characterized in that The quality adjustment parameter includes a texture adjustment parameter; the quality includes clarity; the plurality of sample original images include a plurality of second sample original images; The step of performing degradation processing on the quality of each sample original image among the plurality of sample original images to obtain a sample degraded image corresponding to the sample original image includes: For each second sample original image among the plurality of second sample original images, down-sampling the second sample original image; The downsampled second sample original image is upsampled to obtain a sample degraded image corresponding to the second sample original image; a size of the sample degraded image corresponding to the second sample original image is the same as a size of the second sample original image, and a resolution of the sample degraded image corresponding to the second sample original image is lower than a resolution of the second sample original image.

10. The method according to claim 9, characterized in that The training of the preset model based on each sample original image and the sample degraded image corresponding to each sample original image includes: For each of the second sample original images, calculating the variance of the second sample original image to obtain the complexity corresponding to the second sample original image; Based on each second sample original image, the sample degraded image corresponding to each second sample original image and the complexity corresponding to each second sample original image, the preset model is trained so that the preset model determines the mapping relationship between the complexity and the texture adjustment parameter.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Acquire a video stream; wherein the video stream includes at least one frame of a first image; For each first image in the at least one first image frame, taking the first image as an input parameter of the image processing model, running the image processing model, and obtaining a quality adjustment parameter corresponding to the first image; Adjusting the quality of the first image based on the quality adjustment parameter corresponding to the first image to obtain a target first image; The target first image is displayed.

12. The method according to claim 11, characterized in that The step of using the first image as an input parameter of an image processing model, running the image processing model, and obtaining a quality adjustment parameter corresponding to the first image includes: Obtaining a scene detection result corresponding to the first image based on a difference between a pixel value of a target channel of the first image and a pixel value of a target channel of an image of a previous frame of the first image; wherein the scene detection result corresponding to the first image indicates whether the first image and the image of the previous frame belong to the same scene; When the scene detection result indicates that the first image and the previous frame image do not belong to the same scene, the first image is used as an input parameter of an image processing model, and the image processing model is run to obtain quality adjustment parameters corresponding to the first image.

13. A method for enhancing image quality, characterized in that: include: Acquire a video stream; wherein the video stream includes at least one frame of a first image; For each first image in the at least one first image frame, the first image is used as an input parameter of the image processing model according to any one of claims 1 to 12, and the image processing model is run to obtain a quality adjustment parameter corresponding to the first image; the quality adjustment parameter includes a color adjustment parameter and / or a texture adjustment parameter; the color adjustment parameter includes at least one of a contrast adjustment parameter, a brightness adjustment parameter and a saturation adjustment parameter; the image processing model is a lightweight model; Based on the quality adjustment parameter corresponding to the first image, adjust the quality of the first image to obtain a target first image; the quality includes color and / or clarity, and the color includes at least one of contrast, brightness and saturation; The target first image is displayed.

14. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1 to 13.

15. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 13.

Images