Texture enhancement method and electronic equipment

Abstract

The invention provides a texture enhancement method and electronic equipment, and relates to the technical field of image processing. After an electronic device receives a video stream, scene detection can be performed on a current image in the video stream to determine whether the current image and a previous frame of image belong to the same scene. Under the condition of belonging to the same scene, the electronic equipment can directly take the texture adjustment parameter corresponding to the previous frame of image as the texture adjustment parameter corresponding to the current image, so that the texture adjustment parameter matched with the scene of the current image can be quickly determined. And under the condition that the current image does not belong to the same scene, the electronic equipment determines a texture adjustment parameter matched with the scene of the current image. Afterwards, the electronic equipment utilizes the texture adjustment parameter corresponding to the current image to enhance the texture definition of the current image, realizes real-time enhancement of the image texture, realizes accurate and rapid determination of the texture adjustment parameter, and ensures the texture enhancement effect, thereby ensuring the display effect of the video.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a texture enhancement method and electronic device. Background Art

[0002] Video is one of the ways to record and store information, which can enable users to obtain information more intuitively and conveniently. With the development of technology, video is widely used in different fields such as life, work, education, etc. At the same time, people's requirements for video quality are getting higher and higher.

[0003] At present, in order to improve the video quality, after obtaining the image in the video stream, the electronic device needs to use the specified texture adjustment parameters to adjust the pixel value of each pixel in the image to enhance the texture clarity of the image. However, different video images use the same texture adjustment parameters, which is not adaptive and may cause problems such as over-adjustment or under-adjustment, resulting in poor video display effect. Summary of the invention

[0004] In view of this, the present application provides a texture enhancement method and an electronic device for improving video quality and thus video display effect.

[0005] In a first aspect, the present application provides a texture enhancement method. After acquiring a video stream, the electronic device obtains, for each first image frame in the video stream, a scene detection result corresponding to the first image based on the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image; wherein the scene detection result corresponding to the first image indicates whether the first image and the previous frame of the image belong to the same scene, that is, whether the scene corresponding to the first image has changed.

[0006] When the scene detection result indicates that the first image and the previous frame image belong to the same scene, it means that the scene corresponding to the first image has not changed, and the electronic device can use the texture adjustment parameters corresponding to the previous frame image as the texture adjustment parameters corresponding to the first image; the texture adjustment parameters corresponding to the previous frame image are texture adjustment parameters that match the scene of the previous frame image.

[0007] When the scene detection result indicates that the first image and the previous frame image do not belong to the same scene, it means that the scene corresponding to the first image has changed. The electronic device can re-determine the texture adjustment parameters corresponding to the first image based on the first image, that is, determine that the texture adjustment parameters corresponding to the first image are texture adjustment parameters that match the scene of the first image.

[0008] Based on the texture adjustment parameter corresponding to the first image, the clarity of the high-frequency information of the first image, that is, the texture clarity, is adjusted to obtain a target first image. Afterwards, the electronic device can display the target first image.

[0009] The fact that the first image and the previous frame image belong to the same scene can be understood as the difference in contents (or objects) between the two frames of images is small, or the difference in display effects between the two frames of images is small.

[0010] In the present application, after receiving the current frame image (i.e., the first image), it is determined whether the first image and the previous frame image of the first image belong to the same scene. If they belong to the same scene, the electronic device can use the texture adjustment parameters corresponding to the previous frame image as the texture adjustment parameters corresponding to the first image, without re-predicting the texture adjustment parameters, to achieve rapid determination of the texture adjustment parameters and improve the processing efficiency of the video image. If they do not belong to the same scene, the electronic device can re-predict the texture adjustment parameters corresponding to the first image, and obtain the texture adjustment parameters matching the texture scene of the first image, so that the electronic device can use the texture adjustment parameters matching the texture scene of the first image to adjust the clarity of the high-frequency information of the first image, realize adaptive texture adjustment, improve the quality of the video image, and thus improve the texture display effect of the video. And because only the clarity of the high-frequency information of the first image is enhanced, the data processing amount can be reduced while ensuring the display effect of the first image, and the real-time processing of the video image can be guaranteed. And because there is an error in the prediction of the texture adjustment parameters, that is, for the same scene, there may be an error in the predicted texture adjustment parameters. Therefore, when the current frame image and the previous frame image belong to the same scene, directly using the texture adjustment parameters corresponding to the previous frame image to adjust the current frame image can avoid the different texture display effects of two adjacent frame images belonging to the same scene due to errors, ensure the continuity of the video frame display effect, and improve the user's visual experience.

[0011] In a possible design, the process of determining the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image may include:

[0012] For each pixel in the first image, the electronic device can calculate the difference between the pixel value of the target channel of the pixel and the pixel value of the target channel of the pixel at the corresponding position in the previous frame image, and obtain the difference of the target channel corresponding to the pixel in the first image. In other words, for each first pixel in the first image, the electronic device can calculate the difference between the pixel value of the target channel of the first pixel and the pixel value of the target channel of the corresponding second pixel in the previous frame image, and obtain the difference of the target channel corresponding to the first pixel. Among them, the position of the first pixel on the first image is the same as the position of the second pixel corresponding to the first pixel on the previous frame image.

[0013] The electronic device may obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image based on the average value of the difference of the target channel corresponding to each first pixel point in the first image.

[0014] In the present application, the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is determined based on the difference in the pixel value of the target channel of the pixel point between the current frame and the previous frame image, that is, the scene detection result is determined, and the scene detection result is determined by the overall difference value of two adjacent frames of images, thereby ensuring the accuracy of the scene detection result.

[0015] In a possible design, the number of the target channels is one or more. Accordingly, the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is obtained by averaging the difference values ​​of the target channels corresponding to the pixels in the first image, including:

[0016] For each target channel, the electronic device may calculate the average value of the difference values ​​of the target channels corresponding to each pixel point to obtain the difference value of the target channel corresponding to the first image. Afterwards, the electronic device may obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image by taking the average value of the difference values ​​of each target channel corresponding to the first image.

[0017] In the present application, the electronic device can determine the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image based on the difference between different target channels, that is, determine the scene detection result, and detect the degree of difference between the two frames of images from different dimensions to ensure the accuracy of the scene detection result.

[0018] In a possible design, the process of obtaining the scene detection result corresponding to the first image based on the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image may include:

[0019] When the difference in the target channel between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is less than or equal to the first threshold, it indicates that the difference in the pixel values ​​between the two frames of images is small, that is, the difference in the displayed objects is small. The electronic device can determine that the scene detection result corresponding to the first image indicates that the first image and the previous frame image belong to the same scene, that is, the scene of the first image has not changed.

[0020] When the difference in the target channel between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image is greater than the first threshold, it indicates that the pixel value difference between the two frames of images is large, that is, the display objects are very different. The scene detection result corresponding to the first image is determined to indicate that the first image and the previous frame image do not belong to the same scene, that is, the scene of the first image has changed, thereby achieving accurate detection of the scene of the first image.

[0021] In a possible design, the target channel includes at least one of a hue H channel, a saturation S channel, and a brightness V channel.

[0022] In a possible design, the process of adjusting the clarity of the high-frequency information of the first image may include: adjusting the pixel value of each pixel point (or referred to as a high-frequency pixel point) in the high-frequency information of the first image based on the texture adjustment parameter to obtain a texture adjustment value of each high-frequency pixel point in the high-frequency information;

[0023] For each pixel point in the first image (that is, the first pixel point), a pixel value of the first pixel point after texture enhancement is obtained based on the pixel value of the first pixel point and the texture adjustment value corresponding to the corresponding high-frequency pixel point in the high-frequency information. The position of the first pixel point in the first image is the same as the position of the high-frequency pixel point corresponding to the first pixel point in the high-frequency information of the first image, thereby achieving texture enhancement of the high-frequency information in the first image.

[0024] In one possible design, since the texture is mainly related to the pixel value of the brightness Y channel, the texture adjustment value of the above-mentioned pixel point (that is, the texture adjustment value corresponding to the high-frequency pixel point) includes the texture adjustment value of the brightness Y channel of the high-frequency pixel point, and accordingly, the pixel value after texture enhancement includes the pixel value of the Y channel after texture enhancement.

[0025] The above-mentioned texture enhancement process for the pixels in the first image may include:

[0026] Calculate the product of the pixel value of the Y channel of each high-frequency pixel point in the high-frequency information and the texture adjustment parameter respectively to obtain the texture adjustment value of the Y channel of each high-frequency pixel point in the high-frequency information;

[0027] For each first pixel in the first image, the pixel value of the Y channel of the first pixel and the texture adjustment value of the Y channel of the high-frequency pixel corresponding to the first pixel are calculated to obtain the Y channel pixel value of the first pixel after texture enhancement, thereby achieving accurate enhancement of the high-frequency information of the first image and ensuring the texture enhancement effect of the high-frequency information.

[0028] In a possible design manner, the process of determining the high-frequency information of the first image may include:

[0029] The electronic device may perform mean filtering on the first image to obtain a first filtered image. Afterwards, for each pixel point in the first image, the pixel value of the Y channel of the pixel point is subtracted from the pixel value of the Y channel of the pixel point at the corresponding position in the first filtered image to obtain the high-frequency information of the first image. In the present application, by using mean filtering to determine the high-frequency information of the first image, the speed of determining the high-frequency information can be improved while ensuring the accuracy of the high-frequency information of the determined first image.

[0030] In a possible design manner, the process of determining the texture color adjustment parameter corresponding to the first image may include:

[0031] The electronic device can use the first image as an input parameter of an image processing model, run the image processing model, and obtain texture adjustment parameters corresponding to the first image; the image processing model is a lightweight type of model, which can achieve accurate prediction of texture adjustment parameters that match the scene of the first image, and because the image processing model is a lightweight model, it has high performance, can process images faster, and can reduce power consumption.

[0032] In a possible design, in order to improve the processing speed of the image, the electronic device can downsample the first image to obtain the downsampled first image. Then, based on the difference between the pixel value of the target channel of the downsampled first image and the pixel value of the target channel of the previous frame image after downsampling of the first image, the scene detection result corresponding to the first image is obtained, so as to realize the rapid determination of the scene detection result.

[0033] In a second aspect, the present application provides an electronic device, comprising a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, wherein the computer program code comprises computer instructions; when the processor executes the computer instructions, the electronic device executes the method described above.

[0034] In a third aspect, the present application provides a computer storage medium, comprising computer instructions, which, when executed on an electronic device, enable the electronic device to execute the method as described above.

[0035] In a fourth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0036] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer storage medium described in the third aspect, and the computer program product described in the fourth aspect provided above can be referred to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1A A video scene schematic diagram 1 provided in an embodiment of the present application;

[0038] Figure 1B A video scene diagram provided in an embodiment of the present application Figure 2 ;

[0039] Figure 1C A third schematic diagram of a video scene provided in an embodiment of the present application;

[0040] Figure 1D A video scene diagram provided in an embodiment of the present application Figure 4 ;

[0041] Figure 1E A video scene diagram provided in an embodiment of the present application Figure 5 ;

[0042] Figure 1F A video scene diagram provided in an embodiment of the present application Figure 6 ;

[0043] Figure 1G Schematic diagram 1 of a texture enhancement process provided in an embodiment of the present application;

[0044] Figure 1H A texture enhancement process diagram provided in an embodiment of the present application Figure 2 ;

[0045] Figure 2 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0046] Figure 3A Schematic diagram 3 of a texture enhancement process provided in an embodiment of the present application;

[0047] Figure 3B A schematic diagram of a model training process provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of a mapping curve provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of a texture enhancement method provided in an embodiment of the present application;

[0050] Figure 6A schematic diagram of a scene detection process provided in an embodiment of the present application;

[0051] Figure 7 A texture enhancement process diagram provided in an embodiment of the present application Figure 4 ;

[0052] Figure 8 A texture enhancement process diagram provided in an embodiment of the present application Figure 5 ;

[0053] Fig. 9 A schematic diagram of a segmentation network model structure provided in an embodiment of the present application;

[0054] Fig.10 A texture enhancement process diagram provided in an embodiment of the present application Figure 6 . DETAILED DESCRIPTION

[0055] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0056] Video is composed of multiple frames of images. With the development of technology, video is widely used in different aspects, such as office, movies, conferences, games, collaboration, shooting, etc. Electronic devices can play videos, that is, display the images in the video. For example, when using electronic devices for work, electronic devices (such as user A's device) may share the desktop of other devices (such as user B's device). User B's device sends the desktop display data to user A's device, and user A's device displays the desktop of user B's device (such as user B's device). Figure 1A As shown), the desktop display data can be sent to the device of user A in the form of streaming media, that is, in the form of video stream. The desktop display data is equivalent to an image. It can be simply understood that the device of user B continuously sends an image including the desktop display content to the device of user A, so that the device of user A can display the image. Therefore, when the device of user B displays the content of the PPT file, the desktop display data includes the content of the displayed PPT file, that is, the image includes the content of the PPT file, and the content of the PPT file can be considered as an image. It should be understood that the PPT file can also be considered as an image in other cases, such as recording the content displayed by the electronic device, and the displayed content can include the content of the PPT file.

[0057] For example, in Figure 1BIn the collaborative situation shown, the mobile phone sends screen projection data to the PC in real time. The screen projection data is sent in the form of streaming media. The screen projection data includes the screen data displayed by the mobile phone. The PC displays the screen of the mobile phone according to the received screen projection data. Here, the screen can refer to the image in the video.

[0058] For example, in Figure 1C As shown, the mobile phone displays the game screen. Since the game screen is displayed continuously, one frame of the game screen can be considered as one frame of image in the video.

[0059] For example, Figure 1D As shown, the mobile phone plays XX movie and displays the pictures in XX movie. Here, XX movie is equivalent to a video, and the pictures in the movie are equivalent to images.

[0060] For example, Figure 1E As shown, the mobile phone is in a video recording state, and the mobile phone generates a corresponding video stream, which may include recorded video images. When the user wants to end the recording, the user may click the stop recording button 10 as shown in 1D, and the mobile phone stops recording in response to the click operation of the stop recording button 10, and generates a corresponding video file, which may include multiple frames of images.

[0061] For example, Figure 1F As shown, the mobile phone plays a short video and displays the images in the short video, where the short video can also be called a video.

[0062] Above Figures 1A to 1F The type of video shown is only an example, and other aspects may also involve videos, that is, the video may also be of other types, such as TV series type, etc., which is not limited in this application.

[0063] In order to improve the video quality, the electronic device can enhance the texture (or texture clarity) of the video image to improve the quality of the video image, thereby improving the display effect of the video image. In one implementation, for each frame of the video, the electronic device can directly use the specified adjustment parameter to adjust the pixel value of each pixel in the image to enhance the texture clarity of the image. For example, the pixel value includes the H channel pigment value, and the H channel pigment value of each pixel of the image is multiplied by the specified texture adjustment parameter to enhance the texture clarity of the image. However, different scenes have different requirements for the size of the texture adjustment parameter. For example, when the electronic device plays a movie, the texture clarity of the scene of some pictures in the movie is high, so the required texture adjustment parameter is small. If the texture adjustment parameter is too large, over-adjustment may occur, that is, when the sharpening is excessive, the picture will be distorted and other problems will occur (such as "XX" content is written on the clothes in the image. If the sharpening is excessive, the "XX" content is similar to the effect of pasting, rather than presenting the effect that the "XX" content is printed on the clothes themselves). The texture clarity in some scenes of the picture is low, so the required texture adjustment parameter is larger. If the texture adjustment parameter is too small, the adjustment may be too small, that is, the sharpening degree is too small, the texture clarity of the image is not effectively improved, and the details are still blurred.

[0064] In another implementation, Figure 1G As shown, the electronic device uses a conventional sharpening algorithm, such as an unsharp masking (USM) algorithm, to sharpen the video image to enhance the video clarity. The electronic device can use a sharpening operator (such as Roberts operator, Prewitt operator, Sobel operator, Laplacian operator, Kirsch operator, etc.) to extract the high-frequency information (or edge information) of the image. The sharpening operator can be a mask of size 2x2 or 3x3. The edge information of the image can be obtained by convolving or correlating the mask with the image. Among them, the principle of determining the edge information of the image is to calculate the gradient values ​​of the image in different directions using the first-order differential and / or the second-order differential to achieve edge detection. After obtaining the edge information of the image, the edge information is added or subtracted from the image (i.e., the original image) to achieve sharpening of the image and enhance the texture clarity of the image. However, this implementation method is not adaptive and cannot be adaptively adjusted according to the scene of the image in the video.

[0065] In another implementation, the edge information extraction of the above-mentioned image can be equivalent to high-pass filtering in the frequency domain. The electronic device can use a two-dimensional discrete Fourier transform to transform the image from the spatial domain (or time domain) to the frequency domain. In the frequency domain, places with high frequencies represent areas with relatively complex textures in the image, and places with low frequencies represent areas with relatively smooth textures. Therefore, in order to improve the texture clarity of the image, the electronic device can perform a high-pass filter on the frequency domain image to extract the high-frequency area of ​​the image. Then, the electronic device can use a two-dimensional discrete Fourier inverse transform to transform the high-frequency area of ​​the image from the frequency domain to the time domain. Afterwards, the electronic device can add the high-frequency area of ​​the time domain to the original image to obtain a texture-enhanced image.

[0066] However, the above implementation is not adaptive and cannot be adaptively adjusted according to the scene of the image in the video.

[0067] Therefore, in response to the above problems, the present application proposes a texture enhancement method. After the electronic device obtains the video stream, for each first image in the video stream, the electronic device can perform scene detection, that is, compare the pixel value of the target channel of the first image with the pixel value of the target channel of the previous frame of the first image, and obtain the scene detection result corresponding to the first image to determine whether the scene of the first image has changed, that is, to determine whether the degree of difference in texture information between the first image and the previous frame is large. In the case where the scene detection result corresponding to the first image indicates that the scene has changed, it indicates that the scene of the first image is very different from the scene of the previous frame, that is, the texture information of the scene of the first image is very different from that of the previous frame, and the texture display effect is very different. Therefore, if the scene detection result corresponding to the first image indicates that the scene has changed, it indicates that the scene of the first image is very different from that of the previous frame, that is, the texture information of the scene of the first image is very different from that of the previous frame, and the texture display effect is very different. Figure 1HAs shown, the electronic device needs to reuse the trained first MobileNetV3 model to predict the texture adjustment parameters corresponding to the first image, and obtain the texture adjustment parameters (such as 0.23) adapted to the scene of the first image. When the scene detection result corresponding to the first image indicates that the scene has not changed, it indicates that the scene of the first image is less different from the scene of the previous frame image, that is, the texture information of the scene of the first image and the previous frame image is more different, and the texture display effect is less different. Therefore, the electronic device can directly use the texture adjustment parameters corresponding to the previous frame image as the texture adjustment parameters corresponding to the first image, avoiding unnecessary prediction of texture adjustment parameters. Afterwards, the electronic device can use the texture adjustment parameters corresponding to the first image to adjust the texture information of the first image, such as adjusting the texture information of the first image based on the texture adjustment parameters corresponding to the first image in combination with the sharpening algorithm, obtaining the first image after texture enhancement, improving the texture clarity of the first image, and realizing real-time enhancement of the clarity of the video image. In addition, by determining whether it is necessary to re-predict the texture adjustment parameters through scene detection, avoiding unnecessary prediction of texture adjustment parameters, the data processing amount can be reduced, thereby reducing the power consumption of the electronic device. Afterwards, the electronic device can display the first image after texture enhancement, ensure the texture display effect of the first image, improve the clarity of the first image, thereby improving the video quality and ensuring the user's visual experience.

[0068] Exemplarily, the electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a wearable device, a computer (such as a PC), a personal digital assistant (PDA), or other device capable of displaying images in a video. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0069] For example, Figure 2 FIG. 2 shows a schematic diagram of the structure of the electronic device 200. Figure 2 As shown, the electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 211, a power management module 212, a battery 213, an antenna 1, an antenna 2, a mobile communication module 240, a wireless communication module 250, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, a sensor module 280, a button 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0070] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0071] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0072] The controller may be the nerve center and command center of the electronic device 200. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0073] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0074] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0075] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0076] The charging management module 211 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 211 may receive charging input from a wired charger through the USB interface 230. In some wireless charging embodiments, the charging management module 211 may receive wireless charging input through a wireless charging coil of the electronic device 200. While the charging management module 211 is charging the battery 213, it may also power the electronic device through the power management module 212.

[0077] The wireless communication function of the electronic device 200 can be implemented through the antenna 1, the antenna 2, the mobile communication module 240, the wireless communication module 250, the modem processor and the baseband processor.

[0078] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0079] The mobile communication module 240 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 200. The mobile communication module 240 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 240 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 240 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 240 can be set in the processor 210. In some embodiments, at least some of the functional modules of the mobile communication module 240 can be set in the same device as at least some of the modules of the processor 210.

[0080] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 270A, a receiver 270B, etc.), or displays an image or video through a display screen 294. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 210 and be set in the same device as the mobile communication module 240 or other functional modules.

[0081] The wireless communication module 250 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 200. The wireless communication module 250 can be one or more devices integrating at least one communication processing module. The wireless communication module 250 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 210. The wireless communication module 250 can also receive the signal to be sent from the processor 210, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0082] The electronic device 200 implements the display function through a GPU, a display screen 294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0083] The display screen 294 is used to display images, videos, etc. The display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 294, where N is a positive integer greater than 1.

[0084] The electronic device 200 can realize the shooting function through ISP, camera 293, video codec, GPU, display screen 294 and application processor.

[0085] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0086] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0087] The electronic device 200 can implement audio functions such as music playing and recording through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the headphone jack 270D, and the application processor.

[0088] The buttons 290 include a power button, a volume button, etc. The buttons 290 may be mechanical buttons or touch buttons.

[0089] Indicator 292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0090] The sensor module 280 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0091] The present application embodiment provides a texture enhancement method. An electronic device receives a video stream. Figure 3AAs shown, for each first image in the video stream, the electronic device performs scene detection on the first image, such as comparing the scene of the first image with the scene of the previous frame of the first image to determine whether the scene of the first image has changed. If the scene of the first image has changed, it indicates that the texture information of the first image and the previous frame of the image differ greatly, and it is necessary to determine the texture adjustment parameters corresponding to the first image. Then, the electronic device can input the first image into the trained first MobileNetV3 model, so as to use the trained first MobileNetV3 model to determine the texture adjustment parameters corresponding to the first image, and accurately determine the texture adjustment parameters corresponding to the first image. If the scene of the first image has not changed, it indicates that the texture information of the first image and the previous frame of the image differ little. Therefore, the texture adjustment parameters corresponding to the previous frame of the image can be directly used as the texture adjustment parameters corresponding to the first image, so as to quickly determine the texture adjustment parameters corresponding to the first image. Afterwards, the electronic device can adjust the first image using the texture adjustment parameters corresponding to the first image to enhance the texture clarity of the first image, thereby improving the quality of the first image, that is, improving the video quality, and realizing real-time texture enhancement of the first image.

[0092] The trained first MobileNetV3 model is obtained by training with sample images, and the trained first MobileNetV3 model can predict the texture adjustment parameters corresponding to the image. The process of obtaining the trained first MobileNetV3 model by training with sample images will be described in detail below.

[0093] In an embodiment of the present application, the first device can obtain a first sample image set, which can include multiple first sample original images. The first device can perform texture degradation processing on the first sample original image to reduce the texture clarity of the first sample image, and obtain a sample degraded image corresponding to the first sample original image. Afterwards, the first device can use the first sample original image and its corresponding sample degraded image to train the MobileNetV3 model. The trained MobileNetV3 model can predict the texture adjustment parameters corresponding to the image in the video, thereby using the texture adjustment parameters to improve the texture clarity of the image and enhance the details, that is, to improve the display effect of the image. Specifically, if Figure 3B As shown, the process of training the MobileNetV3 model using the first sample image set may include S301-S306.

[0094] S301: The electronic device obtains a first sample image set, wherein the first sample image set includes a plurality of first sample original images.

[0095] The first sample original image is of higher quality, such as an image with higher definition (or referred to as a standard image).

[0096] In some embodiments, if the format of the first sample original image is not in RGB format, the electronic device may first convert the format of the first sample original image into RGB format for training using the first sample original image in RGB format.

[0097] In some embodiments, the above-mentioned first sample image set can be an image set of multiple types. The types may include collaboration, movies, games, office (such as PPT), meetings, short videos, recordings and the like. Different types of image sets are produced in different ways. For example, a movie-type image set may include: multiple (such as 100) domestic and foreign 1080P / 2K high-definition movies, thereby forming a large data set that is evenly distributed in four aspects: brightness, texture complexity, motion complexity, and color complexity. In other words, it includes movie pictures with different display effects (such as texture display effects), that is, images, to ensure the richness of the movie-type image set and improve the generalization of the model.

[0098] For another example, the above-mentioned office PPT type image set may include: the PPT content in the image includes common content such as text, pictures, tables, statistical charts, etc.

[0099] For another example, the image set of the above-mentioned conference type may include different facial images, such as facial images of people of different genders, facial images of people of different ages, facial images of people of different regions, etc.

[0100] For another example, the image set of the above-mentioned game type may include game screens, that is, images, in different game applications.

[0101] In some embodiments, since model training actually refers to the process of how to restore an image of high quality from an image of low quality. Therefore, after obtaining a first sample original image of high quality, for each first sample original image, the electronic device can perform degradation processing on the clarity of the first sample original image to obtain an image of poor clarity corresponding to the first sample original image, that is, obtain a sample degraded image corresponding to the first sample original image, so that the electronic device can use each first sample original image and each sample degraded image corresponding to the first sample original image to train the model. It should be understood that the content of the first sample original image and the sample degraded image corresponding to the first sample original image, that is, the object is the same, but compared with the first sample original image, the clarity of the object in the sample degraded image corresponding to the first sample original image is poor, such as the object with higher clarity originally becomes blurred. The following will introduce a possible process of degrading the clarity of the first sample original image in combination with S302-S303.

[0102] S302: For each first sample original image among the plurality of first sample original images, the electronic device downsamples the first sample original image to obtain a downsampled first sample original image.

[0103] The size corresponding to the downsampled first sample original image is smaller than the size of the first sample original image.

[0104] S303: The electronic device upsamples the downsampled first sample original image to obtain a sample degraded image corresponding to the first sample original image, wherein the size of the sample degraded image corresponding to the first sample original image is equal to the first sample original image.

[0105] In the embodiment of the present application, for each first sample original image, the electronic device may first downsample the high-quality first sample original image, and then use an image interpolation algorithm (such as a bicubic algorithm) to upsample the downsampled first sample original image to obtain a low-quality image with the same size as the first sample original image but reduced clarity, that is, to obtain a sample degraded image, which simulates the low-definition image corresponding to the first sample original image. In this way, the first sample original image and its corresponding sample degraded image constitute a pair of texture clarity enhanced image sets.

[0106] S303: The electronic device calculates the variance of the first sample original image to obtain the complexity corresponding to the first sample original image.

[0107] For example, the electronic device may first convert the first sample original image into a grayscale image. Afterwards, the electronic device may calculate the variance of the first sample original image using the pixel values ​​in the grayscale image. The variance may reflect the degree of discreteness of the color distribution of the first sample original image. The larger the variance, the more drastic the change in color brightness, and the higher the complexity of the first sample original image; the smaller the variance, the more gradual the change in color brightness, and the lower the complexity of the first sample original image.

[0108] S304, the electronic device trains the first MobileNetV3 model based on each first sample original image, the sample degraded image corresponding to each first sample original image, and the complexity corresponding to each first sample original image to obtain the trained first MobileNetV3 model. The trained first MobileNetV3 model can predict the texture adjustment parameters corresponding to the images in the videos under different scenes.

[0109] Among them, the texture adjustment parameter is used to enhance the texture of the image and improve the clarity of the image.

[0110] In an embodiment of the present application, the electronic device may use the variance corresponding to the first sample original image as a label of the first sample original image. Then, the electronic device may input the labeled first sample original image and its corresponding sample degraded image into the first MobileNetV3 model, so that the first MobileNetV3 model learns how to restore the first sample original image from the sample degraded image corresponding to the first sample original image, and determines the mapping relationship between texture complexity and texture adjustment parameters (or texture clarity strength), such as generating a mapping curve between texture complexity and texture adjustment parameters.

[0111] It should be understood that for scenes with low texture complexity, the required texture clarity strength gradually decreases. If the texture complexity of an image is low, and a larger texture clarity strength is used to adjust the texture of the image, over-sharpening may occur, resulting in problems such as unnatural and distorted images. For scenes with high texture complexity, too high texture clarity strength is also not required. The higher the texture complexity, the smaller the texture clarity strength needs to be. If the texture complexity of an image is large, and a larger texture clarity strength is used to adjust the texture of the image, over-sharpening may also occur. Therefore, the texture clarity strength corresponding to scenes with medium texture complexity should be higher. In order to achieve a mapping curve that satisfies this rule, the first MobileNetV3 model can use four tangent quadratic curves to fit a curve, and obtain the following: Figure 4 The mapping curve between texture complexity and texture clarity is shown in Figure 1. Figure 4 The range of texture complexity in is 0 to 1, which is mapped to 0 to α through a curve. The parameters at the junction between curve segments can be adjusted by adjusting Figure 4 The parameters at the five junction points shown can change the overall shape of the curve. Among them, λ determines the maximum texture clarity strength, and α determines the texture complexity corresponding to the maximum texture clarity strength. X1 and x2 determine the junction position of the curve segment, which is used for fine-tuning the wireless details. μ determines the average texture clarity strength of the wireless as a whole. The first MobileNetV3 model is based on the overall adjustment plus the fine-tuning of the details, and finally determines a suitable mapping curve, thereby ensuring the mapping accuracy between texture complexity and texture clarity strength.

[0112] In some embodiments, the structure of the above-mentioned first MobileNetV3 model may include a first convolutional layer, a batch normalization (BN) layer and an activation function layer, multiple (e.g., 11) encoding blocks, an average pooling layer, and a fully connected layer.

[0113] Optionally, the last fully connected layer of the above-mentioned first MobileNetV3 model may include three fully connected layers, namely fully connected layer a, fully connected layer b and fully connected layer c. Among them, the number of neurons on the relevant feature map of the input of fully connected layer a is 1024, and the output of fully connected layer a is 512 neurons; the Dropout (0.5) of fully connected layer a, that is, the dropout rate is 0.5, and the activation function used by fully connected layer a is the ReLU activation function. The number of neurons on the relevant feature map of the input of fully connected layer b is 512, and the output of fully connected layer b is 256 neurons; the Dropout (0.2) of fully connected layer b, that is, the dropout rate is 0.5, and the activation function used by fully connected layer b is the ReLU activation function. The number of neurons on the relevant feature map of the input of fully connected layer c is 256, and the output of fully connected layer c is 1 neuron; the Dropout (0.1) of fully connected layer c, that is, the dropout rate is 0.1, and the activation function used by fully connected layer c is the ReLU activation function

[0114] During the training process, the electronic device freezes all layers except the last connection layer parameters in the first MobileNetV3 model, and only trains the fully connected layer. The electronic device can first downsample the input image (such as the first sample original image and its corresponding sample degraded image) to 256x256, set the learning rate to 0.0001, the number of iterations (epochs) to 100, and use the Adam optimizer for training. It should be understood that the electronic device can also not downsample the input image, but directly use the original image for training, but the training time is longer.

[0115] In some embodiments, the first MobileNetV3 model is only an example, and other lightweight models may also be used, which are not limited in this application, such as MobileNetV1, MobileNetV2, etc.

[0116] The training process of the first MobileNetV3 model is introduced above. After obtaining the trained first MobileNetV3 model, the electronic device using the trained first MobileNetV3 model can use the trained first MobileNetV3 model to predict the texture adjustment parameters of the image in the video stream, so as to adjust the image using the texture adjustment parameters, realize real-time adjustment of the video image, enhance the details of the video image, improve the clarity of the image, thereby improving the quality of the video image, thereby achieving the improvement of the video quality and ensuring the user's visual experience. Figure 5 , details the process of using the trained first MobileNetV3 model to enhance the clarity of video images. Figure 5 As shown, the process is as follows:

[0117] S401: The electronic device obtains a video stream.

[0118] The video stream includes at least one image. The electronic device acquires the video stream in real time, and the video stream may be generated by the electronic device itself, such as when the electronic device is in a recording state, generating a corresponding video stream. Alternatively, the video stream may be a video stream sent by other devices, such as when the electronic device is in a collaborative state, displaying the display content of other devices, and the image in the video stream includes the display content of other devices.

[0119] S402: For each first image in the video stream, the electronic device downsamples the first image to obtain a downsampled first image.

[0120] In some embodiments, each time the electronic device obtains an image (or referred to as a first image), it can downsample the first image, such as downsampling the first image to 256*256. Of course, the first image can also be downsampled to other sizes, such as 512*512. However, compared to other sizes, 256*256 can require lower computing power for the electronic device while ensuring the accuracy of scene detection.

[0121] S403: The electronic device performs scene detection on the first image using the previous image after downsampling of the first image, and obtains a scene detection result corresponding to the first image, wherein the scene detection result indicates whether the scene of the first image has changed.

[0122] In the embodiment of the present application, the electronic device compares the downsampled first image with the previous downsampled image (or the previous frame image) of the first image to determine whether the first image and the previous frame image belong to the same scene. When the scene detection result indicates that the scene of the first image has changed, it indicates that the texture information between the first image and the previous frame image is quite different, and the texture adjustment parameters corresponding to the previous frame image cannot be used to adjust the current image, that is, adjust the texture of the first image. Therefore, the electronic device needs to determine the new texture adjustment parameters corresponding to the first image, and can execute S404.

[0123] When the scene detection result indicates that the scene of the first image has not changed, it means that the difference in texture information between the first image and the previous image (or the previous frame image) is small. The texture adjustment parameters corresponding to the previous frame image can be used to enhance the texture information of the current image, that is, the first image. The electronic device can execute S405.

[0124] In some embodiments, the electronic device may use the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame of the first image to determine whether the first image and the previous frame of the image belong to the same scene, so as to obtain the scene detection result corresponding to the first image. The following is an introduction to possible implementation methods for determining the scene detection result.

[0125] In one implementation, since the HSV format corresponds to different dimensions, such as hue, saturation, and brightness, it is possible to determine whether the scene of the first image has changed in different dimensions, thereby ensuring that the obtained scene detection result is more accurate. The following will take the above-mentioned target channels including the H channel, the S channel, and the V channel as an example to describe in detail how to use HSV to obtain the scene detection result corresponding to the first image.

[0126] First, for the hue (H) dimension, the electronic device can calculate the difference between the pixel value of the H channel of the downsampled first image and the pixel value of the H channel of the previous frame image after downsampling of the first image, and obtain the difference of the H channel corresponding to the first image, and the difference of the H channel indicates the degree of hue difference between the first image and the previous frame image. The larger the difference of the H channel, the greater the degree of hue difference, the less the first image and the previous frame image include the same content, that is, the less the same texture information, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each pixel point (or first pixel point) in the downsampled first image, the electronic device can calculate the difference between the pixel value of the H channel of the first pixel point and the pixel value of the H channel of the pixel point (or second pixel point) at the corresponding position in the previous frame image, and obtain the difference of the H channel corresponding to the first pixel point. Among them, the position of the second pixel point at the corresponding position in the previous frame image in the previous frame image is the same as the position of the first pixel point in the first image. Afterwards, the electronic device may calculate the difference of the H channel corresponding to all the first pixels in the first image to obtain the difference of the H channel corresponding to the first image. Specifically, the electronic device may use the following formula 1 to calculate the difference of the H channel corresponding to the first image.

[0127]

[0128] Among them, score h represents the difference of the H channel corresponding to the first image, num_piexls represents the total number of pixels in the first image, current_H i represents the i-th pixel in the first image, last_H i Represents the i-th pixel in the previous frame of the first image.

[0129] For the saturation (S) dimension, the electronic device can calculate the difference between the pixel value of the S channel of the downsampled first image and the pixel value of the S channel of the previous frame image after downsampling of the first image, and obtain the difference of the S channel corresponding to the first image, and the difference of the S channel indicates the degree of saturation difference between the first image and the previous frame image. The larger the difference of the S channel, the greater the degree of saturation difference, the less the first image and the previous frame image include the same content, that is, the less the same texture information, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each first pixel point in the downsampled first image, the electronic device can calculate the difference between the pixel value of the S channel of the first pixel point and the pixel value of the H channel of the second pixel point at the corresponding position in the previous frame image, and obtain the difference of the S channel corresponding to the first pixel point. Afterwards, the electronic device can calculate the difference of the S channel corresponding to all the first pixels in the first image, and obtain the difference of the S channel corresponding to the first image. Specifically, the electronic device can use the following formula 2 to calculate the difference of the S channel corresponding to the first image.

[0130]

[0131] Among them, score s Indicates the difference of the S channel corresponding to the first image, current_S i Represents the pixel value of the S channel of the i-th pixel in the first image, last_S i Represents the pixel value of the S channel of the i-th pixel in the previous frame of the first image.

[0132] For the brightness (V) dimension, the electronic device can calculate the difference between the pixel value of the V channel of the downsampled first image and the pixel value of the V channel of the previous frame image after downsampling of the first image, and obtain the difference of the V channel corresponding to the first image, and the difference of the V channel indicates the degree of brightness difference between the first image and the previous frame image. The larger the difference of the V channel, the greater the degree of brightness difference, the less the first image and the previous frame image include the same content, that is, the less the same texture information, and the less likely the first image and the previous frame image belong to the same scene. Exemplarily, for each first pixel point in the downsampled first image, the electronic device can calculate the difference between the pixel value of the V channel of the first pixel point and the pixel value of the V channel of the second pixel point at the corresponding position in the previous frame image, and obtain the difference of the V channel corresponding to the first pixel point. Afterwards, the electronic device can calculate the difference of the V channel corresponding to all the first pixels in the first image, and obtain the difference of the V channel corresponding to the first image. Specifically, the electronic device can use the following formula 3 to calculate the difference of the V channel corresponding to the first image.

[0133]

[0134] Among them, score v represents the difference of the V channel corresponding to the first image, current_V represents the pixel value of the V channel of the i-th pixel in the first image, and last_V i Indicates the pixel value of the V channel of the i-th pixel in the previous frame of the first image.

[0135] It should be noted that the above-described process of determining the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image is only an example. The electronic device may not use each pixel point in the first image to determine it, but may use several pixel points in the first image to determine it. The present application does not limit the method of determining the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image.

[0136] Afterwards, the electronic device can obtain the scene detection result corresponding to the first image according to the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image. Exemplarily, the electronic device can calculate the average value of the difference of the H channel, the difference of the S channel, and the difference of the V channel corresponding to the first image to obtain the scene difference corresponding to the first image. Specifically, the electronic device can use the following formula 4 to calculate the scene difference corresponding to the first image.

[0137]

[0138] Here, score represents the scene difference corresponding to the first image.

[0139] When the average value, that is, the scene difference corresponding to the first image is greater than the threshold value 1, it indicates that the display content of the first image and the previous frame image is relatively different, that is, the texture information difference between the first image and the previous frame image is relatively large, the scene difference degree between the first image and the previous frame image is relatively large, and the first image and the previous frame image do not belong to the same scene. The electronic device can determine that the scene detection result corresponding to the first image indicates that the scene has changed. When the scene difference is less than or equal to the threshold value 1 (such as 27), it indicates that the display content of the first image and the previous frame image is relatively different, that is, the texture information difference between the first image and the previous frame image is relatively small, the scene difference degree between the first image and the previous frame image is relatively small, and the first image and the previous frame image belong to the same scene. The electronic device can determine that the scene detection result corresponding to the first image indicates that the scene has not changed. Of course, the average value of the difference of the H channel, the difference of the S channel, and the difference of the V channel is only an example, and the scene detection result corresponding to the first image can also be determined in other ways. For example, the electronic device may determine that the scene detection result corresponding to the first image indicates that the scene has changed when the difference value of the H channel corresponding to the first image is greater than threshold 2, the difference value of the S channel is greater than threshold 3, and the difference value of the V channel is greater than threshold 4. When the difference value of the H channel corresponding to the first image is less than or equal to threshold 2, the difference value of the S channel is less than or equal to threshold 3, and the difference value of the V channel is less than or equal to threshold 4, the electronic device determines that the scene detection result corresponding to the first image indicates that the scene has not changed.

[0140] In the present application embodiment, Figure 6 As shown, the electronic device downsamples (resizes) the first image in the video stream to improve the image processing speed and obtain the downsampled first image. Afterwards, the electronic device can convert the format of the downsampled first image into the HSV format to obtain the first image in the HSV format. Afterwards, the electronic device can respectively calculate the difference between the pixel value of the target channel of each first pixel point in the first image in the HSV format and the pixel value of the target channel of the second pixel point at the corresponding position of the previous frame image in the HSV format, and obtain the pixel difference value of the target channel corresponding to each first pixel point. Afterwards, the electronic device can calculate the average value of the pixel difference value of the target channel corresponding to each first pixel point to obtain the pixel difference value of the target channel corresponding to the first image. Afterwards, the electronic device can calculate the average value of the pixel difference value of the target channel corresponding to the first image to obtain the scene detection result corresponding to the first image, and the scene detection result indicates whether the scene has changed, so as to achieve accurate determination of the scene detection result. In addition, by downsampling the first image, the number of pixels to be calculated can be reduced, thereby reducing the amount of data processing, and thus improving the determination efficiency of the scene detection result.

[0141] The process of determining the previous frame image in the HSV format is similar to the process of determining the first image in the HSV format.

[0142] In some embodiments, generally speaking, the image format in the video stream is in YUV format, and the electronic device may first convert the YUV format of the downsampled first image into RGB format, and then convert the first image in RGB format into a first image in HSV format.

[0143] It should be understood that the above is an example of how to use HSV to obtain the scene detection result corresponding to the first image, taking the target channel including the H channel, the S channel and the V channel as an example. Of course, the target channel may include one or two channels of the H channel, the S channel and the V channel. The electronic device may calculate the difference of the target channel corresponding to the first image, so as to determine the detection result corresponding to the first image using the difference of the target channel corresponding to the first image. For example, the target channel includes the H channel and the S channel, and the electronic device calculates the difference of the H channel corresponding to the first image and the difference of the S channel corresponding to the first image. Afterwards, the electronic device may calculate the average of the difference of the H channel corresponding to the first image and the difference of the S channel corresponding to the first image. When the average is greater than a threshold value, the electronic device may determine that the scene detection result corresponding to the first image indicates that the scene has changed.

[0144] S404: The electronic device inputs the downsampled first image into the trained first MobileNetV3 model to obtain texture adjustment parameters corresponding to the first image.

[0145] In the embodiment of the present application, since the first image and the previous frame of the first image do not belong to the same scene, different scenes correspond to different texture adjustment parameters. Therefore, the electronic device can use the trained first MobileNetV3 model to process the first image, obtain texture adjustment parameters that match the scene of the first image, and achieve accurate prediction of the texture adjustment parameters. Afterwards, the electronic device can use the texture adjustment parameters that match the scene of the first image to adjust the texture information of the first image, achieve sharpening of the first image, enhance the texture clarity of the first image, avoid the situation where the texture information is enhanced too much or too little, such as the distortion of the first image due to excessive sharpening or the insufficient sharpening degree resulting in the texture clarity of the first image not being effectively improved, thereby ensuring the clarity of the first image.

[0146] Exemplarily, the trained first MobileNetV3 model can calculate the texture complexity corresponding to the downsampled first image. Afterwards, the trained first MobileNetV3 model can use the mapping curve between texture complexity and texture clarity intensity to determine the downsampled first image, that is, the texture clarity intensity corresponding to the first image, and output the texture clarity intensity corresponding to the first image (that is, the texture adjustment parameter corresponding to the first image). Among them, the texture clarity intensity corresponding to the first image represents the texture adjustment parameter that matches the texture information of the first image. When the texture clarity intensity is used to adjust the first image, there will be no over-adjustment or under-adjustment.

[0147] In some embodiments, since the model generally processes images in RGB format, the electronic device can convert the downsampled first image into RGB format. Afterwards, the electronic device can input the downsampled first image in RGB format into the trained first MobileNetV3 model.

[0148] In some embodiments, the above-mentioned use of the trained first MobileNetV3 model to determine the texture adjustment parameters corresponding to the first image is only an example. The present application does not limit the method of predicting the texture adjustment parameters corresponding to the image. For example, the electronic device can use the pixel values ​​of the pixels in the first image (such as YUV pixel values, etc.) to adjust the preset texture adjustment parameters to obtain texture adjustment parameters that match the first image.

[0149] It should be noted that the process of downsampling the first image in S402 is optional, and the electronic device may not downsample the first image, but directly use the first image to perform scene detection to determine whether the scene of the first image has changed. In the case of a change, the electronic device can input the first image into the trained first MobileNetV3 model to determine the texture adjustment parameters corresponding to the first image, but the data processing volume is large and the image processing efficiency is low.

[0150] It should be understood that the first MobileNetV3 model after the above training is only an example of an image processing model, and the image processing model can also be other lightweight types of models obtained through training.

[0151] S405: The electronic device uses the texture adjustment parameter corresponding to the previous image of the first image as the texture adjustment parameter corresponding to the first image.

[0152] In an embodiment of the present application, since the first image and the previous frame image of the first image belong to the same scene, it indicates that the texture information difference between the first image and the previous frame image is small, that is, the difference in texture clarity is small. Therefore, the electronic device can use the texture adjustment parameters corresponding to the previous frame image as the texture adjustment parameters corresponding to the first image.

[0153] In some embodiments, the electronic device uses texture adjustment parameters in combination with a sharpening algorithm to enhance the texture of the first image, that is, to adjust the high-frequency information of the first image, enhance the details of the first image, and improve the clarity of the first image. The high-frequency information forms the edges and details of the image, which is information in the high-frequency area. The high-frequency area refers to an area in which the grayscale value of a pixel in the image changes rapidly, and includes information in the image that needs to be clearly displayed in detail. A possible implementation method of using texture adjustment parameters to improve the clarity of the first image will be introduced below in combination with S406-S407.

[0154] S406: The electronic device determines high-frequency information of the first image.

[0155] S407 . For each pixel point in the high-frequency information of the first image, the electronic device multiplies the pixel value of the Y channel of the pixel point by the texture adjustment parameter to obtain the target first image.

[0156] In some embodiments, the process of determining the high-frequency area of ​​the first image may include: Figure 7 As shown, first, the electronic device performs mean filtering on the pixel values ​​of the Y channel of the pixel points in the first image to obtain the blurred first image (or the first filtered image). Afterwards, the pixel value of the Y channel of each pixel point in the first image is subtracted from the pixel value of the Y channel of the pixel point at the corresponding position in the blurred first image to obtain the high-frequency information of the first image (or called high-frequency edge information, edge information), so as to blur the non-high-frequency information of the first image, such as low-frequency information, that is, the part with less texture information. The blurred first image is the high-frequency information of the first image, that is, the part where the texture needs to be enhanced. For example, the pixel value of the Y channel of the first pixel point in the first image is subtracted from the pixel value of the Y channel of the corresponding position in the blurred first image, that is, the first pixel point, to obtain the pixel difference corresponding to the first pixel point in the first image. By analogy, the pixel difference corresponding to each pixel point in the first image can be obtained, and the pixel difference corresponding to each pixel point in the first image constitutes the high-frequency information of the first image.

[0157] In some embodiments, a possible implementation process of the above S407 may include: the electronic device calculates the product of the high-frequency information of the first image and the texture adjustment parameter λ to obtain the texture adjustment value of the Y channel of each pixel point in the first image. Exemplarily, the electronic device may multiply the pixel value of the Y channel of each pixel point in the high-frequency information of the first image by the texture adjustment parameter, that is, multiply the pixel difference value corresponding to each pixel point in the first image by the texture adjustment parameter to obtain the texture adjustment value of the Y channel corresponding to each pixel point in the first image.

[0158] Afterwards, for each pixel in the first image, the electronic device adds the pixel value of the Y channel of the pixel (i.e., the original pixel value of the Y channel) to the texture adjustment value at the corresponding position to obtain the pixel value of the Y channel of the pixel after texture enhancement. For example, the pixel value of the Y channel of the first pixel in the first image is added to the texture adjustment value at the corresponding position, i.e., the texture adjustment value of the Y channel of the first pixel, to obtain the pixel value of the Y channel of the first pixel in the first image after texture enhancement.

[0159] Afterwards, the electronic device performs image synthesis on the pixel values ​​of the Y channel after texture enhancement and the pixel values ​​of the UV channel of the pixel points of the first image to obtain the first image after texture enhancement, that is, to obtain the target first image, thereby obtaining a texture enhanced video stream.

[0160] In some embodiments, after obtaining the high-frequency information of the first image, a threshold can be used to determine whether to retain part of the high-frequency information of the first image to suppress the noise generated by texture enhancement. Exemplarily, for the pixel value of the Y channel of each pixel in the high-frequency information of the first image, the electronic device can determine whether the pixel value of the Y channel of the pixel is greater than or equal to the threshold 5. If it is greater than or equal to the threshold 5, the electronic device can retain the pixel, that is, the high-frequency information of the first image still includes the pixel value of the pixel. If it is less than the threshold 5, the electronic device can discard the pixel, that is, the high-frequency information of the first image does not include the pixel value of the pixel.

[0161] In an embodiment of the present application, in order to reduce computing power and increase the speed of adjusting the texture clarity of the image, the electronic device can perform real-time texture clarity enhancement processing on the first image in the video stream in the YUV control. And since Y corresponds to brightness, UV corresponds to the chromaticity of color, and UV has little effect on texture clarity, the electronic device can only enhance the pixel value of the Y channel of the first image to ensure the texture enhancement effect of the first image and reduce the amount of data processing. In addition, the electronic device only performs texture enhancement on the high-frequency information of the first image, and does not perform texture enhancement on non-high-frequency information, that is, areas with less texture information, to avoid unnecessary texture adjustment. For example, the first image includes a puppy image portion and a background image portion, and the background image portion refers to the blue sky, which includes less texture information, so it can be blurred. The puppy image portion includes the puppy's hair, which is high-frequency information, so it is necessary to enhance the texture clarity of the puppy's hair to make the detail display effect of the puppy image portion clearer. It can be simply understood that determining the high-frequency information of the first image is to determine the lines of the puppy's hair, then increase the brightness of these lines, and then combine them with the pixel values ​​of the UV channel of the original first image to obtain a first image with higher definition.

[0162] In some embodiments, the high-frequency information may include noise. For example, if the video stream is a video stream generated by recording a video, the image in the captured video stream may include white spots, which are noise. Therefore, the electronic device may perform mean filtering on the high-frequency information of the first image again to obtain the high-frequency information of the filtered first image. Afterwards, for the pixel value of the Y channel of each pixel point in the high-frequency information of the filtered first image, the electronic device may determine whether the pixel value of the Y channel of the pixel point is less than the threshold value 6. If it is less than the threshold value 6, it indicates that the pixel point is likely to be noise, and the electronic device may discard the pixel point, that is, discard the pixel value of the Y channel of the pixel point. If it is greater than or equal to the threshold value 6, the pixel point may be retained. Afterwards, for each pixel point in the high-frequency information of the filtered first image, the electronic device multiplies the pixel value of the Y channel of the pixel point by the texture adjustment parameter to obtain the target first image.

[0163] In some embodiments, the mean filtering method used in the above blur processing is only an example, and the electronic device may also use other methods to blur the first image, such as Gaussian blur (or Gaussian filtering), Laplace operation, median filtering, etc. The difference in high-frequency information of the first image determined by different blur processing methods is small, but compared with other blur processing methods, the performance of mean filtering is higher and the speed of determining the high-frequency information of the first image is faster, so that the texture enhancement rate of the first image can be effectively improved, and the real-time texture enhancement of the video stream can be ensured.

[0164] In the embodiment of the present application, after obtaining the high-frequency information of the first image, the electronic device uses the texture adjustment parameters that match the scene of the first image to adjust the high-frequency information of the first image, thereby achieving adaptive texture adjustment and avoiding excessive or under-enhancement of the texture of the first image. In addition, in order to reduce the influence of noise, after obtaining the high-frequency information of the first image, a mean filter is performed again, and some pixels in the high-frequency information of the first image are discarded according to the threshold value. After two noise reduction processes, the noise generated by texture enhancement is effectively suppressed.

[0165] In some embodiments, the trained first MobileNetV3 model can not only predict the texture adjustment parameters corresponding to the first image, but also predict the color adjustment parameters corresponding to the first image. Exemplarily, when the first image and the previous frame image do not belong to the same scene, the electronic device inputs the downsampled first image into the trained first MobileNetV3 model, and can obtain not only the texture adjustment parameters corresponding to the first image, but also the color adjustment parameters corresponding to the first image. The color adjustment parameters include brightness adjustment parameters, contrast adjustment parameters, and saturation adjustment parameters. In addition, when the first image and the previous frame image belong to the same scene, the electronic device can also directly use the color adjustment parameters corresponding to the previous frame image as the color adjustment parameters corresponding to the first image.

[0166] In the embodiment of the present application, since the first image and the previous frame image of the first image do not belong to the same scene, different scenes correspond to different color adjustment parameters and texture adjustment parameters. Therefore, the electronic device can use the trained first MobileNetV3 model to process the downsampled first image, obtain color adjustment parameters and texture adjustment parameters that match the scene of the first image, and achieve accurate prediction of color adjustment parameters and texture adjustment parameters. Afterwards, the electronic device can use the color adjustment parameters that match the scene of the first image to adjust the color of the first image to avoid excessive or insufficient color enhancement, such as excessive or insufficient brightness increase, excessive or insufficient contrast increase, and excessive or insufficient saturation increase, to ensure the color display effect of the first image. And use the texture adjustment parameters of the first image to adjust the clarity of the first image.

[0167] It is understandable that the first sample image set may also include images with higher color quality and images with lower color quality. The first device trains the first MobileNetV3 model using the images with higher color quality and the images with lower color quality, so that the trained first MobileNetV3 model can predict the color adjustment parameters.

[0168] The above has introduced the detailed process of adjusting the clarity of the first image. The following will introduce the process of adjusting the color of the first image in detail.

[0169] In some embodiments, the electronic device uses a brightness adjustment parameter to increase the brightness of the first image. Also, the electronic device uses a contrast adjustment parameter to increase the contrast of the first image. And the electronic device uses a saturation adjustment parameter to increase the contrast of the first image. Since the brightness is related to the pixel value of the Y channel of the image, the brightness of the first image can be increased by adjusting the pixel value of the Y channel of the pixel point in the first image using the brightness adjustment parameter.

[0170] A possible implementation method of improving the brightness of the first image by using the brightness adjustment parameter will be described below.

[0171] Exemplarily, for each pixel in the first image, the electronic device uses the brightness adjustment parameter as an index of the Y channel pixel of the pixel to obtain a pixel value of the Y channel of the pixel after brightness enhancement.

[0172] In the embodiment of the present application, for each pixel in the first image, the electronic device may use Y L =Y0 L , calculate the pixel value of the Y channel of the pixel after brightness enhancement to enhance the brightness of the first image.

[0173] Among them, Y L represents the pixel value of the Y channel after brightness enhancement of the pixel point, L represents the brightness adjustment parameter, and Y0 represents the original pixel value of the Y channel of the pixel point, that is, the pixel value of the Y channel of the pixel point in the first image.

[0174] In some embodiments, since contrast refers to the degree of difference between brightness values ​​in an image. It reflects the difference in brightness between different areas in an image. It can be seen that the contrast of an image is also related to the pixel value of the Y channel of the image. Therefore, the electronic device can also adjust the pixel value of the Y channel of the pixel point in the first image by using the contrast adjustment parameter to improve the contrast of the first image. Exemplarily, for each pixel point in the first image, the electronic device can obtain the adjustment value of the Y channel of the pixel point based on the average value of the pixel value of the Y channel after brightness enhancement of the pixel point and the pixel value of the Y channel after brightness enhancement corresponding to the first image, combined with the contrast adjustment parameter. Among them, the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image represents the average value of the pixel value of the Y channel after brightness enhancement of each pixel point in the first image. Exemplarily, the electronic device can calculate the difference (or second difference) between the pixel value of the Y channel after brightness enhancement of the pixel point and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image. Then, the electronic device can determine the adjustment value of the Y channel of the pixel point based on the second difference and the contrast adjustment parameter.

[0175] Afterwards, the electronic device obtains the pixel value of the Y channel after contrast enhancement based on the adjustment value of the Y channel of the pixel point and the average value of the pixel value of the Y channel after brightness enhancement corresponding to the first image, and improves the contrast of the first image by reducing the brightness value of the darker pixel points in the first image and increasing the brightness value of the brighter pixel points in the first image, thereby enhancing the contrast of the first image.

[0176] A possible implementation method of improving the brightness of the first image by using the brightness adjustment parameter will be described below.

[0177] For each pixel in the first image, the electronic device calculates a second difference between a pixel value of the Y channel of the pixel after brightness enhancement and an average value of pixel values ​​of the Y channel of the first image after brightness enhancement.

[0178] The electronic device calculates the product of the second difference and the contrast adjustment parameter to obtain an adjustment value of the Y channel of the pixel.

[0179] The electronic device calculates the sum of the adjustment value of the Y channel of the pixel point and the average value of the Y channel pixel value after brightness enhancement corresponding to the first image to obtain the Y channel pixel value after contrast enhancement of the pixel point.

[0180] In the embodiment of the present application, for each pixel point in the first image, the electronic device may use Y C =(Y L-mean0)*C+mean0, calculate the pixel value of the Y channel of the pixel after contrast enhancement to achieve contrast enhancement of the first image.

[0181] Among them, Y C represents the pixel value of the Y channel after contrast enhancement of the pixel point, C represents the contrast adjustment parameter, mean0 represents the average value of the pixel value of the Y channel after contrast enhancement corresponding to the first image, so as to achieve contrast enhancement of the first image.

[0182] It is understandable that, although both contrast and brightness enhancement are for the pixel value of the Y channel, for the same pixel point, the pixel value of the Y channel after contrast enhancement may change compared to the pixel value of the Y channel after brightness enhancement, such as being too large or too small, that is, the pixel value of the Y channel after contrast enhancement may be smaller than the pixel value of the original Y channel. In other words, the brightness of the pixel point will become darker, but it can be determined through experimental tests that, from the perspective of the first image as a whole, the overall brightness of the first image is still enhanced, that is, the brightness has been enhanced. In addition, the pixel value of the Y channel after contrast enhancement may be larger than the pixel value of the Y channel after brightness enhancement. In other words, the brightness of the pixel point will become brighter, but it can be determined through experimental tests that, from the perspective of the first image as a whole, the overall brightness of the first image will not be too bright.

[0183] In some embodiments, since saturation indicates the purity and vividness of a color, and in the YUV color space, the UV channel indicates the chromaticity information of an image, that is, the color in the image, that is, the saturation is related to the pixel values ​​of the U and V channels of the image, the electronic device can adjust the pixel values ​​of the U channel and the pixel values ​​of the V channel of the pixel points in the first image by using the saturation adjustment parameter to improve the saturation of the first image. A possible implementation method of using the saturation adjustment parameter to improve the saturation of the first image will be described below.

[0184] The electronic device calculates the product of the pixel value of the U channel of the pixel point and the saturation adjustment parameter to obtain the pixel value of the U channel of the pixel point after the saturation is enhanced.

[0185] For each pixel in the first image, the electronic device may use U S =U0*S, calculate the pixel value of the U channel after saturation enhancement of the pixel point. S represents the pixel value of the U channel after saturation enhancement of the pixel point, and S represents the saturation adjustment parameter. U0 represents the original pixel value of the U channel of the pixel point, that is, the pixel value of the U channel of the pixel point in the first image.

[0186] The electronic device calculates the product of the pixel value of the V channel of the pixel point and the saturation adjustment parameter to obtain the pixel value of the V channel of the pixel point after the saturation is enhanced.

[0187] In the embodiment of the present application, for each pixel point in the first image, the electronic device may use V S =V0*S, calculate the pixel value of the V channel after saturation enhancement. Where V S V0 represents the pixel value of the V channel after saturation enhancement of the pixel point. V1 represents the original pixel value of the V channel of the pixel point, that is, the pixel value of the V channel of the pixel point in the first image.

[0188] The above description is only an example of improving the saturation of the first image. The electronic device may also improve the saturation of the first image in other ways. For example, for each pixel in the first image, the electronic device uses the saturation as the index of the U channel of the pixel to obtain the pixel value of the U channel after the saturation is enhanced. And the saturation is used as the index of the V channel of the pixel to obtain the pixel value of the V channel after the saturation is enhanced, so as to achieve the saturation enhancement of the first image. However, the saturation enhancement effect of the first image described above is better, and the color display effect of the first image is better.

[0189] In the embodiment of the present application, the electronic device uses the brightness adjustment parameter to enhance the brightness of the first image, uses the saturation adjustment parameter to enhance the saturation of the first image, and uses the contrast adjustment parameter to enhance the contrast of the first image, thereby obtaining a color-enhanced first image. Afterwards, the electronic device can display the color-enhanced first image. Compared with the first image, the color-enhanced first image has a brighter overall color, a larger difference in brightness and darkness, and a more vivid color, and a better color display effect, thereby improving the color quality of the image, thereby improving the video quality and ensuring the user's visual experience.

[0190] In an embodiment of the present application, when the electronic device performs color enhancement on a video stream, it first performs scene detection on the first image in the video stream to determine whether the current frame image (i.e., the first image) and the previous frame image of the first image belong to the same scene. If they belong to the same scene, the electronic device can use the color adjustment parameters corresponding to the previous frame image as the color adjustment parameters corresponding to the first image, without re-predicting the color adjustment parameters, to achieve rapid determination of the color adjustment parameters and improve the processing efficiency of the video image. If they do not belong to the same scene, the electronic device can re-predict the color adjustment parameters corresponding to the first image to obtain color adjustment parameters that match the color scene of the first image, so that the electronic device can use the color adjustment parameters that match the color scene of the first image to adjust the color of the first image and achieve adaptive color adjustment. And because there are errors in the prediction of the color adjustment parameters, that is, for the same scene, the color adjustment parameters that may be predicted may have errors. Therefore, when the current frame image and the previous frame image belong to the same scene, directly using the color adjustment parameters corresponding to the previous frame image to adjust the current frame image can avoid the different color display effects of two adjacent frames belonging to the same scene due to errors, thereby avoiding discontinuities between video frames (such as frame skipping, flickering, etc.).

[0191] In some embodiments, although the pixel values ​​of the Y channel of the first image are processed based on brightness and contrast, and the texture adjustment parameters are also processed based on the pixel values ​​of the Y channel of the first image, the texture adjustment parameters mainly adjust the pixel values ​​of the Y channel in the high-frequency area of ​​the first image, while the brightness and contrast adjustment parameters adjust the pixel values ​​of the Y channel of the entire pixel points of the first image. Therefore, after adjusting the pixel values ​​of the Y channel of the pixel points based on the color adjustment parameters and the contrast adjustment parameters, the color quality (such as brightness, contrast) and clarity of the first image as a whole can still be improved.

[0192] Exemplarily, when the pixel values ​​of the channels of the pixel points in the first image need to be processed using the color adjustment parameters and the texture adjustment parameters, for each pixel point in the image, the color adjustment parameters can be used to determine the color-enhanced pixel value of the corresponding channel of the pixel point, and the texture adjustment parameters can be used to determine the clarity-enhanced pixel value of the Y channel of the pixel point. If the channel corresponding to the pixel point also includes the Y channel, the pixel value of the target Y channel of the pixel point can be determined based on the pixel value of the Y channel after color enhancement and the pixel value of the Y channel after clarity enhancement, such as calculating the average value between the two as the pixel value of the target Y channel, or using the pixel value of the Y channel after color enhancement or the pixel value of the Y channel after clarity enhancement as the pixel value of the target Y channel.

[0193] For example, after adjusting the pixel value of the Y channel of each pixel point in the first image based on the brightness adjustment parameter, the pixel value of the Y channel of each pixel point after brightness enhancement is obtained. The pixel value of the Y channel of the pixel point in the high-frequency part of the first image is adjusted by using the texture adjustment parameter to obtain the pixel value of the Y channel of the pixel point in the high-frequency part after clarity enhancement. For the same pixel point, the average value between the pixel value of the Y channel of the pixel point after brightness enhancement and the pixel value of the Y channel of the pixel point after clarity enhancement can be calculated to obtain the pixel value of the target Y channel of the pixel point. Although the pixel value of the target Y channel of the pixel point may be smaller than the pixel value of the Y channel of the pixel point after brightness enhancement, the texture adjustment is for the high-frequency part of the first image. Overall, the brightness of the first image is still enhanced. And although the pixel value of the target Y channel of the pixel point may be smaller than the pixel value of the Y channel of the pixel point after clarity enhancement, the clarity of the first image can still be improved. In summary, although adjusting the image using both texture adjustment parameters and color adjustment parameters may change the pixel value of the Y channel of the high-frequency part of the pixels, that is, the brightness of the high-frequency part of the pixels changes, it will not affect the overall brightness display effect of the first image. Overall, the brightness of the first image is still enhanced, and the clarity is also enhanced.

[0194] It should be noted that the above is an example of a color adjustment parameter including a saturation adjustment parameter, a contrast adjustment parameter, and a brightness adjustment parameter to introduce a process of enhancing the saturation, contrast, and brightness of a first image to obtain a color-enhanced first image. Of course, the color adjustment parameter may also include only one of the saturation adjustment parameter, the contrast adjustment parameter, and the brightness adjustment parameter, or the color adjustment parameter may also include only two of the saturation adjustment parameter, the contrast adjustment parameter, and the brightness adjustment parameter. In other words, the color adjustment parameter may include at least one of the saturation adjustment parameter, the contrast adjustment parameter, and the brightness adjustment parameter. Correspondingly, the electronic device may use the color adjustment parameter to enhance the color corresponding to the first image to obtain a color-enhanced first image. For example, the color adjustment parameter includes a brightness adjustment parameter, and the electronic device may use the brightness adjustment parameter to enhance the brightness corresponding to the first image to obtain a color-enhanced first image. For another example, the color adjustment parameter includes a brightness adjustment parameter and a saturation adjustment parameter, and the electronic device may use the brightness adjustment parameter to enhance the brightness corresponding to the first image to obtain a color-enhanced first image, and use the saturation adjustment parameter to enhance the saturation corresponding to the first image.

[0195] S408: The electronic device displays a first target image.

[0196] In an embodiment of the present application, after obtaining the first image after texture enhancement, that is, the target first image, the electronic device can display the first image, realize real-time processing and display of the video stream, ensure the clarity of the displayed video image, improve the image quality, thereby improving the video quality and ensuring the user's visual experience.

[0197] In addition, after the color enhancement is performed on the first image, the color display effect of the target first image is also improved.

[0198] In some embodiments, when the first image in the video stream includes a face area, such as a selfie image, the electronic device can determine the high-frequency information of the face image in the first image. Afterwards, for each pixel in the high-frequency information of the face image in the first image, the electronic device multiplies the pixel value of the Y channel of the pixel by the texture adjustment parameter to obtain the target first image, thereby achieving texture enhancement of the portion of interest, ensuring the texture enhancement effect of the image, and reducing the amount of data processing, thereby reducing power consumption. Specifically, Figure 8 As shown, the electronic device can use a segmentation network to segment the face area and the background in the first image to obtain the face image and the background image of the first image, that is, to obtain the pixel values ​​of the Y channel of the pixel points in the face image and the pixel values ​​of the Y channel of the pixel points in the background image.

[0199] Afterwards, after obtaining the facial image, the electronic device may first use mean filtering to blur the facial image to obtain the blurred facial image. Afterwards, the electronic device may determine the high-frequency information of the blurred facial image to obtain the high-frequency information of the facial image. Afterwards, the pixel value of the Y channel of each pixel point in the high-frequency information of the facial image is multiplied by the texture adjustment parameter to obtain the texture adjustment value of the Y channel corresponding to each pixel point in the high-frequency information, that is, the texture adjustment value of the Y channel corresponding to each pixel point in the facial image. Among them, the specific process of determining the high-frequency information of the blurred facial image can refer to the process of determining the high-frequency information of the first image introduced above, and will not be repeated here.

[0200] Afterwards, for each pixel in the facial image of the first image, the electronic device can add the pixel value of the Y channel of the pixel (i.e., the pixel value of the original Y channel) to the texture adjustment value at the corresponding position to obtain the pixel value of the Y channel of the pixel after texture enhancement, thereby achieving texture enhancement of the high-frequency information of the facial part of the first image.

[0201] Afterwards, the electronic device can synthesize the pixel values ​​of the Y channel of the pixel points of the face image of the first image after texture enhancement and the pixel values ​​of the UV channel of the face image of the first image to obtain the face image after texture enhancement. In addition, the electronic device can combine the face image after texture enhancement with the background image to obtain the target first image, thereby achieving texture enhancement of the area of ​​interest of the first image, while enhancing the details of the face, avoiding over-sharpening of the background, thereby avoiding the introduction of excessive background noise, but the amount of data processing will increase and the power consumption will be higher.

[0202] Among them, optionally, the structure of the above segmentation network model is as follows Fig. 9 The segmentation network model outputs a binary image (or binary mask image) corresponding to the first image, wherein the value of the pixel point of the face part in the binary image can be 1, and the value of the pixel point of the background part is 0, thereby achieving face segmentation of the first image and determining the face image of the first image. It should be understood that the above Fig. 9 The segmentation network model shown is only an example, and the structure of the segmentation network model can also be other structures, as long as it can segment the face part of the image.

[0203] In some embodiments, the electronic device can directly use a pure deep learning texture enhancement algorithm, that is, a pure deep learning model, to perform texture enhancement on the first image to obtain a target first image. Fig.10 As shown, the electronic device can input the pixel values ​​of the Y channel of the first image into the pure deep learning model, and the pure deep learning model performs correlation processing (such as downsampling processing, two convolution processing, upsampling processing) on ​​the pixel values ​​of the Y channel of the first image to obtain the pixel values ​​of the Y channel of the first image after texture enhancement. Afterwards, the electronic device can perform image synthesis on the pixel values ​​of the Y channel of the first image after texture enhancement and the pixel values ​​of the UV channel of the first image after texture enhancement to obtain the target first image.

[0204] Among them, optionally, image synthesis can also be performed by a pure deep learning model. Accordingly, the electronic device can input the first image into the pure deep learning model, and the pure deep learning model can directly output the target first image to achieve texture enhancement of the first image.

[0205] In some embodiments, the electronic device can not only perform texture enhancement on images in the video in real time, such as performing texture enhancement on the first image after obtaining the first image in the video stream as mentioned above, but the electronic device can also perform texture enhancement on images in video files that have been stored locally, as long as it can obtain pixel points in the image in the video (such as pixel values ​​of the YUV channel).

[0206] It should be noted that when the first image is received for the first time, that is, the first frame of the first image, the electronic device can directly input it into the trained first MobileNetV3 model to obtain the texture adjustment parameters corresponding to the first frame of the first image, thereby realizing scene detection of the first frame of the first image.

[0207] In some embodiments, the first device and the electronic device may be the same device or different devices, and the present application does not limit them. In some embodiments, the present application provides a computer storage medium including computer instructions, which, when executed on an electronic device, causes the electronic device to execute the method described above.

[0208] In some embodiments, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method described above.

[0209] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.

[0210] It should be understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0211] It should also be understood that in the present application, "when", "if" and "if" all mean that the UE or base station will take corresponding actions under certain objective circumstances. It is not a time limit, and it does not require the UE or base station to make judgments when implementing it, nor does it mean that there are other limitations.

[0212] It will be understood by those skilled in the art that the various digital numbers such as first and second involved in this application are only for the convenience of description, and are not used to limit the scope of the embodiments of this application, and also represent the order of precedence. Elements represented by the singular in this application are intended to represent "one or more" rather than "one and only one", unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to represent "one or more", and "more than one" is intended to represent "two or more".

[0213] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone, where A can be singular or plural, and B can be singular or plural. The term "at least one of..." or "at least one of..." herein represents all or any combination of the listed items. For example, "at least one of A, B, and C" can represent the following six situations: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B, and C exist at the same time, where A can be singular or plural, B can be singular or plural, and C can be singular or plural.

[0214] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0218] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0219] The same or similar parts between the various embodiments in this application can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The above-described implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0220] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A texture enhancement method, characterized in that: include: Acquire a video stream; wherein the video stream includes at least one frame of a first image; For each first image in the at least one first image, based on a difference between a pixel value of a target channel of the first image and a pixel value of a target channel of a previous image of the first image, obtaining a scene detection result corresponding to the first image; wherein the scene detection result corresponding to the first image indicates whether the first image and the previous image belong to the same scene; When the scene detection result indicates that the first image and the previous frame image belong to the same scene, the texture adjustment parameters corresponding to the previous frame image are used as the texture adjustment parameters corresponding to the first image; the texture adjustment parameters corresponding to the previous frame image are texture adjustment parameters that match the scene of the previous frame image; When the scene detection result indicates that the first image and the previous frame image do not belong to the same scene, determining, based on the first image, a texture adjustment parameter corresponding to the first image; the texture adjustment parameter corresponding to the first image is a texture adjustment parameter that matches the scene of the first image; Based on the texture adjustment parameter corresponding to the first image, adjusting the clarity of the high-frequency information of the first image to obtain a target first image; The target first image is displayed.

2. The method according to claim 1, characterized in that The method further comprises: For each pixel point in the first image, calculate the difference between the pixel value of the target channel of the pixel point and the pixel value of the target channel of the pixel point at the corresponding position in the previous frame image, to obtain the difference value of the target channel corresponding to the pixel point in the first image; An average value of the difference of the target channel corresponding to each pixel point in the first image is calculated to obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image.

3. The method according to claim 2, characterized in that The number of the target channels is one or more; The calculating an average value of the difference of the target channel corresponding to each pixel point in the first image to obtain the difference between the pixel value of the target channel of the first image and the pixel value of the target channel of the previous frame image includes: For each of the target channels, calculating an average value of the difference values ​​of the target channels corresponding to the respective pixel points, to obtain the difference value of the target channel corresponding to the first image; An average value of the difference values ​​of each target channel corresponding to the first image is calculated to obtain a difference value between a pixel value of the target channel of the first image and a pixel value of the target channel of the previous frame image.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining a scene detection result corresponding to the first image based on a difference between a pixel value of a target channel of the first image and a pixel value of a target channel of a previous frame of the first image comprises: When a difference in the target channel between a pixel value of the target channel of the first image and a pixel value of the target channel of the previous frame image is less than or equal to a first threshold, determining that the scene detection result corresponding to the first image indicates that the first image and the previous frame image belong to the same scene; When a target channel difference between a pixel value of a target channel of the first image and a pixel value of a target channel of the previous frame image is greater than the first threshold, it is determined that a scene detection result corresponding to the first image indicates that the first image and the previous frame image do not belong to the same scene.

5. The method according to any one of claims 1 to 4, characterized in that The target channel includes at least one of a hue H channel, a saturation S channel, and a brightness V channel.

6. The method according to any one of claims 1 to 4, characterized in that The adjusting the clarity of the high-frequency information of the first image based on the texture adjustment parameter corresponding to the first image includes: Based on the texture adjustment parameter, pixel values ​​of each pixel point in the high-frequency information of the first image are adjusted respectively to obtain texture adjustment values ​​of each pixel point in the high-frequency information; For each pixel in the first image, a pixel value of the pixel after texture enhancement is obtained based on the pixel value of the pixel and the texture adjustment value corresponding to the pixel at the corresponding position in the high-frequency information.

7. The method according to claim 6, characterized in that The texture adjustment value of the pixel point includes the texture adjustment value of the brightness Y channel of the pixel point, and the pixel value after texture enhancement includes the pixel value of the Y channel after texture enhancement; The adjusting the pixel value of each pixel point in the high-frequency information of the first image based on the texture adjustment parameter to obtain the texture adjustment value of each pixel point in the high-frequency information includes: Calculating the product of the pixel value of the Y channel of each pixel point in the high-frequency information and the texture adjustment parameter respectively to obtain the texture adjustment value of the Y channel of each pixel point in the high-frequency information; The step of obtaining, for each pixel point in the first image, a pixel value of the pixel point after texture enhancement based on a pixel value of the pixel point and a texture adjustment value corresponding to a pixel point at a corresponding position in the high-frequency information comprises: For each pixel in the first image, a pixel value of a Y channel of the pixel and a texture adjustment value of a Y channel of a pixel at a corresponding position in the high-frequency information are calculated to obtain a Y channel pixel value of the pixel after texture enhancement.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Performing mean filtering on the first image to obtain a first filtered image; For each pixel point in the first image, a pixel value of a Y channel of the pixel point is subtracted from a pixel value of a Y channel of a pixel point at a corresponding position in the first filtered image to obtain high-frequency information of the first image.

9. The method according to any one of claims 1 to 8, characterized in that The determining, based on the first image, a texture adjustment parameter corresponding to the first image includes: The first image is used as an input parameter of an image processing model, and the image processing model is run to obtain texture adjustment parameters corresponding to the first image; the image processing model is a lightweight type model.

10. The method according to any one of claims 1 to 9, characterized in that The obtaining a scene detection result corresponding to the first image based on a difference in the target channel between a pixel value of the target channel of the first image and a pixel value of the target channel of an image of a previous frame of the first image comprises: Downsampling the first image to obtain a downsampled first image; Based on the difference in the target channel between the pixel value of the target channel of the downsampled first image and the pixel value of the target channel of the previous frame image after downsampling of the first image, a scene detection result corresponding to the first image is obtained.

11. An electronic device, characterized in that: The electronic device includes a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in any one of claims 1 to 10.

12. A computer storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 10.

Images