Image processing method, device and readable storage medium

Abstract

The present application discloses an image processing method comprising: obtaining an image to be processed from a video stream; converting the image to be processed into a YUV format to obtain a first image, and extracting Y channel data corresponding to the first image; using the Y channel data to generate brightness correction parameters corresponding to each pixel of the first image; using the brightness correction parameters to perform gamma brightness correction on the Y channel data corresponding to each pixel to obtain corrected data; replacing the Y channel data with the corrected data to obtain a second image, and converting the second image into an RGB format to obtain a processed image; inputting the processed image and a historical adjacent frame optimized image into an evaluation model to obtain an evaluation parameter for characterizing the brightness difference between the processed image and the historical adjacent frame optimized image; determining that the evaluation parameter is not in a target range, then determining the processed image as the brightness optimized image corresponding to the image to be processed; so that the brightness of the video stream remains generally stable.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Live streaming is a technology that sets up independent signal acquisition equipment on site to collect signals (audio and / or video) and import them into the director end (director device or platform). The director end uploads the signal to the server through the network, and then publishes it to the designated website for people to watch. In some darker scenes, in order to enable users to see the live image clearly, the collected video stream is usually processed to increase its brightness. Currently, the entire interface of the video stream is directly subjected to a unified and fixed brightness boost processing, that is, the brightness of all parts of all pictures generated during the live broadcast is increased by the same amount. However, the darkness of the scene may change, that is, from dim to bright. A unified and fixed brightness boost processing will cause the image after the brightness boost to be too bright, affecting the user's visual experience. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide an image processing method, an electronic device and a computer-readable storage medium, so that users can have a better visual experience.

[0004] To solve the above technical problems, in a first aspect, the present application provides an image processing method, comprising:

[0005] Obtaining an image to be processed from a video stream; the image to be processed is in RGB format;

[0006] Convert the image to be processed into a YUV format to obtain a first image, and extract Y channel data corresponding to the first image;

[0007] Generating brightness correction parameters corresponding to each pixel of the first image using the Y channel data;

[0008] Performing gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain corrected data;

[0009] Replacing the Y channel data with the corrected data to obtain a second image, and converting the second image into an RGB format to obtain a processed image;

[0010] Inputting the processed image and the historical adjacent frame optimized image into an evaluation model to obtain an evaluation parameter for characterizing the brightness difference between the processed image and the historical adjacent frame optimized image;

[0011] If it is determined that the evaluation parameter is not within the target range, the processed image is determined as the brightness optimized image corresponding to the image to be processed.

[0012] Optionally, obtaining the image to be processed from the video stream includes:

[0013] Acquire an initial image from the video stream according to historical adjacent frame images, and perform normalization processing on the initial image to obtain a normalized image;

[0014] Mapping the normalized image to a linear color space to obtain a first intermediate image;

[0015] enhancing the contrast of each color channel in the first intermediate image to obtain a second intermediate image;

[0016] Mapping the second intermediate image to a gamma color space to obtain a third intermediate image;

[0017] The third intermediate image is subjected to inverse normalization processing to obtain the image to be processed.

[0018] Optionally, generating brightness correction parameters corresponding to respective pixels of the first image using the Y channel data includes:

[0019] Sort the Y channel data corresponding to each pixel by size or frequency of occurrence to obtain a data sequence;

[0020] The target data at a preset proportional position in the data sequence is determined as the brightness correction parameter.

[0021] Optionally, generating brightness correction parameters corresponding to respective pixels of the first image using the Y channel data includes:

[0022] Determine the adjacent range corresponding to each pixel;

[0023] Sort the Y channel data corresponding to the pixel and adjacent pixels in the adjacent range by size or frequency of occurrence to obtain a data sequence;

[0024] The target data at a preset proportional position in the data sequence is determined as the brightness correction parameter corresponding to the pixel.

[0025] Optionally, performing gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain corrected data includes:

[0026] A brightness difference between a maximum brightness value and the brightness correction parameter is used to generate a power value inversely proportional to the brightness difference, and a normalized gamma calculation is performed using the power value and the Y channel data to obtain the corrected data.

[0027] Optionally, the using the brightness difference between the maximum brightness value and the brightness correction parameter to generate a power value inversely proportional to the brightness difference, and performing normalized gamma calculation using the power value and the Y channel data to obtain the corrected data includes:

[0028] generating the brightness difference value greater than zero using the maximum brightness value and the brightness correction parameter;

[0029] Generate a power value using the brightness difference and a preset threshold; the power value is inversely proportional to the brightness difference, and the power value is greater than zero;

[0030] Using the ratio of the Y channel data to the maximum brightness value as a base, and obtaining a power function value using the base and the power value;

[0031] The power function value is multiplied by the maximum brightness value to obtain the corrected data.

[0032] Optionally, if the evaluation parameter is within the target range, including:

[0033] The brightness correction parameters are updated so that the Y channel data corresponding to each pixel is re-gamma-corrected using the updated brightness correction parameters to obtain new corrected data, and then a new processed image is obtained using the new corrected data. New evaluation parameters are obtained using the new processed image until the new evaluation parameters are no longer within the target range, thereby obtaining the brightness-optimized image.

[0034] Optionally, updating the brightness correction parameter includes:

[0035] If the processed image is darker than the historical adjacent frame optimized image, reducing the brightness correction parameter;

[0036] If the processed image is brighter than the historical adjacent frame optimized image, the brightness correction parameter is increased.

[0037] Optionally, it also includes:

[0038] Visually outputting the processed image;

[0039] User feedback information in response to the processed image is obtained, training data is generated according to the user feedback information, and the evaluation model is additionally trained using the training data.

[0040] Optionally, it also includes:

[0041] An optimized video stream is formed using processed images corresponding to all video frames in the video stream, and the optimized video stream is output visually.

[0042] In a second aspect, the present application further provides an electronic device, including a memory and a processor, wherein:

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the computer program to implement the above-mentioned image processing method.

[0045] In a third aspect, the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the above-mentioned image processing method when executed by a processor.

[0046] The image processing method provided by the present application obtains an image to be processed from a video stream; the image to be processed is in RGB format; the image to be processed is converted into YUV format to obtain a first image, and Y channel data corresponding to the first image is extracted; the Y channel data is used to generate brightness correction parameters corresponding to each pixel of the first image; the Y channel data corresponding to each pixel is gamma-corrected using the brightness correction parameters to obtain corrected data; the Y channel data is replaced with the corrected data to obtain a second image, and the second image is converted into RGB format to obtain a processed image; the processed image and historical adjacent frame optimized images are input into an evaluation model to obtain evaluation parameters; if it is determined that the evaluation parameters are not in a target range, the processed image is determined to be a brightness-optimized image corresponding to the image to be processed.

[0047] As can be seen, after acquiring the image to be processed, this method converts it to YUV format, where the Y channel data represents the image's brightness. This color format conversion minimizes the impact of brightness correction on image color. Based on the Y channel data corresponding to each pixel in the image to be processed, the overall brightness of the image to be processed can be determined, thereby determining the basis for brightness adjustment, namely, the brightness correction parameters. Using the brightness correction parameters, gamma brightness correction can be performed on the Y channel data based on its specific characteristics. This can significantly enhance dark areas of the image to be processed, while minimizing or eliminating bright areas, resulting in a clear and natural image. The resulting corrected data is then used to generate a second image, which is then restored to RGB format to obtain the processed image. The evaluation model is trained based on user needs and preferences for image brightness and exhibits excellent spatial and temporal adaptability. The evaluation model evaluates whether the processed image responds well to temporal brightness changes compared to historical optimized images of adjacent frames. If the evaluation parameter is in the target interval, the target interval is used to indicate that the brightness change of the processed image is abnormal based on the optimized images of the historical adjacent frames. Therefore, if it is not in the target interval, it means that the brightness optimization of the image to be processed is reasonable. Therefore, the processed image is determined to be a brightness optimized image after brightness optimization, and the optimization process of the image to be processed is determined to be completed. The evaluation parameter can be used to change the gamma brightness correction method for different images to be processed accordingly when the brightness of the scene obtained by the video stream changes, so that the brightness of each processed image is continuous in the time domain. The brightness of the video stream remains generally stable at all times, and each processed image is gamma-brightness corrected, so that the user can have a better visual experience.

[0048] In addition, the present application also provides an electronic device and a computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0050] Figure 1 A schematic diagram of a hardware composition framework applicable to an image processing method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a hardware composition framework applicable to another image processing method provided in an embodiment of the present application;

[0052] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application;

[0053] Figure 4 An image to be processed provided in an embodiment of the present application;

[0054] Figure 5 The embodiment of the present application provides a method for obtaining a processed image by processing in accordance with a related processing method;

[0055] Figure 6 A schematic diagram of a specific process of converting RGB format to YUV format provided in an embodiment of the present application;

[0056] Figure 7 A flowchart of a specific image processing method provided in an embodiment of the present application;

[0057] Figure 8 A comparison chart of the effects before and after image processing provided in an embodiment of the present application;

[0058] Figure 9 Another specific comparison chart of image processing effects before and after provided in the embodiment of the present application;

[0059] Figure 10 A schematic diagram of an optimized video stream effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] For ease of understanding, the hardware composition framework used in the solution corresponding to the image processing method and / or audio processing method provided in the embodiment of the present application is first introduced. Figure 1 , Figure 1 Schematic diagram of a hardware component framework applicable to an image processing method provided in an embodiment of the present application. The electronic device 100 may include a processor 101 and a memory 102, and may further include one or more of a multimedia component 103, an information input / output (I / O) interface 104, and a communication component 105.

[0062] The processor 101 is used to control the overall operation of the electronic device 100 to complete all or part of the steps in the image processing method and / or the audio processing method; the memory 102 is used to store various types of data to support the operation of the electronic device 100. For example, these data may include instructions for any application or method operating on the electronic device 100, as well as application-related data. The memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. In this embodiment, the memory 102 stores at least programs and / or data for implementing the following functions:

[0063] Get the image to be processed from the video stream; the image to be processed is in RGB format;

[0064] Convert the image to be processed into YUV format to obtain a first image, and extract Y channel data corresponding to the first image;

[0065] Generate brightness correction parameters corresponding to each pixel of the first image using the Y channel data;

[0066] Perform gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain the corrected data;

[0067] Replacing the Y channel data with the corrected data to obtain a second image, and converting the second image into RGB format to obtain a processed image;

[0068] The processed image and the historical optimized image of adjacent frames are input into the evaluation model to obtain the evaluation parameters;

[0069] If it is determined that the evaluation parameter is not within the target range, the processed image is determined as a brightness optimized image corresponding to the image to be processed.

[0070] The multimedia component 103 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 102 or sent through the communication component 105. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 104 provides an interface between the processor 101 and other interface modules. The above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 105 is used for wired or wireless communication between the electronic device 100 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 105 may include: a Wi-Fi component, a Bluetooth component, an NFC component.

[0071] The electronic device 100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the image processing method.

[0072] certainly, Figure 1 The structure of the electronic device 100 shown does not constitute a limitation on the electronic device in the embodiment of the present application. In actual applications, the electronic device 100 may include Figure 1 More or fewer components than shown, or combinations of certain components.

[0073] It is understandable that the number of electronic devices is not limited in the embodiments of the present application, and multiple electronic devices may collaborate to complete the image processing method and / or the audio processing method. In a possible implementation, please refer to Figure 2 , Figure 2 A schematic diagram of a hardware composition framework applicable to another image processing method provided in an embodiment of the present application. Figure 2It can be seen that the hardware composition framework may include: a first electronic device 11 and a second electronic device 12 , which are connected via a network 13 .

[0074] In the embodiment of the present application, the hardware structure of the first electronic device 11 and the second electronic device 12 can refer to Figure 1 In other words, the electronic device 100 in this embodiment has two electronic devices 100, and the two devices exchange data. Furthermore, the embodiment of the present application does not limit the form of the network 13, that is, the network 13 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0075] Among them, the first electronic device 11 and the second electronic device 12 can be the same electronic device, such as the first electronic device 11 and the second electronic device 12 are both servers; or they can be different types of electronic devices, for example, the first electronic device 11 can be a smartphone or other smart terminal, and the second electronic device 12 can be a server. In one possible embodiment, a server with strong computing power can be used as the second electronic device 12 to improve data processing efficiency and reliability, thereby improving the processing efficiency of image processing. At the same time, a smartphone with low cost and wide application range is used as the first electronic device 11 to realize the interaction between the second electronic device 12 and the user. It can be understood that the interaction process can be: the smartphone obtains the individual images that make up the video stream, and sends the video stream or image to the server, and the server performs brightness correction. The server sends the obtained brightness optimized image to the smartphone.

[0076] Based on the above description, please refer to Figure 3 , Figure 3 A flowchart of an image processing method provided in an embodiment of the present application is provided. The method in this embodiment includes:

[0077] S101: Obtain an image to be processed from a video stream.

[0078] A video stream refers to a video that is subjected to brightness correction processing, which includes multiple video frames, each of which can be processed as an image to be processed. For example, each video frame is selected as an image to be processed in chronological order. The image to be processed is in RGB format. The RGB format is also called RGB color, which is the so-called optical primary colors, R stands for Red, G stands for Green, and B stands for Blue. Any color that can be seen by the naked eye in nature can be mixed and superimposed by these three colors, so the mode of synthesizing colors in the RGB format is also called the additive color mode.

[0079] In order to ensure that the image brightness at each moment of the video stream meets the requirements, the non-first video frame in the video stream can be used as the image to be processed for brightness correction processing. Specifically, the image to be processed this time can be determined based on the video frame processed last time. Therefore, for the first video frame in the video stream, it does not have a corresponding historical adjacent frame image and does not have a corresponding judgment basis. Therefore, no processing can be done on the first video frame, or the brightness of the first video frame (such as the average brightness) can be adjusted to a preset brightness. In addition, in order to better perform brightness correction, a series of preprocessing can be performed on the original video frame to obtain an object that is more convenient for brightness correction, that is, the image to be processed. Preferably, in one embodiment, the contrast of the video frame can be improved to make the bright areas brighter and the dark areas darker, so that the correction intensity can be accurately determined during subsequent gamma brightness correction. Specifically, the process of obtaining the image to be processed from the video stream may include:

[0080] Step 11: Obtain an initial image from the video stream based on historical adjacent frame images, and normalize the initial image to obtain a normalized image.

[0081] Step 12: Map the normalized image to a linear color space to obtain a first intermediate image.

[0082] Step 13: Enhance the contrast of each color channel in the first intermediate image to obtain a second intermediate image.

[0083] Step 14: Map the second intermediate image to the gamma color space to obtain a third intermediate image.

[0084] Step 15: Perform inverse normalization on the third intermediate image to obtain an image to be processed.

[0085] The initial image refers to the video frame in the video stream that is one moment after the historical adjacent frame image. The historical adjacent frame image refers to the image to be processed from the previous image processing step, that is, the frame immediately preceding the image to be processed in the video stream. Normalization processing involves mapping the pixel values of each color channel of the initial image to the range of 0 to 1. The pixel values obtained after normalization processing can be mapped to a linear color space in a certain manner, resulting in a mapped image, namely the first intermediate image. Image contrast is enhanced by increasing the contrast of the first intermediate image. Contrast enhancement in a linear space is more robust than contrast enhancement in a nonlinear space such as gamma color space, resulting in a better visual appearance. The second intermediate image is mapped back to the gamma color space to obtain the third intermediate image. Denormalization is then performed to restore the pixel values, mapping the pixel values from the range of 0 to 1 back to the original color channel value range, resulting in the image to be processed. This embodiment does not limit the specific method of color space mapping and can be configured as needed.

[0086] S102: Convert the image to be processed into a YUV format to obtain a first image, and extract Y channel data corresponding to the first image.

[0087] Among them, the YUV format is a color encoding method, which has three color channels: Y, U, and V, where Y represents brightness (Luminance or Luma), that is, the grayscale value; U and V represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel. By converting the image to be processed from RGB format to YUV format, Y channel data for representing brightness can be obtained. It should be noted that for the first image, each pixel therein has a set of YUV data, so the number of Y channel data corresponding to the first image extracted is multiple and the same as the number of pixels of the first image. For example, if the long side of the first image (i.e., the image to be processed) has H pixels and the wide side has W pixels, the number of pixels is W*H, and the number of Y channel data is also W*H. This embodiment does not limit the specific method of image format conversion.

[0088] S103: Generate brightness correction parameters corresponding to each pixel of the first image using the Y channel data.

[0089] A single Y channel data can represent the brightness level of a single pixel, and all Y channel data corresponding to the first image can represent the overall brightness level of the first image. Based on the Y channel data, it can be determined whether the first image is darker and the degree of darkness, and the brightness correction parameters of the pixel pair can be determined based on the brightness level. Brightness correction parameters refer to the basic parameters used to generate the gamma parameters used in subsequent gamma brightness correction. Depending on the different ways of generating the gamma parameters, the number and specific determination methods of the brightness correction parameters are not limited. Specifically, when the gamma parameter is less than 1, the lower brightness areas in the image will be stretched, and the higher brightness areas will be compressed, which is manifested externally as a larger brightness increase in the darker parts of the image and a smaller brightness increase in the brighter parts; when the gamma parameter is greater than 1, the higher brightness areas in the image will be stretched, and the lower brightness areas will be compressed, which is manifested as an increase in the contrast of the image. In this embodiment, in order to perform brightness adjustment, the brightness correction parameter should be ensured to be in the range of 0 to 1. Since its size affects the degree of brightness improvement caused by the gamma brightness correction, the brightness correction parameter can be determined using the Y channel data corresponding to all pixels of the first image, thereby achieving the effect of determining the brightness correction parameter based on the overall brightness level of the first image, and then determining the gamma brightness correction amplitude based on the overall brightness level of the first image.

[0090] In one embodiment, the brightness correction parameter is a global parameter, i.e., all pixels of the first image correspond to the same brightness correction parameter. The global brightness correction parameter can be quickly determined, enabling uniform brightness correction of the first image. Specifically, the process of generating the brightness correction parameter corresponding to each pixel of the first image using Y channel data may include:

[0091] Step 21: Sort the Y channel data corresponding to each pixel by size or frequency of occurrence to obtain a data sequence.

[0092] Step 22: Determine the target data at the preset proportional position in the data sequence as the brightness correction parameter.

[0093] When sorting the Y channel data, it can be sorted in order from large to small, or it can be sorted in order from small to large, or its frequency of occurrence can be counted and sorted in order from large to small or from small to large. Among them, the preset proportion position can be the 95% position, or the 80% position, that is, after the number of Y channel data is taken as 95% or 80%, the Y channel data with the sequence number of this value in the data sequence is determined as the target data at the preset proportion position, and the target data is determined as the brightness correction parameter. The brightness correction parameter of the global parameter only needs to be determined once, so its determination speed is faster. However, the global parameter cannot perform separate brightness correction for different parts of the same image. In order to solve this problem, in another embodiment, the brightness correction parameter is a local parameter, that is, the brightness correction parameter corresponding to each pixel can be different, and the brightness correction parameter of the local parameter has local adaptability. Specifically, using the Y channel data, the process of generating the brightness correction parameters corresponding to each pixel of the first image can include:

[0094] Step 31: Determine the adjacent range corresponding to each pixel.

[0095] Step 32: Sort the Y channel data corresponding to the pixel and the adjacent pixels in the adjacent range according to size or frequency of occurrence to obtain a data sequence.

[0096] Step 33: Determine the target data at the preset proportional position in the data sequence as the brightness correction parameter corresponding to the pixel.

[0097] Among them, the adjacent range refers to the range that is close to the specified pixel. The pixels in the adjacent range can be regarded as a part of the image, and the Y channel data of each pixel can be combined to represent the brightness level of the part. The specific size and determination method of the adjacent range are not limited. For example, the range of two pixels away from a certain pixel can be selected as the adjacent range. That is, for a specified pixel, other pixels that are less than or equal to the distance of two pixels are within the adjacent range of the pixel, and these other pixels are adjacent pixels.

[0098] By sorting the Y channel data corresponding to the pixels in the adjacent range and determining the brightness correction parameters, the brightness correction parameters corresponding to a certain part of the image can be determined according to the brightness of the part, so that the subsequent gamma brightness correction process has the ability of local adaptation.

[0099] S104: performing gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain corrected data.

[0100] After obtaining the brightness correction parameters, use them to determine the gamma parameters required for gamma brightness correction, and then use the gamma parameters to perform gamma brightness correction on the Y channel data corresponding to each pixel. Gamma brightness correction, that is, correcting the Y channel data according to the gamma transformation formula, the gamma transformation formula is:

[0101] s=c*r γ

[0102] Where c is a coefficient. In the application scenario of this application, it is the maximum brightness value, i.e., 255. r is the normalized Y channel data. γ is the gamma parameter. s is the corrected data. The specific method for generating the gamma parameters using the brightness correction parameters can be set as needed. The specific method is also related to the specific selection method of the brightness correction parameters. Specifically, in one embodiment, the brightness correction parameters are used to perform gamma brightness correction on the Y channel data corresponding to each pixel. The process of obtaining the corrected data may include:

[0103] Step 41: Using the brightness difference between the maximum brightness value and the brightness correction parameter, a power value inversely proportional to the brightness difference is generated, and the power value and Y channel data are used to perform normalized gamma calculation to obtain corrected data.

[0104] In this embodiment, the brightness difference value can be determined using the maximum brightness value and the brightness correction parameter. The maximum brightness value refers to the maximum value within the optional range of the Y channel parameter. The brightness difference value can represent the brightness level of the first image or a portion of the first image. It can be understood that the smaller the difference value, the higher the brightness, and vice versa. Since the gamma parameter is less than 1, the smaller the gamma parameter, the greater the brightness improvement of the dark area, and the larger the gamma parameter, the smaller the brightness improvement of the dark area. Therefore, when determining the gamma parameter, it is necessary that it is inversely proportional to the brightness difference value, that is, the smaller the brightness difference value, the larger the gamma parameter, and the larger the brightness parameter, the smaller the gamma parameter.

[0105] The gamma transform formula can be considered a power function, with the gamma parameter being the power value. The normalized value of the Y channel data is used as the r value, and the gamma calculation is completed to obtain the corrected data.

[0106] Furthermore, the above step 41 can be further refined to include the following steps:

[0107] Step 51: Generate a brightness difference value greater than zero using the maximum brightness value and the brightness correction parameter.

[0108] Step 52: Generate a power value using the brightness difference and a preset threshold; the power value is inversely proportional to the brightness difference and is greater than zero.

[0109] Step 53: Use the ratio of the Y channel data to the maximum brightness value as the base, and use the base and the power value to obtain the power function value.

[0110] Step 54: Multiply the power function value by the maximum brightness value to obtain corrected data.

[0111] Since the gamma parameter must be greater than zero, when generating the brightness difference value, it is necessary to use the maximum brightness value minus the brightness correction parameter to obtain a brightness difference value greater than zero. After obtaining the brightness difference value, it is calculated with the preset threshold according to the preset formula to obtain the power value. The specific content of the preset formula is not limited, and for example, it can be:

[0112] gamma = 1-β × (255-y_max) / 255

[0113] Among them, gamma is the gamma parameter, β is the preset threshold, and its value is not limited, for example, it can be 0.45, y_max is the brightness correction parameter, and 255-y_max is the brightness difference.

[0114] S105: Replace the Y channel data with the corrected data to obtain a second image, and convert the second image into RGB format to obtain a processed image.

[0115] After obtaining the corrected data, it is used as the new Y channel data to replace the original Y channel data. This data is then combined with the U and V channel data to form a second image. This image is then converted back to RGB format to obtain the processed image. Although the processed image has undergone a brightness correction, it may still not meet the preset requirements (e.g., image quality requirements) or meet the user's needs. To ensure the effectiveness of the brightness correction, the processed image can also be evaluated after it is obtained.

[0116] S106: Inputting the processed image and the historical adjacent frame optimized images into an evaluation model to obtain an evaluation parameter for characterizing the brightness difference between the processed image and the historical adjacent frame optimized images.

[0117] An evaluation model refers to a model that evaluates the brightness level of a processed image. The model's structure and type are not limited. For example, a convolutional neural network model can be used to train the evaluation model. For example, the evaluation model can be obtained by training a convolutional neural network model using training data. The training data comprises multiple groups, each group of training data including two training images, which can be referred to as a first training image and a second training image. The first training image and the second training image correspond to two adjacent moments in time, and the brightness of the first training image and the second training image can be the same or different. Labels are annotated based on the difference in image brightness. For example, the first average brightness of the first training image and the second average brightness of the second training image can be calculated, and the first average brightness can be subtracted from the second average brightness to obtain the brightness difference, which is then used as the label value for the training image group. Alternatively, a weighted average brightness calculation method based on pixel position can be used, and the brightness difference can be obtained by subtracting the first and second average brightness values. Labeling can be performed manually or automatically. The convolutional neural network model, serving as the initial model, is trained using multiple sets of training data. During the training process, the initial model learns to accurately identify the brightness difference between two images based on the labels, thereby generating an evaluation model. Specifically, the evaluation model outputs an evaluation parameter representing the brightness difference between the two images. For example, a larger evaluation parameter indicates a smaller brightness difference, while a smaller evaluation parameter indicates a smaller brightness difference.

[0118] Since the brightness of the environment in which different video frames in the video stream are generated may change, the final effect obtained after brightness correction processing of different images to be processed using the same brightness correction process with the same brightness correction parameters is different. The historical adjacent frame optimized image refers to the brightness optimized image corresponding to the historical adjacent frame image, and the historical adjacent frame image is the historical adjacent frame image mentioned in the above step S101. In order to make the processed image corresponding to each video frame in the video stream maintain a relatively stable brightness in time sequence, the processed image and the historical adjacent frame optimized image can be input into the evaluation model, and the evaluation model can output evaluation parameters. The brightness difference between the processed image and the historical adjacent frame optimized image is characterized in the form of the evaluation parameters, so that the evaluation parameters can be used to evaluate whether the brightness level of the processed image is consistent with that of the historical adjacent frame optimized image. The form of the evaluation parameter is not limited, and for example, it can be a percentage score.

[0119] S107: Determine whether the evaluation parameter is within the target range.

[0120] If it is within the target range, execute step S108; otherwise, execute step S109.

[0121] S108: Update the brightness correction parameters so as to re-perform gamma brightness correction on the Y channel data corresponding to each pixel using the updated brightness correction parameters to obtain new corrected data, and then use the new corrected data to obtain a new processed image, and use the new processed image to obtain new evaluation parameters until the new evaluation parameters are no longer within the target range, thereby obtaining the brightness-optimized image.

[0122] The target interval refers to an interval indicating that the brightness level of the processed image is inconsistent with that of the historically optimized images of adjacent frames, with a significant brightness difference. The specific range and upper and lower limits are not limited. When the evaluation parameter is within the target interval, it indicates that the brightness difference between the historically optimized images of adjacent frames and the processed image is significant, the brightness optimization effect is poor, and re-brightness correction of the processed image is required. Therefore, the brightness correction parameters are updated to re-perform gamma brightness correction. That is, after the brightness correction parameters are updated, step S104 is re-executed to obtain new corrected data, and then a new processed image is obtained using the new corrected data. The new evaluation parameters are then used to obtain new processing parameters, and the process continues until the new evaluation parameters are no longer within the target interval, resulting in the brightness-optimized image.

[0123] In one embodiment, if the brightness correction parameters are determined and gamma brightness correction is performed in the above manner, the process of updating the brightness correction parameters may include:

[0124] Step 61: If the processed image is darker than the historical optimized image of the adjacent frame, then reduce the brightness correction parameter.

[0125] Step 62: If the processed image is brighter than the historical optimized image of the adjacent frame, then increase the brightness correction parameter.

[0126] If the processed image is darker than the optimized image of the adjacent frame, and the evaluation parameters are within the target range, the brightness correction parameters need to be lowered to increase the brightness difference, which in turn leads to a decrease in the gamma parameter, resulting in a greater brightness improvement. Conversely, the brightness correction parameters need to be increased to reduce the brightness difference, thereby increasing the gamma parameter and achieving a smaller brightness improvement.

[0127] Furthermore, in one embodiment, the evaluation model can determine the reference brightness for brightness evaluation according to the user's preferences, that is, the evaluation model needs to ensure that the brightness levels of the historical adjacent frame optimized image and the processed image are consistent, and also needs to ensure that they match the reference brightness. For example, for the first video frame in the video stream, it does not have a historical adjacent frame optimized image. After generating its corresponding processed image, it can be compared with the reference brightness to determine whether it is necessary to update the brightness correction parameters, and then determine the tone of subsequent brightness correction. In addition, the user's feedback on the processed image can be obtained, and then the evaluation model can be trained so that the evaluation model can meet the user's preferences. Specifically, the following steps may also be included:

[0128] Step 71: Visualize the processed image.

[0129] Step 72: Obtain user feedback information in response to the processed image, generate training data based on the user feedback information, and use the training data to perform additional training on the evaluation model.

[0130] The specific form of the user feedback information is not limited. The user feedback information and the processed images can be used to form training data, which can be used to perform additional training on the evaluation model, so that the evaluation model can more accurately understand the user's preference for image brightness.

[0131] Since the image to be processed originates from a video stream, the processed images corresponding to all video frames in the video stream can be used to form an optimized video stream, and the optimized video stream can be output visually so that the user can view the video with appropriate brightness.

[0132] S109: If it is determined that the evaluation parameter is not within the target range, the processed image is determined as a brightness optimized image corresponding to the image to be processed.

[0133] If the evaluation parameter is not within the target range, it means that the brightness difference between the optimized images of adjacent historical frames and the processed image is small, the brightness optimization effect is good, and there is no need to re-correct the brightness of the image to be processed. Therefore, the processed image is determined to be the brightness-optimized image of the image to be processed.

[0134] By applying the image processing method provided in the embodiment of the present application, after obtaining the image to be processed, it is converted into the YUV format, wherein the Y channel data is used to represent the brightness of the image. By converting the color format, the impact of brightness correction on the image color can be reduced as much as possible. According to the Y channel data corresponding to each pixel in the image to be processed, the overall brightness of the image to be processed can be determined, and then the basis for brightness adjustment, i.e., the brightness correction parameters, can be determined. Using the brightness correction parameters, the Y channel data can be gamma-corrected based on the specific situation of the Y channel data. The dark parts of the image to be processed can be greatly improved as needed, while the bright parts can be improved to a lesser extent or not at all, so that the image is clear and natural. After generating a second image using the obtained corrected data, it is restored to the RGB format to obtain the processed image. The evaluation model is trained based on the user's needs and preferences for image brightness, and has good spatial and temporal adaptability. The evaluation model can evaluate whether the brightness change response of the processed image and the optimized image of the historical adjacent frames in the temporal sequence is good. If the evaluation parameter is in the target interval, the target interval is used to indicate that the brightness change of the processed image is abnormal based on the optimized images of the historical adjacent frames. Therefore, if it is not in the target interval, it means that the brightness optimization of the image to be processed is reasonable. Therefore, the processed image is determined to be a brightness optimized image after brightness optimization, and the optimization process of the image to be processed is determined to be completed. The evaluation parameter can be used to change the gamma brightness correction method for different images to be processed accordingly when the brightness of the scene obtained by the video stream changes, so that the brightness of each processed image is continuous in the time domain. The brightness of the video stream remains generally stable at all times, and each processed image is gamma-brightness corrected, so that the user can have a better visual experience.

[0135] Based on the above embodiment, this embodiment will illustrate a specific embodiment. First, please refer to Figure 4 and Figure 5 , Figure 4 An image to be processed provided in an embodiment of the present application, Figure 5 The present application provides an embodiment of a method for processing a processed image using a related processing method. It can be seen that the related processing method increases the brightness of the entire image to be processed, further increasing the brightness of already brighter areas, resulting in poor image quality and loss of bright area information. Furthermore, there is no correlation between the individual image frames in the video stream. When the brightness of the external scene changes, the brightness of the processed video stream will also change accordingly.

[0136] In order to solve the above problems, this application provides an image processing method, please refer to Figure 6 , Figure 6A specific flow chart of converting RGB format to YUV format provided in an embodiment of the present application. First, set the preset threshold β and the preset proportional position (for generating y_max), input the RGB image as the image to be processed, and pre-process the image to be processed. Specifically, obtain the RGB information of each pixel and normalize them to [0,1] respectively to obtain R' data, G' data and B' data, or R1 data, G1 data and B1 data. The R' data, G' data and B' data are converted from gamma space to linear space respectively, and the conversion method is: when the R' data, G' data or B' data is less than or equal to 0.04045, divide by 12.92; when the RGB value is greater than 0.04045, return pow((RGB+0.055) / 1.055,2.4), that is:

[0137]

[0138]

[0139] Among them, color1 is R' data, G' data and B' data, that is, the data corresponding to the normalized image, and color2 is the data corresponding to the first intermediate image, which can be specifically divided into R" data, G" data and B" data, or R2 data, G2 data and B2 data.

[0140] After the conversion is completed, contrast enhancement is performed to improve the contrast of the image. The formula is as follows:

[0141] color3=hdrmode×(color2 / hdr mod e) hdrGamma (3)

[0142] hdrmode and hdrGamma are two fixed parameters used to adjust the image brightness. Their values are greater than zero and can be set as needed. color3 is the data corresponding to the second intermediate image, which can be specifically divided into R'' data, G'' data, and B'' data, or R3 data, G3 data, and B3 data. When the value of color2 is 1.5857, the output image is the same as the original image. Obviously, when RGB < 1.5857, the image brightness decreases, and when RGB > 1.5857, the image brightness increases.

[0143] After processing, convert it to gamma color space:

[0144] color4=12.92×color3,color3≤0.0031308 (4)

[0145]

[0146] Color4 is the data corresponding to the third intermediate image, which can be specifically divided into R”” data, G”” data and B”” data, or R4 data, G4 data and B4 data.

[0147] Multiply color4 by 1.2 and map it to the range of [0, 255] to generate the image to be processed:

[0148] color5=1.2×color4×255 (6)

[0149] Color5 is the data corresponding to the image to be processed, which can be specifically divided into R'' data, G''' data and B''' data, or R5 data, G5 data and B5 data. After obtaining the image to be processed, that is, the rendered RGB image, it is converted to YUV format. Specifically:

[0150]

[0151] The R data, G data and B data in formula (7) specifically refer to R5 data, G5 data and B5 data. Obtain the brightness Y channel information of the image (i.e., Y channel data), analyze the distribution of the brightness values of the image and sort them, take the top 95% value of the brightness sequence from small to large, and record it as the brightness correction parameter, where top 95% is the preset proportional position. Assuming that the width and height of the image are w pixels and h pixels respectively, record the brightness values of w×h pixels, and count the frequency of occurrence of each brightness value. Sort the brightness values of 0-255 and the frequency of occurrence by brightness value from small to large, and take the brightness value of w×h×0.95 from small to large as the maximum brightness value of the entire image and record it as y_max. The reason for not taking the maximum brightness value is that most images may have some pixels with a brightness value of 255, which will make brightness correction impossible. Perform gamma brightness correction:

[0152]

[0153] gamma=1-β×(255-y_maX) / 255 (9)

[0154] Y' is the corrected data. Since the β value is greater than 0, it ranges from 0 to 1. When the entire image is dark, y_max will be smaller, resulting in a larger gamma value and a greater increase in video brightness. When the image is of normal brightness, y_max is close to 255, and the gamma value will approach 1, with little change to the image. In actual use, the default β value is 0.45. The UV channel remains unchanged, and the Y'UV is synthesized and then converted into an RGB image to obtain the rendered image, that is, the processed image. The YUV to RGB conversion method is as follows:

[0155]

[0156] Please refer to Figure 7 , Figure 7 The following is a flowchart of a specific image processing method provided in an embodiment of the present application. The input video is processed by an adaptive algorithm, as shown in Formula 11:

[0157]

[0158] APA represents the image processing by the adaptive algorithm, that is, the brightness correction process. in represents the input image, It represents the iterative image obtained by the adaptive algorithm, that is, the processed image, and then together with the processed image of the previous frame of the image to be processed in the video stream, that is, the optimized image of the historical adjacent frames The evaluation network MASK is input, and the evaluation network outputs a score. Based on this score, it determines whether the processed image has the best quality and / or the brightness that the user prefers. If not, for example, if it is within the target range, the system continues to feed the original video into the adaptive algorithm for optimization, continuously optimizing the video by adjusting β and y_max until the output is the user's favorite video with the best quality and the highest score, and then outputs this video. As shown in Formula 12:

[0159]

[0160] The final output after iteration is the optimized video stream.

[0161] Please refer to Figure 8 and Figure 9 , Figure 8 This is a comparison chart of the effects before and after image processing provided in an embodiment of the present application. Figure 9 Another specific comparison diagram of the effects before and after image processing provided by the embodiment of the present application. Among them, image a and image b can be video frames in the video stream obtained by the anchor end in the video live broadcast scene, and image a1 and image b1 can be video frames in the optimized video stream. It can be seen that Figure 8 Compared with image a, the darker parts of the face in image a1 are brightened, making the picture look clear and natural. For the bright parts of the backlit original image, slight adjustments or no adjustments are made adaptively. Figure 9 Compared with image b, image b1 in the image adaptively enhances the brightness of the darker scene behind the face, making the picture look clear and natural.

[0162] Please refer to Figure 10 , Figure 10This is a schematic diagram of an optimized video stream effect provided by an embodiment of the present application. The curve graph shows the response curve of each pixel in the small image. As can be seen from the curve, the brightness correction method provided by this application has good time domain response characteristics and is continuous and adaptive in the spatial domain.

[0163] The computer-readable storage medium provided in the embodiments of the present application is introduced below. The computer-readable storage medium described below and the image processing method described above can be referenced to each other.

[0164] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image processing method are implemented.

[0165] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0166] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0167] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0168] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0170] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed from the video stream; The image to be processed is in RGB format; Convert the image to be processed into a YUV format to obtain a first image, and extract Y channel data corresponding to the first image; Generating brightness correction parameters corresponding to each pixel of the first image using the Y channel data; Performing gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain corrected data; Replacing the Y channel data with the corrected data to obtain a second image, and converting the second image into an RGB format to obtain a processed image; Inputting the processed image and the historical adjacent frame optimized image into an evaluation model to obtain an evaluation parameter for characterizing the brightness difference between the processed image and the historical adjacent frame optimized image; If it is determined that the evaluation parameter is not within the target range, the processed image is determined as a brightness optimized image corresponding to the image to be processed; If it is within the target range, the brightness correction parameters are updated so as to re-perform gamma brightness correction using the updated brightness correction parameters to obtain a new processed image, until the evaluation parameters of the new processed image are no longer within the target range. The step of generating brightness correction parameters corresponding to respective pixels of the first image using the Y channel data includes: Determine a proximity range corresponding to each pixel; the proximity range is determined based on pixel distance; Sort the Y channel data corresponding to the pixel and adjacent pixels in the adjacent range according to the frequency of occurrence to obtain a data sequence; determining the target data at a preset proportional position in the data sequence as the brightness correction parameter corresponding to the pixel; the brightness correction parameter corresponding to each pixel is different; The evaluation model is obtained by training a convolutional neural network model using training data; the training data includes multiple groups, each group of training data includes a first training image and a second training image, the first training image and the second training image are images corresponding to two adjacent moments in time sequence, and each group of training data includes a label representing the brightness difference between the first training image and the second training image; The method further includes: if there is no historical adjacent frame optimized image corresponding to the processed image, comparing the processed image with a reference brightness to obtain an evaluation parameter characterizing the brightness difference between the processed image and the reference brightness; the reference brightness is determined based on user preference.

2. The image processing method according to claim 1, wherein: The step of obtaining the image to be processed from the video stream includes: Acquire an initial image from the video stream according to historical adjacent frame images, and perform normalization processing on the initial image to obtain a normalized image; Mapping the normalized image to a linear color space to obtain a first intermediate image; enhancing the contrast of each color channel in the first intermediate image to obtain a second intermediate image; Mapping the second intermediate image to a gamma color space to obtain a third intermediate image; The third intermediate image is subjected to inverse normalization processing to obtain the image to be processed.

3. The image processing method according to claim 1, wherein: Generating brightness correction parameters corresponding to respective pixels of the first image using the Y channel data includes: Sort the Y channel data corresponding to each pixel by size or frequency of occurrence to obtain a data sequence; The target data at a preset proportional position in the data sequence is determined as the brightness correction parameter.

4. The image processing method according to claim 1, wherein: The step of performing gamma brightness correction on the Y channel data corresponding to each pixel using the brightness correction parameter to obtain corrected data includes: A brightness difference between a maximum brightness value and the brightness correction parameter is used to generate a power value inversely proportional to the brightness difference, and a normalized gamma calculation is performed using the power value and the Y channel data to obtain the corrected data.

5. The image processing method according to claim 4, characterized in that The step of generating a power value inversely proportional to the brightness difference value by using the brightness difference value between the maximum brightness value and the brightness correction parameter, and performing normalized gamma calculation using the power value and the Y channel data to obtain the corrected data includes: generating the brightness difference value greater than zero using the maximum brightness value and the brightness correction parameter; Generate a power value using the brightness difference and a preset threshold; the power value is inversely proportional to the brightness difference, and the power value is greater than zero; Using the ratio of the Y channel data to the maximum brightness value as a base, and obtaining a power function value using the base and the power value; The power function value is multiplied by the maximum brightness value to obtain the corrected data.

6. The image processing method according to claim 1, wherein: If the evaluation parameter is within the target range, it includes: The brightness correction parameters are updated so that the Y channel data corresponding to each pixel is re-gamma-corrected using the updated brightness correction parameters to obtain new corrected data, and then a new processed image is obtained using the new corrected data. New evaluation parameters are obtained using the new processed image until the new evaluation parameters are no longer within the target range, thereby obtaining the brightness-optimized image.

7. The image processing method according to claim 6, characterized in that: The updating of the brightness correction parameter includes: If the processed image is darker than the historical adjacent frame optimized image, reducing the brightness correction parameter; If the processed image is brighter than the historical adjacent frame optimized image, the brightness correction parameter is increased.

8. The image processing method according to claim 1, wherein: Also includes: Visually outputting the processed image; User feedback information in response to the processed image is obtained, training data is generated according to the user feedback information, and the evaluation model is additionally trained using the training data.

9. The image processing method according to claim 1, wherein: Also includes: An optimized video stream is formed using processed images corresponding to all video frames in the video stream, and the optimized video stream is output visually.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the image processing method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images