Image enhancement processing method and apparatus

Abstract

An image enhancement processing method and apparatus are provided. The image enhancement processing method includes: acquiring a histogram of an image to be processed; dividing the histogram into n sub-histograms, where n is a positive integer; calculating the mapping curve of each of the n sub-histograms, where the first sub-histogram is any one of the n sub-histograms; synthesizing the mapping curves of the n sub-histograms to obtain the histogram mapping curve; and enhancing the image to be processed based on the histogram mapping curve, achieving both fine-tuning and balanced adjustments to brightness and contrast distribution.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image enhancement processing method and apparatus. Background Technology

[0002] The Image Signal Processor (ISP) is primarily responsible for processing the signals output from the image sensor, which are typically raw (RAW) format images. The ISP includes a series of processing submodules for brightness, contrast, color, noise, and scaling, transforming RAW images into images suitable for human vision or display. The brightness and contrast processing submodules are mainly responsible for adjusting the brightness distribution and contrast of the image signal. They typically use a Global Tone Mapping Curve (GTMC) to remap the global histogram of the image, resulting in a more reasonable global brightness distribution. Maximizing the effectiveness of the GTMC is a key factor in improving image quality and ensuring video stability.

[0003] In related technologies, the local brightness distribution information of each image patch is first analyzed to obtain the histogram of each image patch. Then, an equalization operation is performed on each histogram to obtain the tone mapping curve (TMC) of each image patch. The role of TMC is to map the brightness of the input pixel to the output brightness to adjust the brightness distribution of the image. For example, TMC is often used as an adjustment method in image brightness enhancement and contrast adjustment. Least square fitting is performed on the TMC of all image patches in the image to obtain the GTMC of the entire image. Finally, global image enhancement processing is performed on the image based on the GTMC.

[0004] However, the above method adjusts the brightness of all image blocks indiscriminately, which may lead to problems such as excessive local noise and poor controllability in the image. Summary of the Invention

[0005] This application provides an image enhancement processing method and apparatus that can achieve both fine adjustment and balanced adjustment of brightness and contrast distribution.

[0006] In a first aspect, this application provides an image enhancement processing method, comprising: obtaining a histogram of an image to be processed; dividing the histogram into n sub-histograms, where n is a positive integer; calculating a mapping curve for each of the n sub-histograms; synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; and enhancing the image to be processed according to the mapping curve of the histogram.

[0007] This application divides the histogram of the image to be processed into multiple sub-histograms, obtains a corresponding mapping curve for each sub-histogram, and then synthesizes these mapping curves to obtain the mapping curve of the histogram, thereby achieving the enhancement processing of the image to be processed. Processing each sub-histogram separately can achieve the purpose of fine adjustment while taking into account the overall distribution of the image, thus achieving the purpose of balance.

[0008] In one possible implementation, calculating the mapping curve of each of the n sub-histograms includes: extracting multiple features from each sub-histogram; and determining the mapping curve of each sub-histogram based on the multiple features. It should be noted that extracting features from the sub-histograms may also involve extracting at least one feature, without specific limitation.

[0009] In one possible implementation, the plurality of features includes one or more of the following: the mean brightness, the standard deviation of brightness, and the pixel percentage of the sub-histogram, wherein the pixel percentage is the proportion of the number of pixels included in the sub-histogram to the total number of pixels included in the histogram.

[0010] In one possible implementation, determining the mapping curve of each sub-histogram based on the plurality of features includes: calculating the gain curve coefficient and the equalization curve weight of each sub-histogram based on the plurality of features; and determining the mapping curve of each sub-histogram based on the gain curve coefficient and the equalization curve weight.

[0011] The key to obtaining the mapping curve of the sub-histogram in this application lies in the calculation of the gain curve coefficient and the equalization curve weight, which parameterizes the generation process of the mapping curve, improving output efficiency and accuracy. Furthermore, introducing gain curve coefficients and equalization curve weights to each sub-histogram allows for the adjustment of both the brightness and contrast distribution of each sub-histogram, achieving a balanced result.

[0012] In one possible implementation, calculating the gain curve coefficients and equilibrium curve weights of each sub-histogram based on the plurality of features includes: inputting the plurality of features into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and inputting the plurality of features into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0013] In one possible implementation, calculating the gain curve coefficients and equilibrium curve weights of each sub-histogram based on the plurality of features includes: inputting the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and inputting the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0014] In one possible implementation, the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator comprise an adaptive neural fuzzy system or a neural network.

[0015] In one possible implementation, each of the n sub-histograms is a histogram that has undergone low-pass filtering.

[0016] The resulting sub-histogram has a smooth transition on the vertical axis from data to zero, without causing abrupt changes. This ensures a smooth change in the extracted feature values, which in turn ensures a smooth change in the calculated mapping curve. When using this mapping curve to process images, it guarantees the stability of the image brightness.

[0017] In one possible implementation, before dividing the histogram into n sub-histograms, the method further includes: performing a brightness transformation on the histogram; dividing the histogram into n sub-histograms includes: dividing the brightness-transformed histogram into n sub-histograms.

[0018] In one possible implementation, after dividing the histogram into n sub-histograms, the method further includes performing brightness transformation processing on each of the n sub-histograms.

[0019] In one possible implementation, the step of synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram includes: splicing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram, and taking the average, maximum or minimum value at the connection point of the mapping curves of two adjacent sub-histograms, wherein the average, maximum or minimum value is the average, maximum or minimum value of the two values ​​taken at the connection point of the mapping curves of the two adjacent sub-histograms.

[0020] This ensures the monotonicity of GTMC, thereby maintaining the original relative relationship of image brightness and preventing any reversal.

[0021] In one possible implementation, the adaptive neural fuzzy system is further trained according to a first set of parameters, wherein the first set of parameters includes input feature membership function parameters, output membership function parameters, and rule functions.

[0022] In one possible implementation, the neural network is further trained according to a second set parameter, wherein the second set parameter includes the plurality of features, membership function calculation nodes, and output values ​​corresponding to different combinations of rule functions.

[0023] In one possible implementation, the step of calculating the mapping curve of the first sub-histogram based on the gain curve coefficients and the equalization curve weights includes:

[0024] The mapping curve of the first sub-histogram is calculated according to formula (1):

[0025] y i =α i ×y i_gain +β i ×y i_hist (1)

[0026] Among them, y i The mapping curve representing the first sub-histogram, γ i β represents the gain curve coefficient. i α represents the weight of the equilibrium curve. i =1-β i y i_gain This represents the gain curve of the first sub-histogram. j∈[0,m],y i_hist This represents the equilibrium curve of the first sub-histogram.

[0027] Secondly, this application provides an image enhancement processing apparatus, comprising: an acquisition module for acquiring a histogram of an image to be processed; a processing module for dividing the histogram into n sub-histograms, where n is a positive integer; calculating a mapping curve for each of the n sub-histograms; synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; and performing enhancement processing on the image to be processed based on the mapping curve of the histogram.

[0028] In one possible implementation, the processing module is specifically used to extract multiple features of each sub-histogram and determine the mapping curve of each sub-histogram based on the multiple features.

[0029] In one possible implementation, the plurality of features includes one or more of the following: the mean brightness, the standard deviation of brightness, and the pixel percentage of the sub-histogram, wherein the pixel percentage is the proportion of the number of pixels included in the sub-histogram to the total number of pixels included in the histogram.

[0030] In one possible implementation, the processing module is specifically configured to calculate the gain curve coefficient and the equalization curve weight of each sub-histogram based on the plurality of features; and determine the mapping curve of each sub-histogram based on the gain curve coefficient and the equalization curve weight.

[0031] In one possible implementation, the processing module is specifically used to obtain the gain curve coefficients from a first histogram enhancement coefficient generator pre-trained from the plurality of feature inputs, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to obtain the equilibrium curve weights from a second histogram enhancement coefficient generator pre-trained from the plurality of feature inputs, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0032] In one possible implementation, the processing module is specifically configured to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0033] In one possible implementation, the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator comprise an adaptive neural fuzzy system or a neural network.

[0034] In one possible implementation, each of the n sub-histograms is a histogram that has undergone low-pass filtering.

[0035] In one possible implementation, the processing module is further configured to perform brightness transformation processing on the histogram; and divide the brightness-transformed histogram into n sub-histograms.

[0036] In one possible implementation, the processing module is further configured to perform brightness transformation processing on each of the n sub-histograms.

[0037] In one possible implementation, the processing module is specifically used to stitch together the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram, and to take the average, maximum or minimum value at the connection point of the mapping curves of two adjacent sub-histograms. The average, maximum or minimum value is the average, maximum or minimum value of the two values ​​of the mapping curves of the two adjacent sub-histograms at the connection point of each other.

[0038] In one possible implementation, the processing module is further configured to train the adaptive neural fuzzy system according to a first set parameter, wherein the first set parameter includes an input feature membership function parameter, an output membership function parameter, and a rule function.

[0039] In one possible implementation, the processing module is further configured to train the neural network according to a second set parameter, wherein the second set parameter includes the multiple features, membership function calculation nodes, and output values ​​corresponding to different combinations of rule functions.

[0040] In one possible implementation, the processing module is specifically used to calculate the mapping curve of the first sub-histogram according to formula (1):

[0041] y i =α i ×y i_gain +β i ×y i_hist (1)

[0042] Among them, y i The mapping curve representing the first sub-histogram, γ i β represents the gain curve coefficient. i α represents the weight of the equilibrium curve. i =1-β i y i_gain This represents the gain curve of the first sub-histogram. j∈[0,m],y i_hist This represents the equilibrium curve of the first sub-histogram.

[0043] Thirdly, this application provides an image enhancement processing apparatus, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any of the first aspects above.

[0044] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the method described in any one of the first aspects above. Attached Figure Description

[0045] Figure 1 An exemplary structural schematic diagram of a terminal device 100 is shown;

[0046] Figure 2 An exemplary structural schematic diagram of an image acquisition device is shown;

[0047] Figure 3 A flowchart illustrating an embodiment of the image enhancement processing method provided in this application;

[0048] Figure 4 An exemplary schematic diagram of a gamma curve is shown;

[0049] Figure 5 An exemplary coefficient diagram of a filter is shown;

[0050] Figure 6 This is a schematic diagram of the structure of an embodiment of the image enhancement processing apparatus of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0053] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0054] The terminal equipment in this application, also known as user equipment (UE), can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; it can also be deployed on water (such as ships); and it can also be deployed in the air (such as airplanes, balloons, and satellites). The terminal equipment can be a mobile phone, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) device, augmented reality (AR) device, monitoring equipment, smart screen, smart TV, wireless device in remote medical care, or wireless device in a smart home, etc., and this application does not limit this.

[0055] Figure 1 An exemplary schematic diagram of a terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 151, a wireless communication module 152, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, an accelerometer sensor 180B, a proximity sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, and an image sensor 180N (of course, the terminal device 100 may also include other sensors, such as temperature sensors, pressure sensors, distance sensors, magnetic sensors, ambient light sensors, air pressure sensors, bone conduction sensors, etc., which are not shown in the figure).

[0056] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0057] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the central nervous system and command center of the terminal device 100. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0058] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0059] When the processor 110 integrates different devices, such as a CPU and a GPU, the CPU and GPU can work together to execute the methods provided in the embodiments of this application. For example, some algorithms in the method are executed by the CPU and other algorithms are executed by the GPU to achieve faster processing efficiency.

[0060] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0061] Camera 193 (a front-facing camera or a rear-facing camera, or a single camera that can function as both) is used to capture still images or videos. Typically, camera 193 may include a photosensitive element such as a lens assembly and an image sensor. The lens assembly includes multiple lenses (convex or concave lenses) for collecting light signals reflected from the object being photographed and transmitting these signals to the image sensor. The image sensor generates a raw image of the object being photographed based on the light signals.

[0062] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and signal processing of terminal device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area can store the operating system, application code (such as camera applications, WeChat applications, etc.), etc. The data storage area can store data created during the use of terminal device 100 (such as images and videos captured by camera applications, etc.).

[0063] In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0064] Of course, the code for the method provided in this application embodiment can also be stored in external memory. In this case, the processor 110 can run the code stored in external memory through the external memory interface 120.

[0065] The functions of sensor module 180 are described below.

[0066] The gyroscope sensor 180A can be used to determine the motion attitude of the terminal device 100. In some embodiments, the gyroscope sensor 180A can determine the angular velocity of the terminal device 100 about three axes (i.e., the x, y, and z axes). That is, the gyroscope sensor 180A can be used to detect the current motion state of the terminal device 100, such as jitter or stillness.

[0067] Accelerometer 180B can detect the magnitude of acceleration of terminal device 100 in various directions (generally three axes). That is, accelerometer 180B can be used to detect the current motion state of terminal device 100, such as whether it is shaking or stationary.

[0068] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outward through the LED. The terminal device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100.

[0069] The gyroscope sensor 180A (or accelerometer 180B) can send the detected motion state information (such as angular velocity) to the processor 110. The processor 110 determines whether the current state is handheld or tripod based on the motion state information (for example, if the angular velocity is not 0, it means that the terminal device 100 is in handheld state).

[0070] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0071] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0072] For example, the display screen 194 of the terminal device 100 displays a main interface, which includes icons for multiple applications (such as a camera application, a WeChat application, etc.). The user taps the camera application icon on the main interface using the touch sensor 180K, triggering the processor 110 to launch the camera application and open the camera 193. The display screen 194 then displays the camera application's interface, such as the viewfinder.

[0073] The 180N image sensor utilizes the photoelectric conversion function of optoelectronic devices to convert light signals on the photosensitive surface into electrical signals that are proportional to the light signals. It can generate raw image data (RAW format image) of the object to be photographed based on the light signals.

[0074] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 151, wireless communication module 152, modem processor, and baseband processor.

[0075] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0076] The mobile communication module 151 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 100. The mobile communication module 151 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 151 can receive electromagnetic waves via the antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to the modem processor for demodulation. The mobile communication module 151 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna 1. In some embodiments, at least some functional modules of the mobile communication module 151 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 151 and at least some modules of the processor 110 may be housed in the same device.

[0077] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 151 or other functional modules.

[0078] The wireless communication module 152 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 152 can be one or more devices integrating at least one communication processing module. The wireless communication module 152 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 152 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0079] In some embodiments, antenna 1 of terminal device 100 is coupled to mobile communication module 151, and antenna 2 is coupled to wireless communication module 152, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0080] Additionally, terminal device 100 can implement audio functions such as music playback and recording via audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. Terminal device 100 can receive key input 190, generating key signal inputs related to user settings and function control. Terminal device 100 can use motor 191 to generate vibration alerts (e.g., vibration alert for incoming calls). The indicator 192 in terminal device 100 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. The SIM card interface 195 in terminal device 100 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with terminal device 100.

[0081] It should be understood that, in practical applications, terminal device 100 may include more than Figure 1 The number of more or fewer components shown in the embodiments of this application is not limited.

[0082] Figure 2 An exemplary structural schematic diagram of an image acquisition device is shown, such as... Figure 2 As shown, the image acquisition device comprises a lens 201, an image sensor 202, and an image signal processor (ISP) 203. The lens 201 is used to capture still images or videos, collect light signals reflected from the object to be photographed, and transmit the collected light signals to the image sensor. The image sensor 202 generates raw image data (RAW format image) of the object to be photographed based on the light signals. The ISP 203 processes the RAW format image into an image suitable for human eye observation or display, and in this process adjusts the visible light exposure parameters and / or the light intensity of the infrared supplementary light until the convergence condition of the automatic exposure (AE) algorithm is met. In this embodiment, the lens 201 can be... Figure 1 The camera 193 and image sensor 202 in the terminal device shown can be... Figure 1 The image sensor 180N in the sensor module 180 of the terminal device shown, and the ISP 203 can be... Figure 1 The processor 110 in the terminal device shown.

[0083] Optionally, the image acquisition device can adopt a single lens plus a single image sensor, or a dual lens plus a dual image sensor, or a single lens plus a beam splitter and a dual image sensor structure. The single-lens structure can save costs, while the single-image-sensor structure can simplify the camera structure. This application does not impose specific limitations in this regard.

[0084] Figure 3 A flowchart of an embodiment of the image enhancement processing method provided in this application is shown below. Figure 3 As shown, the method of this embodiment can be executed by the aforementioned terminal device, image acquisition device, or processor. This image enhancement processing method may include:

[0085] Step 301: Obtain the histogram of the image to be processed.

[0086] A histogram is a brightness statistical graph. Its horizontal axis represents the brightness value of pixels in the image to be processed, and its vertical axis represents the number of pixels corresponding to a given brightness value (indicating the total number of pixels with a specific brightness value in the image). It should be noted that the brightness values ​​in the histogram can be obtained by sampling at a certain granularity. For example, if the brightness range of the image to be processed is 0-255, sampling at every integer value would result in 256 values ​​on the horizontal axis of the histogram (0, 1, ..., 254, 255); sampling at every interval of an integer value would result in 128 values ​​(0, 2, ..., 252, 254); and sampling at every 0.5 integer value intervals would result in 512 values ​​(0, 0.5, ..., 254.5, 255). This application does not impose specific limitations on this aspect. The number of pixels in the histogram can also be obtained by sampling at a certain granularity. For example, if the pixel array of the image to be processed is 400×600, if sampling is performed on every single pixel, the total number of pixels in the histogram will be 400×600; if sampling is performed on every other pixel, the total number of pixels in the histogram will be half of 400×600. This application does not impose any specific limitations on this.

[0087] Step 302: Divide the histogram into n sub-histograms.

[0088] n is a positive integer. This application divides the histogram based on its x-coordinate. To ensure a smooth transition between sub-histograms, adjacent sub-histograms may or may not overlap. For example, if the histogram's x-coordinate includes 256 values ​​from 0 to 255, and each histogram includes 64 values, the histogram can be divided into four non-overlapping sub-histograms, corresponding to x-coordinate ranges of [0, 63], [64, 127], [128, 191], and [192, 255]. As another example, if the histogram's x-coordinate includes 256 values ​​from 0 to 255, and each histogram includes 70 values, the histogram can be divided into four overlapping sub-histograms, corresponding to x-coordinate ranges of [0, 69], [61, 130], [125, 194], and [186, 255]. For example, if the horizontal axis of a histogram includes 256 coordinate values ​​from 0 to 255, the histogram can be divided into four non-overlapping sub-histograms, corresponding to the horizontal axis ranges of [0, 69], [70, 130], [131, 191], and [192, 255], respectively. The number of coordinate values ​​included in these four sub-histograms is not necessarily equal. It should be noted that the histogram of the image to be processed can be divided using various rules, such as equal / unequal length division, overlapping division, non-overlapping division, etc., and this application does not specifically limit the specific rules in this regard.

[0089] Since many brightness-related processing modules in the ISP have a significant impact on image brightness, the most important one is the gamma curve. Figure 4 An exemplary schematic diagram of a gamma curve is shown, such as... Figure 4 As shown, the horizontal axis represents the original brightness value, the vertical axis represents the mapped brightness value, and the gamma curve is used to represent the correspondence between the original brightness value and the mapped brightness value. Therefore, in order to extract more accurate image features in subsequent steps, the histogram can be subjected to brightness transformation processing. This brightness transformation processing can include gamma transformation or other brightness transformation methods, which are not specifically limited in this application.

[0090] In one possible implementation, the above brightness transformation process can be performed on the histogram of the image to be processed first, and then the brightness-transformed histogram can be divided into n sub-histograms based on the horizontal coordinate of the brightness-transformed histogram.

[0091] In one possible implementation, the histogram can be divided into n sub-histograms based on its horizontal coordinates, and then the brightness transformation process described above can be performed on each sub-histogram.

[0092] Furthermore, this application divides the histogram of the image to be processed into n sub-histograms, and the dynamic range of each sub-histogram can be expanded. As mentioned above, the range of the x-coordinate of each sub-histogram is a subset of the range of the x-coordinate of the histogram of the image to be processed. The range of the x-coordinate of each sub-histogram can be expanded to the range of the x-coordinate of the histogram of the image to be processed, but the value of the y-coordinate is 0 within the expanded range. For example, a sub-histogram with an x-coordinate range of [70, 130] will have an expanded x-coordinate range of [0, 255]. However, the y-coordinate values ​​corresponding to x-coordinates other than [70, 130] are all 0, indicating that the sub-histogram has no corresponding pixels at these brightness values.

[0093] Optionally, this application may also perform enhancement processing on each sub-histogram. This enhancement processing may include equalization (proportionally expanding or shrinking the spacing of the entire histogram on the horizontal axis), left shift (shifting the entire histogram to the left by x + coordinate units), or right shift (shifting the entire histogram to the right by x - coordinate units), etc. The specific method used to enhance a particular sub-histogram can be determined based on the brightness distribution of that sub-histogram, or based on the brightness distribution of that sub-histogram and the brightness distribution of its adjacent sub-histograms. This application does not impose specific limitations in this regard.

[0094] Step 303: Calculate the mapping curve of each sub-histogram in the n sub-histograms.

[0095] The first sub-histogram can be any one of the n sub-histograms. The following explanation uses the first sub-histogram as an example to illustrate step 303.

[0096] Optionally, this application may perform low-pass filtering on the first sub-histogram, for example, by using a Gaussian filter with a mean of 0 and a variance of 10 for the filter coefficients. Figure 5 An exemplary coefficient diagram of a filter is shown, such as... Figure 5 As shown, the horizontal axis represents the brightness of the sub-histogram, and the vertical axis represents the weight of the corresponding brightness. For example, if the sub-histogram is designed to include 16 coordinate values ​​([16,32]), half of the coordinate values ​​are taken from each of the left and right adjacent sub-histograms ([8,16] and [32,40]), and then added to all the coordinate values ​​of this sub-histogram to form a sub-histogram containing 32 coordinate values ​​([8,40]). The weight within this sub-histogram can be set to 1, that is, the vertical axis corresponding to the horizontal axis [16,32] is 1. The weights within adjacent sub-histograms exhibit a Gaussian variation, with the weight decreasing the further away from the original sub-histogram. That is, the smaller the horizontal axis in [8,16], the smaller the weight, and the larger the horizontal axis in [32,40], the smaller the weight. The resulting sub-histogram has a smooth transition on the vertical axis from data to zero, without causing abrupt changes. This ensures a smooth change in the extracted feature values, which in turn ensures a smooth change in the calculated mapping curve. When using this mapping curve to process images, it guarantees the stability of the image brightness.

[0097] This application first extracts multiple features of the first sub-histogram; calculates the gain curve coefficients and equalization curve weights of the first sub-histogram based on the multiple features; and calculates the mapping curve of the first sub-histogram based on the gain curve coefficients and equalization curve weights.

[0098] The features of the first sub-histogram mentioned above may include, but are not limited to, grayscale mean, variance, etc. Preferably, the brightness mean, brightness standard deviation, and pixel percentage of the first sub-histogram are used.

[0099] Here, the mean luminance value refers to the weighted average of all luminance values ​​included in the first sub-histogram, and its calculation formula can be... r represents the average brightness, x i Let a represent the i-th brightness value. i The weight represents the i-th brightness value, and N represents the number of brightness values ​​included in the first sub-histogram. The weight of each brightness value is determined based on the ordinate value corresponding to that brightness value; the higher the ordinate value, the greater the weight. If the dynamic range of each sub-histogram has been extended, then the weight value for the brightness value corresponding to a ordinate value of 0 can be set to 0.

[0100] The standard deviation of luminance refers to the root mean square deviation of all luminance values ​​and the luminance mean included in the first sub-histogram. Its calculation formula can be... δ(r) represents the standard deviation of luminance.

[0101] Pixel percentage refers to the proportion of pixels included in the first sub-histogram to the total number of pixels included in the histogram of the image to be processed. Its calculation formula can be... Where p represents the percentage of pixels, m represents the number of pixels included in the first sub-histogram, and M represents the number of pixels included in the histogram of the image to be processed.

[0102] As mentioned above, when dividing the image into histograms, the sub-histograms may or may not overlap. In the case of no overlap, the range of parameters in the formula for calculating the aforementioned feature values ​​is usually limited to the range of the x-coordinate of the first sub-histogram. For example, if the x-coordinate of the first sub-histogram ranges from [0, 63], then when calculating the mean brightness and standard deviation of brightness, x... i The value range of is [0, 63], specifically 0, 1, ..., 62, 63. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates. Optionally, in this case, the value range of the parameter in the formula can also exceed the range of the x-coordinate of the first sub-histogram. For example, x... i The value range is [0, 65], specifically 0, 1, ..., 64, 65. When calculating m, the value is taken on the ordinate corresponding to the horizontal coordinates of these x-coordinates. Alternatively, the value range of the parameter in the formula is smaller than the range of the horizontal coordinates of the first sub-histogram, for example, x. i The value range of x is [0, 60], specifically 0, 1, ..., 59, 60. When calculating m, the values ​​are taken on the ordinates corresponding to the x-coordinates of 0, 1, ..., 59, 60. However, for cases with overlap, when calculating the above feature values, the range of the parameter in the formula can be limited to the non-overlapping subset of the x-coordinate range of the first sub-histogram. For example, if the x-coordinate range of the first sub-histogram is [0, 69], where [61, 69] overlaps with adjacent sub-histograms, then when calculating the mean brightness and standard deviation of brightness, x... i The value range of is [0, 60], specifically 0, 1, ..., 59, 60. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates. Optionally, in this case, the value range of the parameter in the formula can be either a non-overlapping subset of the x-coordinate range of the first sub-histogram, or an overlapping subset of the x-coordinate range of the first sub-histogram. For example, x... iThe value range of is [0, 69], specifically 0, 1, ..., 68, 69. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates. Optionally, in this case, the value range of the parameter in the formula can be either a non-overlapping subset of the x-coordinate range of the first sub-histogram, or a subset of overlapping subsets of the x-coordinate range of the first sub-histogram. For example, x... i The value range is [0, 69], specifically 0, 1, ..., 64, 65. When calculating m, the value is taken on the vertical axis corresponding to the horizontal axis of 0, 1, ..., 64, 65.

[0103] It should be noted that, when extracting feature values ​​from sub-histograms, this application may use the following parameters in the formula: all or part of the values ​​within the range of the corresponding sub-histogram's x-coordinate; or the values ​​may be extended beyond the range of the corresponding sub-histogram's x-coordinate; or only non-overlapping subsets within the range of the corresponding sub-histogram's x-coordinate; or all or part of both non-overlapping and overlapping subsets within the range of the corresponding sub-histogram's x-coordinate. This application does not impose specific limitations in this regard.

[0104] The calculation of the gain curve coefficients and equalization curve weights of the first sub-histogram based on the above-mentioned multiple features may include: inputting multiple features into a pre-trained first histogram enhancement coefficient generator to obtain gain curve coefficients, the first histogram enhancement coefficient generator being used to calculate gain curve coefficients; inputting multiple features into a pre-trained second histogram enhancement coefficient generator to obtain equalization curve weights, the second histogram enhancement coefficient generator being used to calculate equalization curve weights.

[0105] The first and second histogram enhancement coefficient generators comprise an Adaptive Network-based Fuzzy Inference System (ANFIS) or a neural network. The first histogram enhancement coefficient generator outputs the gain curve coefficients, and the second histogram enhancement coefficient generator outputs the equalization curve weights.

[0106] The following explanation uses ANFIS as an example to illustrate the output process of the gain curve coefficients. Standard ANFIS includes an input layer, a membership layer, a rule layer, and a defuzzification layer.

[0107] (1) The input layer receives multiple features of the first sub-histogram. These multiple features may be, for example, the mean brightness, standard deviation brightness, and pixel percentage of the first sub-histogram. It should be noted that the multiple features are not limited to the three mentioned above, and may include other features, without specific limitations.

[0108] (2) The membership layer has multiple membership functions for each feature. The membership degree of each feature value at the corresponding level is calculated based on the membership function. The membership function can be triangular, trapezoidal, Gaussian, etc., and there can be overlap between membership functions at different levels to ensure a smooth transition. The membership function for each feature value is different, and the number of levels can be designed according to needs. For example, if the designed membership function is trapezoidal, there are three levels: 1-8 is the low level, 12-18 is the medium level, and 22-30 is the high level. 8-12 is the overlapping area of ​​the hypotenuses of the low and medium levels, and 18-22 is the overlapping area of ​​the hypotenuses of the medium and high levels. If the feature value is the average brightness, inputting 5 will calculate the membership degrees for the three levels as 1,0,0; inputting 10 will calculate the membership degrees as 0.5,0.5,0.

[0109] (3) The rule layer adopts the standard rule method of Fuzzy Inference System (FIS), covering all possible combinations of input to membership function. Different feature values ​​at different levels have multiple combinations. The rule layer designs the combination methods of feature values, calculates the combination weights based on membership degree, and designs the corresponding system output coefficient values ​​of the combinations.

[0110] (4) The defuzzification layer adopts the standard FIS implementation method, including but not limited to the maximum membership method, centroid method, and weighted average method. Based on the weight calculation results and the design scheme of the rule layer, the defuzzification layer calculates the gain curve coefficients that the system needs to output. The formula for calculating the gain curve coefficients can be... Where, γ i b represents the gain curve coefficient of the i-th sub-histogram. j (j = 1, 2, ..., k) represents the weight of the combination of the i-th sub-histograms output by the rule layer, γ j This represents the j-th gear value of the output coefficient (gain curve coefficient) required by the corresponding combined system designed in the rule layer.

[0111] The following explanation, using ANFIS as an example, illustrates the output process of the equilibrium curve weights.

[0112] (1) The input layer receives multiple features of the first sub-histogram. These multiple features may be, for example, the mean brightness, standard deviation brightness, and pixel percentage of the first sub-histogram. It should be noted that the multiple features are not limited to the three mentioned above, and may include other features, without specific limitations.

[0113] (2) The membership layer has multiple membership functions for each feature. The membership degree of each feature value at the corresponding level is calculated based on the membership function. The membership function can be triangular, trapezoidal, Gaussian, etc., and there can be overlap between membership functions at different levels to ensure a smooth transition. The membership function for each feature value is different, and the number of levels can be designed according to needs. For example, if the designed membership function is trapezoidal, there are three levels: 1-8 is the low level, 12-18 is the medium level, and 22-30 is the high level. 8-12 is the overlapping area of ​​the hypotenuses of the low and medium levels, and 18-22 is the overlapping area of ​​the hypotenuses of the medium and high levels. If the feature value is the average brightness, inputting 5 will calculate the membership degrees for the three levels as 1,0,0; inputting 10 will calculate the membership degrees as 0.5,0.5,0.

[0114] (3) The rule layer adopts the standard rule method of Fuzzy Inference System (FIS), covering all possible combinations of input to membership function. The difference between calculating the weight of the equilibrium curve and the calculation of the gain curve coefficient mentioned above lies in the rule layer, including the design of the membership function of the membership layer, the combination method of eigenvalues, the calculation method of the combination weight, and the value of the system output coefficient corresponding to the combination.

[0115] (4) The defuzzification layer adopts the standard FIS implementation method, including but not limited to the maximum membership method, centroid method, and weighted average method. Based on the weight calculation results and the design scheme of the rule layer, the defuzzification layer calculates the gain curve coefficients that the system needs to output. The formula for calculating the equalization curve weights can be... Where, β i c represents the equilibrium curve weight of the i-th sub-histogram. j (j = 1, 2, ..., l) represents the weight of the combination of the i-th sub-histograms output by the rule layer, β j This represents the j-th gear value of the output coefficients (equilibrium curve weights) required for the corresponding combined system designed in the rule layer.

[0116] The first and second histogram enhancement coefficient generators mentioned above are both controlled by a series of parameters that govern their input and output characteristics. For ANFIS, this series of parameters may include input feature membership function parameters, output membership function parameters, and rule functions. For neural networks, this series of parameters may include multiple features, membership function calculation nodes, and output values ​​corresponding to different combinations of rule functions. The generation process of these parameters includes: a) acquiring image data training set 1, including dataset construction, capturing natural scenes under different weather and lighting conditions using a terminal device, initializing the above parameters, selecting typical scenes to observe the distribution characteristics of their sub-histograms, initially designing membership functions, initializing each combination value of the first enhancement coefficient rule layer to 1, and initializing each combination value of the second enhancement coefficient rule layer to a small value, such as 5; obtaining parameter set 1, including the membership function of each sub-histogram, and the values ​​of the generator output coefficients in different combinations of rule layer designs; b) using the initial histogram enhancement coefficient generator on training set 1 and parameter set 1 to generate coefficient set 1, including the gain curve output by the first histogram enhancement coefficient generator. c) Use coefficient set 1 to generate GTMC curve set 1, apply GTMC curve set 1 to training set 1 to obtain output set 1, and observe the effect; d) Based on the brightness and contrast effect of output set 1, correct coefficient set 1 to obtain coefficient label set 1; e) Train ANFIS parameters based on training set 1 and coefficient label set 1, using general training methods such as gradient descent and genetic algorithm, etc.; use the trained parameter set 2 to generate coefficient set 2; f) Replace parameter set 1 and coefficient set 1 with parameter set 2 and coefficient set 2, and repeat steps b to e until the image effect reaches the target, to obtain the trained histogram enhancement coefficient generator. It is evident that histogram enhancement coefficient generators, utilizing ANFIS or neural networks as gain curve coefficients and equalization curve weights, offer several advantages. Firstly, the generator itself can optimize its control parameters through a generation-verification-correction-training-correction-training process. These parameters can be locally adjusted to suit the characteristics of histogram distributions in different scenarios. Even modifying control parameters such as membership functions or rules for a specific scenario within a trained histogram enhancement coefficient generator will not affect the curve generation strategy for other scenarios. Secondly, the gain curve coefficients and equalization curve weights obtained through the histogram enhancement coefficient generator better meet the image brightness enhancement requirements, thus contributing to improved image processing efficiency.

[0117] Optionally, when calculating the gain curve coefficients and equalization curve weights, this application can input multiple features and multiple features of adjacent sub-histograms into a pre-trained first histogram enhancement coefficient generator or a second histogram enhancement coefficient generator. In this way, the output data can achieve a smooth transition in brightness when the scene changes because it references the brightness distribution of adjacent sub-histograms.

[0118] After obtaining the gain curve coefficients and equalization curve weights, this application can calculate the mapping curve of the first sub-histogram based on the gain curve coefficients and equalization curve weights. The mapping curve of the i-th sub-histogram is calculated according to formula (1):

[0119] y i =α i ×y i_gain +β i ×y i_hist (1)

[0120] Among them, y i Let β represent the mapping curve of the i-th sub-histogram. i α represents the equilibrium curve weight of the i-th sub-histogram. i =1-β i y i_gain The gain curve representing the i-th sub-histogram is used to adjust image brightness. j∈[0,m],γ i pow(j,γ) represents the gain curve coefficient of the i-th sub-histogram. i )=j∧γ i L represents the maximum grayscale value of the image, y i_hist The equalization curve representing the i-th sub-histogram is used to adjust image contrast.

[0121] Therefore, the key to obtaining the mapping curve of the sub-histogram in this application lies in the calculation of the gain curve coefficient and the equalization curve weight. This parameterizes the generation process of the mapping curve, improving output efficiency and accuracy. Furthermore, introducing gain curve coefficients and equalization curve weights to each sub-histogram allows for the adjustment of both the brightness and contrast distribution of each sub-histogram, achieving a balanced result.

[0122] Step 304: Combine the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram.

[0123] This application can stitch together the mapping curves of n sub-histograms to obtain a complete GTMC, which is the mapping curve of the histogram of the image to be processed. The stitching principle can include: the mapping curve calculated within the extended range (the horizontal coordinate of the sub-histogram is extended to the entire horizontal coordinate range of the image to be processed, the extended range refers to the horizontal coordinate range other than the horizontal coordinate range before the sub-histogram is extended) is only effective within the current sub-histogram. At the stitching point of two adjacent sub-histograms, the average value, the larger value, or the smaller value can be taken to ensure the monotonicity of the GTMC, thereby maintaining the original relative relationship of image brightness and preventing reversal.

[0124] If two adjacent sub-histograms do not overlap, the mapping curves of the two sub-histograms can be joined at the junction using methods such as the median endpoint method (averaging the ordinates of the endpoints of the mapping curves of the two adjacent sub-histograms), the maximum endpoint method (taking the larger ordinate of the endpoints of the mapping curves of the two adjacent sub-histograms), or the minimum endpoint method (taking the smaller ordinate of the endpoints of the mapping curves of the two adjacent sub-histograms). If two adjacent sub-histograms overlap, only the mapping curve of one of the sub-histograms is used in the overlapping portion.

[0125] Example 1: The horizontal axis of the histogram includes 256 coordinate values ​​from 0 to 255. Divide this histogram into four non-overlapping sub-histograms. Figure 1 The x-coordinate ranges from [0, 63], and the sub-histogram... Figure 2 The x-coordinate range is [64, 127], and the sub-histogram... Figure 3 The x-coordinate range is [128, 191], and the sub-histogram... Figure 3 The x-coordinate range is [192, 255]. When extracting features, the range of values ​​for the parameters in the formula is usually limited to the range of the x-coordinate of the sub-histogram, i.e., the sub-histogram... Figure 1 x i The value range is [0, 63], specifically 0, 1, ..., 62, 63. When calculating m, the values ​​are taken on the ordinates corresponding to the x-coordinates of 0, 1, ..., 62, 63; sub-histogram. Figure 2 x i The value range is [64, 127], specifically 64, 65, ..., 126, 127. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates; sub-histogram. Figure 3 x i The value range is [128, 191], specifically 128, 129, ..., 190, 191. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates: 128, 129, ..., 190, 191. (Sub-histogram) Figure 3 x iThe value range is [192, 255], specifically 192, 193, ..., 254, 255. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates. It can be seen that in this embodiment, the x-coordinate range of the sub-histogram and the parameter value range when extracting features are in one-to-one correspondence and completely consistent.

[0126] During splicing, sub-historical Figure 1 Mapping curve and sub-histogram Figure 2 The mapping curves are spliced ​​end to end, and the sub-histogram is... Figure 2 Mapping curve and sub-histogram Figure 3 The mapping curves are spliced ​​end to end, and the sub-histogram is... Figure 3 Mapping curve and sub-histogram Figure 3 The mapping curves are spliced ​​end to end. The endpoint median connection method is used at the splice point; that is, the average value 1 is calculated from the ordinates corresponding to the x-coordinates 63 and 64, and this average value 1 is used as the sub-histogram. Figure 1 Mapping curve and sub-histogram Figure 2 The transition values ​​at the beginning and end of the mapping curve; the average value 2 is calculated by taking the ordinates corresponding to the x-coordinates of 127 and 128 respectively, and the average value 2 is used as the sub-histogram. Figure 2 Mapping curve and sub-histogram Figure 3 The transition values ​​at the beginning and end of the mapping curve; the average value of 3 is calculated by taking the ordinates corresponding to the horizontal coordinates of 191 and 192 respectively, and the average value of 3 is used as the sub-histogram. Figure 3 Mapping curve and sub-histogram Figure 3 The beginning and end transition values ​​of the mapping curve.

[0127] Example 2: The horizontal axis of the histogram includes 256 coordinate values ​​from 0 to 255. This histogram is divided into four adjacent, overlapping sub-histograms. Figure 1 The x-coordinate ranges from [0, 69], and the sub-histogram... Figure 2 The x-coordinate range is [61, 130], and the sub-histogram... Figure 3 The x-coordinate range is [125, 194], and the sub-histogram... Figure 3 The x-coordinate range is [186, 255]. When extracting features, the parameter values ​​in the formula can be taken from a non-overlapping subset of the x-coordinate range of the sub-histogram, i.e., the sub-histogram. Figure 1 x i The value range is [0, 60], specifically 0, 1, ..., 59, 60. When calculating m, the values ​​are taken on the ordinates corresponding to the x-coordinates of 0, 1, ..., 59, 60; sub-histogram. Figure 2 x iThe value range is [70, 124], specifically 70, 71, ..., 123, 124. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates: 70, 71, ..., 123, 124. (Sub-histogram) Figure 3 x i The value range is [131, 185], specifically 131, 132, ..., 184, 185. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates: 131, 132, ..., 184, 185. (Sub-histogram) Figure 3 x i The value range is [195, 255], specifically 195, 196, ..., 254, 255. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates. It can be seen that the parameter value range when extracting features in this embodiment is a subset of the x-coordinate range of the corresponding sub-histogram.

[0128] During splicing, for the overlapping portion of two adjacent sub-histograms, only the mapping curve of the overlapping portion of the rightmost sub-histogram is taken, i.e., the sub-histogram. Figure 1 Take the mapping curve corresponding to the x-coordinate [0, 61], and the sub-histogram. Figure 2 Take the mapping curve corresponding to [61, 125], and the sub-histogram. Figure 3 Take the mapping curve corresponding to [125, 186], and the sub-histogram. Figure 3 Take the mapping curve corresponding to [186,255] and splice these four mapping curves together in sequence.

[0129] Example 3: The horizontal axis of the histogram includes 256 coordinate values ​​from 0 to 255. This histogram is divided into four adjacent, overlapping sub-histograms. Figure 1 The x-coordinate ranges from [0, 69], and the sub-histogram... Figure 2 The x-coordinate range is [61, 130], and the sub-histogram... Figure 3 The x-coordinate range is [125, 194], and the sub-histogram... Figure 3 The x-coordinate range is [186, 255]. When extracting features, the parameter value range in the formula can simultaneously be a subset of the non-overlapping subsets of the x-coordinate range of the sub-histogram and a subset of the overlapping subsets of the x-coordinate range of the sub-histogram, i.e., the sub-histogram... Figure 1 x i The value range is [0, 64], specifically 0, 1, ..., 63, 64. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates: 0, 1, ..., 63, 64. (Sub-histogram) Figure 2 x iThe value range is [65, 127], specifically 65, 66, ..., 126, 127. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates; sub-histogram. Figure 3 x i The value range is [128, 189], specifically 128, 129, ..., 188, 189. When calculating m, the values ​​are taken on the ordinates corresponding to these x-coordinates: 128, 129, ..., 188, 189. (Sub-histogram) Figure 3 x i The value range is [190, 255], specifically 190, 191, ..., 254, 255. When calculating m, the values ​​are taken on the vertical coordinates corresponding to these horizontal coordinates. It can be seen that the parameter value range when extracting features in this embodiment is a subset of the horizontal coordinate range of the corresponding sub-histogram.

[0130] During splicing, for the overlapping portion of two adjacent sub-histograms, only the mapping curve of the overlapping portion of the leftmost sub-histogram is taken, i.e., the sub-histogram. Figure 1 Take the mapping curve corresponding to the x-coordinate [0, 69], and the sub-histogram. Figure 2 Take the mapping curve corresponding to the x-coordinate [69, 130], and the sub-histogram. Figure 3 Take the mapping curve corresponding to the x-coordinate [130, 194], and the sub-histogram. Figure 3 Take the mapping curve corresponding to the horizontal coordinate [194, 255] and splice these four mapping curves together in sequence.

[0131] It should be noted that when synthesizing the above-mentioned mapping curve, the endpoint median connection method is used at the splicing point of the mapping curves of two adjacent sub-histograms. In addition, this application may also use the endpoint maximum value connection method, the endpoint minimum value connection method, etc., without making specific limitations.

[0132] Step 305: Enhance the image to be processed based on the mapping curve of the histogram.

[0133] In step 302 above, a brightness transformation, such as a gamma transformation, is performed on the histogram before or after the histogram partitioning of the image to be processed. Therefore, the mapping curve of the histogram of the image to be processed obtained in step 304 is calculated based on the histogram after the brightness transformation. An inverse transformation, such as an inverse gamma curve mapping, is needed to obtain the final mapping curve. This final mapping curve is then applied to the pixels of the image to be processed, that is, the brightness of each pixel is mapped according to the calculated mapping curve to obtain the enhanced image.

[0134] This application divides the histogram of the image to be processed into multiple sub-histograms, extracts features for each sub-histogram separately, and obtains gain curve coefficients and equalization curve weights. Then, based on the aforementioned parameters, the mapping curve of the sub-histogram is obtained to achieve the enhancement processing of the image to be processed. Since each sub-histogram is processed separately, the brightness distribution and contrast distribution of the sub-histogram can be dynamically adjusted locally for different brightness ranges, achieving both fine adjustment and balanced adjustment of brightness and contrast distribution.

[0135] Figure 6 This is a schematic diagram of the structure of an embodiment of the image enhancement processing apparatus of this application, as shown below. Figure 6 As shown, the device in this embodiment can be applied to Figure 1-3 The end device in the illustrated embodiment. This image enhancement processing apparatus may include: an acquisition module 601 and a processing module 602, wherein the acquisition module 601 is used to acquire a histogram of the image to be processed; the processing module 602 is used to divide the histogram into n sub-histograms, where n is a positive integer; calculate the mapping curve of each of the n sub-histograms; synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; and perform enhancement processing on the image to be processed based on the mapping curve of the histogram.

[0136] In one possible implementation, the processing module 602 is specifically used to extract multiple features of each sub-histogram and determine the mapping curve of each sub-histogram based on the multiple features.

[0137] In one possible implementation, the plurality of features includes one or more of the following: the mean brightness, the standard deviation of brightness, and the pixel percentage of the sub-histogram, wherein the pixel percentage is the proportion of the number of pixels included in the sub-histogram to the total number of pixels included in the histogram.

[0138] In one possible implementation, the processing module is specifically configured to calculate the gain curve coefficient and the equalization curve weight of each sub-histogram based on the plurality of features; and determine the mapping curve of each sub-histogram based on the gain curve coefficient and the equalization curve weight.

[0139] In one possible implementation, the processing module 602 is specifically used to obtain the gain curve coefficients from a first histogram enhancement coefficient generator pre-trained from the plurality of feature inputs, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to obtain the equilibrium curve weights from a second histogram enhancement coefficient generator pre-trained from the plurality of feature inputs, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0140] In one possible implementation, the processing module 602 is specifically used to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

[0141] In one possible implementation, the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator comprise an adaptive neural fuzzy system or a neural network.

[0142] In one possible implementation, each of the n sub-histograms is a histogram that has undergone low-pass filtering.

[0143] In one possible implementation, the processing module 602 is further configured to perform brightness transformation processing on the histogram; and divide the brightness-transformed histogram into n sub-histograms.

[0144] In one possible implementation, the processing module 602 is further configured to perform brightness transformation processing on each of the n sub-histograms.

[0145] In one possible implementation, the processing module 602 is specifically used to splice the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram, and to take the average, maximum or minimum value at the connection point of the mapping curves of two adjacent sub-histograms. The average, maximum or minimum value is the average, maximum or minimum value of the two values ​​of the mapping curves of the two adjacent sub-histograms at the connection point of each other.

[0146] In one possible implementation, the processing module 602 is further configured to train the adaptive neural fuzzy system according to a first set parameter, wherein the first set parameter includes an input feature membership function parameter, an output membership function parameter, and a rule function.

[0147] In one possible implementation, the processing module 602 is further configured to train the neural network according to a second set parameter, wherein the second set parameter includes the multiple features, membership function calculation nodes, and output values ​​corresponding to different combinations of rule functions.

[0148] In one possible implementation, the processing module 602 is specifically used to calculate the mapping curve of the first sub-histogram according to formula (1):

[0149] y i =α i ×y i_gain +β i ×y i_hist (1)

[0150] Among them, y i The mapping curve representing the first sub-histogram, γ i β represents the gain curve coefficient. i α represents the weight of the equilibrium curve. i =1-β i y i_gain This represents the gain curve of the first sub-histogram. j∈[0,m],y i_hist This represents the equilibrium curve of the first sub-histogram.

[0151] The apparatus of this embodiment can be used to perform Figure 3 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0152] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0153] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0159] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image enhancement processing method characterized by, include: Obtain the histogram of the image to be processed; The histogram is divided into n sub-histograms, where n is a positive integer; Calculate the mapping curve of each of the n sub-histograms; The mapping curve of the histogram is obtained by synthesizing the mapping curves of the n sub-histograms; The image to be processed is enhanced based on the mapping curve of the histogram; The step of calculating the mapping curve of each of the n sub-histograms includes: Extract multiple features from each sub-histogram; The gain curve coefficients and equalization curve weights of each sub-histogram are calculated based on the aforementioned features. The mapping curve for each sub-histogram is determined based on the gain curve coefficients and the equalization curve weights.

2. The method according to claim 1, characterized in that, The plurality of features includes one or more of the following: the mean brightness of the sub-histogram, the standard deviation of the brightness of the sub-histogram, and the pixel percentage of the sub-histogram, wherein the pixel percentage of the sub-histogram is the proportion of the number of pixels included in the sub-histogram to the total number of pixels included in the histogram.

3. The method according to claim 1, characterized in that, The calculation of the gain curve coefficients and equalization curve weights of each sub-histogram based on the multiple features includes: The multiple features are input into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients. The first histogram enhancement coefficient generator is used to calculate the gain curve coefficients. The multiple features are input into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights. The second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

4. The method according to claim 1, characterized in that, The calculation of the gain curve coefficients and equalization curve weights of each sub-histogram based on the multiple features includes: The multiple features and the multiple features of adjacent sub-histograms are all input into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients. The first histogram enhancement coefficient generator is used to calculate the gain curve coefficients. The multiple features and the multiple features of adjacent sub-histograms are all input into a pre-trained second histogram enhancement coefficient generator to obtain the balance curve weights. The second histogram enhancement coefficient generator is used to calculate the balance curve weights.

5. The method according to claim 3 or 4, characterized in that, The first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator include an adaptive neural fuzzy system or a neural network.

6. The method according to any one of claims 1-5, characterized in that, Each of the n sub-histograms is a histogram after low-pass filtering.

7. The method according to any one of claims 1-6, characterized in that, Before dividing the histogram into n sub-histograms, the method further includes: The histogram is subjected to brightness transformation processing; The step of dividing the histogram into n sub-histograms includes: Divide the histogram after brightness transformation into n sub-histograms.

8. The method according to any one of claims 1-6, characterized in that, After dividing the histogram into n sub-histograms, the method further includes: Brightness transformation processing is performed on each of the n sub-histograms.

9. The method according to any one of claims 1-8, characterized in that, The process of synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram includes: The mapping curves of the n sub-histograms are spliced ​​together to obtain the mapping curve of the histogram. The values ​​at the connection point of the mapping curves of two adjacent sub-histograms are taken as the average, maximum or minimum values. The average, maximum or minimum values ​​are the average, maximum or minimum values ​​of the two values ​​at the connection point of the mapping curves of the two adjacent sub-histograms.

10. An image enhancement processing apparatus, characterized in that, include: The acquisition module is used to acquire the histogram of the image to be processed; The processing module is used to divide the histogram into n sub-histograms, where n is a positive integer; Calculate the mapping curve of each of the n sub-histograms; The mapping curves of the n sub-histograms are synthesized to obtain the mapping curve of the histogram; the image to be processed is enhanced according to the mapping curve of the histogram; The processing module is specifically used to extract multiple features of each sub-histogram; calculate the gain curve coefficient and equalization curve weight of each sub-histogram based on the multiple features; and determine the mapping curve of each sub-histogram based on the gain curve coefficient and the equalization curve weight.

11. The apparatus according to claim 10, characterized in that, The plurality of features includes one or more of the following: the mean brightness of the sub-histogram, the standard deviation of the brightness of the sub-histogram, and the pixel percentage of the sub-histogram, wherein the pixel percentage of the sub-histogram is the proportion of the number of pixels included in the sub-histogram to the total number of pixels included in the histogram.

12. The apparatus according to claim 10, characterized in that, The processing module is specifically used to obtain the gain curve coefficients from a first histogram enhancement coefficient generator pre-trained from the multiple feature inputs, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to obtain the equilibrium curve weights from a second histogram enhancement coefficient generator pre-trained from the multiple feature inputs, and the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

13. The apparatus according to claim 10, characterized in that, The processing module is specifically used to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficients, wherein the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; and to input the plurality of features and the plurality of features of adjacent sub-histograms into a pre-trained second histogram enhancement coefficient generator to obtain the equilibrium curve weights, wherein the second histogram enhancement coefficient generator is used to calculate the equilibrium curve weights.

14. The apparatus according to claim 12 or 13, characterized in that, The first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator include an adaptive neural fuzzy system or a neural network.

15. The apparatus according to any one of claims 10-14, characterized in that, Each of the n sub-histograms is a histogram after low-pass filtering.

16. The apparatus according to any one of claims 10-15, characterized in that, The processing module is further configured to perform brightness transformation processing on the histogram; and divide the brightness-transformed histogram into n sub-histograms.

17. The apparatus according to any one of claims 10-15, characterized in that, The processing module is also used to perform brightness transformation processing on each of the n sub-histograms.

18. The apparatus according to any one of claims 10-17, characterized in that, The processing module is specifically used to splice the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram. The value at the connection point of the mapping curves of two adjacent sub-histograms is taken as the average, maximum or minimum value. The average, maximum or minimum value is the average, maximum or minimum value of the two values ​​of the mapping curves of the two adjacent sub-histograms at the connection point of each other.

19. An image enhancement processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, perform the method of any one of claims 1-9.

Images