An image processing method, apparatus and system

Abstract

Embodiments of the present application disclose an image processing method, device and system. The image capturing is completed by combining an image acquisition device and an image generation device. The image acquisition device acquires multiple frames of original image data with different exposures, performs Bayer domain processing on each original image data respectively to obtain first intermediate image data corresponding to each original image data, performs reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates a captured image according to the second intermediate image data and the transformation parameters. Thus, the calculation amount of the image acquisition device can be reduced, and the effect of the captured image can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus and system. Background Technology

[0002] The limited size of the image sensors that can be integrated into small shooting devices restricts the original image quality, creating a strong demand for image quality enhancement algorithms. At the same time, as low-power portable products, these devices require algorithms with lightweight characteristics to fit the compact hardware structure.

[0003] Currently, existing technologies mainly employ two high dynamic range (HDR) algorithms: YUV (YUV color space) domain HDR algorithms and RAW (raw image format) domain HDR algorithms. YUV domain algorithms are typically deployed on mobile devices or in the cloud, lacking integration with the shooting terminal, resulting in significant deficiencies in areas such as motion blur and color reproduction. While RAW domain algorithms can achieve superior results in the linear domain, they require extensive computation on the shooting terminal. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an image processing method, apparatus and system that can reduce the computational load of image acquisition devices and improve the quality of captured images.

[0005] In a first aspect, embodiments of the present invention provide an image processing method, the method comprising: The image sensor acquires multiple frames of raw image data with different exposures. Each of the original image data is processed using the Bayer domain to obtain the first intermediate image data corresponding to each original image data. A reversible transformation operation is performed on each of the first intermediate image data to obtain the corresponding second intermediate image data. The reversible transformation operation includes at least gamma transformation and global tone mapping. Obtain transformation parameters, including gamma curves and global tone mapping curves; The second intermediate image data and the transformation parameters are sent to the image generation device so that the image generation device generates a captured image based on the second intermediate image data and the transformation parameters.

[0006] In some embodiments, the original image data is RAW data.

[0007] In some embodiments, the Bayer domain processing includes one or more of black level correction, bad pixel correction, lens shading correction, white balance gain, and de-mosaic processing.

[0008] In some embodiments, the first intermediate image data is linear RGB image data or linear YUV image data.

[0009] In some embodiments, acquiring multiple frames of raw image data with different exposures via an image sensor includes: Acquire automatic exposure information, which includes one or more of automatic exposure histogram, exposure parameters, and automatic dynamic range control parameters; The exposure parameters and total number of frames of the original image data with different exposures for multiple frames are determined based on the automatic exposure information. Based on the exposure parameters and total number of frames of the original image data with different exposures, the image sensor is controlled to acquire original image data.

[0010] In some embodiments, the second intermediate image data is sRGB image data or compressed YUV image data.

[0011] Secondly, embodiments of the present invention provide an image processing method, the method comprising: The system receives second intermediate image data and transformation parameters from each original image data sent by the image acquisition device. The transformation parameters include gamma curves and global tone mapping curves. The second intermediate image data is inversely transformed according to the transformation parameters to obtain the corresponding third intermediate image data. The third intermediate image data are fused together to obtain the fourth intermediate image data; The captured image is generated based on the fourth intermediate image data.

[0012] In some embodiments, the second intermediate image data is sRGB image data or compressed YUV image data.

[0013] In some embodiments, the third intermediate image data is linear RGB image data.

[0014] In some embodiments, generating the captured image based on the fourth intermediate image data includes: The fourth intermediate image data is subjected to dynamic range control transformation, local tone mapping, and local contrast enhancement to obtain the fifth intermediate image data; Gamma transform and global tone mapping are performed on the fifth intermediate image data to obtain the sixth intermediate image data; The captured image is generated based on the sixth intermediate image data.

[0015] Thirdly, embodiments of the present invention provide an image processing system, the system comprising: An image acquisition device is used to acquire multiple frames of raw image data with different exposures through an image sensor, perform Bayer domain processing on each of the raw image data to obtain first intermediate image data corresponding to each raw image data, perform a reversible transformation operation on each of the first intermediate images to obtain corresponding second intermediate image data, the reversible transformation operation includes at least gamma transformation and global tone mapping, obtain transformation parameters, the transformation parameters include gamma curve and global tone mapping curve, and send the second intermediate image data and the transformation parameters to an image generation device. An image generation device is configured to receive second intermediate image data and transformation parameters of each original image data sent by the image acquisition device, perform inverse transformation on each second intermediate image data according to the transformation parameters to obtain corresponding third intermediate image data, fuse each third intermediate image data to obtain fourth intermediate image data, and generate the captured image according to the fourth intermediate image data.

[0016] Fourthly, embodiments of the present invention provide an image processing apparatus, the apparatus comprising: The raw image data acquisition unit is used to acquire multiple frames of raw image data with different exposures through an image sensor; The first intermediate image data acquisition unit is used to perform Bayer domain processing on each of the original image data to obtain the first intermediate image data corresponding to each original image data. The second intermediate image data acquisition unit is used to perform reversible transformation operations on each of the first intermediate images to obtain the corresponding second intermediate image data. The reversible transformation operations include at least gamma transformation and global tone mapping. A transformation parameter acquisition unit is used to acquire transformation parameters, including gamma curves and global tone mapping curves. The data sending unit is used to send the second intermediate image data and the transformation parameters to the image generating device, so that the image generating device generates a captured image based on the second intermediate image data and the transformation parameters.

[0017] Fifthly, embodiments of the present invention provide an image processing method, the method comprising: The data receiving unit is used to receive second intermediate image data and transformation parameters of each original image data sent by the image acquisition device. The transformation parameters include gamma curves and global tone mapping curves. The third intermediate image data acquisition unit is used to perform an inverse transformation on each of the second intermediate image data according to the transformation parameters to obtain the corresponding third intermediate image data. The fourth intermediate image data acquisition unit is used to fuse each of the third intermediate image data to obtain the fourth intermediate image data; An image generation unit is used to generate the captured image based on the fourth intermediate image data.

[0018] In a sixth aspect, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the methods as described in the first and second aspects.

[0019] The technical solution of this invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image. Attached Figure Description

[0020] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of the image processing system according to an embodiment of the present invention; Figure 2 This is a flowchart of an image processing method for an image acquisition device according to an embodiment of the present invention; Figure 3 This is a flowchart of the process of acquiring raw image data according to an embodiment of the present invention; Figure 4 This is a flowchart of an image processing method using an image generation device according to an embodiment of the present invention; Figure 5 This is a flowchart of the process of acquiring captured images according to an embodiment of the present invention; Figure 6 This is a flowchart of an image processing method according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the image processing device of the image acquisition device according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the image processing apparatus of the image generation device according to an embodiment of the present invention; Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0022] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0023] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0024] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0025] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0026] The following explains the terms and techniques involved in the embodiments of the present invention: RAW: Raw image data refers to the unprocessed digital signal directly output by the image sensor. RAW data retains the most original photoelectric conversion information of the scene, has the highest dynamic range and color depth, and is the starting point for ISP processing. It is usually stored in Bayer format (arranged as RGGB / BGGR, etc.).

[0027] RGB: Red, Green, and Blue color model, which represents colors through different intensities of the three channels: Red, Green, and Blue. RGB is the standard color space for display devices and directly corresponds to the colors perceived by the human eye.

[0028] Linear RGB: This refers to RGB data that has a linear relationship with light intensity. After processing RAW data such as depigmentation and white balance, the resulting RGB values ​​are proportional to the actual brightness of the scene. Linear RGB is not suitable for direct display (the human eye is more sensitive to dark areas) and needs to be converted to a non-linear space through gamma transformation.

[0029] sRGB: Standard RGB color space, which uses a non-linear gamma curve, is suitable for direct display on monitors and is also the standard output format for the Internet and digital photography.

[0030] YUV: A color coding system that separates luminance and chrominance. Y represents luminance (Luma), and U and V represent chrominance (Chroma, i.e., color difference).

[0031] Linear YUV: A YUV format in which the Y component has a linear relationship with light intensity. It is typically used in HDR (High Dynamic Range) processing to preserve more brightness details, facilitating subsequent tone mapping and compositing.

[0032] Compressed YUV: A YUV format that has undergone chroma downsampling, reducing the amount of data by lowering the chroma resolution. For example: YUV444: Uncompressed, full-resolution chroma; YUV422: Horizontal chromaticity halved (2:1 compression); YUV420: Both horizontal and vertical chromaticity are halved (4:1 compression).

[0033] ISP (Image Signal Processor): A hardware / software system used to process the output signal of an image sensor. The ISP converts RAW data into a usable image or video stream. An ISP typically consists of two main processing sections: BPS and IPE.

[0034] BPS (Bayer Processing Segment): Responsible for early processing of RAW Bayer format data. Before de-mosaicing, BPS performs black level correction, bad pixel correction, lens shading correction, white balance gain, and noise reduction (temporal / spatial domain). The data processed by BPS is still in Bayer format, maintaining the original color filter array structure.

[0035] IPE (Image Processing Engine): Responsible for RGB / YUV domain processing after demosaicing. IPE receives the demosaiced image output by BPS and performs color space conversion (RGB-YUV), tone mapping (GTM), sharpening, noise reduction (chroma / luminance noise reduction), gamma transformation, 3D-LUT color adjustment, etc.

[0036] Black Level Correction (BLC): Sensors still output non-zero signals (dark current) in the dark. Black Level Correction eliminates this offset by subtracting the dark field reference value, ensuring that the black area of ​​the image is truly zero and restoring the correct dynamic range.

[0037] Defect Pixel Correction (DPC): Sensors may have defective pixels (pixels that are always bright or dead). DPC detects statistical anomalies in neighboring pixels and replaces defective pixels with interpolated surrounding valid pixels to avoid bright or dark spots in the image.

[0038] Lens Shading Correction (LSC): Due to the optical characteristics of lenses, there is unevenness in brightness and color between the center and edges of an image (bright center, dark edges, and possible color deviation). LSC eliminates this unevenness through pixel-by-pixel gain compensation, typically using a pre-calibrated gain table for correction.

[0039] White Balance Gain: Different light sources have different color temperatures, causing white objects to appear off-white. White balance adjusts the gain of the R, G, and B channels (usually using the G channel as the baseline, adjusting R and B) to ensure that white objects appear neutral white under any light source, thus ensuring accurate color reproduction.

[0040] Demosaicing: Bayer sensors capture only one color (R / G / B) per pixel. Demosaicing reconstructs the two missing colors for each pixel using interpolation algorithms, generating a complete RGB image. Common algorithms include bilinear interpolation, edge-aware interpolation, and adaptive gradient.

[0041] Noise Reduction: Image sensors contain various noise sources (photon shot noise, readout noise, thermal noise, etc.). Noise reduction involves multiple stages: RAW domain noise reduction: Processing on Bayer data to preserve details; YUV domain noise reduction: Separate processing of luminance and chrominance noise; Temporal noise reduction: Utilizing multi-frame information to reduce random noise; Spatial noise reduction: Smoothing noise by utilizing spatial neighborhood information.

[0042] Gamma Correction: The human eye is more sensitive to changes in brightness in dark areas, while sensor output is linear. Gamma correction uses a nonlinear power function to compress highlights and expand shadows, making the encoded image more consistent with human visual perception while adapting to the nonlinear response of display devices.

[0043] GTM (Global Tone Mapping): Used for high dynamic range (HDR) image processing, it compresses linear data with a wide dynamic range to the limited dynamic range that the display device can render (typically 8-bit sRGB). GTM adjusts the overall contrast and brightness distribution through global curves (such as S-curves and logarithmic curves), preserving details while avoiding overexposure or underexposure.

[0044] Sharpening: Enhances image edge contrast and improves visual clarity. It is typically achieved through high-pass filtering or Unsharp Masking (USM) algorithms to enhance the steepness of edge transitions.

[0045] Photography has become a core application of various smart devices. From smartphones to professional imaging equipment, continuous improvement in image quality has always been the main theme of technological evolution. With the continuous development of consumer electronics, the form of photography devices is evolving from handheld to wearable devices. Wearable devices, due to their ability to free up hands and enable real-time recording from a first-person perspective, are gradually becoming an important new carrier of imaging technology. Among many wearable devices, smart glasses, with their first-person shooting capabilities, provide users with an excellent recording and interactive experience. Therefore, photography is widely recognized as one of the core and most frequently used application scenarios for smart glasses. However, the compact form and size requirements of smart glasses severely limit the size of the image sensors that can be integrated within them. Given the difficulty in overcoming physical optical bottlenecks, the limitation of sensor size directly leads to inherent defects such as low signal-to-noise ratio and insufficient dynamic range in the original images, making the image quality of smart glasses a key bottleneck restricting user experience and product implementation. Therefore, developing efficient and high-quality image quality enhancement algorithms, especially high dynamic range (HDR) imaging algorithms, to address the hardware limitations of smart glasses has become a key investment focus and a core direction for technological breakthroughs in product development in this field.

[0046] Currently, the industry's efforts to improve image dynamic range primarily rely on traditional HDR algorithms, including YUV domain HDR and RAW domain HDR algorithms. However, these existing solutions all exhibit significant limitations when applied to smart glasses. Specifically, traditional YUV domain HDR algorithms are typically deployed on mobile devices or in the cloud, lacking deep integration between their processing flow and the front-end shooting terminal. Because YUV domain algorithms convert the image from the linear RAW domain to the non-linear YUV domain during processing, early tone mapping and color synthesis operations inevitably result in image information loss. This leads to severe ghosting artifacts when processing moving scenes and significant limitations in color reproduction and expressiveness, making it difficult to meet the high-quality imaging requirements of smart glasses.

[0047] Theoretically, RAW domain HDR algorithms can complete all processing steps, including multi-frame fusion, ghosting removal, and tone mapping, within the original linear domain, thus achieving theoretically superior image quality. However, this approach requires massive data computation at the front-end shooting terminal (i.e., the glasses). For portable devices like smart glasses, which are extremely sensitive to power consumption, heat generation, and response latency, the high-intensity RAW domain algorithm computation will lead to a sharp increase in terminal power consumption and a significant increase in shooting time, severely impacting the device's basic battery life and user experience, making it virtually impossible to implement in actual product development. Therefore, this invention provides an image processing method and a solution to the above problems.

[0048] Figure 1 This is a schematic diagram of an image processing system according to an embodiment of the present invention. Figure 1 As shown, the image processing system of this embodiment includes an image acquisition device 1 and an image generation device 2.

[0049] The image acquisition device 1 includes an image sensor for acquiring raw image data. After partially processing the raw image data, the sensor sends it to the image generation device 2. The image generation device 2 further processes the received image data to generate the final captured image.

[0050] Image acquisition device 1 can be any device with an image sensor, such as wearable devices (smart glasses, smart helmets, smartwatches, smart bracelets), mobile phones, and dashcams. This embodiment of the invention uses smart glasses as an example for illustration.

[0051] Specifically, the image acquisition device 1 acquires multiple frames of raw image data with different exposures through an image sensor, performs Bayer domain processing on each of the raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each of the first intermediate images to obtain corresponding second intermediate image data, the reversible transformation operation includes at least gamma transformation and global tone mapping, obtains transformation parameters, the transformation parameters include gamma curve and global tone mapping curve, and sends the second intermediate image data and the transformation parameters to the image generation device.

[0052] The image generating device 2 can be any device with data processing capabilities, such as a mobile phone, laptop, desktop computer, tablet computer, or server. This embodiment of the invention uses a mobile phone or server as an example for illustration. The image generating device 2 can communicate with the image acquisition device 1 to exchange data. The communication connection method can be set according to actual needs. For example, when the image generating device 2 is a mobile phone, it can connect to the image acquisition device 1 via Bluetooth or other communication methods; when the image generating device 2 is a server, it can connect to the image acquisition device 1 via wireless LAN, mobile communication network, or other communication methods, or it can use a mobile phone as an intermediary device for relay connection.

[0053] The image generation device 2 is used to receive second intermediate image data and transformation parameters of each original image data sent by the image acquisition device, perform inverse transformation on each second intermediate image data according to the transformation parameters to obtain corresponding third intermediate image data, fuse each third intermediate image data to obtain fourth intermediate image data, and generate the captured image according to the fourth intermediate image data.

[0054] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0055] Figure 2 This is a flowchart of an image processing method using an image acquisition device according to an embodiment of the present invention. Figure 2 As shown, the image processing method of this embodiment of the invention is executed by an image acquisition device, and specifically includes the following steps: Step S110: Acquire multiple frames of raw image data with different exposures using an image sensor.

[0056] In this embodiment, the image sensor is controlled by AE (Auto Exposure) to acquire multiple frames of raw image data with different exposures. The raw image data is RAW data.

[0057] Specifically, Figure 3 This is a flowchart illustrating the acquisition of raw image data according to an embodiment of the present invention. Figure 3As shown, acquiring multiple frames of raw image data with different exposures using an image sensor includes the following steps: Step S111: Obtain automatic exposure information.

[0058] In this embodiment, automatic exposure information is obtained through AE (Auto Exposure), which includes one or more of the following: automatic exposure histogram, exposure parameters, and automatic dynamic range control parameters.

[0059] Auto Exposure (AE) is a technique that allows a camera or image processing system to automatically adjust exposure parameters based on ambient light to obtain an image with suitable brightness. AE systems typically follow a closed-loop feedback control process, which can be summarized as follows: First, the brightness information of the current image frame is analyzed, and the current brightness is compared with a preset target brightness. If the current image is too dark, the system automatically increases the exposure; if it is too bright, the exposure is decreased. After adjustment, the system re-evaluates the brightness of the new image and repeats the above process until the image brightness reaches the ideal state. The AE system can be deployed in an image generation device, and the image acquisition device obtains the automatic exposure information from the image generation device.

[0060] The automatic exposure histogram (AE) is a statistical chart used by after-effects (AE) algorithms to evaluate the brightness distribution of an image. It shows the brightness distribution of all pixels in an image. The horizontal axis represents the brightness level of a pixel, from 0 (pure black) to 255 (pure white), and the vertical axis represents the number of pixels at a certain brightness level. By observing the histogram, the AE algorithm can determine the image's exposure status. If underexposed, the peak of the AE histogram is concentrated on the left (dark areas); if overexposed, the peak is concentrated on the right (bright areas); if well-exposed, the peak is distributed in the middle region. The AE algorithm uses histogram information to more intelligently adjust exposure, rather than simply calculating the average brightness of the entire image, thus avoiding being misled by large areas of black or white objects.

[0061] Exposure parameters are variables that are directly controlled and adjusted by the AE algorithm, and they determine the brightness of the final image. Exposure parameters mainly include shutter speed, aperture, and ISO.

[0062] Shutter speed refers to the length of time an image sensor is exposed to light. A slower shutter speed (longer exposure time) allows more light to enter, resulting in a brighter image, but may cause moving objects to blur. Conversely, a faster shutter speed can freeze motion, but the image will be darker.

[0063] Aperture refers to the size of the opening in a lens that controls the amount of light entering, usually expressed as an f-number. The smaller the f-number, the larger the aperture, the more light enters per unit time, and the brighter the image.

[0064] ISO / Gain refers to the sensitivity of an image sensor to light or the signal amplification factor. The higher the ISO value, the more sensitive the sensor is to light, and the brighter the image. However, the side effect is that it introduces more image noise, making the image grainy.

[0065] Automatic Dynamic Range Control (ADRC) is an image processing technique designed to solve exposure challenges in high-contrast scenes (such as backlit scenes). When a scene contains both very bright and very dark areas, traditional after-effect (AE) may fail to capture both. ADRC compresses the dynamic range of the entire scene by adjusting target values ​​for different brightness areas, ensuring that details in both bright and dark areas are preserved. ADRC parameters include target brightness values ​​for dark areas (brightlow target) and target brightness values ​​for bright areas (bright high target). Specifically, the ADRC algorithm attempts to increase the brightness of dark areas to near the bright low target to enhance shadow details and prevent them from becoming completely black, and attempts to suppress or preserve the brightness of bright areas to prevent them from exceeding the bright high target, thus avoiding overexposure of bright areas and preserving highlight details.

[0066] Step S112: Determine the exposure parameters and total number of frames of the original image data with different exposures for multiple frames based on the automatic exposure information.

[0067] In this embodiment, the exposure parameters and total number of frames of original image data with different exposures for multiple frames are determined based on the automatic exposure information.

[0068] Specifically, the total number of frames can be determined based on the complexity of the scene and the distribution width of the automatic exposure histogram. For example, in a normal scene, the total number of frames is determined to be 3 frames; in a high dynamic range or night scene, the total number of frames can be determined to be 4 frames or more; and in a fast-moving scene, the total number of frames can be determined to be 2 frames.

[0069] To determine the exposure parameters for multiple frames of raw image data with different exposures, a level-based approach can be used. Specifically, a baseline exposure EV0 is first calculated based on the auto-exposure histogram. Then, exposure values ​​for other levels are calculated based on EV0. Changes in exposure values ​​can be achieved by adjusting the integration time or analog gain. For example, levels can be categorized as follows: Normal exposure (EV0) = 1 × EV0; Under-explosion (EV-) = 0.5 × EV0; Overexposure (EV+) = 2 × EV0; Severe underexposure (EV--) = 0.25 × EV0; Severe overexposure (EV++) = 4 × EV0.

[0070] The embodiments of the present invention can determine the exposure parameters based on a preset method. For example, assuming a total of 3 frames, any three levels can be selected from "EV+", "EV0", "EV-" and "EV--" as the exposure parameters.

[0071] Step S113: Control the image sensor to acquire raw image data according to the exposure parameters and total number of frames of the raw image data with different exposures of the multiple frames.

[0072] In this embodiment, based on the exposure parameters and the total number of frames, the image sensor is controlled to capture multiple frames of raw image data with different exposures.

[0073] Step S120: Perform Bayer domain processing on each of the original image data to obtain the first intermediate image data corresponding to each original image data.

[0074] In this embodiment, Bayer domain processing is performed on each of the original image data to obtain first intermediate image data corresponding to each original image data. The first intermediate image data is either linear RGB image data or linear YUV image data.

[0075] The Bayer domain processing includes one or more of the following: black level correction, bad pixel correction, lens shading correction, white balance gain, and de-mosaic processing. Specifically, the image processing flow of the image acquisition device in this embodiment of the invention is executed by an ISP (Image Signal Processor). That is, the image acquisition device includes an ISP, which includes a BPS (Bayer Processing Segment) and an IPE (Image Processing Engine). The Bayer domain processing of each of the original image data is implemented by the BPS.

[0076] Specifically, when the first intermediate image data is linear RGB image data, the Bayer domain processing of each of the original image data using BPS includes the following steps: Performing black level correction (BLC) on the original image data can reduce the dark current noise of the sensor, making true black zero.

[0077] The original image data is subjected to Defect Pixel Correction (DPC) to repair dead or hot spots on the sensor and prevent abnormal noise from being generated during fusion.

[0078] Lens Shading Correction (LSC) is performed on the original image data to compensate for the physical vignetting where the center of the lens is bright and the edges are dark, ensuring consistent edge brightness during multi-frame fusion.

[0079] The original image data is subjected to white balance gain (AWB Gain). Based on the color temperature, the R, G, and B channels are multiplied by gain coefficients respectively to make white objects appear white.

[0080] The original image data is de-mosaiced to convert the single-channel data into linear RGB image data, which is also the first intermediate image data.

[0081] In other words, the BPS receives raw image data from the image sensor, performs black level correction, bad pixel correction, lens shading correction, white balance, etc. on the raw image data, and finally performs de-mosaic to translate the single-channel data into full-color linear RGB image data.

[0082] Specifically, when the first intermediate image data is linear YUV image data, linear RGB image data is first obtained through the BPS execution flow described above. Then, a linear transformation is performed on the linear RGB image data to convert it into the YUV color space, resulting in linear YUV image data, which is the first intermediate image data. This linear transformation can be achieved by multiplying the linear RGB image data by a predetermined linear transformation matrix.

[0083] Step S130: Perform a reversible transformation operation on each of the first intermediate image data to obtain the corresponding second intermediate image data.

[0084] In this embodiment, after obtaining the first intermediate image data, the first intermediate image data is processed by IPE to obtain the corresponding second intermediate image data. The reversible transformation operation includes at least gamma transformation and global tone mapping.

[0085] Specifically, under normal circumstances, IPE can perform functions such as CCM (Color Correction Matrix), AWB (Auto White Balance), GTM (Global Tone Mapping), LTM (Local Tone Mapping), Local Contrast Enhancement (LCE), 2D LUT (2-Dimensional Look-Up Table), 3D LUT (3-Dimensional Look-Up Table), Gamma Correction, Sharpening, and format conversion. In this embodiment of the invention, the irreversible transformation operations of these IPE operations are disabled, and only the reversible transformation operations are enabled.

[0086] Irreversible transformation operations include LTM, LCE, 2D LUT, 3D LUT, etc.

[0087] The reversible transformation operations include CCM, AWB, GTM, gamma transform, sharpening, format conversion, etc.

[0088] In other words, noise reduction, gamma transformation, GTM, sharpening and other operations are performed on each of the first intermediate image data to obtain the corresponding second intermediate image data.

[0089] Noise reduction is used to remove noise (color distortion, graininess) from the first intermediate image data. Sensors generate random noise during light sensing; noise reduction smooths out these random signals that are not inherent to the image itself, making the image appear cleaner. Noise reduction can be achieved using methods such as temporal domain noise reduction or spatial domain noise reduction.

[0090] Gamma transform is used to adjust the brightness curve of the first intermediate image data to conform to human visual perception or the display characteristics of the monitor. Specifically, gamma transform uses a gamma curve to transform the first intermediate image data. The gamma curve is a curve describing the non-linear relationship between the input signal and the output brightness, and it can be expressed by the following formula:

[0091] in, For input signal, For output signal, This is the gamma value.

[0092] in, This determines the curvature of the curve; if If, then the curve is a straight line; if The curve bends downwards, the dark areas are compressed, and the bright areas are expanded; if The curve bends upwards, expanding the dark areas and compressing the bright areas. Thus, by performing a specific exponential operation on the pixel values ​​of the first intermediate image data through gamma transformation, the grayscale distribution of the image is altered.

[0093] GTM uses a global tone mapping curve to map all pixels of the first intermediate image data. Specifically, the global tone mapping curve defines the correspondence between input luminance values ​​and output luminance values. For each pixel in the first intermediate image data, the corresponding output luminance value is found from the global tone mapping curve based on the input luminance value, and each pixel is adjusted according to the output luminance value.

[0094] Sharpening is used to enhance the edge sharpness of the first intermediate image data. It can extract the high-frequency components (edges) of the image and then overlay them back onto the original image, using visual illusion to make the edges appear sharper.

[0095] Therefore, after performing reversible transformation operations such as noise reduction, gamma transformation, GTM, and sharpening on each of the first intermediate image data, the corresponding second intermediate image data is obtained.

[0096] The second intermediate image data is either sRGB image data or compressed YUV image data. sRGB (standard Red Green Blue) is a color space standard. Linear RGB image data represents the true intensity of light in the physical world, while sRGB is a color space processed to suit human perception and efficient storage. Linear RGB image data undergoes a gamma transformation to obtain sRGB image data. Linear YUV image data refers to complete YUV data converted from linear RGB image data without compression. Compressed YUV image data refers to a YUV format that downsamples the chromaticity components of linear YUV image data to reduce the data size.

[0097] Meanwhile, as mentioned above, the first intermediate image data is either linear RGB image data or linear YUV image data. Therefore, converting the first intermediate image data into the second intermediate image data can actually be divided into four cases: converting linear RGB to sRGB, converting linear RGB to compressed YUV, converting linear YUV to sRGB, and converting linear YUV to compressed YUV. The common processes involved in these four conversion cases include noise reduction, gamma transformation, GTM, and sharpening.

[0098] The process for converting linear RGB to sRGB is as follows: First, the linear RGB image data is denoised; then, a gamma transform is performed to convert the linear RGB data into non-linear RGB data; next, a GTM is performed to adjust the brightness and contrast of the image; finally, sharpening is performed to enhance image details and output sRGB image data.

[0099] The process for converting linear RGB to compressed YUV is as follows: First, the linear RGB image data is denoised; then, a gamma transform is performed to convert the linear RGB data into non-linear RGB data; next, GTM is used to adjust the brightness and contrast of the image; then, the RGB data is converted to YUV format and the chromaticity components are compressed; finally, sharpening is performed to output compressed YUV image data.

[0100] The process for converting linear YUV to sRGB is as follows: First, the linear YUV image data is denoised; then, a gamma transform is performed to convert the linear YUV data into non-linear YUV data; next, a GTM is performed to adjust the dynamic range of the luminance component; then, the YUV data is converted to the RGB color space; finally, sharpening is performed to output sRGB image data.

[0101] The process for converting linear YUV to compressed YUV is as follows: First, the linear YUV image data is denoised; then, a gamma transform is performed to convert the linear YUV data into non-linear YUV data; next, a GTM is performed to adjust the dynamic range of the luminance component; then, the chrominance component is compressed; finally, sharpening is performed to output the compressed YUV image data.

[0102] Step S140: Obtain the transformation parameters.

[0103] In this embodiment, the transformation parameters of the above-mentioned reversible transformation are obtained, and the transformation parameters include the gamma curve and the global tone mapping curve.

[0104] Step S150: Send the second intermediate image data and the transformation parameters to the image generation device, so that the image generation device generates a captured image based on the second intermediate image data and the transformation parameters.

[0105] In this embodiment, the generated second intermediate image data and the transformation parameters are sent to the image generation device so that the image generation device generates a captured image based on the second intermediate image data and the transformation parameters.

[0106] Furthermore, sending the second intermediate image data to the image generating device includes the following steps: Step S151: Encode the second intermediate image data into compressed image data.

[0107] Step S152: Send the compressed image data to the image generation device.

[0108] Specifically, the second intermediate image data is encoded into compressed image data. The compressed image data can be in any existing format; this embodiment uses JPEG as an example. First, the second intermediate image data is divided into multiple n×n pixel blocks. A DCT (Discrete Cosine Transform) is performed on each block. The DCT transforms the image from the spatial domain to the frequency domain.

[0109] Finally, the compressed image data and transformation parameters are sent to the image generation device, which then performs the subsequent operations.

[0110] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0111] Figure 4 This is a flowchart of an image processing method using an image generation device according to an embodiment of the present invention. Figure 4 As shown, the image processing method of this embodiment of the invention is executed by an image generating device, and specifically includes the following steps: Step S210: Receive the second intermediate image data and transformation parameters of each original image data sent by the image acquisition device.

[0112] In this embodiment, the image generation device receives transformation parameters and second intermediate image data corresponding to each original image data sent by the image acquisition device. The transformation parameters include gamma curves and global tone mapping curves.

[0113] The second intermediate image data is either sRGB image data or compressed YUV image data.

[0114] More specifically, as described above, when the image acquisition device sends the second intermediate image data, it encodes the second intermediate image data into compressed image data before sending it. Thus, after receiving the compressed image data sent by the image acquisition device, the image generation device decodes the compressed image data to obtain sRGB image data or compressed YUV image data.

[0115] Step S220: Perform an inverse transformation on each of the second intermediate image data according to the transformation parameters to obtain the corresponding third intermediate image data.

[0116] In this embodiment, each second intermediate image data is inversely transformed according to the transformation parameters to obtain the corresponding third intermediate image data.

[0117] Specifically, as described above, the gamma transform uses a gamma curve to transform the first intermediate image data, and the GTM uses a global tone mapping curve to map all pixels of the first intermediate image data. Therefore, after obtaining the second intermediate image data, the gamma curve, and the global tone mapping curve, the second intermediate image data can be inversely transformed using the global tone mapping curve and the gamma curve to obtain the third intermediate image data. The third intermediate image data is linear RGB image data.

[0118] In one alternative implementation, the second intermediate image data can be converted into the third intermediate image data through two inverse transformations. Specifically, firstly, a first inverse transformation is performed on the second intermediate image data based on the global tone mapping curve, and then a second inverse transformation is performed on the image data after the first inverse transformation based on the gamma curve to obtain the third intermediate image data.

[0119] In another alternative implementation, the second intermediate image data can be converted into third intermediate image data through an inverse transform. Specifically, firstly, a fusion transform curve is calculated based on the global tone mapping curve and the gamma curve, and then the second intermediate image data is inversely transformed based on the fusion transform curve to obtain the third intermediate image data.

[0120] This invention does not limit the method of calculating the fusion transformation curve; it can be implemented based on various existing methods. In a specific implementation, it can be achieved in the following way: Let x be the pixel value of the input image; Let the gamma curve be denoted as the function f1(x); The global tone mapping curve is denoted as the function f2(x); The fusion transformation curve is denoted as function f3(x).

[0121] To ensure that the transformation effect of the blended transform curve is the same as that of the global tone mapping curve plus the gamma curve, the function f3(x) can be expressed as:

[0122] Furthermore, when the second intermediate image data is sRGB image data, the inverse transformation yields linear RGB image data, which is the third intermediate image data. When the second intermediate image data is compressed YUV image data, the compressed YUV image data is first upsampled or interpolated to restore the resolution of the U and V components to the same level as the Y component. Then, the restored image undergoes an inverse transformation to obtain image data in linear YUV format. Next, the linear YUV format is converted to linear RGB image data, which is the third intermediate image data. The process of converting the linear YUV format to linear RGB image data is the reverse of the process of converting the linear RGB format to linear YUV image data, and will not be described in detail here.

[0123] Step S230: Fuse the third intermediate image data to obtain the fourth intermediate image data.

[0124] In this embodiment, after obtaining the third intermediate image data corresponding to each original image data, DHR (High Dynamic Range) fusion is performed on each of the third intermediate image data to obtain the fourth intermediate image data.

[0125] HDR is a technology designed to enhance the brightness range and color performance of images, allowing the image to simultaneously display brighter highlights and darker shadow details, thus more closely resembling the visual effect seen by the human eye in the real world.

[0126] Specifically, fusing the third intermediate image data to obtain the fourth intermediate image data includes the following steps: Step S231: Align each of the third intermediate image data.

[0127] In this embodiment, image alignment is first performed. One of the third intermediate image data (e.g., a normal exposure image) is selected as the reference image. Then, the offset of the other third intermediate image data relative to the reference image is calculated. The pixel remapping technique is used to align each third intermediate image data to the same position to ensure the accuracy of subsequent fusion.

[0128] Step S232: Determine the weight value of each pixel in each of the third intermediate image data.

[0129] In this embodiment, after aligning the various third intermediate image data, the weight value of each pixel in each third intermediate image data is determined. Specifically, pixels with normal brightness and clear details are assigned a higher weight (close to 1) and dominate in the fusion process. Pixels that are overexposed or underexposed are assigned a very low weight (close to 0) and are basically discarded in the fusion process.

[0130] Step S233: Fuse each of the third intermediate image data according to the weight values.

[0131] In this embodiment, each pixel of each of the third intermediate image data is weighted and fused according to the weight value to obtain the fourth intermediate image data.

[0132] Step S240: Generate the captured image based on the fourth intermediate image data.

[0133] In this embodiment, after obtaining the fused fourth intermediate image data, the fourth intermediate image data is further processed to generate the captured image.

[0134] in, Figure 5 This is a flowchart illustrating the acquisition of captured images according to an embodiment of the present invention. For example... Figure 5 As shown, generating the captured image based on the fourth intermediate image data includes the following steps: Step S241: Perform dynamic range control transformation, local tone mapping, and local contrast enhancement processing on the fourth intermediate image data to obtain the fifth intermediate image data.

[0135] In this embodiment, the fourth intermediate image data is sequentially processed using Dynamic Range Control (DRC), Local Tone Mapping (LTM), and Local Contrast Enhancement (LCE) to obtain the fifth intermediate image data. The fifth intermediate image data is in RGB format. More specifically, the fifth intermediate image data is linear RGB image data.

[0136] The DRC transformation analyzes the brightness distribution of the entire image (e.g., by constructing a brightness histogram) to identify extremely dark shadows and extremely bright highlights. Then, a tone curve (TC) is constructed according to a pre-defined strategy, which determines how the input brightness is mapped to the output brightness. Finally, the DRC transformation is performed on the fourth intermediate image data based on the tone curve. This allows for the compression and adaptation of a wide range of real-world brightness to the narrow capabilities of a device, while preserving as much important visual detail as possible.

[0137] LTM dynamically determines how to adjust the brightness of each pixel based on its surrounding local environment (such as the brightness, edges, and texture of its neighbors). Specifically, the image is first divided into many small, potentially overlapping regions, or a neighborhood window is defined for each pixel. For each region or pixel, a corresponding gain value is calculated. For example, in a generally dark region, a gain greater than 1 is calculated to boost the brightness of that region, thus revealing details in the shadows. To avoid obvious boundaries or halo artifacts between adjacent regions, the LTM algorithm uses predefined filtering techniques (such as bilateral filtering and guided filtering) to ensure that the gain changes are smooth and natural. Finally, the brightness value of each pixel is processed by a mapping function jointly determined by itself and its surrounding pixels.

[0138] The principle behind LCE (Low-Frequency Image Conversion) is based on the assumption that an image can be decomposed into low-frequency and high-frequency components. The goal of LCE is to selectively enhance the high-frequency components. Specifically, firstly, algorithms such as Laplacian pyramids and wavelet transforms are used to decompose the image into high-frequency and low-frequency layers of different scales. A gain coefficient α is applied to the decomposed high-frequency layer. When α > 1, the high-frequency signal is amplified, making the edges and textures of the image sharper and clearer. When α < 1, the high-frequency signal is suppressed, which can smooth and reduce noise. The processed high-frequency layer is then recombine with the original low-frequency layer to obtain the final image.

[0139] Step S242: Perform gamma transformation and global tone mapping on the fifth intermediate image data to obtain the sixth intermediate image data.

[0140] In this embodiment, the fifth intermediate image data is subjected to gamma transformation and global tone mapping using gamma curves and global tone mapping curves to obtain the sixth intermediate image data. The gamma curves and global tone mapping curves used for the gamma transformation and global tone mapping processing are those in the transformation parameters sent by the image acquisition device. The sixth intermediate image data is in sRGB format.

[0141] Step S243: Generate the captured image based on the sixth intermediate image data.

[0142] In this embodiment, the sixth intermediate image data in RGB format is converted into image data in YUV format, and then the YUV format image data is encoded and compressed into the captured image.

[0143] In this embodiment of the invention, no specific implementation method is limited for converting RGB format image data to YUV format image data. For encoding and compressing YUV format image data into a captured image, the YUV format image data can first be downsampled to obtain compressed YUV format image data, and then the compressed YUV format image data can be encoded into a captured image of a specified format, such as JPEG format.

[0144] In summary, traditional RAW domain HDR algorithms process RAW data directly output from image sensors without deep processing. Because RAW data is linear, with pixel values ​​proportional to scene brightness and without non-linear processing such as gamma correction, multi-frame fusion calculations are more accurate and produce better results. However, due to the large volume of RAW data and its Bayer format, processing it demands very high processor computing power. Furthermore, processing Bayer format images requires specially designed alignment and fusion operations, increasing algorithm complexity and making it unsuitable for deployment in image acquisition equipment. Traditional YUV domain HDR algorithms, on the other hand, process YUV data that has already undergone preliminary ISP processing. However, the conversion from RAW to YUV is a lossy process, resulting in the loss of some original information and a lower dynamic range potential compared to the RAW domain. Therefore, this embodiment of the invention disables irreversible transformation operations in the ISP within the image acquisition device, performs a reversible transformation operation on the acquired raw image data to obtain second intermediate image data, and sends the second intermediate image data and transformation parameters together to the image generation device. On the image generation device side, an inverse transformation is performed on the second intermediate image data according to the transformation parameters to obtain linear RGB image data, and an HDR algorithm is performed based on the linear RGB image data. Thus, by adjusting the specific functions of the ISP, a reversible transformation from a compressed sRGB domain image to a linear RGB domain image is achieved. This migrates the linear HDR method, which typically only processes RAW domain images, to the YUV domain on the image generation device side, achieving HDR fusion ghosting performance and color advantages comparable to pure RAW algorithms, while reducing the amount of data that needs to be transmitted. Simultaneously, since the algorithm does not require processing in the RAW domain at all, the huge amount of data transmitted in RAW images is reduced, and since RAW domain processing is also unnecessary on the image acquisition device side, no additional power consumption of the image acquisition device is added.

[0145] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0146] Figure 6 This is a flowchart of an image processing method according to an embodiment of the present invention. Figure 6 The structure and image processing flow of the image acquisition device and image generation device are shown. The image acquisition device 1 includes an image sensor and an ISP, wherein the ISP includes a BPS and an IPE.

[0147] The image sensor acquires multiple frames of raw image data with different exposures. The raw image data is RAW data. Specifically, based on the platform's AE statistics, including AE histogram, exposure parameters, ADRC parameters, etc., the exposure parameters and total number of frames for the multiple images are calculated. Typically, it is any three levels from EV+, EV0, EV-, and EV--, but it can also be four or more levels. The image sensor is then instructed to capture frames as needed.

[0148] The raw image data is processed by the BPS to obtain first intermediate image data, which is linear RGB or linear YUV data. Specifically, multiple exposure frames acquired by the image sensor are sent to the platform BPS to complete all operations in the Bayer domain to generate linear RGB or linear YUV data.

[0149] The second intermediate image data is obtained by performing a reversible transformation operation on the first intermediate image data using IPE. The second intermediate image data is either sRGB or compressed YUV. Specifically, local operation modules such as LTM, LCE, and 2DLUT in the platform's IPE are turned off. The linear RGB or linear YUV obtained in the previous step is processed by IPE through noise reduction, Gamma transformation, GTM, sharpening, etc., to obtain sRGB or compressed YUV, which is then encoded into a compressed image.

[0150] The processed multi-exposure images, along with the GTM and Gamma curves used by the platform, are transmitted to image generation device 2.

[0151] The image processing device performs an inverse transformation on the received image based on the GTM curve and Gamma curve to obtain linear RGB image data.

[0152] A fused image is obtained by performing linear domain HDR fusion on linear RGB image data.

[0153] The fused image is subjected to DRC transformation, local tone mapping, and local contrast enhancement to generate LDR (Low Dynamic Range) linear RGB image data with more suitable local brightness, color, and contrast. After platform Gamma and GTM processing, the sRGB color gamut result image is obtained, and the final result, i.e., the captured image, is obtained through encoding and compression.

[0154] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0155] Figure 7 This is a schematic diagram of the image processing device of the image acquisition device according to an embodiment of the present invention. Figure 7 As shown, the image processing device of the image acquisition apparatus of this embodiment includes a raw image data acquisition unit 71, a first intermediate image data acquisition unit 72, a second intermediate image data acquisition unit 73, a transformation parameter acquisition unit 74, and a data transmission unit 75. The raw image data acquisition unit 71 is used to acquire multiple frames of raw image data with different exposures using an image sensor. The first intermediate image data acquisition unit 72 is used to perform Bayer domain processing on each of the raw image data to obtain first intermediate image data corresponding to each raw image data. The second intermediate image data acquisition unit 73 is used to perform a reversible transformation operation on each of the first intermediate images to obtain corresponding second intermediate image data; the reversible transformation operation includes at least gamma transformation and global tone mapping. The transformation parameter acquisition unit 74 is used to acquire transformation parameters, including a gamma curve and a global tone mapping curve. The data transmission unit 75 is used to send the second intermediate image data and the transformation parameters to an image generation device, so that the image generation device generates a captured image based on the second intermediate image data and the transformation parameters.

[0156] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0157] Figure 8 This is a schematic diagram of the image processing apparatus of the image generation device according to an embodiment of the present invention. Figure 8 As shown, the image processing apparatus of the image generation device in this embodiment of the invention includes a data receiving unit 81, a third intermediate image data acquisition unit 82, a fourth intermediate image data acquisition unit 83, and a captured image generation unit 84. The data receiving unit 81 receives second intermediate image data and transformation parameters from each original image data sent by the image acquisition device. The transformation parameters include a gamma curve and a global tone mapping curve. The third intermediate image data acquisition unit 83 performs an inverse transformation on each of the second intermediate image data according to the transformation parameters to obtain corresponding third intermediate image data. The fourth intermediate image data acquisition unit 83 fuses each of the third intermediate image data to obtain fourth intermediate image data. The captured image generation unit 84 generates the captured image based on the fourth intermediate image data.

[0158] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0159] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 9 includes an image acquisition device, an image generation device, etc. Figure 9As shown, the electronic device 9 includes at least one processor 91; a memory 92 communicatively connected to at least one processor 91; and a communication component 93 communicatively connected to a scanning device, the communication component 93 receiving and transmitting data under the control of the processor 91; wherein the memory 92 stores instructions executable by at least one processor 91, the instructions being executed by at least one processor 91 to implement the above-described image processing method.

[0160] Specifically, the electronic device includes: one or more processors 91 and a memory 92. Figure 9 Taking a processor 91 as an example, the processor 91 and the memory 92 can be connected via a bus or other means. Figure 9 Taking a bus connection as an example, memory 92, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 91 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 92, thereby realizing the above-mentioned image processing method.

[0161] The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, the memory 92 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 92 may optionally include memory remotely located relative to the processor 91, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] One or more modules are stored in memory 92 and, when executed by one or more processors 91, perform the image processing method in any of the above method embodiments.

[0163] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0164] This invention combines an image acquisition device and an image generation device to capture images. The image acquisition device acquires multiple frames of raw image data with different exposures, performs Bayer domain processing on each raw image data to obtain first intermediate image data corresponding to each raw image data, performs a reversible transformation operation on each first intermediate image to obtain corresponding second intermediate image data, and sends the second intermediate image data and transformation parameters to the image generation device. The image generation device generates the captured image based on the second intermediate image data and transformation parameters. This reduces the computational load of the image acquisition device and improves the quality of the captured image.

[0165] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0166] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0167] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The image sensor acquires multiple frames of raw image data with different exposures. Each of the original image data is processed using the Bayer domain to obtain the first intermediate image data corresponding to each original image data. A reversible transformation operation is performed on each of the first intermediate image data to obtain the corresponding second intermediate image data. The reversible transformation operation includes at least gamma transformation and global tone mapping. Obtain transformation parameters, including gamma curves and global tone mapping curves; The second intermediate image data and the transformation parameters are sent to the image generation device so that the image generation device generates a captured image based on the second intermediate image data and the transformation parameters.

2. The method according to claim 1, characterized in that, The original image data is RAW data.

3. The method according to claim 1, characterized in that, The Bayer domain processing includes one or more of the following: black level correction, bad pixel correction, lens shading correction, white balance gain, and de-mosaic processing.

4. The method according to claim 1, characterized in that, The first intermediate image data is linear RGB image data or linear YUV image data.

5. The method according to claim 1, characterized in that, The acquisition of multiple frames of raw image data with different exposures via an image sensor includes: Acquire automatic exposure information, which includes one or more of automatic exposure histogram, exposure parameters, and automatic dynamic range control parameters; The exposure parameters and total number of frames of the original image data with different exposures for multiple frames are determined based on the automatic exposure information. Based on the exposure parameters and total number of frames of the original image data with different exposures, the image sensor is controlled to acquire original image data.

6. The method according to claim 1, characterized in that, The second intermediate image data is sRGB image data or compressed YUV image data.

7. An image processing method, characterized in that, The method includes: The system receives second intermediate image data and transformation parameters from each original image data sent by the image acquisition device. The transformation parameters include gamma curves and global tone mapping curves. The second intermediate image data is inversely transformed according to the transformation parameters to obtain the corresponding third intermediate image data. The third intermediate image data are fused together to obtain the fourth intermediate image data; The captured image is generated based on the fourth intermediate image data.

8. The method according to claim 7, characterized in that, The second intermediate image data is sRGB image data or compressed YUV image data.

9. The method according to claim 7, characterized in that, The third intermediate image data is linear RGB image data.

10. The method according to claim 7, characterized in that, The step of generating the captured image based on the fourth intermediate image data includes: The fourth intermediate image data is subjected to dynamic range control transformation, local tone mapping, and local contrast enhancement to obtain the fifth intermediate image data; Gamma transform and global tone mapping are performed on the fifth intermediate image data to obtain the sixth intermediate image data; The captured image is generated based on the sixth intermediate image data.

11. An image processing system, characterized in that, The system includes: An image acquisition device is used to acquire multiple frames of raw image data with different exposures through an image sensor, perform Bayer domain processing on each of the raw image data to obtain first intermediate image data corresponding to each raw image data, perform a reversible transformation operation on each of the first intermediate images to obtain corresponding second intermediate image data, the reversible transformation operation includes at least gamma transformation and global tone mapping, obtain transformation parameters, the transformation parameters include gamma curve and global tone mapping curve, and send the second intermediate image data and the transformation parameters to an image generation device. An image generation device is configured to receive second intermediate image data and transformation parameters of each original image data sent by the image acquisition device, perform inverse transformation on each second intermediate image data according to the transformation parameters to obtain corresponding third intermediate image data, fuse each third intermediate image data to obtain fourth intermediate image data, and generate a captured image based on the fourth intermediate image data.

12. An image processing apparatus, characterized in that, The device includes: The raw image data acquisition unit is used to acquire multiple frames of raw image data with different exposures through an image sensor; The first intermediate image data acquisition unit is used to perform Bayer domain processing on each of the original image data to obtain the first intermediate image data corresponding to each original image data. The second intermediate image data acquisition unit is used to perform reversible transformation operations on each of the first intermediate images to obtain the corresponding second intermediate image data. The reversible transformation operations include at least gamma transformation and global tone mapping. A transformation parameter acquisition unit is used to acquire transformation parameters, including gamma curves and global tone mapping curves. The data sending unit is used to send the second intermediate image data and the transformation parameters to the image generating device, so that the image generating device generates a captured image based on the second intermediate image data and the transformation parameters.

13. An image processing method, characterized in that, The method includes: The data receiving unit is used to receive second intermediate image data and transformation parameters of each original image data sent by the image acquisition device. The transformation parameters include gamma curves and global tone mapping curves. The third intermediate image data acquisition unit is used to perform an inverse transformation on each of the second intermediate image data according to the transformation parameters to obtain the corresponding third intermediate image data. The fourth intermediate image data acquisition unit is used to fuse each of the third intermediate image data to obtain the fourth intermediate image data; An image generation unit is used to generate a captured image based on the fourth intermediate image data.

14. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.

Images