Image local tone mapping method and device based on image segmentation
Through the image segmentation and local tone mapping methods based on SAM model, the problem of insufficient image area division is solved, efficient tone mapping and contrast adjustment is achieved, and the color expression and visual effect of the image is improved.
Patent Information
- Application Number
- CN202410959771.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, the image area division is not accurate enough, resulting in block effects and loss of details.
Using an image segmentation method, the image is segmented through the SAM model, an accurate area mask is generated, and local tone mapping is performed according to the characteristics of the area.
It realizes a high degree of locality of tone mapping, improves the color expression and visual effects of the image, simplifies the prior knowledge requirements for semantic segmentation, and maintains the order of image contrast.
Smart Images
Figure CN119478068A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image local tone mapping method and device based on image segmentation. Background Art
[0002] Tone mapping is a computer graphics technique used to approximate the display of high dynamic range images on limited dynamic range display devices. High dynamic range images typically contain a larger range of brightness and color than can be displayed on conventional display devices. The goal of tone mapping is to compress these high dynamic range images to a lower dynamic range that can be displayed on standard display devices while maintaining the visual quality of the image.
[0003] Among the existing mainstream tone mapping methods, local tone mapping is widely favored due to its excellent effect. This method refers to the neighborhood information of pixels during the mapping process and applies different mapping functions to different local areas of the image. This processing method makes two pixels with the same grayscale value appear different brightness or color after mapping, while two pixels with different grayscale values may have the same brightness or color after mapping. This change not only enhances the local contrast of the image, but also makes the details of the image richer and more vivid.
[0004] There are many ways to implement local tone mapping, two common methods include:
[0005] 1. Contrast-limited adaptive histogram equalization (CLAHE): This method divides the image into fixed-size blocks and performs histogram equalization on each block. This method can effectively improve the contrast of the image. However, due to the fixed block division, it may cause obvious visual discontinuity at the block boundary, which is the so-called "block effect".
[0006] 2. Image layering method: This method divides the image into different layers of sub-images. Each layer of sub-images contains different information and dynamic ranges, so different processing strategies are used. This method can better preserve the details and color information of the image, but it is relatively complex to implement.
[0007] Although these methods have achieved remarkable results in improving image quality, some challenges still exist. One of the biggest challenges is how to effectively divide the image region. Traditional local tone mapping methods usually divide the image into fixed blocks or use filter windows to divide the region. However, these methods may cause block effects in the image after stretching, and cannot effectively distinguish different objects in the region, nor can they independently adjust their contrast and brightness.
[0008] To solve this problem, some methods try to distinguish different objects by segmenting the image, thereby achieving more accurate tone mapping. However, traditional image segmentation methods often have difficulty in flexibly segmenting images, and may still be affected by block effects. On the other hand, using advanced semantic segmentation models (such as CNN, UNet, MaskRCNN, etc.) to segment images and generate region masks is a theoretically feasible method. However, these methods usually require a lot of prior knowledge to train the model, so they may be limited in practical applications.
[0009] In addition, local tone mapping faces another problem: how to determine the stretch target value. After determining the objects in the image that need to be contrast stretched, how to stretch the grayscale values of these objects to the appropriate brightness and contrast is a key issue. If the stretching is insufficient, the image details may be lost; if the stretching is excessive, the image may be distorted. Therefore, determining the appropriate stretch target value is one of the keys to achieving high-quality tone mapping. Summary of the invention
[0010] In view of this, the present invention proposes a method and device for local tone mapping of an image based on image segmentation, so as to solve the problem in the prior art that the image area division is not accurate enough, resulting in block effect and detail loss.
[0011] The specific technical solutions of the present invention are as follows:
[0012] An image local tone mapping method based on image segmentation comprises the following steps:
[0013] Step 1: The image processing algorithm receives the original image I as input and performs brightness domain conversion on it to generate a brightness image L;
[0014] Step 2: Input the brightness image L into the SAM model for image segmentation. The SAM model outputs the segmented image, including different regions and their corresponding grayscale means.
[0015] Step 3, the image processing algorithm performs mapping transformation on the segmented image according to the segmentation result output by the SAM model to obtain the mapped image I';
[0016] Step 4, the image processing algorithm converts the mapped image I' from the brightness domain back to the original color domain and outputs the final displayed image I".
[0017] Furthermore, the brightness domain conversion in step 1 is achieved by using the formula L=f(R,G,B), where R, G, B represent the values of the red, green, and blue color channels of each pixel in the image, respectively, and L represents the brightness value after conversion.
[0018] Furthermore, the SAM model includes a feature extraction and conversion stage and a combined feature decoding and segmentation stage, wherein the feature extraction and conversion stage is responsible for converting the brightness image into the feature space and receiving key guidance information and converting it into a feature form that the model can understand; the combined feature decoding and segmentation stage combines the above two feature maps to decode the final segmentation result.
[0019] Furthermore, the mapping transformation in step 3 includes pixel value and coordinate acquisition and regional stretching processing. The pixel value and coordinate acquisition is achieved by checking the value of each pixel in the image, dividing it into different regions, and obtaining its coordinates and grayscale values; the regional stretching processing is to perform different stretching transformations on pixels falling into different regions.
[0020] Furthermore, the formula of the stretching transformation is I'=g(L,λ), where λ represents the ratio of different regions when the mapping changes are performed, and the ratio is calculated by the preset parameters and the variance value of the region.
[0021] Furthermore, the mask image is mapped at different scales to obtain the stretched image M′, which is expressed as
[0022] M′=f(L (x,y) ,M i_grey ,sigma Mi )
[0023]
[0024] where sigma Mi Represents the ratio of mapping changes in different areas, which is determined by the preset parameters a, b, h, and M i The variance value of the region is calculated.
[0025] Furthermore, the SAM-based image segmentation process in step 2 is expressed by the following formula:
[0026] SAM(L)=[M i ,M i_grey ],
[0027] Through SAM image segmentation, we can get M representing different areas of the image. i (i represents different regions of the image), and for the image region M i The grayscale mean value M of all pixels in i_grey .
[0028] An image local tone mapping device based on image segmentation, comprising:
[0029] The image processing module is used to receive the original image I as input, and perform brightness domain conversion on it to generate a brightness image L;
[0030] An image segmentation module, including a SAM model, is used to receive a brightness image L as input, perform image segmentation processing on it, and output a segmented image, wherein the segmented image includes different regions and their corresponding grayscale means;
[0031] A tone mapping module, connected to the image processing module and the image segmentation module, is used to perform mapping transformation on the segmented image according to the segmentation result output by the image segmentation module to obtain a mapped image I';
[0032] The color conversion module is connected to the tone mapping module, and is used to convert the mapped image I' from the brightness domain back to the original color domain, and output the final displayed image I".
[0033] The beneficial effects of the present invention are:
[0034] 1. Achieve high locality of tone mapping: By adopting an image segmentation-based approach, the present invention can accurately segment objects in an image, so that different tone mapping can be performed on different objects. This feature significantly improves the locality and accuracy of tone mapping, making the processed image more refined and realistic in color expression.
[0035] 2. Simplify the prior knowledge requirements for semantic segmentation: Traditional semantic segmentation methods usually require a lot of prior knowledge. However, the present invention uses a SAM model-based approach to segment images, effectively reducing the reliance on prior knowledge and making the image segmentation process simpler and more efficient.
[0036] 3. Maintaining the order of image contrast: In order to solve the problem of local mapping stretching target value, the present invention performs "S" curve stretching on the grayscale of the divided objects based on the global grayscale mean. This stretching method not only maintains the order of image contrast, but also avoids the contrast reversal phenomenon that may occur during the stretching process, thereby ensuring the quality of image processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0038] Figure 1 It is a flowchart of the image local tone mapping method based on image segmentation of the present invention. DETAILED DESCRIPTION
[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0040] The present invention proposes an image local tone mapping method based on image segmentation, which mainly includes the following steps:
[0041] Step 1: Perform brightness domain conversion on image I to obtain brightness image L, as shown in the formula: L = Light (I).
[0042] In step 1, the specific conversion formula depends on the color space conversion method used. Taking the conversion from RGB to YCbCr as an example, the formula for brightness domain conversion is as follows: L = 0.299 × R + 0.587 × G + 0.114 × B. This formula is used to convert the values of the red, green, and blue color channels in the RGB image to the values of the brightness channel L in the YCbCr color space. Among them, R, G, and B represent the values of the red, green, and blue color channels of each pixel in the image, respectively, and L represents the brightness value after conversion.
[0043] When performing brightness domain conversion, image processing libraries or functions are usually used to perform this calculation process. These libraries or functions have implemented the corresponding conversion formulas and provide interfaces for developers to call. By calling these functions, the original image can be easily converted from RGB color space to YCbCr color space and the value of the brightness channel L can be extracted.
[0044] It should be noted that in addition to the YCbCr color space, there are other luminance-chrominance color spaces that can also be used for luminance domain conversion, such as YUI, YIQ, etc. The specific choice of color space depends on the application requirements and the purpose of image processing.
[0045] After completing the brightness domain conversion, we get an image L that only contains brightness information. Each pixel in this image only represents the brightness value of the corresponding position in the original image, and does not contain color information. Such brightness images are very useful in certain image processing and analysis tasks, such as face recognition, image segmentation, image enhancement, etc.
[0046] In summary, step 1.1 is the key step to convert image I into brightness domain. By applying the corresponding conversion formula, we can extract brightness information from the original image and obtain an image L containing only brightness information.
[0047] Step 2: Perform SAM-based image segmentation on the brightness image L to obtain masks of different areas in the image and the grayscale mean level of the area.
[0048] The SAM model is a deep learning model for image segmentation, which mainly consists of three parts: image encoder, prompt and mask. The entire segmentation process can be divided into two stages: stage one is the encoder stage, and stage two is the decodbr stage.
[0049] Phase 1: encoder phase
[0050] Image encoder: This part is responsible for converting (or mapping) the brightness image L to be segmented into the image feature space. It is usually implemented through a series of convolutional layers, pooling layers, etc., with the aim of extracting useful feature information from the original image.
[0051] prompt: This part is responsible for converting (or mapping) the input prompt to the feature space of the prompt. The input prompt is the key guiding information for the model when outputting the final mask. It can be of sparse type (such as sparse refers to the target box or text describing the target) or dense type (such as denseprompt refers to a mask type prompt). The prompt converts the input prompt into a feature form that the model can understand through a series of operations (such as linear transformation, attention mechanism, etc.).
[0052] Phase 2: decoder phase
[0053] Mask: After getting the output of imageencoder and prompt, mask
[0054] Responsible for combining these two feature maps to decode the final segmentation mask. Specifically, the final segmentation mask is decoded by using the two embedding feature maps output by image encoder and prompt respectively. It usually includes a series of upsampling operations, convolution operations, etc., the purpose is to convert the feature map obtained in the encoder stage back to a mask image of the same size as the original image. Each pixel in this mask image represents the area to which the corresponding pixel in the original image belongs.
[0055] The SAM-based image segmentation process can be vividly expressed by the following formula:
[0056] SAM(L)=[M i ,M i_grey ]
[0057] The above formula means that through SAM image segmentation, we can get M representing different areas of the imagei (i represents different regions of the image), and for the image region M i The grayscale mean value M of all pixels in i_grey .
[0058] Step 3: Stretch the mask image (M) divided into different regions using an “s” curve to obtain image M ′ .
[0059] Step 3.1: Obtain pixel values and coordinates. The M of the mask image obtained in step 2 i Segment the image pixels and obtain the coordinates and grayscale values of the image pixels (L (x,y) represents the grayscale value of the image pixel at (x,y).
[0060] In image processing, a mask is usually a two-dimensional matrix of the same size as the original image, in which the value of each pixel represents the region or category to which the corresponding pixel in the original image belongs. Through step 2, we have obtained such a mask image.
[0061] In this step, we use the mask image to segment the original image at the pixel level. Specifically, for each pixel in the mask image, we check its value, and then divide the pixels at the corresponding position in the original image into different regions based on this value. In this way, we can obtain the grayscale value of each pixel and their coordinates (x, y) in the original image.
[0062] Step 3.2: Regional stretching. The image pixels in different regions of the mask image are stretched differently. That is, the mask image is mapped at different proportions to obtain the stretched image M′, which is expressed as
[0063] M′=f(L (x,y) ,M i_grey ,sigma Mi )
[0064]
[0065] where sigma Mi Represents the ratio of mapping changes in different areas, which is determined by the preset parameters a, b, h, and M i The variance value of the region is calculated.
[0066] After obtaining the grayscale value and coordinates of each pixel, we can perform the "S" curve stretching operation. The goal of this step is to apply different stretching ratios to the pixels in each area, thereby enhancing the contrast of the image.
[0067] The specific way of stretching is through a mapping function. This function maps the original grayscale value to a new grayscale value. The mapping ratio (or stretching strength) is calculated based on the preset parameters a, b, h and the variance value of each area. The variance value reflects the distribution of pixel grayscale values in the area, which can help us determine whether a stronger stretching is needed for the area.
[0068] The formula of the mapping function is usually a nonlinear function, shaped like an "S" curve. The characteristic of this function is that the mapping ratio is larger at the low and high ends of the grayscale value, and smaller in the middle. In this way, after mapping, the contrast between the dark and bright areas of the image will be enhanced, while the contrast in the middle area will remain unchanged or change slightly.
[0069] Specifically, for each pixel, we find the corresponding new grayscale value in the mapping function according to the region it belongs to and the original grayscale value, and replace the original grayscale value with this new value. In this way, after processing in step 3.2, we get a new image stretched by the "S" curve. This new image has higher contrast and better visual effects while maintaining the original image information.
[0070] Step 4: Convert the image to its native color domain for display.
[0071] After stretching the image, we get an image representation in the brightness domain. However, this representation only contains the brightness information of the image, and loses the color information in the original image. In order to restore the original appearance of the image, we need to convert the image from the brightness domain back to its original color domain.
[0072] Detailed explanation of the conversion process:
[0073] 1. Determine the original color domain: First, we need to know what color domain the original image is in. Common color domains include RGB, CMYK, HSV, etc. In this example, we assume that the original image is in the RGB domain.
[0074] 2. Mapping brightness to color: Since brightness domain images only contain brightness information, we need to convert the brightness value into the values of the three RGB channels according to a certain mapping relationship. This mapping relationship may be a fixed formula or determined according to the color distribution of the original image.
[0075] 3. Color interpolation: In some cases, it may be necessary to interpolate the brightness values to get a smoother color transition. This can be achieved through interpolation algorithms such as linear interpolation, bilinear interpolation, or cubic interpolation.
[0076] 4. Apply Color Lookup Table (CLUT): In some advanced image processing systems, a Color Lookup Table (CLUT) can be used for color conversion. A CLUT is a predefined mapping table that can directly map brightness values to RGB values.
[0077] After the above conversion process, we get an image representation in the RGB domain. This image not only contains the original brightness information, but also restores the original color information. In this way, we can accurately display the processed image on the display while retaining its original color characteristics.
[0078] In general, step 4 is a process of converting the brightness domain image back to the original color domain, which ensures that the color information of the image is accurately restored and displayed, thereby improving the accuracy and readability of image processing.
[0079] The scheme of the present invention is further described in detail below with reference to specific embodiments.
[0080] Embodiment 1:
[0081] Step 1: First, convert the input image I into the brightness domain to obtain the image L.
[0082] L=Light(I)=0.2126×R+0.7152×G+0.0722×B
[0083] Step 2: Perform SAM segmentation on the brightness image L to obtain the mask of each area, thereby obtaining the image pixel position and pixel grayscale value of each area.
[0084] SAM(L)=[M i ,M i_grey ]
[0085] Step 3: Perform different stretches on the image pixels that fall into different areas of the image, that is, perform mapping changes of different proportions to obtain the stretched image M′, the formula is:
[0086]
[0087] Where L (x,y) For area M i The grayscale value of the image pixel at position (x, y) is the grayscale value of the image pixel at position (x, y) in region M. i The gray value mean; greylevel is the gray value range of the image. Taking 12 images as an example, greylevel=4096; sigma Mi Represents the ratio of mapping changes in different regions, which is determined by the preset parameters a=0.01, b=0, h=0.01, and M i The variance value of the region is calculated.
[0088] Step 4: Finally, convert the image M′ to the original color domain for display.
[0089] Embodiment 2:
[0090] Step 1: First, convert the input image I into the brightness domain to obtain the image L.
[0091] L=Light(I)=0.2126*R+0.7152*G+0.0722*B
[0092] Step 2: Perform SAM segmentation on the brightness image L to obtain the mask of each area, thereby obtaining the image pixel position and pixel grayscale value of each area.
[0093] SAM(L)=[M i ,M i_grey ]
[0094] Step 3: Perform different stretches on the image pixels that fall into different areas of the image, that is, perform mapping changes of different proportions to obtain the stretched image M′, the formula is:
[0095]
[0096]
[0097] Where L (x,y) For area M i The grayscale value of the image pixel at position (x, y) is the grayscale value of the image pixel at position (x, y) in region M. i The gray value mean; π is the circular constant; greylevel is the gray value range of the image, taking 12 images as an example, greylevel=4096; sigma Mi Represents the ratio of mapping changes in different regions, which is determined by the preset parameters a=0.1, b=0.3, h=0.1, and M i The variance value of the region is calculated.
[0098] Step 4: Finally, convert the image M′ to the original color domain for display.
[0099] The present invention also proposes an image local tone mapping device based on image segmentation, comprising: an image processing module, used to receive an original image I as input, and perform brightness domain conversion on it to generate a brightness image L; an image segmentation module, including a SAM model, used to receive the brightness image L as input, and perform image segmentation processing on it, and output a segmented image, wherein the segmented image includes different regions and their corresponding grayscale means; a tone mapping module, connected to the image processing module and the image segmentation module, used to perform mapping transformation on the segmented image according to the segmentation result output by the image segmentation module, and obtain a mapped image I'; a color conversion module, connected to the tone mapping module, used to convert the mapped image I' from the brightness domain back to the original color domain, and output the final displayed image I".
[0100] The beneficial effects of the present invention are:
[0101] 1. Achieve high locality of tone mapping:
[0102] Through image segmentation based on the SAM model, the present invention can accurately identify different regions in the image. These regions can be objects with similar colors, textures or shapes, or specific parts of the image, such as the sky, the ground, buildings, etc.
[0103] Once the image is segmented into different regions, the present invention can perform independent tone mapping on each region. This means that each region can have a different tone adjustment, thereby achieving highly localized tone mapping. This localized processing method can ensure that each part of the image is properly color-adjusted, improving the overall color expression and visual effect.
[0104] More importantly, the regions segmented according to the SAM model have precise boundaries and shapes, which makes the tone mapping process more accurate and detailed. By accurately stretching the tone of each local area, the present invention can preserve the details and textures in the image and avoid color distortion or blurring that may be caused by overall tone mapping.
[0105] 2. For different areas, "S" curve stretching is performed to ensure the orderliness of grayscale while achieving controllable adjustment of different grayscale segments:
[0106] In the process of realizing tone mapping, the present invention uses an "S" curve stretching method to adjust the grayscale of the image. This method can maintain the orderliness of the image grayscale and avoid the contrast inversion phenomenon that may occur during the stretching process.
[0107] For different regions, the present invention can perform "S" curve stretching according to the characteristics and requirements of each region. This means that different regions can have different stretching parameters and curve shapes, thereby achieving controllable adjustment of different grayscale segments.
[0108] By adjusting the parameters of the "s" curve, the present invention can control the contrast, brightness, color saturation and other properties of each area. This adjustment method is not only flexible and changeable, but also can accurately control the color performance of each area, making the color effect of the overall image more coordinated and natural.
[0109] In summary, the present invention ensures the accuracy and controllability of image processing by achieving high localization of tone mapping and performing "S" curve stretching for different regions. This not only improves the quality of image processing, but also provides strong technical support for various practical application scenarios.
[0110] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A local tone mapping method for an image based on image segmentation, characterized in that: Includes steps: Step 1: The image processing algorithm receives the original image I as input and performs brightness domain conversion on it to generate a brightness image L; Step 2: Input the brightness image L into the SAM model for image segmentation. The SAM model outputs the segmented image, including different regions and their corresponding grayscale means. Step 3, the image processing algorithm performs mapping transformation on the segmented image according to the segmentation result output by the SAM model to obtain the mapped image I'; Step 4, the image processing algorithm converts the mapped image I' from the brightness domain back to the original color domain and outputs the final displayed image I".
2. The image local tone mapping method based on image segmentation according to claim 1, characterized in that: The brightness domain conversion in step 1 is achieved by using the formula L = f(R, G, B), where R, G, B represent the values of the red, green, and blue color channels of each pixel in the image, respectively, and L represents the brightness value after conversion.
3. The image local tone mapping method based on image segmentation according to claim 1, characterized in that: The SAM model includes a feature extraction and conversion stage and a combined feature decoding and segmentation stage. The feature extraction and conversion stage is responsible for converting the brightness image into the feature space and receiving key guidance information and converting it into a feature form that the model can understand; the combined feature decoding and segmentation stage combines the above two feature maps to decode the final segmentation result.
4. The image local tone mapping method based on image segmentation according to claim 1, characterized in that: The mapping transformation in step 3 includes pixel value and coordinate acquisition and regional stretching processing. Pixel value and coordinate acquisition is achieved by checking the value of each pixel in the image, dividing it into different regions, and obtaining its coordinates and grayscale values; regional stretching processing is to perform different stretching transformations on pixels falling into different regions.
5. The image local tone mapping method based on image segmentation according to claim 1, characterized in that: The formula of the stretch transformation is I'=g(L,λ), where λ represents the ratio of different regions when the mapping changes, and the ratio is calculated by the preset parameters and the variance value of the region.
6. The image local tone mapping method based on image segmentation as claimed in claim 5, characterized in that: The mask image is mapped at different scales to obtain the stretched image M′, which is given by the formula: M′=f(L (x,y) ,M i_grey ,sigma Mi ) where sigma Mi Represents the ratio of mapping changes in different areas, which is determined by the preset parameters a, b, h, and M i The variance value of the region is calculated.
7. The image local tone mapping method based on image segmentation according to claim 1, characterized in that: The SAM-based image segmentation process in step 2 is expressed by the following formula: SAM(L)=[M i ,M i_grey ], Through SAM image segmentation, we can get M representing different areas of the image. i (i represents different regions of the image), and for the image region M i The grayscale mean value M of all pixels in i_grey .
8. An image local tone mapping device based on image segmentation, characterized in that: include: The image processing module is used to receive the original image I as input, and perform brightness domain conversion on it to generate a brightness image L; An image segmentation module, including a SAM model, is used to receive a brightness image L as input, perform image segmentation processing on it, and output a segmented image, wherein the segmented image includes different regions and their corresponding grayscale means; A tone mapping module, connected to the image processing module and the image segmentation module, is used to perform mapping transformation on the segmented image according to the segmentation result output by the image segmentation module to obtain a mapped image I'; The color conversion module is connected to the tone mapping module, and is used to convert the mapped image I' from the brightness domain back to the original color domain, and output the final displayed image I".