Image processing methods and apparatuses, electronic devices, storage media
By converting the image color space and searching for the minimum color difference, the problem of inconsistent color gamut between the image display device and the image to be displayed is solved, achieving high-quality color reproduction and memory color retention on different devices.
Patent Information
- Application Number
- CN202210772660.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The color gamut of the image display device is inconsistent with that of the image to be displayed, resulting in color distortion and aberration, especially when displaying memory colors such as skin tones and skies.
By performing color space conversion on the image to be processed, identifying and memorizing color pixels and maintaining their display parameters, performing minimum color difference search on pixels that are outside the color gamut of the display device, replacing them with color pixels supported by the display device, and performing inverse conversion to output the image.
It improves the display quality of images on different display devices, maintains the color memory effect, and enhances the user experience.
Smart Images

Figure CN117392023B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to image display technology, and more particularly to an image processing method and apparatus, electronic device, and storage medium. Background Technology
[0002] Due to the complexity of display devices and standards, color distortion can easily occur when an image is displayed on a particular device, and even memory colors like skin tones and skies can be distorted. Furthermore, the color gamut of an image and the color gamut of a display device often differ; when neither the image nor the display device's color gamut is a standard color gamut, inconsistencies in color gamut can easily arise. Summary of the Invention
[0003] This disclosure provides an image processing method and apparatus, an electronic device, and a storage medium, which can at least solve the problem of inconsistent color gamut between the display device and the image to be displayed in an electronic device.
[0004] According to a first aspect of the present disclosure, an image processing method is provided, applied to a display device, the method comprising:
[0005] The first non-linear mapping process is performed on the pixels of the image to be processed, and the mapped pixels are converted from the first color space to the second color space to obtain the converted image pixel data.
[0006] In the converted image pixel data, the first feature color pixel of the target object is determined, and the remaining pixels in the image are treated as non-first feature color pixels.
[0007] Determine a target pixel whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixels, and find a target color point with the smallest color difference from the target pixel within the color gamut range, and replace the display parameters of the target pixel with the display parameters of the target color point;
[0008] The image output will show the pixels corresponding to the replaced display parameters.
[0009] Optionally, the image processing method further includes:
[0010] For the first characteristic color pixel of the target object in the image, preserve the display parameters; and / or
[0011] For non-first characteristic color pixels located within the color gamut of the display device, the display parameters are maintained.
[0012] Optionally, determining the first feature color pixel of the target object in the converted image pixel data includes:
[0013] Obtain the first training image;
[0014] The pixel data of the target object in the first training image is first labeled, and the first detection network is trained based on the first labeled pixel data to obtain the first detection network model;
[0015] The target object in the image to be processed is identified based on the first detection network model.
[0016] Optionally, determining the first feature color pixel of the target object in the converted image pixel data includes:
[0017] Obtain a second training image; the second training image includes the first feature color pixel data of the target object;
[0018] The first feature color pixel data of the target object and the non-first feature color pixels in the second training image are labeled a second time. The second detection network is trained based on the second labeled pixel data to obtain the second detection network model.
[0019] The first feature color pixel point of the region where the target object is located is identified based on the second detection network model.
[0020] Optionally, finding the target color gamut point with the smallest color difference from the target pixel point outside the color gamut range within the color gamut range includes:
[0021] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0022] Within the circle and the color gamut, search for color points, calculate the color difference between the target pixel and any color point, determine the search direction based on the magnitude relationship between two adjacent color differences, and search for the next color point in the determined search direction.
[0023] When a color point with a color difference less than the first set threshold is found, and / or the number of searches reaches the second set threshold, the color point with the smallest color difference found is taken as the target color point.
[0024] Optionally, finding the target color gamut point with the smallest color difference from the target pixel point outside the color gamut range within the color gamut range includes:
[0025] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0026] Within the circle and the color gamut, traverse each color point, calculate the color difference between the target pixel and the color point, and take the color point with the smallest color difference as the target color point.
[0027] Optionally, the method further includes:
[0028] The color gamut information of the image is determined through the header file of the image to be processed; and
[0029] The color gamut range supported by the display device is determined by the display parameters configured on the display device.
[0030] Optionally, the step of outputting the image corresponding to the pixels after the display parameters have been replaced includes:
[0031] The pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space, and then the pixels of the image are subjected to a second nonlinear mapping process to output the image processed by the second nonlinear mapping; wherein, the first nonlinear mapping is the inverse process of the second nonlinear mapping.
[0032] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:
[0033] The mapping unit is used to perform a first non-linear mapping process on the pixels of the image to be processed;
[0034] The conversion unit is used to convert the mapped pixels from the first color space to the second color space to obtain the converted image pixel data.
[0035] The first determining unit is used to determine the first feature color pixel of the target object in the converted image pixel data, and the remaining pixels in the image are treated as non-first feature color pixels.
[0036] The second determining unit is used to determine the target pixel point whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixels, and to find the target color point with the smallest color difference with the target pixel point within the color gamut range, and to replace the display parameters of the target pixel point with the display parameters of the target color point.
[0037] The replacement unit is used to find the target color gamut point with the smallest color difference from the target pixel point outside the color gamut within the color gamut range, and replace the display parameters of the target pixel point outside the color gamut range with the display parameters of the target color gamut point.
[0038] The output unit is used to output the image corresponding to the pixels after the display parameters have been replaced.
[0039] Optionally, the device further includes:
[0040] The holding unit is used to hold display parameters for the first characteristic color pixel of the target object in the image; and / or
[0041] For non-first characteristic color pixels located within the color gamut of the display device, maintain the display parameters.
[0042] Optionally, the second determining unit includes:
[0043] The first acquisition subunit is used to acquire the first training image;
[0044] The first training subunit is used to perform a first annotation on the pixel data of the target object in the first training image, and to train a first detection network based on the first annotated pixel data to obtain a first detection network model.
[0045] The first identification subunit is used to identify the target object in the image to be processed based on the first detection network model.
[0046] Optionally, the second determining unit further includes:
[0047] The second acquisition subunit is used to acquire a second training image; the second training image includes the first feature color pixel data of the target object;
[0048] The second training subunit is used to perform a second annotation on the first feature color pixel data of the target object and the non-first feature color pixels in the second training image, and to train the second detection network based on the second annotated pixel data to obtain the second detection network model.
[0049] The second recognition subunit is used to identify the first feature color pixel point in the area where the target object is located based on the second detection network model.
[0050] Optionally, the replacement unit is further configured to:
[0051] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0052] Within the circle and the color gamut, search for color points, calculate the color difference between the target pixel and any color point, determine the search direction based on the magnitude relationship between two adjacent color differences, and search for the next color point in the determined search direction.
[0053] When a color point with a color difference less than a first preset threshold is found, and / or the number of searches reaches a second preset threshold, the color point with the smallest color difference found is taken as the target color point.
[0054] Optionally, the replacement unit is further configured to:
[0055] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0056] Within the circle and the color gamut, traverse each color point, calculate the color difference between the target pixel and the color point, and take the color point with the smallest color difference as the target color point.
[0057] Optionally, the device further includes:
[0058] The third determining unit is used to determine the color gamut information of the image based on the header file of the image to be processed; and
[0059] The fourth determining unit is used to determine the color gamut range supported by the display device through the display parameters configured in the display device.
[0060] Optionally, the output unit is further configured to:
[0061] The pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space, and then the pixels of the image are subjected to a second nonlinear mapping process to output the image processed by the second nonlinear mapping; wherein, the first nonlinear mapping is the inverse process of the second nonlinear mapping.
[0062] According to a third aspect of the present disclosure, an electronic device is provided, including a processor and a memory for storing processor-executable instructions, the processor being configured to perform the steps of the image processing method when the executable instructions in the memory are invoked.
[0063] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the image processing method.
[0064] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:
[0065] In the embodiments of this disclosure, by performing color space conversion on the image to be processed, and then replacing the non-first feature color pixels in the image that do not support the color gamut with the closest color gamut supported by the display device, and then performing inverse color space conversion on the replaced image, the display quality of the image is improved and the user experience is enhanced.
[0066] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0067] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0068] Figure 1 This is a schematic flowchart illustrating an image processing method according to an embodiment of the present disclosure;
[0069] Figure 2 This is a schematic flowchart illustrating an image processing method according to an embodiment of the present disclosure;
[0070] Figure 3 This is a schematic diagram illustrating an image skin tone annotation method according to an embodiment of this disclosure;
[0071] Figure 4 This is a schematic diagram illustrating the search process for the target pixel of the pixel to be replaced in an image, as shown in an embodiment of this disclosure.
[0072] Figure 5 This is a schematic diagram of the search algorithm for the target pixel of the pixel to be replaced in an image, as shown in an embodiment of this disclosure.
[0073] Figure 6 This is a schematic diagram of the composition structure of an image processing apparatus according to an embodiment of the present disclosure;
[0074] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0075] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0076] Currently, color gamut mapping and color reproduction techniques in image processing typically only perform color gamut mapping for the display device, such as using the same set of parameters for display. This makes it impossible to distinguish between skin tones or sky colors that are memorized. Furthermore, during color gamut mapping, colors outside the gamut are usually directly compressed and cropped. Generally, color gamut mapping algorithms are divided into point-to-point mapping algorithms and spatial mapping algorithms. Point-to-point mapping algorithms are device-to-device mapping algorithms, pursuing pixel accuracy, while spatial mapping algorithms consider the impact of image content and the spatial relationship between pixels during the mapping process, ignoring the accuracy of individual colors. Point-to-point mapping algorithms include two core techniques: color gamut cropping and color gamut compression. Color gamut cropping algorithms can maintain color saturation, but they result in significant loss of detail in colors outside the gamut and can also introduce noise.
[0077] The image processing method of this disclosure addresses the technical problems existing in the existing color gamut mapping algorithm. By detecting the areas where the memory colors such as skin color or sky color are located in the image, and then using an iterative search algorithm, the goal is to find the minimum color difference between the pixel color in the image to be replaced and the target color point. This allows the image to be displayed according to the image color gamut on different display devices. The above method can adaptively perform skin color protection and color reproduction according to the color gamut range of the display device.
[0078] After image acquisition devices such as cameras, video recorders, and mobile phones complete image acquisition, they need to be displayed on different display devices, such as the image acquisition device itself or other output devices. However, due to the differences and complexity of display device hardware characteristics, the same image acquired by an image acquisition device cannot maintain a consistent effect when displayed on different display devices, thus failing to accurately represent the creator's intention and resulting in significant deviations in display effects. Furthermore, because the human eye has inherent impressions of colors like the sky and skin tones, it is necessary to maintain the display effect of these memorized colors as much as possible during the display process across different display devices.
[0079] The image processing method of this disclosure differs from existing color gamut mapping methods in that its color reproduction means can not only realize color gamut mapping when displaying an image, but also maintain and preserve memory colors such as skin tones. By searching for the smallest color difference pixel in the entire color space, a color that is closest to the source color is finally found as the target color. This enables accurate color reproduction on the display device, fully maintaining the high quality of the image displayed on the display device and improving the user's viewing experience of the image.
[0080] Figure 1 This is a schematic flowchart illustrating an image processing method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the image processing method of this disclosure embodiment specifically includes the following processing steps:
[0081] Step 101: Perform a first nonlinear mapping process on the pixels of the image to be processed, and convert the mapped pixels from the first color space to the second color space to obtain the converted image pixel data.
[0082] In this embodiment of the disclosure, the first nonlinear mapping process includes inverse gamma correction, which is the inverse transformation of gamma correction and is a type of nonlinear mapping such as inverse gamma curve mapping, used for color reconstruction. The first color space includes the RGB color space, and the second color space includes the XYZ color space.
[0083] In this embodiment of the disclosure, before displaying the image to be displayed on the display device, it is necessary to convert the color space of the pixels of the image to be processed, and adjust the display parameters based on the pixels in the converted image pixel data, so that the image to be displayed is displayed on the display device in a way that is more in line with the viewer's viewing experience.
[0084] Step 102: Determine the first feature color pixel of the target object in the converted image pixel data, and treat the remaining pixels in the image as non-first feature color pixels.
[0085] In this embodiment, the first characteristic color is primarily a memory color, i.e., a color that people have a fixed memory of for different objects, including but not limited to the color of the sky, the skin tone of a person's portrait, etc. The target objects mainly include portraits, the sky, etc. In this embodiment, when adjusting the pixel display parameters of the image to be displayed, it is necessary to identify the memory colors in the image to determine these memory colors, so that the memory colors are displayed on the display device with their inherent display parameters, thus better conforming to people's image viewing habits and improving the image viewing experience. As one implementation method, the display parameters include at least one of the following parameters: contrast, brightness, saturation, resolution, etc.
[0086] In this embodiment, it is also necessary to identify the region where the target object is located in the image to be processed. Specifically, as one implementation, the region where the target object is located in the image can be identified using a neural network model. The target objects here mainly include objects such as the sky and human figures. Specifically, a first training image is acquired; the pixel data of the target object in the first training image is first labeled; a first detection network is trained based on the first labeled pixel data to obtain a first detection network model; and the target object in the image to be processed is identified based on the first detection network model. Here, the first training image can be an image captured by a camera of an electronic device or an image published on the internet. As an example, in this embodiment, some pixel data of the region where the target object, such as the sky or human figure, is located in the image to be processed can also be first labeled and used as labeled data for training to train the relevant parameters of the first detection network model to obtain a first detection network model with better accuracy. The first detection network here includes neural networks such as Unet. Those skilled in the art should understand that Unet is only an example, and other neural network models such as radial basis function neural networks, feedforward neural networks, and feedback neural networks are also possible. Based on the first detection network model, the target object in the image to be processed is identified, thereby determining the region of the target object in the image to be processed.
[0087] In this embodiment, after identifying the area where the target object, such as the sky or a person, is located in the image, it is also necessary to identify specific characteristic colors in the area, such as skin color or other memory colors in the person's image. One implementation method is to use a neural network model to identify specific characteristic colors in the image. Specifically, a second training image is obtained; the first characteristic color pixel data of the target object and the non-first characteristic color pixels in the second training image are second-labeled; a second detection network is trained based on the second-labeled pixel data to obtain a second detection network model; and the first characteristic color pixels of the area where the target object is located are identified based on the second detection network model. The second training image here includes the first characteristic color pixel data of the target object. The second training image can be an image published online, or an image stored locally on an electronic device or captured by an electronic device. In this embodiment, some pixel data of the first characteristic color region of the target object in the image to be processed and some non-first characteristic color pixels in the image to be processed can also be used as training data for second labeling, which can also train a second detection network model; the second detection network here includes a Support Vector Machine (SVM) binary classifier model. The second detection network model identifies the first feature color pixels in the region where the target object is located. Here, the first feature color mainly refers to a memory color such as sky color or skin color.
[0088] Step 103: Determine the target pixel whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixels, and find the target color point with the smallest color difference from the target pixel within the color gamut range, and replace the display parameters of the target pixel with the display parameters of the target color point.
[0089] In the field of color, every pixel in an image has a corresponding position in the color gamut. That is, once the color gamut range of the display device is determined, it can be determined whether each pixel in the image is within the color gamut range of the display device.
[0090] In this embodiment, the display parameters of pixels with a first characteristic color, such as a memory color, are not adjusted. Only the display parameters of pixels that are not memory colors and are not within the color gamut of the display device are adjusted to ensure the image to be displayed is better presented on different display devices. For pixels with the first characteristic color of the target object in the image, the display parameters are maintained. That is, the display characteristics of memory colors, etc., in the image are preserved so that the image is more in line with people's viewing habits when displayed on different display devices.
[0091] In this embodiment of the disclosure, for non-memory color regions in an image, it is necessary to determine whether the color gamut of the non-memory color is within the color gamut range supported by the display device. If the color gamut of the pixels in the non-memory color region is within the color gamut range supported by the display device, then the display parameters of the pixels in the non-memory color region are maintained. That is, pixels that are not the first feature color and are located within the color gamut range of the display device retain their display parameters.
[0092] If the color gamut of a pixel in a non-memory color region of an image exceeds the color gamut range of the image's display device, a target color gamut point with the smallest color difference from the target pixel exceeding the color gamut range is found within the color gamut range, and the display parameters of the target pixel exceeding the color gamut range are replaced with the display parameters of the target color gamut point.
[0093] One implementation involves finding the target color gamut point with the smallest color difference from a target pixel outside the color gamut within the color gamut range. This includes: for a target pixel outside the color gamut range, drawing a perpendicular line to the color gamut boundary closest to the target pixel, and drawing a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius; searching for color points within the circle and the color gamut range, calculating the color difference between the target pixel and any color point, and determining the search direction based on the magnitude relationship between two adjacent color differences, and searching for the next color point in the determined search direction; when a color point with a color difference less than a first preset threshold is found, and / or the number of searches reaches a second preset threshold, the searched color point with the smallest color difference is taken as the target color point. The first preset threshold includes 0.1, 0.08, etc., meaning that when the total color difference of the pixels is less than 0.1 or 0.08, the color gamut corresponding to the searched point with the smallest color difference is taken as the display parameter of the pixel. The second set threshold includes 3000, 3500, 4000, 4500, etc. That is, when the number of searches reaches 3000, in order not to increase the amount of data processing, the point with the smallest color difference in the 3000 searches is determined as the target color point.
[0094] Alternatively, as one implementation, the target color point with the smallest color difference between the target pixel within the color gamut and the target pixel outside the color gamut can be found. This includes: for a target pixel outside the color gamut, drawing a perpendicular line to the color gamut boundary closest to the target pixel, and drawing a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius; within the circle and the color gamut, traversing each color point, calculating the color difference between the target pixel and the target color point, and selecting the color point with the smallest color difference as the target color point. Here, each color point is directly traversed within the circle and the color gamut until all color points have been traversed, and the target color point is finally determined. This method does not consider the consumption of computing resources.
[0095] Step 104 will output the image corresponding to the pixels after the display parameters have been replaced.
[0096] Specifically, the pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space. Then, the pixels of the image are subjected to a second non-linear mapping process to restore the contrast of the image before the first non-linear mapping process. The image processed by the second non-linear mapping is then output.
[0097] In this embodiment of the disclosure, the first nonlinear mapping is the inverse process of the second nonlinear mapping. The second nonlinear mapping process includes Gamma mapping. The first color space includes the RGB color space, and the second color space includes the XYZ color space.
[0098] The following specific examples further illustrate the essence of the technical solutions in the embodiments of this disclosure.
[0099] Figure 2 This is a schematic flowchart illustrating an image processing method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the technical solution of this disclosure includes the following processing steps:
[0100] Step 1: Input an image to be processed.
[0101] For the input image to be processed, its color gamut information (P3 / sRGB) is determined by its header file. Here, the header file of the image to be processed includes a file with the .hdr extension, which serves as the file description area. The header file information contains two parts: an information area and a palette area. The color gamut information of the image can be determined by examining the header file of the image to be processed.
[0102] In this embodiment of the disclosure, it is also necessary to determine the color gamut range of the display device, such as a liquid crystal display. This is so that in subsequent image processing, it can be determined whether the color gamut of the pixels in the image is within the color gamut of the display device and whether it can be accurately displayed by the display device.
[0103] Step 2: Perform inverse gamma correction on the input image to linear space, and then use a transformation matrix to transform the pixels of the input image into an XYZ space that is independent of the current processing device.
[0104] Step 3: Label the human image regions in the input image, and train a detection network based on the labeled data. This labeled human image region data is used to train a human image region detection network, Unet, to identify human image regions in the input image. For example... Figure 3As shown, the labeled human region is "person," and a large amount of such data can be used to train a Unet segmentation network. Those skilled in the art should understand that it is possible to train the parameters of the Unet human region detection network and obtain the Unet model based on the training data; the implementation details will not be elaborated here.
[0105] In this embodiment of the disclosure, memory colors such as skin tone and sky color in the image are also identified. This example uses skin tone as an example for illustration. A batch of skin tone pixel data and non-skin tone pixel data are labeled in the input image, and an SVM binary classifier is trained. This SVM is then connected to UNet to detect skin tone pixels from the segmented portrait region. Figure 3 As shown, in the annotation of skin color pixels, 1 represents skin color pixels and 2 represents non-skin color pixels. This type of data is used to train an SVM binary classifier.
[0106] Step 4: The input image is processed using the Unet segmentation network model and the SVM binary classifier model to detect target objects and feature colors. The Unet human region detection network and the SVM classifier determine whether pixels in the image represent skin color.
[0107] Step 5: Traverse each pixel of the image and perform the operation in Step 4. If the traversed pixel meets the skin color pixel judgment condition, keep the color value of the pixel unchanged; otherwise, proceed to Step 6.
[0108] Step 6: If the pixel is not a skin color pixel, determine whether the color of the pixel is within the color gamut range of the display device. If it is within the color gamut range, keep the color value unchanged; otherwise, proceed to step 7.
[0109] Step 7: When the color point is outside the color gamut of the target display device, use a search algorithm to search within the entire color gamut space of the display device to find a color point that has the smallest color difference from the source color and is located within the color gamut of the target display device.
[0110] Figure 4 This is a schematic diagram illustrating the search for the target pixel in an image to be replaced, as shown in the embodiments of this disclosure. Figure 4 As shown, as one implementation method, when displaying images on different display devices, color gamut mapping is required to achieve accurate color reproduction. During color gamut mapping, for the colors of pixels to be replaced that are not skin tones or outside the color gamut range, a search algorithm is used to search the entire color space until a color point with the smallest color difference to be replaced is found. Specific steps include: when the color of a pixel to be replaced is outside the color gamut range of the target display device, such as... Figure 5As shown by black point A in the diagram, the area formed by triangles CDE represents the color gamut of the target display device. First, draw a perpendicular line from point A to the nearest color gamut boundary CD. With the intersection point B of the perpendiculars as the center and the length of line segment AB as the radius, construct a circle O. Search within the entire circle (the color gamut range). Whenever a color point is determined, calculate the color difference (ΔE2000) between that color point and point A. Compare the color difference between the currently found color point and point A with the color difference between the previously found color point and point A. Use the color point with the smaller color difference to determine the next search direction. Repeat the aforementioned search steps until a color point within the color gamut with the smallest color difference from point A is found, which is then designated as the target color point X.
[0111] One implementation method is to determine the target color point X if the color difference between the searched color point and the pixel in the image is less than a first preset threshold, and the color point with the smallest color difference from point A within the color gamut is found. Alternatively, if the number of searches reaches a second preset threshold, the color point with the smallest color difference found so far is taken as the target color point X within the color gamut and with the smallest color difference from point A. Another implementation method is to iterate through each color point within the circle O, calculate the color difference between the pixel and the color point, and take the color point with the smallest color difference from point A within the color gamut as the target color point X.
[0112] Step 8: After all pixels of the input image have been traversed, the pixels of the image are uniformly converted from XYZ space to RGB color space, and then gamma correction is used to restore and maintain the contrast of the source image.
[0113] Step 9: After the above processing steps, the skin tone of the input image can be effectively preserved, and the display effect can be dynamically adjusted on the display device according to the content of the input image.
[0114] The technical solution of this disclosure embodiment can maintain a consistent display effect based on the color gamut of the image when displaying the same image on different display devices. During the image display process, through portrait detection and skin color thresholding, the skin color and sky color parts in the image can be effectively identified. Based on different color gamut conditions and the color gamut range of the display device, no display parameter mapping is performed on the skin color part to maintain the original display effect and preserve the original color of the memory color. For pixels outside the color gamut range of the display device, display parameter mapping is performed to make them more compatible with the display capabilities of the display device, thereby maximizing the reproduction of the display color of the input image.
[0115] Figure 6 This is a schematic diagram of the composition structure of an image processing apparatus shown in an embodiment of the present disclosure, such as... Figure 6As shown, the image processing apparatus of this disclosure includes:
[0116] The mapping unit 60 is used to perform a first non-linear mapping process on the pixels of the image to be processed;
[0117] The conversion unit 61 is used to convert the mapped pixels from the first color space to the second color space to obtain the converted image pixel data.
[0118] The first determining unit 62 is used to determine the first feature color pixel of the target object in the converted image pixel data, and the remaining pixels in the image are treated as non-first feature color pixels.
[0119] The second determining unit 63 is used to determine the target pixel whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixels, and to find the target color point with the smallest color difference with the target pixel within the color gamut range, and to replace the display parameters of the target pixel with the display parameters of the target color point.
[0120] Replacement unit 64 is used to find the target color gamut point with the smallest color difference from the target pixel point outside the color gamut within the color gamut range, and replace the display parameters of the target pixel point outside the color gamut range with the display parameters of the target color gamut point.
[0121] Output unit 65 is used to output the image corresponding to the pixels after the display parameters are replaced.
[0122] As one implementation method, in Figure 6 Based on the image processing apparatus shown, the image processing apparatus of this disclosure embodiment further includes:
[0123] Holding unit ( Figure 6 (not shown in the image), used to maintain display parameters for the first characteristic color pixel of the target object in the image; and / or
[0124] For non-first characteristic color pixels located within the color gamut of the display device, maintain the display parameters.
[0125] As one implementation, the second determining unit 63 of the image processing apparatus shown in this embodiment includes:
[0126] First acquisition subunit ( Figure 6 (Not shown in the image), used to obtain the first training image;
[0127] First training subunit ( Figure 6 (not shown in the image) is used to perform a first annotation on the pixel data of the target object in the first training image, and to train a first detection network based on the first annotated pixel data to obtain a first detection network model;
[0128] First identification subunit ( Figure 6 (not shown in the image), used to identify target objects in the image to be processed based on the first detection network model.
[0129] As one implementation, the second determining unit of the image processing apparatus in this embodiment further includes:
[0130] Second acquisition subunit ( Figure 6 (Not shown in the image), used to acquire a second training image; the second training image includes the first feature color pixel data of the target object;
[0131] Second training subunit ( Figure 6 (not shown in the image), used to perform a second annotation on the first feature color pixel data of the target object and the non-first feature color pixels in the second training image, and to train a second detection network based on the second-annotated pixel data to obtain a second detection network model;
[0132] Second identification subunit ( Figure 6 (not shown in the image), used to identify the first feature color pixel in the region where the target object is located based on the second detection network model.
[0133] In one implementation, the replacement unit 64 is further configured to:
[0134] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0135] Within the circle and the color gamut, search for color points, calculate the color difference between the target pixel and any color point, determine the search direction based on the magnitude relationship between two adjacent color differences, and search for the next color point in the determined search direction.
[0136] When a color point with a color difference less than the first set threshold is found, and / or the number of searches reaches the second set threshold, the color point with the smallest color difference found is taken as the target color point.
[0137] In one implementation, the replacement unit 64 is further configured to:
[0138] For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius.
[0139] Within the circle and the color gamut, traverse each color point, calculate the color difference between the target pixel and the color point, and take the color point with the smallest color difference as the target color point.
[0140] As one implementation method, in Figure 6 Based on the image processing apparatus shown, the image processing apparatus of this disclosure embodiment further includes:
[0141] Third Determined Unit ( Figure 6 (Not shown in the image header), used to determine the color gamut information of the image through the image header file; and
[0142] Fourth determination unit ( Figure 6 (not shown in the image), used to determine the color gamut range supported by each display device for the image.
[0143] In one implementation, the output unit 66 is further configured to:
[0144] The pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space, and then the pixels of the image are subjected to a second nonlinear mapping process to output the image processed by the second nonlinear mapping; wherein, the first nonlinear mapping is the inverse process of the second nonlinear mapping.
[0145] In an exemplary embodiment, the mapping unit 60, conversion unit 61, first determining unit 62, second determining unit 63, replacement unit 64, output unit 65, holding unit, first training subunit, first identification subunit, second training subunit, second identification subunit, third determining subunit, and fourth determining subunit may be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), baseband processors (BPs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components.
[0146] In this embodiment of the disclosure, Figure 6 The specific manner in which each unit in the image processing apparatus performs its operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.
[0147] Figure 7 This is a block diagram illustrating a post-processing device 800 according to an exemplary embodiment, such as... Figure 7 As shown, the post-processing device 800 supports multi-screen output and may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.
[0148] Processing component 802 typically controls the overall operation of post-processing device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
[0149] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on post-processing device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0150] The power supply component 806 provides power to the various components of the post-processing device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the post-processing device 800.
[0151] The multimedia component 808 includes a screen that provides an output interface between the post-processing device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0152] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when post-processing device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
[0153] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0154] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of the post-processing device 800. For example, sensor assembly 814 may detect the on / off state of the device 800, the relative positioning of components such as the display and keypad of the post-processing device 800, changes in the position of the post-processing device 800 or a component thereof, the presence or absence of user contact with the post-processing device 800, the orientation or acceleration / deceleration of the post-processing device 800, and temperature changes of the post-processing device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0155] Communication component 816 is configured to facilitate wired or wireless communication between post-processing device 800 and other devices. Post-processing device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0156] In an exemplary embodiment, the post-processing device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the steps of the image processing method of the above embodiments.
[0157] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of a post-processing device 800 to complete the steps of the image processing method of the above embodiments. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0158] This disclosure also describes a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform the steps of the image processing method of the foregoing embodiments.
[0159] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0160] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that, The method includes: A first nonlinear mapping process is performed on the pixels of the image to be processed, and the mapped pixels are converted from a first color space to a second color space to obtain the converted image pixel data; wherein, the second color space is a device-independent color space; In the converted image pixel data, the first feature color pixel of the target object is determined, and the remaining pixels in the image are treated as non-first feature color pixels. Identify target pixels whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixels, and find the target color point with the smallest color difference from the target pixel within the color gamut range. Replace the display parameters of the target pixel with the display parameters of the target color point; and maintain the display parameters of the first feature color pixels of the target object in the image. The image output will show the pixels corresponding to the replaced display parameters.
2. The method according to claim 1, characterized in that, The method further includes: For non-first characteristic color pixels located within the color gamut of the display device, the display parameters are maintained.
3. The method according to claim 1 or 2, characterized in that, The step of determining the first feature color pixel of the target object in the converted image pixel data includes: Obtain the first training image; The pixel data of the target object in the first training image is first labeled, and the first detection network is trained based on the first labeled pixel data to obtain the first detection network model; The target object in the image to be processed is identified based on the first detection network model.
4. The method according to claim 3, characterized in that, The step of determining the first feature color pixel of the target object in the converted image pixel data includes: Obtain a second training image; the second training image includes the first feature color pixel data of the target object; The first feature color pixel data of the target object and the non-first feature color pixels in the second training image are labeled a second time. The second detection network is trained based on the second labeled pixel data to obtain the second detection network model. The first feature color pixel point of the region where the target object is located is identified based on the second detection network model.
5. The method according to claim 1 or 2, characterized in that, The step of finding the target color point with the smallest color difference from the target pixel point within the color gamut includes: For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius. Within the circle and the color gamut, search for color points, calculate the color difference between the target pixel and any color point, determine the search direction based on the magnitude relationship between two adjacent color differences, and search for the next color point in the determined search direction. When a color point with a color difference less than the first set threshold is found, and / or the number of searches reaches the second set threshold, the color point with the smallest color difference found is taken as the target color point.
6. The method according to claim 1 or 2, characterized in that, The step of finding the target color point with the smallest color difference from the target pixel point within the color gamut includes: For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius. Within the circle and the color gamut, traverse each color point, calculate the color difference between the target pixel and the color point, and take the color point with the smallest color difference as the target color point.
7. The method according to claim 1 or 2, characterized in that, The method further includes: The color gamut information of the image is determined through the header file of the image to be processed; and The color gamut range supported by the display device is determined by the display parameters configured on the display device.
8. The method according to claim 1 or 2, characterized in that, The step of outputting the image corresponding to the pixels after the display parameters have been replaced includes: The pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space, and then the pixels of the image are subjected to a second nonlinear mapping process to output the image processed by the second nonlinear mapping; wherein, the first nonlinear mapping is the inverse process of the second nonlinear mapping.
9. An image processing apparatus, characterized in that, The device includes: The mapping unit is used to perform a first non-linear mapping process on the pixels of the image to be processed; A conversion unit is used to convert the mapped pixels from a first color space to a second color space to obtain converted image pixel data; wherein, the second color space is a device-independent color space; The first determining unit is used to determine the first feature color pixel of the target object in the converted image pixel data, and the remaining pixels in the image are treated as non-first feature color pixels. The second determining unit is used to determine the target pixel points whose color gamut exceeds the color gamut range of the display device among the non-first feature color pixel points; The replacement unit is used to find the target color gamut point with the smallest color difference from the target pixel point outside the color gamut within the color gamut range, and replace the display parameters of the target pixel point outside the color gamut range with the display parameters of the target color gamut point. The holding unit is used to hold the display parameters for the first characteristic color pixel of the target object in the image; The output unit is used to output the image corresponding to the pixels after the display parameters have been replaced.
10. The apparatus according to claim 9, characterized in that, The holding unit is also used to hold display parameters for non-first characteristic color pixels located within the color gamut of the display device.
11. The apparatus according to claim 9 or 10, characterized in that, The second determining unit includes: The first acquisition subunit is used to acquire the first training image; The first training subunit is used to perform a first annotation on the pixel data of the target object in the first training image, and to train a first detection network based on the first annotated pixel data to obtain a first detection network model. The first identification subunit is used to identify the target object in the image to be processed based on the first detection network model.
12. The apparatus according to claim 11, characterized in that, The second determining unit further includes: The second acquisition subunit is used to acquire a second training image; the second training image includes the first feature color pixel data of the target object. The second training subunit is used to perform a second annotation on the first feature color pixel data of the target object and the non-first feature color pixels in the second training image, and to train the second detection network based on the second annotated pixel data to obtain the second detection network model. The second recognition subunit is used to identify the first feature color pixel point of the area where the target object is located based on the second detection network model.
13. The apparatus according to claim 9 or 10, characterized in that, The replacement unit is further configured to: For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius. Within the circle and the color gamut, search for color points, calculate the color difference between the target pixel and any color point, determine the search direction based on the magnitude relationship between two adjacent color differences, and search for the next color point in the determined search direction. When a color point with a color difference less than the first set threshold is found, and / or the number of searches reaches the second set threshold, the color point with the smallest color difference found is taken as the target color point.
14. The apparatus according to claim 9 or 10, characterized in that, The replacement unit is further configured to: For a target pixel that is outside the color gamut range, draw a perpendicular line to the color gamut boundary closest to the target pixel, and draw a circle with the foot of the perpendicular as the origin and the length of the line segment from the foot of the perpendicular to the target pixel as the radius. Within the circle and the color gamut, traverse each color point, calculate the color difference between the target pixel and the color point, and take the color point with the smallest color difference as the target color point.
15. The apparatus according to claim 9 or 10, characterized in that, The device further includes: The third determining unit is used to determine the color gamut information of the image based on the header file of the image to be processed; and The fourth determining unit is used to determine the color gamut range supported by the display device through the display parameters configured in the display device.
16. The apparatus according to claim 9 or 10, characterized in that, The output unit is also used for: The pixels of the image after the display parameters are replaced are converted from the second color space back to the first color space, and then the pixels of the image are subjected to a second nonlinear mapping process to output the image processed by the second nonlinear mapping; wherein, the first nonlinear mapping is the inverse process of the second nonlinear mapping.
17. An electronic device comprising a processor and a memory for storing processor-executable instructions, characterized in that, The processor is configured to perform the steps of the image processing method as described in any one of claims 1 to 8 when executing executable instructions in memory.
18. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the steps of the image processing method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Image processing apparatus and method
CN1643894A
Color mapping techniques for color imaging devices
US20080043260A1
