Image processing method and device, electronic equipment and storage medium
By semantic segmentation and color mapping of RGB images, the problem of insufficient color of electronic device cameras during image acquisition is solved, the bright color of image and color calibration accuracy is achieved, and the visual effect is improved.
Patent Information
- Application Number
- CN202411017593.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2025-07-25
AI Technical Summary
In the prior art, the electronic device camera has problems such as insufficient saturation, graying and light colors when collecting images, and color adjustments cannot be performed by region.
By acquiring the RGB image and its grayscale image for semantic segmentation, the semantic type of each pixel unit is determined, and the matching target 3D LUT table is obtained according to the semantic type, and the RGB value of the pixel unit is color mapped.
The image color is flexibly adjusted, making the image color more vivid, and the color correction results are more accurate, improving the visual effect of the image.
Smart Images
Figure CN120378756A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of image color enhancement and image processing, and particularly to an image processing method, apparatus, electronic device, and storage medium. Background Art
[0002] Due to the diversity of electronic device cameras, some cameras have problems such as insufficient saturation, grayish pictures, and dull colors when collecting images and videos. Color enhancement can be performed on the collected data to adjust the saturation characteristics of the image and video pictures, making the picture colors more rich and vivid, and enhancing the subjective visual experience of people. Summary of the Invention
[0003] The present disclosure provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product to at least solve the problem in the related art that images cannot be color-adjusted in regions. The technical solution of the present disclosure is as follows:
[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including: obtaining a first RGB image and a grayscale image of the first RGB image, and performing semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block; for each pixel unit, obtaining a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit; and performing color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image.
[0005] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including: a segmentation module, configured to obtain a first RGB image and a grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block; an obtaining module, configured to obtain a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit for each pixel unit; and a mapping module, configured to perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image.
[0006] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method according to the first aspect of the embodiments of the present disclosure.
[0007] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the method described in the first aspect of the embodiments of the present disclosure are implemented.
[0008] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, the steps of the method described in the first aspect of the embodiments of the present disclosure are implemented.
[0009] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: obtaining a first RGB image and a grayscale image of the first RGB image, and determining a semantic segmentation result of the grayscale image. Further, according to the semantic type of each pixel unit in the semantic segmentation result, a target 3DLUT table matching the pixel unit can be determined. Further, according to the target 3DLUT table, color mapping can be performed on the first RGB value of the pixel unit to obtain a second RGB image corresponding to the first RGB image. In this application, through semantic segmentation, the semantic types of different pixel units can be determined, and then the target 3D LUT tables corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, and the color calibration result more accurate, further improving the visual effect of the image.
[0010] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.
[0012] Figure 1 is a flowchart of an image processing method shown according to an exemplary embodiment.
[0013] Figure 2 is a flowchart of an image processing method shown according to another exemplary embodiment.
[0014] Figure 3 is a flowchart of an image processing method shown according to another exemplary embodiment.
[0015] Figure 4 is a flowchart of processing an image shown according to an exemplary embodiment.
[0016] Figure 5It is a flowchart of an image processing method shown according to another exemplary embodiment.
[0017] Figure 6 It is a flowchart of processing an image shown according to another exemplary embodiment.
[0018] Figure 7 It is a block diagram of an image processing apparatus shown according to an exemplary embodiment.
[0019] Figure 8 It is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners
[0020] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data used can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0022] In the technical solutions of the present disclosure, the acquisition, storage, use, processing, etc. of data all comply with the provisions of relevant laws and regulations.
[0023] The following describes the image processing method and apparatus of the embodiments of the present disclosure with reference to the drawings.
[0024] Figure 1 It is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 1 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0025] S101, obtain a first RGB image and a grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block.
[0026] It should be noted that the execution subject of the image processing method in the embodiments of the present disclosure is an electronic device such as a mobile phone, a notebook, a desktop computer, a vehicle-mounted terminal, a smart home appliance, a wearable device, etc. Among them, the wearable device may include a wrist-worn device (such as a smart watch, a smart bracelet), a head-mounted device, a foot-worn device, etc. The image processing method in the embodiments of the present disclosure may be executed by the image processing device in the embodiments of the present disclosure, and the image processing device in the embodiments of the present disclosure may be configured in any electronic device to execute the image processing method in the embodiments of the present disclosure.
[0027] In some implementations, the electronic device obtains perception data based on its own image sensor, performs operations such as denoising and color enhancement on the obtained perception data, and performs format conversion on the perception data to convert it into data in RGB format, that is, the first RGB image.
[0028] Exemplarily, the electronic device obtains perception data through an image sensor, such as a camera. The perception data is an image in RAW format, and the RAW image is denoised and image data is converted to convert the RAW image into an RGB image to obtain the first RGB image.
[0029] Further, by obtaining the gray values of the first RGB image in the red (Red) channel, the gray values in the green (Green) channel, and the gray values in the blue (Blue) channel respectively, and performing weighted calculation on each gray value, a gray image of the first RGB image is obtained.
[0030] Further, by obtaining a preset semantic type and performing semantic recognition on the gray image, the semantic type of each pixel unit is obtained, and then the gray image is segmented according to the semantic type of each pixel unit to obtain a semantic segmentation result.
[0031] Optionally, the semantic type may be determined according to the name of the object in the physical world in the image. For example, a cat, a dog, a plant, a person, etc. may be determined as the semantic type.
[0032] Optionally, based on a target detection neural network, according to a preset semantic type, semantic segmentation may be performed on the gray image to obtain a semantic segmentation result. For example, a region-based convolutional neural network (Mask Region-based Convolutional Neural Network, MASK-RCNN), a segmentation network (Segmentation Network, SegNet) may be used for semantic segmentation.
[0033] S102, for each pixel unit, obtain a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit.
[0034] Optionally, an association relationship table can be established in advance according to the association relationship between multiple semantic types and a set of three-dimensional look-up tables (3D LUTs), where each set of 3D LUTs includes multiple candidate 3D LUT tables.
[0035] In some implementations, after determining the semantic type of a pixel unit, the corresponding set of 3D LUTs for the pixel unit can be determined by querying the association relationship table. By performing interpolation calculations on the candidate 3D LUT tables in the set of 3D LUTs, a target 3D LUT table that matches the pixel unit can be obtained.
[0036] Optionally, two candidate 3D LUT tables can be determined from the set of 3D LUTs according to the parameter information of the pixel, and the weighting coefficients for interpolation can be determined according to the parameter information of the pixel. Then, according to the weighting coefficients, interpolation calculations are performed on the candidate 3D LUT tables to obtain the target 3D LUT table. Optionally, the parameter information of the pixel can be the global color temperature and the global illumination.
[0037] Optionally, the set of 3D LUTs can also be input into a neural network, and the neural network performs interpolation calculations on the candidate 3D LUT tables to obtain the target 3D LUT table.
[0038] S103, perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image.
[0039] In some implementations, the pixel unit can look up the first RGB value in the target 3D LUT table to determine the second RGB value corresponding to the first RGB value, and then update the first RGB image based on the second RGB value to obtain the second RGB image.
[0040] That is to say, by performing color mapping on the first RGB value of each pixel unit in the first RGB image, the second RGB value corresponding to each first RGB value is obtained, and then each pixel unit in the first RGB image is updated based on the second RGB value to obtain the second RGB image.
[0041] The image processing method provided by the embodiments of the present disclosure obtains a first RGB image and a grayscale image of the first RGB image, and determines the semantic segmentation result of the grayscale image. Furthermore, according to the semantic type of each pixel unit in the semantic segmentation result, the target 3D LUT table matched by the pixel unit can be determined. Further, according to the target 3D LUT table, the first RGB value of the pixel unit can be color-mapped to obtain a second RGB image corresponding to the first RGB image. In this application, through semantic segmentation, the semantic types of different pixel units can be determined, and then the target 3D LUT tables corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, and the color calibration result more accurate, further improving the visual effect of the image.
[0042] Figure 2 is a flowchart of an image processing method shown according to an exemplary embodiment, as Figure 2 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0043] S201, obtain a first RGB image and a grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block.
[0044] For the related content of step S201, reference can be made to the above embodiments, which will not be elaborated here.
[0045] S202, obtain a 3D LUT set associated with the semantic type, where the 3D LUT set includes multiple candidate 3D LUT tables.
[0046] In some implementations, an association relationship table can be established in advance according to the association relationship between different semantic types and the 3D LUT set. Among them, each 3D LUT set includes multiple candidate 3D LUT tables. Furthermore, after determining the semantic type of the pixel unit, by querying the association relationship table, the 3D LUT set associated with the semantic type can be determined.
[0047] S203, perform interpolation on the candidate 3D LUT table to obtain a target 3D LUT table.
[0048] In some implementations, the candidate 3D LUT table can be interpolated according to the color parameters of the first RGB image to obtain the target 3D LUT table. Among them, the color parameters can be the global color temperature and the global illumination. Optionally, while performing format conversion on the perceptual data to obtain the first RGB image, the global color temperature and the global illumination of the first RGB image can be obtained, and then the candidate 3D LUT table in the 3D LUT set can be interpolated according to the global color temperature and the global illumination to obtain the target 3D LUT table.
[0049] Optionally, the first candidate 3D LUT table and the second candidate 3D LUT table can be determined from the candidate 3D LUT tables in the 3D LUT set according to the first parameter in the global color temperature and the global illumination. For example, the global illumination can be used as the first parameter, and then from the candidate 3D LUT tables in the 3D LUT set, the candidate 3D LUT tables that meet the requirements of the global illumination and satisfy the color correction target are selected as the first candidate 3D LUT table and the second candidate 3D LUT table.
[0050] Furthermore, according to the second parameter in the global color temperature and the global illumination, the interpolation coefficients of the first candidate 3D LUT table and the second candidate 3D LUT table are determined, and based on the interpolation coefficients, the first candidate 3D LUT table and the second candidate 3D LUT table are interpolated to obtain the target 3D LUT table. For example, the global color temperature can be used as the second parameter, and the color temperature range in which the value of the global color temperature is located is determined, and then the interpolation coefficients can be determined according to the color temperature range.
[0051] Optionally, the formula for calculating the target 3D LUT table is as follows:
[0052] L out = Inter({L in} c , cct, lux) (1)
[0053] Among them, L out represents the target 3D LUT table, Inter represents the interpolation function, cct represents the global color temperature, lux represents the global illumination, L in represents the 3D LUT set, and c represents the semantic type.
[0054] S204, perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain the second RGB image corresponding to the first RGB image.
[0055] In some implementations, the second RGB value matching the first RGB value can be obtained by looking up in the target 3D LUT table. That is, using the first RGB value of the pixel unit as an index, look up in the target 3D LUT table to obtain the second RGB value that matches the first RGB value.
[0056] Furthermore, the second RGB image can be obtained according to the second RGB value of the pixel unit. That is, by looking up the second RGB value that matches the first RGB value of each pixel unit of the first RGB image and using the second RGB value to update the first RGB value of the first RGB image, the second RGB image can be obtained.
[0057] Furthermore, after obtaining the second RGB image, the second RGB image can be processed to obtain the target RGB image with vivid colors. For example, operations such as denoising and image enhancement can be performed on the second RGB image to obtain the target image.
[0058] The image processing method provided by the embodiments of the present disclosure obtains the first RGB image and the grayscale image of the first RGB image, and determines the semantic segmentation result of the grayscale image. Furthermore, according to the semantic type of each pixel unit in the semantic segmentation result, the 3D LUT set associated with the pixel unit can be determined, and the first candidate 3D LUT table and the second candidate 3D LUT table can be determined from the 3D LUT set. Furthermore, the target 3D LUT table can be determined according to the first candidate 3D LUT table and the second candidate 3D LUT table. Further, according to the target 3D LUT table, color mapping can be performed on the first RGB value of the pixel unit to obtain the second RGB image corresponding to the first RGB image. In this application, through semantic segmentation, the semantic types of different pixel units can be determined, and then the target 3D LUT tables corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, the color calibration result more accurate, and further improve the visual effect of the image.
[0059] Figure 3 is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 3 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0060] S301, obtain the first RGB image.
[0061] For the relevant content of step S301, reference can be made to the above embodiments, which will not be elaborated here.
[0062] S302. Extract the grayscale values of each pixel in different color channels, and weight the grayscale values of the color channels to obtain the initial grayscale value of the pixel, thereby obtaining the initial grayscale image.
[0063] In some implementations, the grayscale values of each pixel under the Red channel, the Green channel, and the Blue channel are respectively obtained, and then the grayscale values of each color channel are weighted and calculated to obtain the initial grayscale image.
[0064] Optionally, the formula for calculating the initial grayscale image is as follows:
[0065]
[0066] where represents the initial grayscale image, represents the grayscale value under the Red channel, represents the grayscale value under the Green channel, represents the grayscale value under the Blue channel.
[0067] S303. Perform interpolation processing and downsampling on the initial grayscale image to obtain the grayscale image of the first RGB image.
[0068] In some implementations, in order to further reduce the computational complexity and the amount of data transmitted, so as to improve the processing efficiency, interpolation processing and downsampling can be performed on the initial grayscale image, and then the grayscale image of the first RGB image can be obtained. For example, interpolation calculations can be performed on the initial grayscale image according to its calculation accuracy. For instance, the grayscale image can be interpolated into grayscale images of three scales: 128*128, 256*256, and 512*512. S304. Send the grayscale image to the semantic segmentation NPU model, and the NPU model performs semantic segmentation on the grayscale image to obtain the semantic segmentation result.
[0069] In some implementations, by transmitting the grayscale image to the semantic segmentation (Neural Processing Unit, NPU) model, the NPU model performs semantic segmentation on the grayscale image based on the semantic segmentation algorithm to obtain the semantic segmentation result.
[0070] Optionally, network structures such as MASK-RCNN and SegNet can be used to perform semantic segmentation on the grayscale image. For example, assuming the grayscale image is it can be segmented into the corresponding semantic segmentation result S pre . S pre is integer data of size H×W×C, where H and W are the length and width, and C is the semantic type.
[0071] S305. Receive the compressed package sent by the NPU model and decode the compressed package to obtain the semantic segmentation result.
[0072] In some implementations, to reduce the amount of data transmitted, the NPU model compresses the semantic segmentation result and transmits the compressed package. That is, by receiving the compressed package sent by the NPU model, the semantic segmentation result is obtained from the compressed package.
[0073] In some implementations, during the compression process, the NPU model can use intra-frame coding technology to compress the semantic segmentation result. Optionally, by performing block processing on the semantic segmentation result and performing predictive coding on the blocks, a compressed package of the semantic segmentation result is obtained.
[0074] That is, by receiving the block predictive coding result and residual information of the semantic segmentation result sent by the NPU model, and based on the block predictive coding result and residual information, the compressed package is decoded to obtain the semantic segmentation result.
[0075] In some implementations, to make the size of the semantic segmentation result the same as that of the grayscale image, after obtaining the semantic segmentation result, the first size of the first RGB image and the second size of the semantic segmentation result can be determined, and whether the first size and the second size are the same is compared.
[0076] Optionally, in response to the first size and the second size being different, and according to the first size, the second size of the semantic segmentation result is interpolated and adjusted.
[0077] S306. For each pixel unit, obtain the target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit.
[0078] S307. Perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain the second RGB image corresponding to the first RGB image.
[0079] For the relevant content of steps S306 - S307, refer to the above embodiments and will not be elaborated here.
[0080] The image processing method provided by the embodiments of the present disclosure obtains a first RGB image and a grayscale image of the first RGB image, and sends the grayscale image to an NPU model. The NPU model performs semantic segmentation on the grayscale image to obtain a semantic segmentation result. Further, according to the semantic type of each pixel unit in the semantic segmentation result, the target 3D LUT table matched by the pixel unit can be determined. Further, according to the target 3D LUT table, color mapping is performed on the first RGB value of the pixel unit to obtain a second RGB image corresponding to the first RGB image. In this application, the semantic type of different pixel units can be determined through semantic segmentation, and then the target 3D LUT table corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, the color calibration result more accurate, and further improve the visual effect of the image.
[0081] Figure 4 The figure shows a flowchart for processing an image. Figure 4 It includes a preamble module, a 3D LUT module, a semantic segmentation module, and a subsequent module. Based on the preamble module, a first RGB image and a grayscale image of the first RGB image can be obtained, and then the grayscale image is transmitted to the semantic segmentation module. The NPU model performs semantic segmentation on the grayscale image to obtain a semantic segmentation result. Further, the semantic segmentation result and the first RGB image are input into the 3D LUT module. In the 3D LUT module, the target 3D LUT table is determined according to the semantic segmentation result, and color mapping is performed on the first RGB value of each pixel unit of the first RGB image based on the target 3D LUT table to obtain a second RGB image. Further, the second RGB image is passed into the subsequent module, and operations such as denoising and image enhancement are performed on the second RGB image to obtain a target image.
[0082] Figure 5 is a flowchart of an image processing method shown according to an exemplary embodiment. As Figure 5 shown, the image processing method of the embodiments of the present disclosure includes the following steps:
[0083] S501, obtain a first RGB image.
[0084] S502, extract the grayscale values of each pixel in different color channels, and weight the grayscale values of the color channels to obtain the initial grayscale value of the pixel, and obtain the initial grayscale image.
[0085] S503, perform interpolation processing and downsampling on the initial grayscale image to obtain the grayscale image of the first RGB image.
[0086] S504. Send the grayscale image to the NPU model, and the NPU model performs semantic segmentation on the grayscale image to obtain the semantic segmentation result.
[0087] S505. Receive the compressed package sent by the NPU model and decode the compressed package to obtain the semantic segmentation result.
[0088] S506. Obtain the 3D LUT set associated with the semantic type. The 3D LUT set includes multiple candidate 3D LUT tables.
[0089] S507. Interpolate the candidate 3D LUT tables to obtain the target 3D LUT table.
[0090] S508. Use the first RGB value of the pixel unit as an index to look up in the target 3D LUT table to obtain the second RGB value that matches the first RGB value.
[0091] S509. Obtain the second RGB image according to the second RGB value of the pixel unit.
[0092] For the relevant content of steps S501 - S509, refer to the above embodiments and will not be elaborated here.
[0093] The image processing method provided by the embodiments of the present disclosure obtains the first RGB image and the grayscale image of the first RGB image, and determines the semantic segmentation result of the grayscale image. Furthermore, according to the semantic type of each pixel unit in the semantic segmentation result, the target 3D LUT table matched by the pixel unit can be determined. Further, according to the target 3D LUT table, color mapping can be performed on the first RGB value of the pixel unit to obtain the second RGB image corresponding to the first RGB image. In this application, through semantic segmentation, the semantic types of different pixel units can be determined, and then the target 3D LUT tables corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, the color correction result more accurate, and further improve the visual effect of the image.
[0094] Figure 6 The figure shows a flowchart for processing an image. Figure 6It includes an NPU side and an Image Signal Processor (ISP) side. The ISP side obtains the first RGB image, calculates the initial grayscale image corresponding to the first RGB image according to the grayscale value of the first RGB image, and performs interpolation processing and downsampling on the initial grayscale image to obtain a grayscale image with a lower resolution. Further, the grayscale image is sent to the NPU side, and the NPU model is used to perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, and the semantic segmentation result is compressed to obtain a compressed package corresponding to the semantic segmentation result.
[0095] Further, the NPU side transmits the compressed package to the ISP side. The ISP side decodes the compressed package to obtain the semantic segmentation result, determines the semantic type of each pixel unit according to the semantic segmentation result, and then can determine the target 3D LUT table matched by the pixel unit according to the semantic type, and uses the target 3D LUT table to perform color adjustment on the first RGB image to obtain the second RGB image.
[0096] Figure 7 It is a block diagram of an image processing device shown according to an exemplary embodiment. Refer to Figure 7 In the image processing device 700 according to the embodiment of the present disclosure, it includes:
[0097] A segmentation module 701, configured to obtain the first RGB image and the grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block;
[0098] An acquisition module 702, configured to, for each pixel unit, obtain a target three-dimensional lookup 3D LUT table matched with the pixel unit according to the semantic type of the pixel unit;
[0099] A mapping module 703, configured to perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain the second RGB image corresponding to the first RGB image.
[0100] In an embodiment of the present disclosure, the mapping module 703 is further configured to: use the first RGB value of the pixel unit as an index to perform a lookup in the target 3D LUT table to obtain a second RGB value matched with the first RGB value; and obtain the second RGB image according to the second RGB value of the pixel unit.
[0101] In one embodiment of the present disclosure, the obtaining module 702 is further configured to: obtain a set of 3D LUTs associated with the semantic type, where the set of 3D LUTs includes a plurality of candidate 3D LUT tables; perform interpolation on the candidate 3D LUT tables to obtain the target 3D LUT table.
[0102] In one embodiment of the present disclosure, the obtaining module 702 is further configured to: obtain the global color temperature and global illumination of the first RGB image; perform interpolation on the candidate 3D LUT tables in the set of 3D LUTs according to the global color temperature and global illumination to obtain the target 3D LUT table.
[0103] In one embodiment of the present disclosure, the obtaining module 702 is further configured to: determine a first candidate 3D LUT table and a second candidate 3D LUT table from the candidate 3D LUT tables in the set of 3D LUTs according to a first parameter in the global color temperature and global illumination; determine the interpolation coefficients of the first candidate 3D LUT table and the second candidate 3D LUT table according to a second parameter in the global color temperature and global illumination; perform interpolation on the first candidate 3D LUT table and the second candidate 3D LUT table based on the interpolation coefficients to obtain the target 3D LUT table.
[0104] In one embodiment of the present disclosure, the segmentation module 701 is further configured to: extract the gray values of each pixel in different color channels, and weight the gray values of the color channels to obtain the initial gray value of the pixel, thereby obtaining an initial gray image; perform interpolation processing and downsampling on the initial gray image to obtain the gray image of the first RGB image.
[0105] In one embodiment of the present disclosure, the segmentation module 701 is further configured to: send the gray image to a semantic segmentation NPU model, and the NPU model performs semantic segmentation on the gray image to obtain the semantic segmentation result; receive a compressed package sent by the NPU model and decode the compressed package to obtain the semantic segmentation result.
[0106] In one embodiment of the present disclosure, the segmentation module 701 is further configured to: receive the block prediction coding result and residual information of the semantic segmentation result sent by the NPU model; decode the compressed package based on the block prediction coding result and residual information to obtain the semantic segmentation result.
[0107] In one embodiment of the present disclosure, the segmentation module 701 is further configured to: determine a first size of the first RGB image and a second size of the semantic segmentation result; and in response to the first size and the second size being different, and based on the first size, perform interpolation adjustment on the second size of the semantic segmentation result.
[0108] Regarding the device in the above embodiment, the specific manners in which each module performs operations have been described in detail in the embodiment related to the method, and will not be elaborated herein.
[0109] The image processing device provided by the embodiment of the present disclosure acquires a first RGB image and a grayscale image of the first RGB image, and determines a semantic segmentation result of the grayscale image. Further, according to the semantic type of each pixel unit in the semantic segmentation result, a target 3D LUT table matched by the pixel unit can be determined. Further, according to the target 3D LUT table, color mapping can be performed on the first RGB value of the pixel unit to obtain a second RGB image corresponding to the first RGB image. In this application, the semantic type of different pixel units can be determined through semantic segmentation, and then the target 3D LUT table corresponding to different semantic types can be determined, so as to perform color adjustment on different pixel units according to different target 3D LUT tables, so as to flexibly adjust the image color, make the image color more vivid, the color calibration result more accurate, and further improve the visual effect of the image.
[0110] Figure 8 is a block diagram of an electronic device shown according to an exemplary embodiment.
[0111] As Figure 8 shown, the above electronic device 800 includes:
[0112] A memory 801 and a processor 802, a bus 803 connecting different components (including the memory 801 and the processor 802), and the memory 801 stores a computer program, and when the processor 802 executes the program, the image processing method described in the embodiment of the present disclosure is implemented.
[0113] The bus 803 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in a variety of bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
[0114] The electronic device 800 typically includes a variety of electronically readable media. These media can be any available media accessible by the electronic device 800, including volatile and non-volatile media, removable and non-removable media.
[0115] The memory 801 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 804 and / or cache memory 805. The electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 806 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 8 not shown, commonly referred to as a "hard disk drive"). Although Figure 8 not shown in the figure, a disk drive for reading and writing on a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing on a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to the bus 803 through one or more data media interfaces. The memory 801 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present disclosure.
[0116] A program / utility 808 having a set (at least one) of program modules 807 can be stored, for example, in the memory 801. Such program modules 807 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 807 generally execute the functions and / or methods in the embodiments described in the present disclosure.
[0117] The electronic device 800 can also communicate with one or more external devices 809 (such as a keyboard, a pointing device, a display 891, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 892. Moreover, the electronic device 800 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through the network adapter 893. As Figure 8As shown, network adapter 893 communicates with other modules of electronic device 800 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0118] Processor 802 executes various functional applications and data processing by running programs stored in memory 801.
[0119] It should be noted that for the implementation process and technical principle of the electronic device in this embodiment, refer to the foregoing explanation of the image processing method of the present disclosure, which will not be elaborated here.
[0120] To implement the above embodiments, the present disclosure also proposes a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the image processing method provided by the present disclosure are implemented.
[0121] Optionally, the computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.
[0122] To implement the above embodiments, the present disclosure also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, the image processing method as described above is implemented.
[0123] Those skilled in the art will readily conceive of other implementations of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and embodiments are only to be regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.
[0124] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.
Claims
1. An image processing method, characterized in that, The method includes: Obtain a first RGB image and a grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block; For each pixel unit, obtain a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit; Perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image.
2. The method according to claim 1, characterized in that, The performing color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image includes: Use the first RGB value of the pixel unit as an index to look up in the target 3D LUT table to obtain a second RGB value that matches the first RGB value; Obtain the second RGB image according to the second RGB value of the pixel unit.
3. The method according to claim 1, wherein The obtaining a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit includes: Obtain a 3D LUT set associated with the semantic type, where the 3D LUT set includes multiple candidate 3D LUT tables; Interpolate the candidate 3D LUT tables to obtain the target 3D LUT table.
4. The method according to claim 3, wherein The interpolating the candidate 3D LUT tables to obtain the target 3D LUT table includes: Obtain the global color temperature and global illumination of the first RGB image; Interpolate the candidate 3D LUT tables in the 3D LUT set according to the global color temperature and global illumination to obtain the target 3D LUT table.
5. The method according to claim 4, wherein The interpolating the candidate 3D LUT tables in the 3D LUT set according to the global color temperature and global illumination to obtain the target 3D LUT table includes: Determine a first candidate 3D LUT table and a second candidate 3D LUT table from the candidate 3D LUT tables in the 3D LUT set according to a first parameter in the global color temperature and global illumination; Determine the interpolation coefficients of the first candidate 3D LUT table and the second candidate 3D LUT table according to a second parameter in the global color temperature and global illumination; Interpolate the first candidate 3D LUT table and the second candidate 3D LUT table based on the interpolation coefficients to obtain the target 3D LUT table.
6. The method according to claim 1, wherein Obtaining the grayscale image of the first RGB image includes: Extract the grayscale values of each pixel in different color channels, and weight the grayscale values of the color channels to obtain the initial grayscale value of the pixel, and obtain an initial grayscale image; Perform interpolation processing and downsampling on the initial grayscale image to obtain the grayscale image of the first RGB image.
7. The method according to claim 1, characterized in that The performing semantic segmentation on the grayscale image to obtain a semantic segmentation result includes: Send the grayscale image to the semantic segmentation NPU model, and perform semantic segmentation on the grayscale image by the NPU model to obtain the semantic segmentation result; Receive the compressed package sent by the NPU model, and decode the compressed package to obtain the semantic segmentation result.
8. The method according to claim 7, wherein The decoding the compressed package to obtain the semantic segmentation result includes: Receive the block prediction coding result and residual information of the semantic segmentation result sent by the NPU model; Based on the block prediction coding result and residual information, decode the compressed package to obtain the semantic segmentation result.
9. The method according to claim 7, characterized in that, After obtaining the semantic segmentation result, it further includes: Determine the first size of the first RGB image and the second size of the semantic segmentation result; In response to the first size and the second size being different, and according to the first size, perform interpolation adjustment on the second size of the semantic segmentation result.
10. An image processing apparatus, characterized in that, The device includes: A segmentation module, configured to obtain a first RGB image and a grayscale image of the first RGB image, and perform semantic segmentation on the grayscale image to obtain a semantic segmentation result, where the semantic segmentation result includes the semantic type of each pixel unit, and the pixel unit is a single pixel or a pixel block; An acquisition module, configured to obtain a target three-dimensional lookup 3D LUT table that matches the pixel unit according to the semantic type of the pixel unit for each pixel unit; A mapping module, configured to perform color mapping on the first RGB value of the pixel unit based on the target 3D LUT table to obtain a second RGB image corresponding to the first RGB image.
11. An electronic device, characterized in that, It includes: A processor; A memory for storing processor-executable instructions; Wherein, the processor is configured to: Implement the steps of the method according to any one of claims 1-9.
12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instruction is executed by the processor, the steps of the method according to any one of claims 1-9 are implemented.