A neural template based image correction method and system

Abstract

An apparatus and method for performing image correction may include: receiving an input image to be corrected; generating a plurality of template images based on the input image, wherein each of the plurality of template images is generated based on at least one of image feature transformations; providing the plurality of template images to an artificial intelligence (AI) module; using the AI ​​module, assigning zero-amplitude and non-zero-amplitude weights to each template image based on the degree and type of correction to be applied to each template image; and applying the assigned weights to the plurality of template images to obtain a corrected output image.

Description

Technical Field

[0001] This invention relates to image processing, and more specifically, to an image correction method and system based on neural templates. Background Technology

[0002] Automatic white balance (AWB) technology can involve setting the white point of an image to match the color temperature of the ambient light illuminating a scene included in the image, without receiving input from the user, thereby ensuring the resulting image looks like the scene. Some AWB techniques may involve scene light source estimation and color balance correction of the image. Automatic exposure technology can involve automatically adjusting the image brightness based on the amount of light reaching the camera sensor.

[0003] Some methods of AWB may involve computing a set of transforms from the input image. Then, nearest neighbor matching can be performed on each pixel to perform AWB in the input image. However, when the number of neighboring pixels is relatively large, nearest neighbor matching methods can be time-consuming, slowing down the AWB process. Furthermore, the computations involved in nearest neighbor matching methods may require a large number of training images, which can lead to increased memory requirements. Therefore, these AWB methods may not be scalable.

[0004] Furthermore, some AWB methods may consider each pixel in the image independently, and the output of each template may depend only on the intensity of the values ​​across channels for the same pixel. Additionally, some AWB methods may not use image attributes such as race or hair texture, or background category attributes such as green or brown grass, which could be important clues when performing AWB.

[0005] Therefore, there is a need for a method and system to overcome these problems by providing a memory-efficient and time-efficient method and system for image correction.

[0006] The above information is intended only to enhance the understanding of the general context of this disclosure and is not intended to acknowledge or suggest in any way that the information is prior art known to those skilled in the art. Summary of the Invention

[0007] According to one aspect of this disclosure, a method for performing image correction is provided. The method may include receiving an input image to be corrected. The method may include generating a plurality of template images based on the input image, wherein each of the plurality of template images is generated based on at least one of image feature transformations. The method may include providing the plurality of template images to an artificial intelligence (AI) module. The method may include using the AI ​​module to assign weights in zero and non-zero amplitudes to each template image based on the degree and type of correction to be applied to each template image. The method may include applying the assigned weights to the plurality of template images to obtain a corrected output image.

[0008] According to one aspect of this disclosure, an apparatus for performing image correction is provided. The apparatus may include at least one processor. The apparatus may include a memory configured to store instructions that, when executed by the at least one processor, cause the apparatus to receive an input image to be corrected. The at least one processor may cause the apparatus to generate a plurality of template images based on the input image, wherein each of the plurality of template images is generated based on at least one of image feature transformations. The at least one processor may cause the apparatus to provide the plurality of template images to an artificial intelligence (AI) module. The at least one processor may cause the apparatus to use the AI ​​module to assign weights in zero and non-zero amplitudes to each template image based on the degree and type of correction to be applied to each template image. The at least one processor may cause the apparatus to apply the assigned weights to the plurality of template images to obtain a corrected output image.

[0009] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores one or more instructions, wherein the instructions, when executed by at least one processor of the device, cause the device to perform operations. These operations may include receiving an input image to be corrected. These operations may include generating a plurality of template images based on the input image, wherein each of the plurality of template images is generated based on at least one of image feature transformations. These operations may include providing the plurality of template images to an artificial intelligence (AI) module. These operations may include using the AI ​​module to assign weights in zero and non-zero amplitudes to each template image based on the degree and type of correction to be applied to each template image. These operations may include applying the assigned weights to the plurality of template images to obtain a corrected output image.

[0010] According to one aspect of this disclosure, a computer-readable storage medium storing instructions is provided. When executed by at least one processor, these instructions can cause the at least one processor to perform a corresponding method.

[0011] The foregoing overview is illustrative only and is not intended to be limiting in any way. Other aspects, embodiments, and features will become apparent from the accompanying drawings and the following detailed description, in addition to the illustrative aspects, embodiments, and features described above. Attached Figure Description

[0012] The above and other aspects, features, and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 An environment for image correction according to an embodiment of the present disclosure is shown;

[0014] Figure 2 A block diagram of a data flow for image correction according to an embodiment of the present disclosure is shown;

[0015] Figure 3A Template image generation based on color kernel transformation according to embodiments of the present disclosure is illustrated;

[0016] Figure 3B Template image generation based on attribute kernel transformation according to embodiments of the present disclosure is illustrated;

[0017] Figure 3C Template image generation based on shift-based pixel kernel transformation according to embodiments of the present disclosure is illustrated;

[0018] Figure 4 A training environment for training an AI model / neural network according to an embodiment of the present disclosure is shown;

[0019] Figure 5 Image correction based on a trained AI model / neural network according to embodiments of the present disclosure is illustrated;

[0020] Figure 6 A block diagram of a system for image correction according to an embodiment of the present disclosure is shown; and

[0021] Figure 7 A flowchart of a method for image correction according to an embodiment of the present disclosure is shown.

[0022] Those skilled in the art will understand that any block diagram herein represents a conceptual view of an illustrative system embodying the principles of the subject matter. Similarly, it should be understood that any flowchart, diagrammatic representation, state transition diagram, pseudocode, etc., represents various processes that can be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown. Detailed Implementation

[0023] In this disclosure, the term "exemplary" is intended to mean "used as an example, instance, or illustration." Any embodiment or implementation of the subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0024] Although specific embodiments of this disclosure are described herein and illustrated by way of example in the accompanying drawings, this disclosure is not intended to be limited to the particular forms disclosed, but is intended to cover a variety of modifications, equivalents and alternatives that fall within the spirit and scope of this disclosure.

[0025] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a setup, device, or method that includes a list of components or steps may include not only those components or steps but also other components or steps not expressly listed or inherent to such setup, device, or method. In other words, without further constraints, the inclusion of one or more elements in a device, system, or apparatus that begins with “comprising…a” does not exclude the presence of other elements or additional elements in that device, system, or apparatus.

[0026] In the following detailed description of embodiments of this disclosure, reference is made to the accompanying drawings, which form part of this disclosure, and embodiments in which this disclosure can be practiced are illustrated by way of example. The embodiments have been described in sufficient detail to enable those skilled in the art to practice this disclosure, and it should be understood that other embodiments may be utilized and changes may be made without departing from the scope of this disclosure. Therefore, the following description should not be considered limiting.

[0027] It should be understood that the boxes and combinations of flowcharts in each flowchart can be executed by one or more computer programs including instructions. The entirety of one or more computer programs can be stored on a single storage device, or the one or more computer programs can be divided into different parts stored on multiple different storage devices.

[0028] Any function or operation described herein can be processed by one or a combination of processors. This one or a combination of processors is the circuitry that performs the processing and includes, for example, an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, Bluetooth, etc. ®Circuits including chips, GPS chips, NFC chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio CODEC chips, Universal Serial Bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), and other ICs.

[0029] A processor may include various processing circuits and / or multiple processors. For example, as used herein, including in the claims, the term "processor" may include various processing circuits, including at least one processor, wherein one or more of the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform multiple functions, these terms cover, for example, but not limited to, a situation where one processor performs some of the stated functions while another(s) processor(s) perform other stated functions, and a situation where a single processor can perform all of the stated functions. Furthermore, the at least one processor may include, for example, a combination of processors performing various stated / disclosed functions in a distributed manner. The at least one processor may execute program instructions to implement or perform various functions.

[0030] The terms "artificial intelligence (AI) model" and "neural network" are used interchangeably throughout this disclosure. According to one embodiment, an AI module can be used to implement the AI ​​model and / or neural network. The AI ​​module may be or may include a combination of hardware and software modules. The hardware module may include circuitry for performing the functions discussed in the embodiments described herein.

[0031] Figure 1 An environment 100 for image correction according to an embodiment of the present disclosure is shown.

[0032] Environment 100 may include an image sensor 101, a template generation unit 103, a preprocessing unit 105, an AI module 107, a correction unit 109, and an image output unit 111. In an embodiment, AI module 107 may be, or may include, for example, a neural network module. In an embodiment, AI module 107 may include at least one of an AI model and a neural network.

[0033] Image sensor 101 can be configured to capture images. Template generation unit 103 can be configured to apply at least one of image feature transformations. Image feature transformations may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel transformation on the captured image to generate multiple template images. (Refer to the following...) Figures 3A-3CExamples of color kernel transformation, attribute kernel transformation, and shifted pixel transformation are discussed in more detail.

[0034] The preprocessing unit 105 can be configured to perform preprocessing operations on the captured image for the AI ​​module 107 and send the preprocessed captured image to the AI ​​module 107. The AI ​​module 107 can be configured to estimate corrections to be applied to the multiple generated template images. The correction unit 109 can be configured to apply the estimated corrections to the multiple generated template images and forward the corrected images to the image output unit 111. The image output unit 111 can be configured to display the corrected images to a user or otherwise output the corrected images.

[0035] Figure 2 A block diagram of the data flow for performing an image correction process 200 according to an embodiment of the present disclosure is shown.

[0036] In operation 201, an input image captured by an image sensor can be input for image correction. The input image frame can be a digital image with values ​​ranging from 0 to 255, and can have a height H and a width W. The input image frame can be represented using a three-channel matrix (e.g., corresponding to the blue channel B, green channel G, and red channel R of a blue-green-red (BGR) image). However, embodiments are not limited to this; in embodiments, a three-channel matrix corresponding to the red channel R, green channel G, and blue channel B of a red-green-blue (RGB) image can be used to represent the input image frame. The input image frame can be represented with a shape of (H, W, 3).

[0037] In operation 203, the BGR image can be segmented into three single-channel images (e.g., BGR images). In operation 205, multiple template images can then be generated using the three-channel images, which can be stacked in shapes H, W, N, where N refers to the number of template images. In operation 207, the input image can be preprocessed before being passed to the AI ​​module 107.

[0038] In operation 209, AI module 107 can perform a correction estimate for each color contribution component of each of the multiple template images. This correction estimate can be generated in the form of a transformation matrix having weights for each color contribution component of each of the multiple template images. In operation 211, the AI ​​module can output a transformation matrix of shape (3, N).

[0039] In operation 213, matrix multiplication between the transformation matrix and the template image can be performed. In operation 215, the corrected BGR image with shape (H, W, 3) can be output to the user. Therefore, image correction performed using multiple template images can contribute to memory-efficient and time-efficient image correction.

[0040] Figure 3A Template image generation based on color kernel transformation 300a according to an embodiment of the present disclosure is illustrated.

[0041] The color kernel transform can receive an input image 301 comprising a three-channel BGR image of shape (H, W, 3). The color kernel transform may include template initialization according to Equations 1, 2, and 3 below:

[0042] B = I[:,:,0] / 255 Equation 1

[0043] G = I[:,:,1] / 255 Equation 2

[0044] R = I[:,:,2] / 255 Equation 3

[0045] Based on the above initialization, three component images can be generated, and they can be scaled by dividing each component image by the maximum value of 255. Multiple template images can be generated based on the component images.

[0046] exist Figure 3A In the example shown, multiple template images may include eleven template images. The template image may include one or more component images. For example, the eleven template images can be defined according to Equations 4 through 14 below:

[0047] T1 = Equation 4

[0048] T2 = G (Equation 5)

[0049] T3 = R (Equation 6)

[0050] T4 = G Equation 7

[0051] T5 = R Equation B, 8

[0052] T6 = B Equation 9

[0053] T7 = R Equation 10

[0054] T8 = G Equation G, 11

[0055] T9 = B Equation B, 12

[0056] T 10 = R G Equation B, 13

[0057] T 11 = 1 Equation 14

[0058] However, the number of template images is not limited to eleven, and its number can be increased or decreased to improve the efficiency of image correction. In this embodiment, having an equal number of template images (eleven) can contribute to optimal image correction.

[0059] exist Figure 3A In this example, the template image is represented as a BGR matrix in color format. However, the embodiments are not limited to this, and during computation, the template image can be flattened into an array format with values ​​scaled between 0 and 1 for normalization purposes.

[0060] Figure 3B Template image generation based on attribute kernel transformation 300b according to embodiments of the present disclosure is illustrated. In embodiments of the present disclosure, a semantic model can be used to extract semantic details. In embodiments, the semantic model can be at least one of a pre-trained machine learning (ML) model, a pre-trained AI model, and a pre-trained neural network. The extracted semantic details can be used to generate one or more template images.

[0061] like Figure 3B As shown, input image 302a can be fed to a semantic model, which can be trained for attribute prediction. The semantic model can understand semantic segmentation regions and attribute information (e.g., sunny sky, cloudy sky / black hair / Asian face / brown grass). For example, the semantic model can identify attributes corresponding to sky 310a, hair 310b, face 310c, and grass 310d in input image 302a, and can generate template image 302b based on the identified attributes.

[0062] Figure 3C Template image generation based on a shifted pixel kernel transform 300c according to an embodiment of the present disclosure is illustrated. In the embodiment, a template image can be generated by applying a shifted pixel kernel transform. Pixel shifting operations can be used to shift pixels to the right, up, and left. Pixel shifting operations can stack pixels together to obtain neighborhood pixel information in the template image generation. Each channel of the template image can have a pixel value corresponding to one pixel and four neighboring pixels. The template image can include information about spatial correlation and the correlation between different channels, which can be used for correction.

[0063] In embodiments, multiple template images can be generated based on at least one of image feature transformations. Image feature transformations may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation. In embodiments, a combination of color kernel transformation and attribute kernel transformation can be used to generate multiple template images. In embodiments, a combination of color kernel transformation and shifted pixel kernel transformation can be used to generate multiple template images. In embodiments, a combination of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation can be used to generate multiple template images.

[0064] Figure 4 A training environment 400 for training at least one of an AI model and a neural network according to an embodiment of the present disclosure is shown.

[0065] In embodiments of this disclosure, the training environment 400 may include an AI model 407 to be trained to generate accurate image correction parameters. In embodiments, the AI ​​model 407 may be or may include a neural network, but embodiments are not limited thereto. An input image 401 may be provided in parallel to the AI ​​model 407 and the kernel transformation unit 403. In embodiments, the AI ​​model 407 may correspond to the AI ​​module 107, and the kernel transformation unit 403 may correspond to the template generation unit 103, but embodiments are not limited thereto. In embodiments, the kernel transformation unit 403 may apply a color kernel transformation and may generate an image with three color channels (e.g., as shown in the image). Figure 4 Multiple template images with different combinations of B, G, and R (as shown). For example, kernel transform unit 403 can output a template image corresponding to the blue channel B. Template image corresponding to the green channel G Template image corresponding to the red channel R Template image corresponding to the combination of green channel G and red channel R And the template image corresponding to the combination of blue channel B and red channel R. However, the embodiments are not limited to this. However, the kernel transformation unit 403 is not limited to color kernel transformation, and may include, but is not limited to, one or more of attribute kernel transformation, shifted pixel kernel transformation, and color kernel transformation.

[0066] AI model 407 can be configured to generate features for each template image corresponding to weights selected from zero and non-zero amplitudes, based on the degree and type of correction required in each template image. In an embodiment, weights can be assigned in the form of a transformation matrix by assigning weights to each color contribution component of each of the multiple template images.

[0067] In the example, the input image can be a three-channel input image, and the number of template images can be N. Therefore, the size of the transformation matrix can be "3xN". The transformation matrix can be applied to multiple template images using matrix multiplication 405. Matrix multiplication 405 can generate the predicted image 409.

[0068] In one embodiment, the predicted image 409 can be compared with the true image 411 (which may be, for example, an actual image) to perform loss calculation. The training loss determined by the loss calculation can be used as feedback to the AI ​​model 407. The AI ​​model 407 can be configured to update the previously generated weight / transformation matrix based on this training loss. In one embodiment, transformation matrix generation, matrix multiplication, and loss calculation can be repeated until the training loss decreases below a training loss threshold. In another embodiment, transformation matrix generation, matrix multiplication, and loss calculation can be repeated to minimize the training loss or to make the value of the training loss relatively close to zero.

[0069] Therefore, training the AI ​​model 407 does not require a large number of training images to predict the corrected images, thereby reducing the overall memory requirements for training and making the above solution scalable. Furthermore, training the AI ​​model 407 can be performed in a time-saving manner. Additionally, the resources used to train the AI ​​model 407 can be reduced, making it possible to train it using devices with limited resources (e.g., mid- and low-level devices with fewer resources than high-level devices).

[0070] Figure 5 Image correction based on a trained AI model 407 according to an embodiment of the present disclosure is illustrated.

[0071] In embodiments of this disclosure, an input image 401 may be provided to a kernel transformation unit 403. The kernel transformation unit 403 may be configured to apply one or more image feature transformations. Image feature transformations may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation to generate an image as described above. Figures 3A-3C The multiple template images discussed.

[0072] The input image 401 can be provided to the trained AI model 407. The trained AI model 407 can be configured to generate or assign weights selected from zero and non-zero amplitudes to each template image based on the degree and type of correction required for each of the multiple template images generated by the kernel transformation unit 403.

[0073] The assigned weights can be provided as a transformation matrix, where weights are assigned to each color contribution component of each of the multiple template images. Matrix multiplication 405 can be performed between the template images and the transformation matrix to generate a BGR color correction channel 413. The BGR color correction channel 413 can be combined to form a color-corrected image 415, for example, which could be an image with proper white balance.

[0074] exist Figure 5 In the example shown, the template image may include a template image corresponding to the blue channel B. Template image corresponding to the green channel G Template image corresponding to the red channel R Template image corresponding to the combination of green channel G and red channel R And the template image corresponding to the combination of blue channel B and red channel R. .

[0075] In an embodiment, AI model 407 can generate images including template images. arrive Corresponding weight vector arrive The transformation matrix. In the embodiment, the weight vector... arrive This can be represented by equations 15 to 19 below.

[0076] Equation 15 can be used to represent the template image. Corresponding weight vector The template image can be represented according to Equation 16. Corresponding weight vector The template image can be represented according to Equation 17. Corresponding weight vector The template image can be represented according to Equation 18. Corresponding weight vector And it can be represented by Equation 19 with respect to the template image. Corresponding weight vector .

[0077] Equation 15

[0078] Equation 16

[0079] Equation 17

[0080] Equation 18

[0081] Equation 19

[0082] As shown above, template image It can correspond to weights Where 0.55 is the final weight contribution of the blue channel, 0.014 is the final weight contribution of the green channel, and 0.037 is the final weight contribution of the red channel. Template image It can correspond to weights Where -0.22 is the final weight contribution of the blue channel, 0.936 is the final weight contribution of the green channel, and 0.266 is the final weight contribution of the red channel. Template image It can correspond to weights Where -0.014 is the final weight contribution of the blue channel, -0.084 is the final weight contribution of the green channel, and 1.115 is the final weight contribution of the red channel. Template image It can correspond to weights Where -0.202 is the final weight contribution of the blue channel, 0.292 is the final weight contribution of the green channel, and -0.218 is the final weight contribution of the red channel. Template image It can correspond to weights Where -0.039 is the final weight contribution of the blue channel, 0.000 is the final weight contribution of the green channel, and -0.004 is the final weight contribution of the red channel.

[0083] AI model 407 can be configured to generate a weight vector that specifies the weight contribution of each channel included in the remaining template image. For example, based on the original image with a blue tint, the color-corrected image can have more green and red channel components added to obtain color-corrected image 415, for example, as shown in bold. Therefore, the template image contributes to the final red channel R. The highest value in the transformation vector is 1.115, contributing to the template image of the final green channel G. The second highest value in the transformation vector is 0.936, and so on.

[0084] In the embodiments, it is possible to generate, such as Figure 3A The diagram shows eleven template images, and a transformation matrix of size "3x11" can be generated to correct each template image. A matrix multiplication 405 between the eleven template images and the transformation matrix of size "3x11" can be performed to apply the correction to each template image.

[0085] BGR color correction channel 413 can be generated as a result of matrix multiplication 405. The correction weights from the trained AI model 407 can help to perform accurate automatic white balance correction on the input image.

[0086] Figure 6 A block diagram of a system 600 for image correction according to an embodiment of the present disclosure is shown.

[0087] In embodiments of this disclosure, system 600 may include a memory 602, at least one processor 604, an image sensor 606, a training unit 608, and an AI module 610, which are communicatively coupled to each other. AI module 610 may further include an AI model 612 and a processing unit 614. In embodiments, system 600 may include an input / output (I / O) module and a communication interface.

[0088] It can be noted that, in the embodiments, system 600 may include more or fewer components than those described herein. The various components of system 600 may be implemented using hardware, software, firmware, or any combination thereof. Furthermore, the various components of system 600 may be operatively coupled to each other. More specifically, the various components of system 600 may communicate with each other using a communication channel medium (such as a bus, interconnect, etc.).

[0089] In embodiments, at least one processor 604 and processing unit 614 may be or may include a multi-core processor, a single-core processor, or a combination of one or more multi-core processors and one or more single-core processors. For example, at least one processor 604 and processing unit 614 may be implemented as one or more of various processing devices, such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing circuitry with or without an accompanying DSP, or various other processing devices including a microcontroller unit (MCU), hardware accelerator, dedicated computer chip, etc.

[0090] In this embodiment, memory 602 is capable of storing machine-executable instructions, referred to herein as instructions. In this embodiment, at least one processor 604 and processing unit 614 are implemented as executors of software instructions. Therefore, at least one processor 604 and processing unit 614 are capable of executing the instructions stored in memory 602 to perform one or more of the operations described herein.

[0091] The memory 602 can be any type of storage device accessible to at least one processor 604 and processing unit 614 to perform corresponding functions. For example, the memory 602 may include one or more volatile or non-volatile memories, or combinations thereof. For example, the memory 602 may be implemented as a semiconductor memory, such as flash memory, mask ROM, PROM (programmable ROM), EPROM (erasable PROM), RAM (random access memory), etc.

[0092] In an embodiment, AI model 612 may be configured in the internal memory or storage device of AI module 610 to predict correction weights to achieve automatic white balance in an image. Some examples of one or more AI models include, but are not limited to, neural networks, deep neural networks, ML models, etc.

[0093] In embodiments of this disclosure, an input image may be received for correction. The correction facilitated by system 600 may be automatic white balance correction. At least one image sensor 606 may be used to capture at least one input image. In embodiments, the input image may be pre-stored in memory 602 and may be retrieved from memory 602 for correction. In embodiments, the input image may be received from the I / O module of system 600.

[0094] After the received image is corrected, at least one processor 604 can be configured to generate a plurality of template images. In an embodiment, at least one processor 604 can be configured to generate each of the plurality of template images based on at least one nonlinear combination from one or more channels in the RGB channels. (See reference...) Figure 3A The discussion focused on how multiple template images can be generated by applying color kernel transformation.

[0095] In embodiments of this disclosure, to generate multiple template images from an input image, at least one processor 604 may be configured to use an AI module 610 to extract semantic segmentation information included in the input image and generate one or more template images corresponding to one or more attributes included in the semantic segmentation information. The AI ​​module 610 may be trained based on multiple images and their corresponding attributes to understand the semantic segmentation information included in the images. Each template image generated by the AI ​​module 610 may include at least one attribute. In embodiments, as referenced above... Figure 3B The discussion focused on how multiple template images can be generated by applying attribute kernel transformations.

[0096] In embodiments of this disclosure, to generate multiple template images from an input image, at least one processor 604 can be configured to generate multiple template images by performing pixel shifting operations on multiple pixels present in the input image. Pixel shifting operations may include right shifting operations, left shifting operations, up shifting operations, and down shifting operations. In embodiments, as referenced... Figure 3C The proposed method involves generating multiple template images by applying a shifted pixel kernel transformation.

[0097] In embodiments of this disclosure, at least one processor 604 may be configured to generate a plurality of template images based on one or more applications of image feature transformations. Image feature transformations may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation of the input image.

[0098] After generating multiple template images, at least one processor 604 can be configured to assign zero-amplitude and non-zero-amplitude weights to each template image based on the degree and type of correction required in each template image. These weights can be assigned by the at least one processor 604 using the AI ​​module 610. To assign zero-amplitude and non-zero-amplitude weights to each template image, the at least one processor 604 can be configured to generate a transformation matrix by assigning weights to each color contribution component of each of the multiple template images. For example, for an RGB image, weights can be assigned to each color contribution component, as referenced above. Figure 5 The subject of discussion.

[0099] In embodiments of this disclosure, at least one processor 604 may be configured to apply assigned weights to multiple template images to obtain a corrected output image. Applying the assigned weights to multiple template images may involve matrix multiplication between the multiple template images and a transformation matrix generated by the AI ​​module, such as... Figure 5 As shown, the output of the matrix multiplication operation is the corrected output image.

[0100] In embodiments of this disclosure, at least one processor 604 may be configured to provide an AI module 610 with a plurality of sample template images corresponding to the sample image. The AI ​​module 610 may be configured to predict the weight of each color contribution component of each of the plurality of sample template images.

[0101] At least one processor 604 can be configured to apply the predicted weights to each color contribution component of each of a plurality of sample template images to generate a predicted output image. At least one processor 604 can be configured to compare the predicted output image with a ground truth image of the sample images. At least one processor 604 can be configured to compute a training loss between the predicted output image and the ground truth image.

[0102] Training unit 608 can be configured to train AI module 610 based on a computed training loss, which can be provided as feedback to AI module 610, as referenced above. Figure 4 The training unit 608 can be configured to train the AI ​​module 610 until the training loss is below a predetermined threshold.

[0103] The training of AI module 610 may include training the AI ​​model 612 included in AI module 610. AI model 612 may include at least one of neural network, deep neural network, ML model, etc.

[0104] Training the AI ​​model 612 does not require multiple training images to predict the corrected image, thus reducing the overall memory requirements for training and making the above solution scalable. Training the AI ​​model 612 can be performed in a time-saving manner. The above solution may be feasible for devices with limited resources, such as mid- and low-level devices with fewer resources than high-level devices.

[0105] Figure 7 A flowchart of a method 700 for image correction according to an embodiment of the present disclosure is shown. The method 700 depicted in the flowchart can be performed by, for example, system 600. The operations of the flowchart and combinations thereof can be implemented by, for example, hardware, firmware, processor, circuitry, and / or various devices associated with the execution of software including one or more computer program instructions.

[0106] The operation of method 700 can be described and / or practiced by using at least one processor 604 of a system or device other than system 600 (e.g., an image processing system).

[0107] At operation 702, method 700 may include receiving an input image for correction. At least one image sensor may be used to capture at least one input image. In embodiments, the input image may be pre-stored in memory and may be retrieved from memory for correction. In embodiments, the input image may be received from an external source via an I / O module.

[0108] At operation 704, method 700 may include generating a plurality of template images. In an embodiment, generation may include generating each of the plurality of template images based on at least one non-linear combination from one or more channels in the RGB channels. (Refer to the above...) Figure 3A The discussion focused on how multiple template images can be generated by applying color kernel transformation.

[0109] In embodiments of this disclosure, generating multiple template images based on an input image may include using an AI module to extract semantic segmentation information present in the input image and generating one or more template images corresponding to one or more attributes included in the semantic segmentation information. The AI ​​module may be trained based on multiple images and their corresponding attributes to understand the semantic segmentation information included in the images. Each template image generated by the AI ​​module may include at least one attribute. In embodiments, as referenced above... Figure 3B The discussion focused on how multiple template images can be generated by applying attribute kernel transformations.

[0110] In embodiments of this disclosure, generating multiple template images based on an input image may include generating multiple template images by performing pixel shift operations on multiple pixels present in the input image. Pixel shift operations may include right shift operations, left shift operations, up shift operations, and down shift operations. In embodiments, as referenced above... Figure 3C The proposed method involves generating multiple template images by applying a shifted pixel kernel transformation.

[0111] In embodiments of this disclosure, generating multiple template images based on an input image may include generating multiple template images based on one or more image feature transformations. Image feature transformations may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation of the input image.

[0112] At operation 706, method 700 may include providing multiple template images to the AI ​​module. At operation 708, method 700 may include assigning zero-amplitude and non-zero-amplitude weights to each template image based on the degree and type of correction required for each template image. The AI ​​module 610 may be used to assign the weights. Assigning zero-amplitude and non-zero-amplitude weights to each template image may include generating a transformation matrix by assigning weights to each color contribution component of each of the multiple template images. For example, for an RGB image, weights may be assigned to each color contribution component, as referenced above. Figure 5 The subject of discussion.

[0113] At operation 710, method 700 may include applying assigned weights to multiple template images to obtain a corrected output image. Applying the assigned weights to multiple template images may involve matrix multiplication between the multiple template images and a transformation matrix generated by the AI ​​module, such as... Figure 5 As shown. The output of matrix multiplication can be a corrected output image.

[0114] In embodiments of this disclosure, method 700 may include a method for training an AI module. Method 700 may include providing sample images to the AI ​​module for training. Method 700 may then include using the AI ​​module to predict the weights of each color contribution component for each of a plurality of sample template images.

[0115] Method 700 may include applying the predicted weights to each color contribution component of each of a plurality of sample template images to generate a predicted output image. Method 700 may include comparing the predicted output image with a ground truth image of the sample images. Method 700 may include calculating a training loss between the predicted output image and the ground truth image.

[0116] Method 700 may include training the AI ​​module based on a computational training loss, which can be provided as feedback to the AI ​​module, as referenced above. Figure 4 The AI ​​module can be trained until the training loss falls below a predetermined threshold.

[0117] Training the AI ​​model can be done without requiring multiple training images to predict the corrected image, thus reducing the overall memory requirements for training and making the solution scalable. This solution may be feasible for devices with limited resources, such as mid- and low-level devices with fewer resources than high-level devices.

[0118] The operation sequence of method 700 can be compared with... Figure 7 The operations are executed in different orders as presented in the code. In addition, one or more operations can be combined and executed as a single step, or an operation can have several sub-steps that can be executed in parallel or sequentially.

[0119] refer to Figure 7 The methods disclosed, or references Figure 7 One or more operations of the explained system 600 can be implemented using software that includes computer-executable instructions stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more optical media disks), volatile memory components (e.g., DRAM or SRAM), or non-volatile memory or storage components (e.g., hard disk drives or solid-state non-volatile memory components, such as flash memory components), and executed on a computer (e.g., any suitable computer, such as a laptop computer, netbook, webbook, tablet computing device, smartphone, or other mobile computing device). For example, such software can be executed on a single local computer.

[0120] Furthermore, embodiments consistent with this disclosure can be implemented using one or more computer-readable storage media. A computer-readable storage medium refers to any type of physical memory on which processor-readable information or data can be stored. Therefore, a computer-readable storage medium can store instructions executable by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" can be understood to include tangible articles and exclude carrier waves and transient signals, such as non-transient media. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disk drives, CD (optical disc) ROMs, DVDs, flash drives, magnetic disks, and any other known physical storage media.

[0121] Those skilled in the art will understand that, in general, the terminology used herein is intended to be “open” (e.g., the term “comprising” can be interpreted as “including but not limited to,” the term “having” can be interpreted as “having at least,” and the term “comprising” can be interpreted as “including but not limited to,” etc.). For example, to aid understanding, detailed descriptions may include the use of introductory phrases “at least one” and “one or more” to introduce statements. However, the use of such phrases should not be construed as implying that introducing a statement with the indefinite article “a” or “an” limits the description including any particular portion of such introduced statement to disclosures including only one such statement, even when such statements include the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” and “an” (e.g., “a” and / or “an” can generally be interpreted as meaning “at least one” or “one or more”); the same applies to the use of definite articles to introduce such statements. Furthermore, even if specific portions of the descriptive statements are explicitly stated, those skilled in the art will recognize that such statements can generally be interpreted as indicating at least the quantity mentioned (e.g., the phrase "two statements" without other embellishments generally means at least two statements or two or more statements).

[0122] As used herein, terms and phrases such as “have,” “may have,” “include,” or “may include” a feature (such as a number, function, operation, or component, such as a part) indicate the presence of that feature without excluding the presence of other features. Furthermore, as used herein, the phrases “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” can include all possible combinations of A and B. For example, “A or B,” “at least one of A and B,” and “at least one of A or B” can indicate all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0123] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes and are not intended to limit the scope and spirit of this disclosure.

[0124] In an embodiment, the weight allocation in this method may include generating a transformation matrix by assigning weights to each color contribution component of each of a plurality of template images.

[0125] In an embodiment, applying the assigned weights in this method may include performing matrix multiplication between the transformation matrix and multiple template images.

[0126] In an embodiment, generating multiple template images in this method may include generating each of the multiple template images based on at least one non-linear combination of one or more channels from the red, green, and blue (RGB) color channels.

[0127] In an embodiment, generating multiple template images in this method may include using an AI module to extract semantic segmentation information included in the input image. Generating multiple template images may also include generating one or more template images corresponding to one or more attributes included in the semantic segmentation information, wherein each of the one or more template images may include at least one attribute.

[0128] In an embodiment, generating multiple template images in this method may include generating multiple template images by performing at least one pixel shift operation on multiple pixels included in an input image, wherein the at least one pixel shift operation may include at least one of a right shift operation, a left shift operation, an up shift operation, and a down shift operation.

[0129] In an embodiment, the image feature transformation in this method may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation.

[0130] In an embodiment, the method may include providing an AI module with multiple sample template images corresponding to sample images. The method may include generating predicted weights for color contribution components corresponding to the multiple sample template images. The method may include applying the predicted weights to the color contribution components corresponding to the multiple sample template images to generate a predicted output image. The method may include comparing the predicted output image with a ground truth image to calculate a training loss. The method may include training the AI ​​module based on the calculated training loss.

[0131] In one embodiment, the training of the AI ​​module may include performing loss calculations until the training loss falls below a predetermined threshold.

[0132] In an embodiment, in order to assign weights, the device can be instructed to generate a transformation matrix by assigning weights to each color contribution component of each of the multiple template images.

[0133] In one embodiment, in order to apply the assigned weights, the device can be instructed to perform matrix multiplication between the transformation matrix and multiple template images.

[0134] In an embodiment, in order to generate multiple template images, the device can be instructed to generate each of the multiple template images based on at least one non-linear combination of one or more channels from the red, green, and blue (RGB) color channels.

[0135] In this embodiment, to generate multiple template images, the device can use an AI module to extract semantic segmentation information included in the input image. Furthermore, to generate multiple template images, the device can generate one or more template images corresponding to one or more attributes included in the semantic segmentation information, wherein each of the one or more template images can include at least one attribute.

[0136] In an embodiment, in order to generate multiple template images, the device can be instructed to generate multiple template images by performing at least one pixel shift operation on multiple pixels included in the input image. The at least one pixel shift operation may include at least one of a right shift operation, a left shift operation, an up shift operation, and a down shift operation.

[0137] In an embodiment, the image feature transformation in the device may include at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation.

[0138] In this embodiment, the device can be instructed to provide the AI ​​module with multiple sample template images corresponding to the sample image. The device can also be instructed to generate predicted weights for the color contribution components corresponding to the multiple sample template images. Furthermore, the device can apply these predicted weights to the color contribution components corresponding to the multiple sample template images to generate a predicted output image. Finally, the device can compare the predicted output image with a real image to calculate a training loss. Finally, the device can train the AI ​​module based on the calculated training loss.

[0139] In one embodiment, in order to train the AI ​​module, instructions can cause the device to perform loss calculations until the training loss falls below a predetermined threshold.

[0140] In an embodiment, generating a plurality of template images in one or more non-transitory computer-readable storage media may include generating each of the plurality of template images based on at least one non-linear combination of one or more channels from the red, green and blue (RGB) color channels.

Claims

1. A method for performing image correction, the method comprising: Receive the input image to be corrected; Multiple template images are generated based on the input image, wherein each of the multiple template images is generated based on at least one of the image feature transformations; The multiple template images are provided to the artificial intelligence (AI) module; Using an AI module, weights are assigned to each template image in terms of zero and non-zero amplitude values ​​based on the degree and type of correction to be applied to each template image; and The assigned weights are applied to the multiple template images to obtain the corrected output image.

2. The method according to claim 1, wherein, The weight allocation includes: A transformation matrix is ​​generated by assigning weights to each color contribution component of each of the plurality of template images.

3. The method according to any one of claims 1 and 2, wherein, Generating the plurality of template images includes: Each of the plurality of template images is generated based on at least one non-linear combination of one or more channels in the red-green-blue (RGB) color channel.

4. The method according to any one of claims 1 to 3, wherein, Generating the plurality of template images includes: Use an AI module to extract semantic segmentation information from the input image; and Generate one or more template images corresponding to one or more attributes included in the semantic segmentation information. Each of the one or more template images includes at least one attribute.

5. The method according to any one of claims 1 to 4, wherein, Generating the plurality of template images includes: The plurality of template images are generated by performing at least one pixel shift operation on a plurality of pixels included in the input image. The at least one pixel shift operation includes at least one of a right shift operation, a left shift operation, an up shift operation, and a down shift operation.

6. The method according to any one of claims 1 to 5, wherein, Image feature transformation includes at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation.

7. The method according to any one of claims 1 to 6, further comprising: Provide the AI ​​module with multiple sample template images corresponding to the sample images; Generate predicted weights for the color contribution components corresponding to the plurality of sample template images; The predicted weights are applied to the color contribution components corresponding to the plurality of sample template images to generate the predicted output image; The predicted output image is compared with the real image to calculate the training loss; as well as The AI ​​module is trained using computational training loss.

8. An apparatus for performing image correction, the apparatus comprising: At least one processor; as well as The memory is configured to store instructions that, when executed by the at least one processor, cause the device to: Receive the input image to be corrected; Multiple template images are generated based on the input image, wherein each of the multiple template images is generated based on at least one of the image feature transformations; The multiple template images are provided to the artificial intelligence (AI) module; Using an AI module, weights are assigned to each template image in terms of zero and non-zero amplitude values ​​based on the degree and type of correction to be applied to each template image; and The assigned weights are applied to the multiple template images to obtain the corrected output image.

9. The device according to claim 8, wherein, In order to assign weights, the instructions also cause the device to: A transformation matrix is ​​generated by assigning weights to each color contribution component of each of the plurality of template images.

10. The device according to any one of claims 8 and 9, wherein, In order to generate the plurality of template images, the instructions also cause the device to: Each of the plurality of template images is generated based on at least one non-linear combination of one or more channels in the red-green-blue (RGB) color channel.

11. The device according to any one of claims 8 to 10, wherein, In order to generate the plurality of template images, the instructions also cause the device to: Use an AI module to extract semantic segmentation information from the input image; and Generate one or more template images corresponding to one or more attributes included in the semantic segmentation information, wherein each of the one or more template images includes at least one attribute.

12. The device according to any one of claims 8 to 11, wherein, In order to generate the plurality of template images, the instructions also cause the device to: The plurality of template images are generated by performing at least one pixel shift operation on a plurality of pixels included in the input image. The at least one pixel shift operation includes at least one of a right shift operation, a left shift operation, an up shift operation, and a down shift operation.

13. The device according to any one of claims 8 to 12, wherein, Image feature transformation includes at least one of color kernel transformation, attribute kernel transformation, and shifted pixel kernel transformation.

14. The device according to any one of claims 8 to 13, wherein, The instruction also causes the device to: Provide the AI ​​module with multiple sample template images corresponding to the sample images; Generate predicted weights for the color contribution components corresponding to the plurality of sample template images; The predicted weights are applied to the color contribution components corresponding to the plurality of sample template images to generate the predicted output image; The predicted output image is compared with the real image to calculate the training loss; as well as The AI ​​module is trained using computational training loss.

15. A computer-readable storage medium for storing instructions, wherein, When executed by at least one processor, the instructions cause the at least one processor to perform the method of any one of claims 1 to 7.

Images