Image processing method and device, electronic equipment, storage medium and program product

Abstract

The application relates to an image processing method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of frames of RGB images to be fused, the exposure degrees of the RGB images being different; converting each RGB image into a grayscale image, and performing first fusion processing on the grayscale images to obtain a fused grayscale image; and performing color restoration processing on the fused grayscale image to obtain a fused RGB image corresponding to the plurality of frames of RGB images. The method can reduce the calculation amount and improve the image processing efficiency in the fusion process of the plurality of frames of RGB images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In some special shooting scenarios, such as backlighting, sunset, and night scenes, the images captured by the terminal often suffer from overexposure of highlights or loss of detail in shadows, making the images unable to truly record the scene seen by the human eye.

[0003] To improve this phenomenon, related technologies typically involve fusing multiple color images with different exposures to obtain a single color image with better quality.

[0004] However, the above image processing methods suffer from high computational load and low processing efficiency. Summary of the Invention

[0005] This application provides an image processing method, apparatus, electronic device, storage medium, and program product that can reduce computational load and improve image processing efficiency during the fusion of multiple RGB images.

[0006] Firstly, this application provides an image processing method. The method includes:

[0007] Acquire multiple RGB images to be fused, each with a different exposure level;

[0008] Each of the RGB images is converted into a grayscale image, and a first fusion process is performed on each of the grayscale images to obtain a fused grayscale image;

[0009] Color restoration processing is performed on the fused grayscale image to obtain the fused RGB image corresponding to the multi-frame RGB images.

[0010] Secondly, this application also provides an image processing apparatus. The apparatus includes:

[0011] The acquisition module is used to acquire multiple frames of RGB images to be fused, each of which has a different exposure.

[0012] A fusion module is used to convert each of the RGB images into a grayscale image and perform a first fusion process on each of the grayscale images to obtain a fused grayscale image;

[0013] The color restoration module is used to perform color restoration processing on the fused grayscale image to obtain the fused RGB image corresponding to the multi-frame RGB images.

[0014] Thirdly, this application also provides an electronic device. The electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method described in the first aspect above.

[0015] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0016] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0017] The aforementioned image processing methods, apparatus, electronic devices, storage media, and program products, in the process of fusing multiple frames of RGB images, acquire multiple frames of RGB images to be fused, each RGB image having different exposure levels. Then, each RGB image is converted into a grayscale image, and a first fusion process is performed on each grayscale image to obtain a fused grayscale image. Next, color restoration processing is performed on the fused grayscale image to obtain the fused RGB image corresponding to the multiple frames of RGB images. Thus, the embodiments of this application convert each RGB image into a grayscale image and perform fusion processing in the grayscale domain. Compared with the traditional method of directly fusing multiple frames of RGB images in the RGB domain, the embodiments of this application can greatly reduce the amount of computation in the process of fusing multiple frames of RGB images and improve image processing efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0020] Figure 2 This is a flowchart of an image processing method in one embodiment;

[0021] Figure 3 Here is a flowchart of step 203 in another embodiment;

[0022] Figure 4 Here is a flowchart of step 301 in another embodiment;

[0023] Figure 5 A flowchart of an image processing method in another embodiment;

[0024] Figure 6 This is a structural block diagram of an image processing device in one embodiment;

[0025] Figure 7 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] In the field of imaging, dynamic range refers to the range of light intensity distribution in an image, from the darkest shadows to the brightest highlights. The dynamic range in nature is very wide. However, current camera devices (such as mobile phones, tablets, and cameras) have limited dynamic range (usually only 0~255) due to insufficient hardware performance. Therefore, in some special high dynamic range scenarios, such as backlighting, sunset, and night scenes, details that exceed the effective dynamic range of the camera device (i.e., the range of brightness changes that the camera device's photosensitive element can accept) often cannot be displayed normally in the image. The captured images often have overexposed highlights or lost details in shadows, making the image unable to truly record the scene seen by the human eye.

[0028] To address this issue and improve the dynamic range of captured images, the relevant techniques typically involve fusing multiple color images with different exposures. The resulting image has a better dynamic range, color, and detail, thereby improving the overall quality of the captured image.

[0029] However, in the above method, multiple RGB images are directly fused in the RGB domain. That is, the input of the image fusion algorithm is a three-channel RGB image, and multiple RGB images are fused in the RGB domain to output a fused RGB image. However, the fusion process involves a large amount of time-consuming computation. This method of directly fusing multiple RGB images in the RGB domain has the problems of large computational load and low processing efficiency.

[0030] Therefore, embodiments of this application provide an image processing method, apparatus, electronic device, storage medium, and program product that can reduce the amount of computation and improve image processing efficiency during the fusion processing of multiple RGB images.

[0031] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 The implementation environment shown includes electronic device 102, which may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, smart cars, etc. Portable wearable devices may include smartwatches, smart bracelets, head-mounted devices, etc.

[0032] In one embodiment, such as Figure 2 As shown, an image processing method is provided, which is applied to... Figure 1 Taking electronic device 102 as an example, it can be understood that this method can also be applied to servers, and can also be applied to systems that include electronic devices and servers, and can be implemented through the interaction between electronic devices and servers.

[0033] Please see Figure 2 The image processing method includes the following steps:

[0034] Step 201: The electronic device acquires multiple RGB images to be fused, each with a different exposure.

[0035] In this embodiment, the multi-frame RGB images can be a set of color images captured by an electronic device on a subject, such as a person, animal, natural scenery, or still life. The multi-frame RGB images can be obtained through continuous shooting, or by selecting multiple RGB images that meet the requirements from a sequence of continuously shot images.

[0036] The exposure of each RGB image is different. Exposure refers to the degree to which the photosensitive element in the shooting component of an electronic device is illuminated by light. That is, how much light the photosensitive element receives. The more light it receives, the higher the exposure and the brighter the image. Conversely, the lower the exposure, the darker the image. In other words, exposure controls the brightness of the image.

[0037] Underexposure or overexposure will affect image quality. Therefore, this application embodiment obtains multiple RGB images with different exposures, each RGB image is of equal size, and the multiple RGB images are fused to obtain a fused RGB image with good image quality.

[0038] Optionally, the multi-frame RGB images may also be captured by one or more other devices with capturing components. The electronic device may acquire the multi-frame RGB images from the device that captured the RGB images, or the electronic device may acquire the multi-frame RGB images from a specified database, and so on.

[0039] Step 202: The electronic device converts each RGB image into a grayscale image and performs a first fusion process on each grayscale image to obtain a fused grayscale image.

[0040] For each RGB image, the electronic device converts the RGB image into a grayscale image with the same size as the RGB image.

[0041] An RGB image comprises three color channels (R, G, and B channels) in an image layer of equal size. Optionally, the electronic device can average the three pixel values ​​at the same pixel location in the three image layers to obtain the pixel value at that location in the grayscale image. Alternatively, the electronic device can perform a weighted sum of the three pixel values ​​at the same pixel location in the three image layers according to a preset weight ratio to obtain the pixel value at that location in the grayscale image.

[0042] In this way, the electronic device can obtain the grayscale image corresponding to each RGB image through conversion. Then, the electronic device performs a first fusion process on each grayscale image. The process of the first fusion process is briefly introduced below.

[0043] To increase image detail, firstly, for each grayscale image, the electronic device generates an image pyramid corresponding to that grayscale image.

[0044] An image pyramid is a collection of images arranged in a pyramid shape with progressively decreasing resolution. The bottom layer (first layer) of the image pyramid is the original grayscale image. The second layer of the image pyramid, from bottom to top, is obtained by downsampling the original grayscale image (for example, called the second layer downsampled grayscale image). The third layer of the image pyramid is obtained by further downsampling the second layer downsampled grayscale image, and so on. The higher the layer, the smaller the image size and the lower the image resolution.

[0045] In this way, the electronic device obtains the image pyramid corresponding to each grayscale image. Then, for each image pyramid, the electronic device calculates the weight image of the grayscale image of that layer based on the grayscale image of each layer of the image pyramid.

[0046] For example, for the original grayscale image at the bottom layer of the image pyramid, the electronic device can input the grayscale image into a pre-set Gaussian function, and calculate the fusion weight of each pixel in the grayscale image through the Gaussian function. The fusion weight is between [0, 1]. The electronic device uses the fusion weight of each pixel as the pixel value in the weight image of the grayscale image.

[0047] Similarly, for the downsampled grayscale image of the second layer in the image pyramid, the electronic device can input the downsampled grayscale image into a pre-set Gaussian function, and calculate the fusion weight of each pixel in the downsampled grayscale image through the Gaussian function. By analogy, the weight image of the grayscale image of each layer of the image pyramid is finally calculated.

[0048] In this embodiment, the fusion weights of each pixel are normally distributed in the interval [0, 255]. For example, in a grayscale image, if a pixel has a value of 255, then the pixel is overexposed (too bright), and the corresponding fusion weight is 0. If a pixel has a value of 0, then the pixel is underexposed (too dark), and the corresponding fusion weight is also 0. For midtones (i.e., areas with appropriate exposure) in the interval [0, 255], the fusion weight can be set to 1.

[0049] Finally, the electronic device combines the various image pyramids and the weight images corresponding to each grayscale image in the image pyramid to perform Laplacian fusion, resulting in a fused grayscale image.

[0050] The following is a brief introduction to the Laplace fusion process.

[0051] In an image pyramid, the weight image corresponding to each grayscale image can also be viewed as a weight pyramid as a whole, with each image pyramid corresponding to one weight pyramid. During Laplacian reconstruction, for a set of image pyramids and weight pyramids, each layer performs pixel-wise multiplication of the corresponding grayscale image and weight image, resulting in a multiplication result for each layer.

[0052] Then, the multiplication results of the same layer in each image pyramid are added together, thus obtaining a fused image pyramid, which is a brightness-fused image pyramid.

[0053] Next, detail layers are obtained from each grayscale image. The image pyramid after brightness fusion is superimposed from the top to the bottom (i.e., the image is from small to large) and integrated into the detail layers. Finally, the fused grayscale image in step 202 is obtained. The size of the fused grayscale image is the same as that of each RGB image and the grayscale image obtained by converting the RGB image.

[0054] Step 203: The electronic device performs color restoration processing on the fused grayscale image to obtain the fused RGB image corresponding to the multiple RGB images.

[0055] Through the first fusion process described in step 202, the brightness and image details of the fused grayscale image have been fused. In step 203, the electronic device performs color restoration processing on the fused grayscale image, that is, restores the color of the fused grayscale image, so as to obtain a color fused RGB image.

[0056] In this embodiment of the application, the electronic device can input the fused grayscale image into a pre-trained image generation model, and output a fused RGB image based on the fused grayscale image through the image generation model.

[0057] Other possible implementations of step 203 will be described in the examples below.

[0058] In this embodiment, each RGB image is converted into a grayscale image and fused in the grayscale domain. Compared to the traditional method of directly fusing multiple RGB images in the RGB domain, the computational cost of color image processing in the traditional method is about three times that of single-channel grayscale image processing. In this embodiment, the time-consuming processes such as image pyramid generation, weight image generation, and Laplacian fusion are all implemented in the grayscale domain. After the first fusion process is completed, color restoration is performed on the fused grayscale image, which can greatly reduce the computational cost in the multi-frame RGB image fusion process and improve image processing efficiency.

[0059] In one embodiment, based on Figure 2 The illustrated embodiment can be found in [reference]. Figure 3 This embodiment relates to the process by which an electronic device performs color restoration processing on a fused grayscale image to obtain a fused RGB image corresponding to multiple RGB images. For example... Figure 3 As shown, step 203 may include Figure 3 Steps 301 and 302 are shown below:

[0060] Step 301: The electronic device obtains the color information of each color channel based on the weighted images of each RGB image and each grayscale image.

[0061] Here, grayscale image refers to grayscale image directly obtained by electronic device from RGB image conversion. As described in step 202 above, for each grayscale image, electronic device inputs the grayscale image into a pre-set Gaussian function, calculates the fusion weight of each pixel in the grayscale image through the Gaussian function, and uses the fusion weight of each pixel as the pixel value in the weight image of the grayscale image, thereby obtaining the weight image of the grayscale image. That is, the pixel value of each pixel in the weight image is the fusion weight of the corresponding pixel in the grayscale image in the first fusion process.

[0062] The electronic device first obtains the color information of each color channel based on multiple frames of RGB images and the weight image corresponding to the grayscale image converted from each RGB image.

[0063] For example, the electronic device can filter each weighted image to obtain a filtered weighted image, and then the electronic device can obtain the color information of each color channel based on each RGB image and each filtered weighted image.

[0064] In this embodiment of the application, the electronic device can use Gaussian filtering, mean filtering or guided filtering to filter each weight image. For example, according to a certain radius, the fusion weight in the weight image can be smoothed to eliminate edges, making the boundary of the weight image smoother and avoiding discrete and uneven locations from affecting the fusion quality.

[0065] The following describes the process by which an electronic device obtains color information for each color channel based on the weighted images of each RGB image and each grayscale image. It should be noted that the weighted image mentioned below refers to the filtered weighted image described above.

[0066] See Figure 4 Step 301 may include Figure 4 Steps 401 and 402 are shown below:

[0067] Step 401: The electronic device uses each weighted image to perform a second fusion process on each RGB image in each color channel to obtain the fused image corresponding to the color channel.

[0068] Electronic devices can first use weighted images to perform weighted fusion of multiple RGB images to obtain the fused result img_f. The fusion formula is as follows:

[0069]

[0070] Where N is the number of RGB images in multiple frames, RGB i For the i-th RGB image in a multi-frame RGB image, weight i This is the weighted image of the grayscale image corresponding to the i-th frame RGB image.

[0071] The fusion result, img_f, is a three-channel result, meaning it includes the fused image corresponding to each color channel. The fusion process is described below for each color channel.

[0072] In each color channel (R channel, G channel, or B channel), the electronic device uses weighted images to perform a second fusion process on each RGB image in that color channel to obtain the fused image corresponding to that color channel. For example, for the R channel, the electronic device uses weighted images to perform a second fusion process on each RGB image and the image layer corresponding to the R channel to obtain the fused image corresponding to the R channel. Similarly, for the G channel, the fused image corresponding to the G channel can be obtained, and for the B channel, the fused image corresponding to the B channel can be obtained.

[0073] In an exemplary embodiment of step 401, for each color channel, the electronic device can perform pixel multiplication processing on the image layer corresponding to the color channel and the weight image corresponding to the RGB image in each RGB image (i.e., multiply the pixel values ​​at the corresponding pixel positions of the image layer and the weight image) to obtain the multiplication result. Then, the electronic device adds the multiplication results of each RGB image on the color channel to realize the process of using each weight image to perform a second fusion processing on each RGB image on the color channel to obtain the fused image corresponding to the color channel.

[0074] Step 402: The electronic device obtains color information based on the fused image corresponding to each color channel.

[0075] For example, the electronic device can perform a third fusion process on the fused image corresponding to each color channel to obtain a single-channel image. It should be noted that the size of the single-channel image, the fused image and the RGB image are all equal. Then, the electronic device performs pixel division processing on the fused image and the single-channel image corresponding to each color channel to obtain the color information of the color channel, so as to realize the process of step 402.

[0076] That is, the electronic device merges the three color channels of the above fusion result img_f into a single-channel image, and then uses the fusion image of each color channel included in the fusion result img_f to perform pixel division processing with the single-channel image (dividing the pixel value at the corresponding pixel position of the single-channel image corresponding to the color channel) to obtain the color information of each color channel.

[0077] The following is an exemplary description of how an electronic device performs a third fusion process on the fused images corresponding to each color channel to obtain a single-channel image.

[0078] In one possible implementation, the electronic device determines the corresponding pixel processing method for different image regions in the fused image.

[0079] The pixel processing methods include:

[0080] 1) For the same pixel location, the electronic device uses the maximum value among the pixel values ​​corresponding to the pixel location in each fused image as the pixel value corresponding to the pixel location in the single-channel image.

[0081] Suppose that for a certain pixel location, the fused image of the R channel has the pixel value of img corresponding to that pixel location. r The fused image of the G channel and the pixel value corresponding to that pixel position is img. g The fused image of the B channel and the pixel value corresponding to this pixel position is img. b The pixel value of img1 corresponding to that pixel position in a single-channel image can be calculated using the following formula:

[0082] img1=max{img r img g img b}

[0083] 2) For the same pixel location, the electronic device uses the average of the pixel values ​​corresponding to the pixel location in each fused image as the pixel value corresponding to the pixel location in the single-channel image.

[0084] Continuing with the example above, the pixel value of img2 corresponding to that pixel position in a single-channel image can be calculated using the following formula:

[0085] img2=(img r +img g +img b ) / 3

[0086] 3) For the same pixel location, the electronic device uses the statistical values ​​of the maximum and average values ​​as the pixel values ​​corresponding to the pixel location in the single-channel image.

[0087] Continuing with the example above, and assuming that the weight of a certain region in the third fusion process is w, then for a certain pixel position in that region, the pixel value of the single-channel image corresponding to that pixel position can be calculated using the following formula:

[0088] img3 = img1 * w + img2 * (1 - w)

[0089] In this way, after the electronic device determines one of the above pixel processing methods for different image regions in the fused image, for each image region, the electronic device performs fusion processing on the pixel values ​​corresponding to each fused image and the image region according to the pixel processing method of the image region to obtain a single-channel image.

[0090] Next, the electronic device divides the fused image corresponding to each color channel by the single-channel image pixel by pixel to obtain the color information for each color channel. Assuming that the color information for a pixel location in a single-channel image is represented by `color`, then:

[0091] Color=img_f / img1, or Color=img_f / img2, or Color=img_f / img3

[0092] In this embodiment of the application, for an image region, if the image region is a highlight region and the color saturation of the highlight region is low, the color of the highlight region is not preferred to be used in the color restoration process, so the first maximum pixel processing method can be selected; for other non-highlight regions, the colors of these regions are preferred to be used in the color restoration process, so the second maximum pixel processing method can be selected.

[0093] For certain areas in specific scenarios, such as night scenes, the color saturation of bright light signs can be better when combined with the color information of other dark frames. Therefore, the third maximum pixel processing method can be selected.

[0094] In the color restoration process, this application embodiment can control the proportion of the maximum value used by setting weights, thereby making more use of the most accurate and expressive colors from the input frame, thus improving the color restoration effect. Even when local weights (such as the weights for highlight areas) are not optimal, it can still effectively reconstruct image colors.

[0095] In another possible implementation, the electronic device may also directly use any of the pixel processing methods described above to fuse the fused images corresponding to each color channel without distinguishing image regions, thus obtaining a single-channel image. Then, the electronic device performs pixel-wise division between the fused image corresponding to each color channel and the single-channel image to obtain the color information of the color channel.

[0096] Step 302: The electronic device uses the color information to perform color restoration processing on the fused grayscale image to obtain a fused RGB image.

[0097] After the electronic device acquires the color information of each color channel, it can be understood that, as mentioned above, the electronic device performs pixel division processing on the fused image corresponding to each color channel and the single-channel image. The color information of each color channel includes the color information of each pixel position in the grayscale image in that color channel. In this way, the electronic device performs pixel multiplication processing on the color information of each color channel with the fused grayscale image to obtain the fused RGB image, thereby realizing the color restoration process of the fused grayscale image. Compared with the computational workload of calculating the image pyramid of the three channels in the RGB domain, calculating the weight image for each layer of the image pyramid, and performing Laplacian fusion of the three channels, the color restoration process calculation method of this application embodiment is simple, involving only simple pixel multiplication and division operations, which greatly reduces the amount of computation and improves the image processing efficiency.

[0098] In one embodiment, see Figure 5 The image processing method in this embodiment includes the following steps:

[0099] Step 501: The electronic device acquires multiple RGB images to be fused, each with a different exposure.

[0100] Step 502: The electronic device converts each RGB image into a grayscale image and performs a first fusion process on each grayscale image to obtain a fused grayscale image.

[0101] Step 503: The electronic device performs filtering on the weighted images of each grayscale image to obtain the filtered weighted image.

[0102] The pixel value of each pixel in the weighted image is the fusion weight of the corresponding pixel in the grayscale image during the first fusion process.

[0103] Step 504: For each color channel, the electronic device performs pixel-wise multiplication of the image layer corresponding to that color channel and the corresponding filtered weight image in each RGB image to obtain the multiplication result.

[0104] Step 505: The electronic device adds the multiplication results of each RGB image on the color channel to obtain the fused image corresponding to the color channel.

[0105] Step 506: The electronic device determines the corresponding pixel processing method for different image regions in the fused image.

[0106] Step 507: For each image region, the electronic device performs fusion processing on the pixel values ​​corresponding to each fused image and the image region according to the pixel processing method of the image region to obtain a single-channel image.

[0107] Pixel processing methods include:

[0108] For the same pixel location, the maximum value among the pixel values ​​corresponding to the pixel location in each fused image is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the average value of the pixel values ​​corresponding to the pixel location in each fused image is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the statistical value of the maximum value and the average value is taken as the pixel value corresponding to the pixel location in the single-channel image.

[0109] Step 508: The electronic device performs pixel division between the fused image corresponding to each color channel and the single-channel image to obtain the color information of that color channel.

[0110] Step 509: The electronic device performs pixel-wise multiplication of the color information of each color channel with the fused grayscale image to obtain the fused RGB image.

[0111] In the image processing method proposed in this application embodiment, the main idea of ​​color restoration processing is to use the fusion weights in the weight images of each grayscale image in the blurring (i.e. filtering) brightness fusion stage (i.e. the first fusion process), and use the filtered weight images to fuse the original multi-frame RGB images with different exposures to obtain a fusion result (the fusion result includes the fusion images corresponding to the three color channels mentioned above). Based on the fusion result, a single-channel image is obtained (obtained through the pixel processing method mentioned above). The fusion result is then divided by the single-channel image to obtain color information. Finally, the fused grayscale image (i.e. the grayscale image after brightness fusion) is multiplied by the color information to obtain the fused RGB image after color reconstruction.

[0112] The process involves dividing the fused result (three channels) by the single-channel image to obtain the division result, and then multiplying the division result (i.e., color information) by the fused grayscale image. This is essentially a reversible color restoration process. During color restoration, the fused grayscale image replaces the single-channel image mentioned above to recover the fused RGB image of the three channels, thus achieving color restoration. This application provides a lightweight color restoration scheme for multi-frame grayscale fusion, reducing the computational load of multi-frame fusion while reconstructing colors more efficiently.

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0114] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0115] In one embodiment, such as Figure 6 As shown, an image processing apparatus is provided, comprising:

[0116] The acquisition module 601 is used to acquire multiple RGB images to be fused, each of which has a different exposure.

[0117] The fusion module 602 is used to convert each of the RGB images into grayscale images and perform a first fusion process on each of the grayscale images to obtain a fused grayscale image;

[0118] The color restoration module 603 is used to perform color restoration processing on the fused grayscale image to obtain the fused RGB image corresponding to the multi-frame RGB image.

[0119] In one embodiment, the color restoration module 603 includes:

[0120] The acquisition unit is used to acquire color information of each color channel based on the weight images of each RGB image and each grayscale image, wherein the pixel value of each pixel in the weight image is the fusion weight of the corresponding pixel in the grayscale image in the first fusion process;

[0121] The processing unit is used to perform color restoration processing on the fused grayscale image using the color information to obtain the fused RGB image.

[0122] In one embodiment, the acquisition unit is specifically used to perform a second fusion process on each of the RGB images in each of the color channels using each of the weighted images to obtain a fused image corresponding to each color channel; and to acquire each of the color information based on the fused image corresponding to each of the color channels.

[0123] In one embodiment, the acquisition unit is specifically used to perform pixel multiplication on the image layer corresponding to the color channel and the corresponding weight image in each RGB image for each color channel to obtain a multiplication result; and to add the multiplication results of each RGB image on the color channel to obtain the fused image corresponding to the color channel.

[0124] In one embodiment, the acquisition unit is specifically used to perform a third fusion process on the fused image corresponding to each color channel to obtain a single-channel image; and to perform pixel division processing between the fused image corresponding to each color channel and the single-channel image to obtain the color information of the color channel.

[0125] In one embodiment, the acquisition unit is specifically used to determine the corresponding pixel processing method for different image regions in the fused image; for each image region, according to the pixel processing method of the image region, the pixel values ​​corresponding to each fused image and the image region are fused to obtain the single-channel image;

[0126] In one embodiment, the pixel processing method includes:

[0127] For the same pixel location, the maximum value among the pixel values ​​corresponding to the pixel location in each of the fused images is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the average value of the pixel values ​​corresponding to the pixel location in each of the fused images is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the statistical value of the maximum value and the average value is taken as the pixel value corresponding to the pixel location in the single-channel image.

[0128] In one embodiment, the acquisition unit is specifically used to filter each of the weighted images to obtain a filtered weighted image; and to acquire the color information of each of the color channels based on each of the RGB images and each of the filtered weighted images.

[0129] In one embodiment, the processing unit is specifically used to perform pixel-wise multiplication of the color information of each color channel with the fused grayscale image to obtain the fused RGB image.

[0130] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0131] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0132] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0133] This application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of an image processing method.

[0134] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0136] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0137] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: Acquire multiple RGB images to be fused, each with a different exposure level; Each of the RGB images is converted into a grayscale image, and a first fusion process is performed on each of the grayscale images to obtain a fused grayscale image; Using the weighted images of each grayscale image, a second fusion process is performed on each of the RGB images in each color channel to obtain a fused image corresponding to each color channel; based on the fused image corresponding to each color channel, the color information of each color channel is obtained; the pixel value of each pixel in the weighted image is the fusion weight of the corresponding pixel in the grayscale image in the first fusion process; Using the color information, color restoration processing is performed on the fused grayscale image to obtain the fused RGB image corresponding to the multi-frame RGB images.

2. The method according to claim 1, characterized in that, The step of using the weighted images to perform a second fusion process on each of the RGB images in each of the color channels to obtain the fused image corresponding to the color channel includes: For each color channel, the image layer corresponding to the color channel in each RGB image and the corresponding weight image are multiplied pixel by pixel to obtain the multiplication result; The multiplication results of the RGB images in the color channels are added together to obtain the fused image corresponding to the color channels.

3. The method according to claim 1, characterized in that, The step of obtaining each color information based on the fused image corresponding to each color channel includes: A third fusion process is performed on the fused images corresponding to each of the color channels to obtain a single-channel image; The color information of each color channel is obtained by dividing the fused image corresponding to each color channel by the single-channel image by pixels.

4. The method according to claim 3, characterized in that, The third fusion process, which involves performing a third fusion process on the fused images corresponding to each of the color channels to obtain a single-channel image, includes: For different image regions in the fused image, determine the corresponding pixel processing method; For each image region, the pixel values ​​corresponding to each fused image and the image region are fused according to the pixel processing method of the image region to obtain the single-channel image.

5. The method according to claim 4, characterized in that, The pixel processing methods include: For the same pixel location, the maximum value among the pixel values ​​corresponding to the pixel location in each of the fused images is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the average value of the pixel values ​​corresponding to the pixel location in each of the fused images is taken as the pixel value corresponding to the pixel location in the single-channel image; or, the statistical value of the maximum value and the average value is taken as the pixel value corresponding to the pixel location in the single-channel image.

6. The method according to claim 1, characterized in that, Each of the weight images is a filtered weight image.

7. The method according to any one of claims 2-6, characterized in that, Using the color information described above, color restoration processing is performed on the fused grayscale image to obtain the fused RGB image, including: The color information of each color channel is multiplied pixel by pixel with the fused grayscale image to obtain the fused RGB image.

8. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire multiple frames of RGB images to be fused, each of which has a different exposure. A fusion module is used to convert each of the RGB images into a grayscale image and perform a first fusion process on each of the grayscale images to obtain a fused grayscale image; The color restoration module is used to perform a second fusion process on each color channel of each RGB image using the weight images of each grayscale image to obtain a fused image corresponding to each color channel; and to obtain the color information of each color channel based on the fused image corresponding to each color channel; the pixel value of each pixel in the weight image is the fusion weight of the corresponding pixel in the grayscale image in the first fusion process. The color restoration module is further configured to use the color information to perform color restoration processing on the fused grayscale image to obtain the fused RGB image corresponding to the multi-frame RGB images.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images