Image fusion method and device

Abstract

The invention discloses an image fusion method and device, and belongs to the technical field of industrial detection. The image fusion method comprises the steps that multiple first images corresponding to a to-be-detected object are acquired, and the multiple first images are acquired from the to-be-detected object in different stroboscopic areas under at least two exposure intensities; performing channel separation processing on the plurality of first images under the same exposure intensity to obtain second images corresponding to each channel in each stroboscopic region; generating a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel; and performing fusion processing on the third images under the channels to obtain a target fusion image corresponding to the to-be-detected object. According to the image fusion method, the definition and detail performance of the fused image are improved, and the overexposure removal effect is good.

Description

Technical Field

[0001] The present application belongs to the field of industrial detection technology, and in particular, relates to an image fusion method and device. Background Art

[0002] In industrial inspection scenarios, due to factors such as lighting and the surface material of the object to be tested, there may be overexposed areas of highlights in the image of the object to be tested, thus interfering with normal visual stability detection. In the related art, there is an overexposure removal algorithm based on digital image enhancement, which usually adjusts the brightness of the overexposed area of ​​the image from the perspective of the image grayscale to improve the overall contrast of the image. However, this method may cause the image to be over-enhanced, making the image look unnatural or have artifacts, and the overexposure removal effect is not good. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes an image fusion method and device, which improves the clarity and detail expression of the fused image and has a better effect of removing overexposure.

[0004] In a first aspect, the present application provides an image fusion method, the method comprising:

[0005] Acquire a plurality of first images corresponding to the object to be measured, wherein the plurality of first images are images acquired under at least two exposure intensities and in different stroboscopic regions of the object to be measured;

[0006] Performing channel separation processing on the plurality of first images under the same exposure intensity to obtain second images corresponding to each channel under each stroboscopic area;

[0007] Based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel, a third image under the same channel is generated;

[0008] The third images under each of the channels are fused to obtain a target fused image corresponding to the object to be measured.

[0009] According to the image fusion method provided in the embodiment of the present application, the partitioned stroboscopic and multi-exposure lighting methods are utilized to collect partitioned stroboscopic images (i.e., first images) under different exposures, and then the channels of the first images under the same exposure intensity are separated to obtain the second image, and a third image is generated based on the pixel values ​​of the second image under the same channel to obtain the target fused image. This method can better remove the overexposed areas in the image, improve the clarity and detail expression of the fused image, and achieve better removal of overexposure.

[0010] An image fusion method according to an embodiment of the present application, generating a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel, comprises:

[0011] Determine the minimum value of pixel values ​​at the same pixel position as the target pixel value at the pixel position;

[0012] The third image is obtained based on the target pixel value at each of the pixel positions.

[0013] In an image fusion method according to an embodiment of the present application, determining the minimum value of pixel values ​​at the same pixel position as the target pixel value at the pixel position includes:

[0014] Performing stacking processing on the multiple second images under the same channel to obtain an image matrix corresponding to the same channel; the first dimension of the image matrix is ​​the width of the second image, the second dimension of the image matrix is ​​the height of the second image, and the third dimension of the image matrix is ​​the number of the multiple second images under the same channel;

[0015] The minimum value of the pixel values ​​at the same pixel position in the image matrix is ​​determined as the target pixel value at the pixel position.

[0016] The image fusion method of one embodiment of the present application, wherein the third images under each channel are fused to obtain a target fused image corresponding to the object to be measured, comprises:

[0017] Performing fusion processing on the third images corresponding to the channels under the same exposure intensity to obtain a fourth image corresponding to the object to be measured under the same exposure intensity;

[0018] The fourth images under the exposure intensities are fused to obtain a target fused image corresponding to the object to be measured.

[0019] In an image fusion method according to an embodiment of the present application, the fourth images at the exposure intensities are fused to obtain a target fused image corresponding to the object to be measured, including:

[0020] Processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each of the fourth images to obtain a weight matrix corresponding to each of the fourth images;

[0021] Based on each of the weight matrices, weighted fusion processing is performed on at least two fourth images under the exposure intensity to obtain the target fused image.

[0022] In an image fusion method according to an embodiment of the present application, processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each of the fourth images to obtain a weight matrix corresponding to each of the fourth images includes:

[0023] Obtain at least one of a first weight corresponding to the contrast matrix of the fourth image, a second weight corresponding to the saturation matrix, and a third weight corresponding to the exposure matrix; based on the first weight, the second weight, and at least one of the third weights, perform weighted fusion processing on at least one of the contrast matrix, the saturation matrix, and the exposure matrix of the fourth image to obtain a weight matrix corresponding to the fourth image.

[0024] In an image fusion method according to an embodiment of the present application, before acquiring a plurality of first images corresponding to the object to be measured, the method further includes:

[0025] The image acquisition device is controlled to perform at least two image acquisitions on the object to be measured in different stroboscopic areas; the image acquisition device is arranged above the object to be measured, the field of view of the image acquisition device covers the object to be measured, and the irradiation direction of the light source in each stroboscopic area is toward the object to be measured.

[0026] In a second aspect, the present application provides an image fusion device, the device comprising:

[0027] A first processing module is used to obtain a plurality of first images corresponding to the object to be measured, wherein the plurality of first images are images acquired under at least two exposure intensities and in different stroboscopic regions of the object to be measured;

[0028] A second processing module is used to perform channel separation processing on the multiple first images under the same exposure intensity to obtain a second image corresponding to each channel under each stroboscopic area;

[0029] A third processing module, configured to generate a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel;

[0030] The fourth processing module is used to perform fusion processing on the third images under each of the channels to obtain a target fused image corresponding to the object to be measured.

[0031] According to the image fusion device provided in the embodiment of the present application, the partitioned stroboscopic and multi-exposure lighting methods are used to collect partitioned stroboscopic images (i.e., first images) under different exposures, and then the channels of the first images under the same exposure intensity are separated to obtain the second image, and the third image is generated based on the pixel values ​​of the second image under the same channel to obtain the target fused image. This can better remove the overexposed area in the image, improve the clarity and detail expression of the fused image, and have a better effect of removing overexposure.

[0032] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image fusion method as described in the first aspect above when executing the computer program.

[0033] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image fusion method as described in the first aspect above.

[0034] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image fusion method as described in the first aspect above.

[0035] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0036] By using the zoned stroboscopic and multi-exposure lighting method, the zoned stroboscopic images (i.e., the first image) under different exposures are collected, and then the channels of the first images under the same exposure intensity are separated to obtain the second image, and the third image is generated based on the pixel value of the second image under the same channel to obtain the target fused image. This method can better remove the overexposed area in the image, improve the clarity and detail performance of the fused image, and has a better effect of removing overexposure.

[0037] Furthermore, by sorting the pixel values ​​of the stroboscopic images in different directions at the same position in the same channel and taking the smallest pixel value to obtain the third image, the high brightness impact caused by the stroboscopic images can be reduced, which helps to retain the detail information in the image and reduces the loss of details caused by overexposure. The same operation is performed on each set of stroboscopic images of exposure intensity, which can further enhance the detail performance of the image under different exposure conditions.

[0038] Furthermore, by performing multi-exposure fusion processing on the first de-overexposed image (i.e., the fourth image) corresponding to each exposure intensity obtained, calculating the fusion weight based on the contrast, color saturation and exposure of the multi-exposure images, and using the feature pyramid for fusion, the final exposure-suppressed image (i.e., the target fused image) can be obtained, which further enhances the image quality and achieves a better de-overexposure effect.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 This is one of the flowcharts of the image fusion method provided in the embodiment of the present application;

[0042] Figure 2 This is one of the principle schematic diagrams of the image fusion method provided in the embodiment of the present application;

[0043] Figure 3 This is the second principle schematic diagram of the image fusion method provided in the embodiment of the present application;

[0044] Figure 4 This is the third principle schematic diagram of the image fusion method provided in the embodiment of the present application;

[0045] Figure 5 This is the fourth principle schematic diagram of the image fusion method provided in the embodiment of the present application;

[0046] Figure 6 This is the fifth principle schematic diagram of the image fusion method provided in the embodiment of the present application;

[0047] Figure 7 This is the sixth principle schematic diagram of the image fusion method provided in the embodiment of the present application;

[0048] Figure 8 This is the second flow chart of the image fusion method provided in the embodiment of the present application;

[0049] Fig. 9 is a structural schematic diagram of an image fusion device provided in an embodiment of the present application;

[0050] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0052] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0053] The image fusion method, image fusion device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0054] The image fusion method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.

[0055] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0056] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.

[0057] The image fusion method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the image fusion method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The image fusion method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.

[0058] like Figure 1 As shown, the image fusion method includes: step 110, step 120, step 130 and step 140.

[0059] Step 110: Acquire a plurality of first images corresponding to the object to be measured;

[0060] In this step, the image fusion method can be applied to industrial inspection scenarios, and the objects to be inspected may include industrial equipment and products, etc. For example, the objects to be inspected may include mechanical equipment, automation equipment, and metal materials, etc.

[0061] The multiple first images are images acquired under at least two exposure intensities and in different stroboscopic regions of the object to be measured.

[0062] The overexposure degree of each first image may be different.

[0063] The exposure intensity may be customized based on the user, for example, at least two exposure intensities may be set to two levels of low and high, or at least two exposure intensities may be set to three levels of low, medium and high.

[0064] Different stroboscopic areas have different lighting conditions, and the exposure areas of the images of the object to be measured collected in different stroboscopic areas are also different.

[0065] For example, two exposure intensities (a and b) can be set to capture images of the object to be measured in two stroboscopic areas (area one and area two) to obtain multiple first images; at exposure intensity a, image capture can be performed on the object to be measured in area one and area two respectively to obtain two first images; and at exposure intensity b, image capture can be performed on the object to be measured in area one and area two respectively to obtain two first images.

[0066] like Figure 2 As shown, in some embodiments, before step 110, the image fusion method may further include:

[0067] The image acquisition device is controlled to acquire images of the object to be measured at least twice in different stroboscopic areas.

[0068] In this embodiment, the image acquisition device may be disposed above the object to be measured.

[0069] The field of view of the image acquisition device covers the object to be measured.

[0070] The irradiation direction of the light source in each stroboscopic area is toward the object to be measured.

[0071] In one stroboscopic region, at least two image acquisitions may be performed on the object to be measured, and the acquired images correspond to the same overexposed region but have different exposure intensities.

[0072] The zone strobe can be divided into two zones, three zones or four zones, such as Figure 2An example of a lighting structure with four-zone stroboscopic lighting is given, and the light source plane can be divided into four zones (zone one, zone two, zone three and zone four). The lights in zone one can be lit in sequence, while the lights in other zones are turned off, and image collection is carried out; the lights in zone two can be lit, while the lights in other zones are turned off, and image collection is carried out; the lights in zone three can be lit, while the lights in other zones are turned off, and image collection is carried out; the lights in zone four can be lit, while the lights in other zones are turned off, and image collection is carried out; when light sources in different stroboscopic zones illuminate the object to be measured, the overexposed areas of the first image collected are different.

[0073] For example, three exposure intensities can be used to shoot four stroboscopic images. The three levels of exposure intensity, low, medium and high, can be represented by a, b and c respectively, and 12 first images can be collected. The first zone illumination image under the exposure intensity of a can be named a1, the second zone illumination image can be named a2, the third zone illumination image can be named a3, and the fourth zone illumination image can be named a4. Similarly, the images under the two exposure intensities of b and c can be named b1, b2, b3, b4 and c1, c2, c3, c4 respectively.

[0074] like Figure 3 As shown, when collecting images based on zoned stroboscopic lighting, the positions of the highlights on the non-extreme curved surface are different, that is, the overexposed areas on the object to be measured are different.

[0075] like Figure 4 and Figure 5 The following are examples of images obtained by stroboscopic photography in different areas. Figure 4 This is the image obtained by lighting the upper left corner of the object to be measured. Figure 5 This is an image obtained by lighting the upper right side of the object to be tested. The over-exposed areas of the stroboscopic images in different directions are different.

[0076] Step 120, performing channel separation processing on the plurality of first images at the same exposure intensity to obtain a second image corresponding to each channel in each stroboscopic area;

[0077] In this step, under the same exposure intensity, the image acquisition device can be controlled to acquire multiple first images of the object to be measured.

[0078] The overexposed areas corresponding to the multiple first images at the same exposure intensity are different.

[0079] The first image is a color image, and each first image can be separated into different color channels. For example, the first image can be separated into a red channel (R channel), a green channel (G channel), and a blue channel (B channel).

[0080] For example, a1, a2, a3 and a4 can be used to represent the first image captured by strobing the bright light of each area at an exposure intensity of a. For example, a1 represents the image captured by the bright light of the first area at an exposure intensity of a, and a2 represents the image captured by the bright light of the second area at an exposure intensity of a.

[0081] By performing channel separation on a1, we can obtain the second image Ra1 in the R channel, the second image Ga1 in the G channel, and the second image Ba1 in the B channel of the image captured under the exposure intensity a.

[0082] After channel separation of a1, a2, a3 and a4, the second images Ra1, Ga1 and Ba1 corresponding to a1, the second images Ra2, Ga2 and Ba2 corresponding to a2, the second images Ra3, Ga3 and Ba3 corresponding to a3, and the second images Ra4, Ga4 and Ba4 corresponding to a4 can be obtained respectively.

[0083] Step 130, generating a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel;

[0084] In this step, the same channel corresponds to the second images of each stroboscopic area, and the overexposed areas of each second image are different.

[0085] Each pixel position represents the same physical point in each second image.

[0086] The second images have the same size, the width of the second image is m, and the height of the second image is n. For example, the position information of the upper left corner of each second image can be represented as (0, 0), and the position information of the lower right corner of each second image can be represented as (m-1, n-1).

[0087] In the actual implementation process, multiple second images under the same channel can be stacked, each second image can be placed in the same coordinate system, and then the third image can be generated based on the pixel value at the same pixel position in each second image.

[0088] Based on the pixel value at the same pixel position in each second image, the pixel average value, maximum value, minimum value or pixel weighted average value at the pixel position can be calculated, and then the third image can be generated based on the pixel value at each pixel position.

[0089] Step 140: perform fusion processing on the third images under each channel to obtain a target fused image corresponding to the object to be measured.

[0090] In this step, after obtaining the third image in each channel, the third images of multiple channels may be fused to obtain a candidate fused image.

[0091] Based on the image quality of the candidate fusion images corresponding to each exposure intensity, a candidate fusion image with better quality can be obtained from multiple candidate fusion images and determined as the target fusion image.

[0092] Alternatively, the candidate fused images corresponding to the exposure intensities may be further fused, and the fused image may be determined as the target fused image.

[0093] According to the image fusion method provided in the embodiment of the present application, the partitioned stroboscopic and multi-exposure lighting methods are utilized to collect partitioned stroboscopic images (i.e., first images) under different exposures, and then the channels of the first images under the same exposure intensity are separated to obtain the second image, and a third image is generated based on the pixel values ​​of the second image under the same channel to obtain the target fused image. This method can better remove the overexposed areas in the image, improve the clarity and detail expression of the fused image, and achieve better removal of overexposure.

[0094] In some embodiments, step 130 may include:

[0095] The minimum value of the pixel values ​​at the same pixel position is determined as the target pixel value at the pixel position;

[0096] Based on the target pixel value at each pixel position, a third image is obtained.

[0097] In this embodiment, the pixel values ​​of the same pixel position in different second images may be different.

[0098] For each pixel position, the pixel value of the position may be acquired from all the second images. For example, when the number of the second images is N, each pixel position has N corresponding pixel values.

[0099] In all second images, for each pixel position, the minimum pixel value can be selected as the target pixel value of the position, and the above process is repeated for all pixel positions to obtain a third image consisting of the target pixel values ​​of all pixel positions.

[0100] The third image may reflect the minimum brightness information of all the second images under the same channel.

[0101] In some embodiments, determining the minimum value of pixel values ​​at the same pixel position as the target pixel value at the pixel position may include:

[0102] Stacking multiple second images under the same channel to obtain an image matrix corresponding to the same channel;

[0103] The minimum value of the pixel values ​​at the same pixel position in the image matrix is ​​determined as the target pixel value at the pixel position.

[0104] In this embodiment, for the multiple second images obtained in step 120, the second images of the same channel under the same exposure intensity can be stacked to obtain an image matrix of the same channel.

[0105] The first dimension of the image matrix is ​​the width of the second image, the second dimension of the image matrix is ​​the height of the second image, and the third dimension of the image matrix is ​​the number of multiple second images under the same channel.

[0106] For example, the width of the second image is m, the height of the second image is n, and the number of the second images in the same channel is x, then the first dimension of the image matrix is ​​m, the second dimension is n, and the third dimension is x.

[0107] Based on the multiple second images corresponding to each channel at each exposure intensity, multiple image matrices corresponding to each exposure intensity can be obtained.

[0108] Each image matrix may be sorted in the third dimension according to the size of the pixel value, and then a matrix slice with the smallest pixel value is obtained to obtain the third image.

[0109] In the actual execution process, for the multiple second images corresponding to the exposure intensity a: Ra1, Ga1, Ba1, Ra2, Ga2, Ba2, Ra3, Ga3, Ba3, Ra4, Ga4 and Ba4, the second images of the same channel can be stacked to obtain three image matrices: MatrixR = [Ra1, Ra2, Ra3, Ra4], MatrixG = [Ga1, Ga2, Ga3, Ga4] and MatrixB = [Ba1, Ba2, Ba3, Ba4], assuming that the width and height of the image are m and n respectively, the size of each image matrix is ​​m×n×4 (that is, the first dimension is m, the second dimension is n, and the third dimension is 4).

[0110] Sort MatrixR, MatrixG and MatrixB by pixel value in the third dimension, such as Figure 6 As shown, all pixels are sorted along the direction of the arrow.

[0111] For example, in the matrix before sorting, the pixel value at position (0, 0, 0) is 5, the pixel value at position (0, 0, 1) is 6, the pixel value at position (0, 0, 2) is 8, and the pixel value at position (0, 0, 3) is 7; in the matrix after sorting, the pixel value at position (0, 0, 0) is 5, the pixel value at position (0, 0, 1) is 6, the pixel value at position (0, 0, 2) is 7, and the pixel value at position (0, 0, 3) is 8.

[0112] like Figure 7As shown, the slice with the smallest pixel value in the four slice matrices of the third dimension included in the sorted MatrixR, MatrixG and MatrixB can be taken out and recorded as MatR, MatG and MatB (ie, the third image) respectively.

[0113] According to the image fusion method provided in the embodiment of the present application, by sorting the pixel values ​​of the stroboscopic images in different directions at the same position in the same channel and taking the smallest pixel value to obtain the third image, the high brightness impact caused by the stroboscopic images can be reduced, which helps to retain the detail information in the image and reduces the loss of details caused by overexposure. The same operation is performed on the stroboscopic images of each group of exposure intensities, which can further enhance the detail performance of the image under different exposure conditions.

[0114] In some embodiments, step 140 may include:

[0115] Performing fusion processing on the third images corresponding to each channel under the same exposure intensity to obtain a fourth image corresponding to the object to be measured under the same exposure intensity;

[0116] The fourth images at each exposure intensity are fused to obtain a target fused image corresponding to the object to be measured.

[0117] In this embodiment, the third images corresponding to the channels can be used as new RGB three channels to synthesize a color image, namely, the fourth image.

[0118] The fourth image is an image obtained by performing the initial overexposure removal process.

[0119] For each exposure intensity, a corresponding fourth image can be obtained, and the fourth images corresponding to each exposure intensity can be fused to obtain a target fused image corresponding to the object to be measured.

[0120] The target fused image is an image obtained by performing secondary de-exposure processing.

[0121] In some embodiments, performing fusion processing on the fourth images at each exposure intensity to obtain a target fused image corresponding to the object to be measured may include:

[0122] Processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each fourth image to obtain a weight matrix corresponding to each fourth image;

[0123] A weighted fusion process is performed on at least two fourth images under the exposure intensities based on each weight matrix to obtain a target fused image.

[0124] In this embodiment, the contrast matrix is ​​used to characterize how fast the brightness in the fourth image changes.

[0125] The saturation matrix is ​​used to represent the color vividness of the fourth image.

[0126] The exposure matrix is ​​used to represent the overall brightness level of the fourth image.

[0127] Based on each weight matrix, a weighted fusion processing is performed on the fourth image under at least two exposure intensities. For example, for each pixel position, the pixel value of the corresponding pixel position under different exposure intensities can be multiplied by the respective weights (which can be obtained based on the weight matrix), and then the weighted pixel values ​​are added to obtain the final pixel value of the pixel position in the target fused image.

[0128] In some embodiments, processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each fourth image to obtain a weight matrix corresponding to each fourth image may include:

[0129] Obtain at least one of a first weight corresponding to a contrast matrix of the fourth image, a second weight corresponding to a saturation matrix, and a third weight corresponding to an exposure matrix;

[0130] Based on at least one of the first weight, the second weight and the third weight, at least one of the contrast matrix, the saturation matrix and the exposure matrix of the fourth image is weighted fused to obtain a weight matrix corresponding to the fourth image.

[0131] In this embodiment, the first weight is used to characterize the importance of contrast to the target fused image, the second weight is used to characterize the importance of saturation to the target fused image, and the third weight is used to characterize the importance of exposure to the target fused image.

[0132] In some embodiments, for each fourth image, a corresponding weight matrix can be calculated.

[0133] Each weight matrix may be normalized. For example, the sum of multiple weight matrices may be calculated, and then a normalized weight matrix may be obtained based on a ratio of each weight matrix to the sum of the multiple weight matrices.

[0134] In the actual implementation process, for each fourth image: IMGa, IMGb and IMGc, the contrast matrix, saturation matrix and exposure matrix of each fourth image can be calculated, and then the weight matrix corresponding to the fourth image can be obtained.

[0135] The fourth image can be converted to a grayscale image, with ImgR, ImgG, and ImgB representing the red, green, and blue channels:

[0136] ImgGray=ImgR*0.299+ImgG*0.587+ImgB*0.114

[0137] Then you can use Laplace filtering to process the grayscale image ImgGray and take the absolute value to get the contrast matrix ImgW contrast :

[0138] IUsQ contrast =abs(Filter Laplace (ImgGray)

[0139] Among them, ImgW contrast is the contrast matrix, ImgGray is the grayscale image, Filter Laplace For the Laplace filter, the Laplace kernel can be set to

[0140] The saturation matrix can be obtained based on the average brightness of each pixel in the fourth image and the difference of each channel from the average brightness:

[0141]

[0142]

[0143] Among them, ImgR, ImgG and ImgB represent the red, green and blue channels, mean c is the average brightness of each pixel, ImgW saturation is the saturation matrix.

[0144] A Gaussian curve can be used to obtain the exposure of each pixel near the optimal exposure:

[0145]

[0146] Among them, the optimal exposure μ can be set to 0.5, and the Gaussian standard deviation σ can be set to 0.2. x is the pixel value.

[0147] The pixel values ​​of each channel of the fourth image may be set to IR, IG, and IB.

[0148] The pixel exposure of each channel can be obtained as follows:

[0149]

[0150] Then, the exposure value matrix of the fourth image can be calculated:

[0151] IUsQ exposedness =IR exposedness *IG exposedness *IB exposedness

[0152] The first weight exponents1, the second weight exponents2 and the third weight exponents3 corresponding to the contrast matrix, the saturation matrix and the exposure matrix can be obtained, and each weight is an adjustable power index. For example, each power index can be set to 1.

[0153] Based on the first weight, the second weight and the third weight, the contrast matrix, the saturation matrix and the exposure matrix can be weighted fused to obtain a fused weight matrix:

[0154] Wmap=(ImgW contrast ) exponents1 *(ImgW saturation ) exponents2 *(ImgW exposedness ) exponents3

[0155] Among them, Wmap is the fusion weight matrix, ImgW contrast is the contrast matrix, exponents1 is the first weight, ImgW saturation is the saturation matrix, exponents2 is the second weight, ImgW exposedness is the exposure matrix, and exponents3 is the third weight.

[0156] For each of the fourth images IMGa, IMGb and IMGc, its corresponding fusion weight matrix Wmap1, Wmap2 and Wmap3 can be calculated.

[0157] Then each fusion weight matrix can be normalized to obtain the normalized weight matrix:

[0158] Wmap sum =Wmap1+Wmap2+Wmap3

[0159]

[0160] Then, the Laplacian pyramid can be used to decompose the different fourth images IMGa, IMGb and IMGc into detailed information of multiple different resolution levels, and Wmap1, Wmap2 and Wmap3 can be mapped into multiple resolutions using a Gaussian pyramid for weighted fusion of images at different levels.

[0161] The Laplacian pyramid decomposition of the lth layer of the image Img can be defined as L{Img} l , the Gaussian pyramid decomposition of the weight Wmap layer l is defined as G{Wmap} l .

[0162] The weighted Laplacian pyramid decomposition of the lth layer can be expressed as:

[0163] L{R} l =L{Img} l *G{Wmap} l

[0164] Obtain weighted Laplacian pyramid decomposition layer groups L{R} corresponding to the fourth images IMGa, IMGb, and IMGc, and fuse them at the same layer to obtain a fused Laplacian pyramid decomposition layer group L{R} Fusion For example, for layer l, the weighted Laplacian pyramid decomposition layer L{R} of each fourth image can be l Perform direct addition and fusion.

[0165] Decompose the fused Laplacian pyramid into a layer group L{R} Fusion , we can upsample and accumulate layer by layer starting from the bottom layer, and finally get the final target fused image IMG.

[0166] According to the image fusion method provided in the embodiment of the present application, by performing multi-exposure fusion processing on the initial de-overexposed image (i.e., the fourth image) corresponding to each exposure intensity obtained, calculating the fusion weight based on the contrast, color saturation and exposure of the multi-exposure images, and using the feature pyramid for fusion, the final exposure-suppressed image (i.e., the target fused image) can be obtained, thereby further enhancing the image quality and achieving a better de-overexposure effect.

[0167] The image fusion device provided by the present application is described below. The image fusion device described below and the image fusion method described above can be referenced to each other.

[0168] The image fusion method provided in the embodiment of the present application may be executed by an image fusion device. In the embodiment of the present application, the image fusion device performing the image fusion method is taken as an example to illustrate the image fusion device provided in the embodiment of the present application.

[0169] The embodiment of the present application also provides an image fusion device.

[0170] like Fig. 9 As shown, the image fusion device includes: a first processing module 910 , a second processing module 920 , a third processing module 930 and a fourth processing module 940 .

[0171] A first processing module 910 is used to obtain a plurality of first images corresponding to the object to be measured, where the plurality of first images are images acquired from the object to be measured in different stroboscopic regions under at least two exposure intensities;

[0172] The second processing module 920 is used to perform channel separation processing on the multiple first images under the same exposure intensity to obtain the second image corresponding to each channel under each stroboscopic area;

[0173] A third processing module 930 is used to generate a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel;

[0174] The fourth processing module 940 is used to perform fusion processing on the third images under each channel to obtain a target fused image corresponding to the object to be measured.

[0175] According to the image fusion device provided in the embodiment of the present application, the partitioned stroboscopic and multi-exposure lighting methods are used to collect partitioned stroboscopic images (i.e., first images) under different exposures, and then the channels of the first images under the same exposure intensity are separated to obtain the second image, and the third image is generated based on the pixel values ​​of the second image under the same channel to obtain the target fused image. This can better remove the overexposed area in the image, improve the clarity and detail expression of the fused image, and have a better effect of removing overexposure.

[0176] In some embodiments, the third processing module 930 may also be used to:

[0177] The minimum value of the pixel values ​​at the same pixel position is determined as the target pixel value at the pixel position;

[0178] Based on the target pixel value at each pixel position, a third image is obtained.

[0179] In some embodiments, the third processing module 930 may also be used to:

[0180] The multiple second images under the same channel are stacked to obtain an image matrix corresponding to the same channel; the first dimension of the image matrix is ​​the width of the second image, the second dimension of the image matrix is ​​the height of the second image, and the third dimension of the image matrix is ​​the number of the multiple second images under the same channel;

[0181] The minimum value of the pixel values ​​at the same pixel position in the image matrix is ​​determined as the target pixel value at the pixel position.

[0182] In some embodiments, the fourth processing module 940 may also be used to:

[0183] Performing fusion processing on the third images corresponding to each channel under the same exposure intensity to obtain a fourth image corresponding to the object to be measured under the same exposure intensity;

[0184] The fourth images at each exposure intensity are fused to obtain a target fused image corresponding to the object to be measured.

[0185] In some embodiments, the fourth processing module 940 may also be used to:

[0186] Processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each fourth image to obtain a weight matrix corresponding to each fourth image;

[0187] A weighted fusion process is performed on at least two fourth images under the exposure intensities based on each weight matrix to obtain a target fused image.

[0188] In some embodiments, the fourth processing module 940 may also be used to:

[0189] Obtain at least one of a first weight corresponding to the contrast matrix of the fourth image, a second weight corresponding to the saturation matrix, and a third weight corresponding to the exposure matrix; based on at least one of the first weight, the second weight, and the third weight, perform weighted fusion processing on at least one of the contrast matrix, the saturation matrix, and the exposure matrix of the fourth image to obtain a weight matrix corresponding to the fourth image.

[0190] In some embodiments, the image fusion device may further include a fifth processing module, which is used to control the image acquisition device to perform at least two image acquisitions on the object to be measured in different stroboscopic areas before acquiring multiple first images corresponding to the object to be measured; the image acquisition device is arranged above the object to be measured, the field of view of the image acquisition device covers the object to be measured, and the illumination direction of the light source in each stroboscopic area is toward the object to be measured.

[0191] The image fusion device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0192] The image fusion device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0193] The image fusion device provided in the embodiment of the present application can achieve Figures 1 to 8 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0194] In some embodiments, Fig.10 As shown, an embodiment of the present application further provides an electronic device 1000, including a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the program is executed by the processor 1001, each process of the above-mentioned image fusion method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0195] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0196] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-mentioned image fusion method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0197] On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to perform the various processes of the above-mentioned image fusion method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0198] On the other hand, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image fusion method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0199] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0200] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image fusion method, characterized in that: include: Acquire a plurality of first images corresponding to the object to be measured, wherein the plurality of first images are images acquired under at least two exposure intensities and in different stroboscopic regions of the object to be measured; Performing channel separation processing on the plurality of first images under the same exposure intensity to obtain second images corresponding to each channel under each stroboscopic area; Based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel, a third image under the same channel is generated; The third images under each of the channels are fused to obtain a target fused image corresponding to the object to be measured.

2. The image fusion method according to claim 1, characterized in that: The step of generating a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel comprises: Determine the minimum value of pixel values ​​at the same pixel position as the target pixel value at the pixel position; The third image is obtained based on the target pixel value at each of the pixel positions.

3. The image fusion method according to claim 2, characterized in that: The step of determining the minimum value of pixel values ​​at the same pixel position as the target pixel value at the pixel position includes: Performing stacking processing on the multiple second images under the same channel to obtain an image matrix corresponding to the same channel; the first dimension of the image matrix is ​​the width of the second image, the second dimension of the image matrix is ​​the height of the second image, and the third dimension of the image matrix is ​​the number of the multiple second images under the same channel; The minimum value of the pixel values ​​at the same pixel position in the image matrix is ​​determined as the target pixel value at the pixel position.

4. The image fusion method according to any one of claims 1 to 3, characterized in that: The step of fusing the third images under each of the channels to obtain a target fused image corresponding to the object to be measured includes: Performing fusion processing on the third images corresponding to the channels under the same exposure intensity to obtain a fourth image corresponding to the object to be measured under the same exposure intensity; The fourth images under the exposure intensities are fused to obtain a target fused image corresponding to the object to be measured.

5. The image fusion method according to claim 4, characterized in that: The step of fusing the fourth images at the exposure intensities to obtain a target fused image corresponding to the object to be measured includes: Processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each of the fourth images to obtain a weight matrix corresponding to each of the fourth images; Based on each of the weight matrices, weighted fusion processing is performed on at least two fourth images under the exposure intensity to obtain the target fused image.

6. The image fusion method according to claim 5, characterized in that: The step of processing at least one of a contrast matrix, a saturation matrix, and an exposure matrix of each of the fourth images to obtain a weight matrix corresponding to each of the fourth images includes: Obtain at least one of a first weight corresponding to the contrast matrix of the fourth image, a second weight corresponding to the saturation matrix, and a third weight corresponding to the exposure matrix; based on the first weight, the second weight, and at least one of the third weights, perform weighted fusion processing on at least one of the contrast matrix, the saturation matrix, and the exposure matrix of the fourth image to obtain a weight matrix corresponding to the fourth image.

7. The image fusion method according to any one of claims 1 to 3, characterized in that: Before acquiring a plurality of first images corresponding to the object to be measured, the method further includes: The image acquisition device is controlled to perform at least two image acquisitions on the object to be measured in different stroboscopic areas; the image acquisition device is arranged above the object to be measured, the field of view of the image acquisition device covers the object to be measured, and the irradiation direction of the light source in each stroboscopic area is toward the object to be measured.

8. An image fusion device, characterized in that: include: A first processing module is used to obtain a plurality of first images corresponding to the object to be measured, wherein the plurality of first images are images acquired under at least two exposure intensities and in different stroboscopic regions of the object to be measured; A second processing module is used to perform channel separation processing on the multiple first images under the same exposure intensity to obtain a second image corresponding to each channel under each stroboscopic area; A third processing module, configured to generate a third image under the same channel based on the pixel value at the same pixel position in the second image corresponding to each stroboscopic area under the same channel; The fourth processing module is used to perform fusion processing on the third images under each of the channels to obtain a target fused image corresponding to the object to be measured.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image fusion method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image fusion method according to any one of claims 1 to 7 is implemented.

Images