Image generation method and system based on multi-exposure and multi-focal-length fusion

Abstract

The invention discloses an image generation method and system based on multi-exposure and multi-focal-length fusion, and the method comprises the steps: obtaining multi-exposure image data, carrying out the Laplacian pyramid decomposition of each image, and fusing the Laplacian pyramids of a plurality of images into a new Laplacian pyramid; carrying out nonlinear stretching on the fused top layer image of the new Laplacian pyramid; carrying out pyramid inverse decomposition to obtain a recovered and fused multi-exposure fusion image; obtaining a plurality of images after multi-exposure fusion of the cameras with different focal lengths, and zooming the images of the cameras with different focal lengths into the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length; matching the zoomed images with different focal lengths with the image blocks of the shortest focal length image to obtain a final image estimation block; and combining and generating a new clear image according to all the image estimation blocks. According to the invention, high-definition, wide-view and high-dynamic-range images can be generated.

Description

Technical Field

[0001] The present invention mainly relates to the field of image processing technology, and in particular to an image generation method and system based on multi-exposure and multi-focal length fusion. Background Art

[0002] Vehicle-grade cameras for rail transit scenarios often require superior photosensitivity (sensitivity, signal-to-noise ratio, and dynamic range). In theory, the higher the signal-to-noise ratio, the higher the contrast of the image, the more detailed the presentation of details, and the more conducive to the subsequent positioning detection algorithm to identify. In order to adapt to some extreme application scenarios, such as driving in tunnels, vehicle-mounted cameras have very high requirements for dynamic range, which often need to reach more than 120dB. In order to achieve such a high dynamic range, it is necessary to use HDR mode for multiple exposures and superimpose them into an image output, that is, multi-exposure fusion. Laplace pyramid fusion is one of the simple and easy-to-implement effective methods in multi-exposure fusion. However, if the exposure parameters of the input multi-exposure image are not properly selected, the output image may have low contrast and unclear details in dark areas; secondly, if the number of pyramid layers is not within a reasonable range, the generated fused image is prone to lose some textures in the original image, and overexposure occurs in bright places such as the sky and lights.

[0003] Multi-sensor fusion is a more common development direction of intelligent driving. When information alignment and fusion based on multiple sensors is used for applications such as target detection and segmentation, higher accuracy and safety can often be achieved. Among them, fusion based on multi-focal length cameras is a fusion method that takes into account both a large field of view and small targets in the distance. How to give full play to the advantages of cameras with different focal lengths and fuse images of telephoto cameras with those of short-focal length cameras is a research hotspot.

[0004] Multi-exposure fusion can improve the dynamic range of the image, and multi-focal length fusion can obtain a clear image with a wide field of view. Currently, there is little research on the integration of multi-exposure and long- and short-focal length fusion. How to combine the advantages of both is a topic worth studying. Summary of the invention

[0005] In view of the technical problems existing in the prior art, the present invention provides an image generation method and system based on multi-exposure and multi-focal length fusion for generating high-definition, wide-viewing-angle and high-dynamic-range images.

[0006] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0007] An image generation method based on multi-exposure and multi-focal length fusion comprises the steps of:

[0008] Multi-exposure image Laplacian pyramid decomposition and fusion: Get multi-exposure image data, perform Laplacian pyramid decomposition on each image, and weightedly fuse the Laplacian pyramids of multiple images into a new Laplacian pyramid;

[0009] Nonlinear enhancement of the top image of the Laplacian pyramid: nonlinear stretching of the top image of the fused new Laplacian pyramid;

[0010] Multi-exposure image fusion reconstruction: According to the Laplacian pyramid of the nonlinear enhancement of the top image, the pyramid is inversely decomposed to obtain the restored and fused multi-exposure fused image;

[0011] Multi-focal length camera image alignment: obtain multiple multi-exposure fused images from cameras with different focal lengths, and scale the images from cameras with different focal lengths to the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length;

[0012] Similar image block matching: Match the image blocks of the scaled images with different focal lengths with the image blocks of the shortest focal length image to obtain the final image estimation block;

[0013] Multi-focal length fusion image solution: Based on all image estimation blocks, a new clear image is generated.

[0014] Preferably, the specific process of multi-focal length camera image alignment is:

[0015] The multi-exposure fused images of multiple cameras with different focal lengths are used to calculate the coordinate transformation relationship between the cameras with different focal lengths according to the joint calibration information of the cameras, and the images of the cameras with other focal lengths are scaled to the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length.

[0016] Preferably, the four vertices of the images of any other focal length camera except the shortest focal length camera are converted into the shortest focal length camera image coordinate system according to the joint calibration information to obtain the corresponding four-point coordinates, the minimum circumscribed rectangle of the four points is obtained on the shortest focal length camera image, the size of the non-shortest focal length camera image is scaled to the minimum circumscribed rectangle, and the alignment with the shortest focal length camera image is completed.

[0017] Preferably, the specific formula for nonlinearly stretching the top image of the fused new Laplacian pyramid is:

[0018]

[0019] Among them, R k represents the top layer of the Laplacian pyramid, i.e., the kth layer image; L is the top layer image of the Laplacian pyramid after nonlinear enhancement; the parameter z is related to the image R k The histogram of C is related to R k The parameter value for which the cumulative distribution function is equal to 0.1.

[0020] Preferably, the specific process of similar image block matching is:

[0021] Take any image block obtained from the shortest focal length camera image, and let the i-th block be y i,o ; Correspondingly, in any other zoomed image of a corresponding focal length camera, a neighboring window Ω of size m×m is set with the i-th image block as the center, and the j-th image block in the window Ω is represented by y i,j Represents, calculate its difference with the image block y in the shortest focal length camera image i,o The Euclidean distance of

[0022] Select the window Ω and y i,o The t image blocks with the smallest Euclidean distance are taken as the most similar blocks, and the Euclidean distance D i,j To divide the weights, we can get the image block y in the shortest focal length image i,o The clear image estimation block i .

[0023] Preferably, the calculation formula of the Euclidean distance is:

[0024]

[0025] Among them, p and q represent the horizontal and vertical coordinates respectively, D i,j represents the calculated y i,o With y i,j The Euclidean distance between .

[0026] Preferably, the clear image estimation block i for:

[0027]

[0028] where ω i,j represents the weight, c represents the number of camera images with non-shortest focal lengths and image block y i,o After the coordinate transformation, there is a corresponding relationship, that is, the final clear image estimation block is obtained by weighted averaging of small image blocks from c cameras.

[0029] Preferably, the specific process of solving the multi-focal length fusion image is:

[0030] For the pixel value at the coordinate (p′, q′) on the shortest focal length camera image, if the point is covered by the clear image block estimated after matching similar image blocks, the pixel value after fusion is calculated as:

[0031]

[0032] Where f(p′,q′) is the pixel value with coordinates (p′,q′) in the shortest focal length camera image calculated after fusion; N means that there are N clear estimation blocks covering the pixel point (p′,q′); e x (p′, q′) is the pixel value of all the x-th clear estimated blocks covering the point at (p′, q′); if the point at (p′, q′) is not covered by the clear image block estimated after matching similar image blocks, the original value is retained.

[0033] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0034] The present invention further discloses an image generation system based on multi-exposure and multi-focal length fusion, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] The present invention nonlinearly enhances the top-level image of the Laplace pyramid so that the generated image absorbs more low-frequency information in dark areas, enhances the brightness of dark areas of the image while not affecting the brightness of bright areas of the image to avoid overexposure, and makes the algorithm more tolerant to the selection of exposure parameters for multi-exposure images; secondly, the present invention uses multi-focal length camera fusion to generate an image that takes into account a large field of view and small targets in the distance, and generates clearer image textures; finally, the present invention connects multi-exposure fusion and multi-focal length fusion in series to combine the advantages of the two and generate high-definition, wide-field of view and high dynamic range images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure is a flow chart of an embodiment of the image generation method based on multi-exposure and multi-focal length fusion of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0039] like Figure 1 As shown, the image generation method based on multi-exposure and multi-focal length fusion provided by the embodiment of the present invention includes the steps of:

[0040] Multi-exposure image Laplacian pyramid decomposition and fusion: Get multi-exposure image data, perform Laplacian pyramid decomposition on each image, and weightedly fuse the Laplacian pyramids of multiple images into a new Laplacian pyramid;

[0041] Nonlinear enhancement of the top image of the Laplacian pyramid: Nonlinearly stretch the top image of the fused new Laplacian pyramid, which is a small-sized image containing only low-frequency information, to increase the brightness of the dark area of ​​the image.

[0042] Multi-exposure image fusion reconstruction: Use the enhanced Laplacian pyramid, upsample from the top layer, superimpose Laplacian details of the same scale, then upsample and superimpose details, recursively to the highest resolution to obtain the fused image;

[0043] Multi-focal length camera image alignment: For the multi-exposure fused images obtained by the above steps, calculate the coordinate transformation relationship between the cameras with different focal lengths according to the joint calibration information of the cameras, and scale the images of the cameras with other focal lengths to the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length;

[0044] Similar image block matching: For the area in the shortest focal length image that has a corresponding relationship with other focal length images, it is divided into multiple image blocks, and then block matching is performed with the image blocks in the different focal length images after scaling in the previous step, and similar texture image blocks are searched. Different weights are assigned according to the similarity between the searched image blocks and the small image blocks in the shortest focal length image, and the final image estimation block with clear texture is solved by weighted average;

[0045] Multi-focal length fusion image solution: Based on all similar image blocks obtained, a weighted average solution is performed according to their positions in the image of the camera with the shortest focal length.

[0046] Specifically, the Laplacian pyramid decomposition and fusion of multi-exposure images refers to converting multiple RGB images with different exposures into HSV color space, performing Laplacian pyramid decomposition on the brightness V components therein respectively, and merging the Laplacian pyramids of multiple images into a new Laplacian pyramid by assigning weights according to contrast, saturation and brightness values.

[0047] The decomposition, weight solution and pyramid fusion methods are conventional methods and will not be described in detail here.

[0048] In the prior art, if the exposure parameters of the multi-exposure fusion input image are improperly selected, or the ambient light environment of the environment scene itself is dark, the brightness of the dark area of ​​the fused image will be low, and the texture details will be easily lost. Ordinary enhancement methods enhance the dark area while easily leading to excessive enhancement of the bright area and overexposure. In the Laplace pyramid fusion process, the present invention performs nonlinear stretching on the top layer of the Laplace pyramid, that is, a small-sized image containing only low-frequency information, to replace the original top layer image and improve the brightness of the dark area of ​​the image; the brightness of the dark area of ​​the image is enhanced, and the exposure parameter selection for the multi-exposure image is more compatible; the generated image absorbs more low-frequency information in the dark area, enhances the brightness of the dark area of ​​the image, and does not affect the brightness of the bright area of ​​the image to avoid overexposure, making the exposure parameter selection for the multi-exposure image more tolerant. Specifically, the nonlinear enhancement of the top layer image of the Laplace pyramid refers to nonlinear stretching of the top layer image of the fused Laplace pyramid to improve the brightness of the dark area of ​​the image. The specific formula is as follows:

[0049]

[0050] In the above formula, R k represents the image at the top layer of the Laplace pyramid, i.e., the kth layer, which is the standardized image; L is the top layer image of the Laplace pyramid after nonlinear enhancement, and the parameter z is related to the image R k The histogram of C is related to R k The parameter value for which the cumulative distribution function is equal to 0.1.

[0051] Specifically, the multi-exposure image fusion reconstruction is to perform pyramid inverse decomposition according to the Laplacian pyramid whose top layer has been nonlinearly enhanced to obtain the restored and fused multi-exposure fused image.

[0052] Specifically, multi-focal length camera image alignment refers to calculating the coordinate transformation relationship between the shortest focal length camera image and other focal length camera images according to the joint calibration information of the cameras for the multi-exposure fused images of cameras with different focal lengths, and scaling the other focal length camera images to the same size as the image blocks of the corresponding areas in the shortest focal length camera image. The method is as follows:

[0053] Suppose that the four vertices of any focal length camera image other than the shortest focal length camera are converted into the shortest focal length camera image coordinate system according to the joint calibration information to obtain the corresponding four-point coordinates, and the minimum circumscribed rectangle of the four points is obtained on the shortest focal length camera image. Taking one of the non-shortest focal length cameras as an example, the vertex coordinates of the minimum circumscribed rectangle corresponding to it on the shortest focal length camera image are (x1, y1), (x1+w, y1), (x1, y1+h), (x1+w, y1+h), where (x1, y1) is the coordinate of the upper left corner of the rectangle, and w and h are the width and height of the rectangle respectively. The image of the non-shortest focal length camera is scaled to a rectangle of size w×h to complete the alignment with the shortest focal length camera image. The other non-shortest focal length camera images are aligned with the shortest focal length camera image using the same method.

[0054] Specifically, similar image block matching refers to the process of searching for similar image blocks in images of cameras with different focal lengths and solving for clear estimated image blocks. The mutual mapping relationship between images is determined through the joint calibration information of multiple cameras with different focal lengths. Based on this, the images of other cameras except the camera with the shortest focal length are reduced to the same size as the corresponding area block of the image of the camera with the shortest focal length. For small image blocks in the image of the camera with the shortest focal length, the window range of the image block in the scaled images of cameras with other focal lengths is determined through the mapping relationship, and similar texture image blocks are searched in the window. Different weights are assigned according to the similarity between the searched image blocks and the small image blocks in the image of the camera with the shortest focal length, and the final image block with clear texture is solved by weighted method. The specific process is as follows:

[0055] After the images of cameras with different focal lengths are aligned, the regions with coordinates (x1, y1), (x1+w, y1), (x1, y1+h), (x1+w, y1+h) on the image of the camera with the shortest focal length, i.e., the corresponding rectangular regions after scaling of the image acquired by a camera with other focal lengths, are divided into (w-n+1)×(h-n+1) n×n image blocks. Similarly, the corresponding images of cameras with other focal lengths with a scaled size of w×h are also divided into (w-n+1)×(h-n+1) n×n image blocks. It can be seen that there is a one-to-one correspondence between the small blocks in the non-shortest focal length camera image and the shortest focal length camera image.

[0056] For any image patch obtained from the shortest focal length camera image, let the i-th patch be y i,o Correspondingly, in any other zoomed image of a focal length camera, a neighboring window Ω of size m×m is set with the i-th image block as the center, and the j-th image block in the window Ω is represented by y i,j Represents, calculate its difference with the image block y in the shortest focal length camera image i,o The Euclidean distance is as follows:

[0057]

[0058] In the above formula, p and q represent the horizontal and vertical coordinates respectively, D i,j represents the calculated y i,o With y i,j The Euclidean distance between .

[0059] Select the window Ω and y i,o The t image blocks with the smallest Euclidean distance are taken as the most similar blocks, and the Euclidean distance D i,j To divide the weight:

[0060] ω i,j =exp(-D i,j / h)

[0061] Among them, h is the filtering parameter.

[0062] Therefore, the image block y in the shortest focal length image i,o The clear estimation block i for:

[0063]

[0064] In the above formula, c represents the number of camera images with non-shortest focal lengths and image block y i,o After the coordinate transformation, there is a corresponding relationship, that is, the final clear image estimation block is obtained by weighted averaging of small image blocks from c cameras.

[0065] Specifically, the multi-focal length fusion image solution refers to combining all the solved clear image estimation blocks to generate a new clear image. The method is as follows:

[0066] For the pixel value at the coordinate (p′, q′) on the shortest focal length camera image, if the point is covered by the clear image block estimated after matching similar image blocks, the pixel value after fusion is calculated as:

[0067]

[0068] In the above formula, f(p′,q′) is the pixel value with coordinates (p′,q′) in the shortest focal length camera image calculated after fusion; N means that there are N clear estimation blocks covering the pixel point (p′,q′); e x (p′, q′) is the pixel value of all the x-th clear estimated blocks covering the point at (p′, q′); if the point at (p′, q′) is not covered by the clear image block estimated after matching similar image blocks, the original value is retained.

[0069] For cameras with two or more different focal lengths, the present invention fuses the multi-exposure images of each camera and then fuses the fused images of the cameras with different focal lengths again to achieve the fusion of multi-exposure and multi-focal length images, thereby obtaining new image data with wide viewing angle, clear texture and wide dynamic range.

[0070] The present invention determines the image scaling size of cameras with different focal lengths and the search range of similar image blocks based on joint calibration information, searches for similar texture image blocks and generates image estimation blocks with clearer texture by weighted averaging according to the degree of similarity, and combines them to generate new images with a wide viewing angle and clearer texture. The generated images take into account both a large field of view and small targets in the distance, and have clearer textures.

[0071] The present invention improves the dynamic range of an image through multi-exposure fusion, and then uses the multi-exposure fused image as input for multi-focal length fusion to generate new image data in combination, thereby obtaining new image data with a wide viewing angle, clear texture and a wide dynamic range.

[0072] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0073] like Figure 1 As shown, the image generation method based on multi-exposure and multi-focal length fusion provided by the embodiment of the present invention includes the steps of:

[0074] Step 1) Laplacian pyramid decomposition and fusion of multi-exposure images

[0075] For two cameras with different focal lengths, multiple RGB images with different exposures are obtained at similar times. Taking the short-focus camera image as an example, the multiple RGB images with different exposures obtained by the camera are converted into HSV color space, and the brightness V component is decomposed by Laplacian pyramid respectively, and the Laplacian pyramids of multiple images are fused into a new Laplacian pyramid by assigning weights according to contrast, saturation and brightness values. The decomposition, weight solution and pyramid fusion methods are conventional methods and will not be repeated here.

[0076] Similarly, for telephoto images, Laplace decomposition and fusion are performed in the same way.

[0077] Step 2) Nonlinear enhancement of the top image of the Laplacian pyramid

[0078] Taking the short-focus camera as an example, after obtaining the fused new Laplacian pyramid, the top layer of the fused Laplacian pyramid is nonlinearly stretched to increase the brightness of the dark area of ​​the image. The specific formula is as follows:

[0079]

[0080]

[0081] In the above formula, R k represents the image at the top layer of the Laplacian pyramid, i.e., the kth layer, which is the standardized image; L is the top layer image of the Laplacian pyramid after nonlinear enhancement; the parameter z is related to the image R k The histogram of k The parameter value for which the cumulative distribution function is equal to 0.1.

[0082] Similarly, the Laplacian pyramid after telephoto camera fusion is processed in the same way.

[0083] Step 3) Multi-exposure image fusion and reconstruction

[0084] After the top layer of the Laplacian pyramid is nonlinearly enhanced, the pyramid is inversely decomposed to obtain the restored fused multi-exposure fusion image. Since there are two cameras with different focal lengths, two multi-exposure images are obtained.

[0085] Step 4) Multi-focal length camera image alignment

[0086] For the multi-exposure fusion images of the telephoto and short-focus cameras, the coordinate transformation relationship between the short-focus camera image and the telephoto camera image is calculated according to the joint calibration information of the cameras, and the telephoto camera image is scaled to the same size as the image block of the corresponding area in the short-focus camera image. The method is as follows:

[0087] Suppose the four vertices of the telephoto camera image are converted into the short-focus camera image coordinate system according to the joint calibration information to obtain the corresponding four-point coordinates, and the minimum enclosing rectangle of the four points is obtained on the short-focus camera image. Suppose the vertex coordinates of the minimum enclosing rectangle are (x1, y1), (x1+w, y1), (x1, y1+h), (x1+w, y1+h), where (x1, y1) is the coordinate of the upper left corner of the rectangle, and w and h are the width and height of the rectangle respectively; scale the telephoto camera image to a rectangle of size w×h to complete the alignment with the short-focus camera image.

[0088] Step 5) Similar image block matching

[0089] The clear image patch searched in the telephoto camera is used to estimate the clear estimation patch of the short-focus image as follows:

[0090] After the long-focus and short-focus camera images are aligned, the area with coordinates (x1, y1), (x1+w, y1), (x1, y1+h), (x1+w, y1+h) on the short-focus camera image, that is, the rectangular area corresponding to the image acquired by the long-focus camera after scaling, is divided into (w-n+1)×(h-n+1) n×n image blocks. Similarly, the long-focus camera image with a scaled size of w×h is also divided into (w-n+1)×(h-n+1) n×n image blocks. It can be seen that there is a one-to-one correspondence between the small blocks in the short-focus camera image and the long-focus camera image.

[0091] For any image patch obtained from a short-focus camera image, let its i-th patch be y i,o ; Correspondingly, in the corresponding telephoto camera zoomed image, a neighboring window Ω of size m×m is set with the i-th image block as the center, and the j-th image block in the window Ω is represented by y i,j Represents, calculate its difference with the image block y in the shortest focal length camera image i,o The Euclidean distance is as follows:

[0092]

[0093] In the above formula, p and q represent the horizontal and vertical coordinates respectively, D i,j represents the calculated y i,o With y i,o The Euclidean distance between .

[0094] Select the window Ω and y i,o The t image blocks with the smallest Euclidean distance are taken as the most similar blocks, and the Euclidean distance D i,j To divide the weight:

[0095] ω i,j =exp(-D i,j / h)

[0096] Among them, h is the filtering parameter.

[0097] Therefore, the image block y in the short-focus image i,o The clear estimation block i for:

[0098]

[0099] Step 6) Multi-focal length fusion image solution

[0100] After solving the obtained clear image estimation blocks, combine them to generate a new clear image as follows:

[0101] For the pixel value at the coordinate (p′, q′) on the short-focus camera image, if the point is covered by the clear image block estimated after matching similar image blocks, the pixel value after fusion is calculated as:

[0102]

[0103] In the above formula, f(p′,q′) is the pixel value with coordinates (p′,q′) in the short-focus camera image calculated after fusion; N means that there are N clear estimation blocks covering the pixel point (p′,q′); e x (p′, q′) is the pixel value of all the x-th clear estimated blocks covering the point at (p′, q′); if the point at (p′, q′) is not covered by the clear image block estimated after matching similar image blocks, the original value is retained.

[0104] The present invention nonlinearly enhances the top-level image of the Laplace pyramid so that the generated image absorbs more low-frequency information in dark areas, enhances the brightness of dark areas of the image while not affecting the brightness of bright areas of the image to avoid overexposure, and makes the algorithm more tolerant to the selection of exposure parameters for multi-exposure images; secondly, the present invention uses multi-focal length camera fusion to generate an image that takes into account a large field of view and small targets in the distance, and generates clearer image textures; finally, the present invention connects multi-exposure fusion and multi-focal length fusion in series to combine the advantages of the two and generate high-definition, wide-field of view and high dynamic range images.

[0105] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0106] The present invention further discloses an image generation system based on multi-exposure and multi-focal length fusion, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0107] The medium and system of the present invention correspond to the above method and also have the advantages described in the above method.

[0108] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] Glossary:

[0110] RGB image: A representation of color images that represents a variety of colors by changing the values ​​of the three channels red (R), green (G), and blue (B) and their superposition.

[0111] Image HSV space: image color model, including image hue (H), saturation (S) and value (V).

[0112] Camera focal length: Camera focal length refers to the distance from the optical center of the lens to the imaging plane such as film, CCD or CMOS.

[0113] Exposure time: The time interval from the opening to the closing of the camera shutter determines the total length of time the photosensitive element receives light.

[0114] Joint calibration between cameras: By calculating the relative spatial position relationship between two or more cameras, their image data can be fused and processed in a unified coordinate system.

[0115] Gaussian Pyramid: Gaussian Pyramid is a multi-resolution image representation method generated by Gaussian smoothing and downsampling operations.

[0116] Laplacian Pyramid: Laplacian Pyramid is a multi-resolution image representation method based on Gaussian pyramid. It generates the next layer (low-resolution layer) from each layer of Gaussian pyramid, upsamples the next layer and calculates the difference between the current layer and the upsampled image to obtain the Laplacian layer of the current layer.

[0117] Image dynamic range: The difference between the maximum and minimum brightness in an image, usually used to measure the image sensor's ability to record the brightest and darkest parts of a scene.

[0118] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. An image generation method based on multi-exposure and multi-focal length fusion, characterized in that: Includes steps: Multi-exposure image Laplacian pyramid decomposition and fusion: Get multi-exposure image data, perform Laplacian pyramid decomposition on each image, and weightedly fuse the Laplacian pyramids of multiple images into a new Laplacian pyramid; Nonlinear enhancement of the top image of the Laplacian pyramid: nonlinear stretching of the top image of the fused new Laplacian pyramid; Multi-exposure image fusion reconstruction: According to the Laplacian pyramid of the nonlinear enhancement of the top image, the pyramid is inversely decomposed to obtain the restored and fused multi-exposure fused image; Multi-focal length camera image alignment: obtain multiple multi-exposure fused images from cameras with different focal lengths, and scale the images from cameras with different focal lengths to the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length; Similar image block matching: Match the image blocks of the scaled images with different focal lengths with the image blocks of the shortest focal length image to obtain the final image estimation block; Multi-focal length fusion image solution: Based on all image estimation blocks, a new clear image is generated.

2. The image generation method based on multi-exposure and multi-focal length fusion according to claim 1, characterized in that: The specific process of multi-focal length camera image alignment is: The multi-exposure fused images of multiple cameras with different focal lengths are used to calculate the coordinate transformation relationship between the cameras with different focal lengths according to the joint calibration information of the cameras, and the images of the cameras with other focal lengths are scaled to the same size as the image blocks of the corresponding areas in the image of the camera with the shortest focal length.

3. The image generation method based on multi-exposure and multi-focal length fusion according to claim 2, characterized in that: The four vertices of the images of any other focal length camera except the shortest focal length camera are converted into the shortest focal length camera image coordinate system according to the joint calibration information to obtain the corresponding four-point coordinates. The minimum enclosing rectangle of the four points is obtained on the shortest focal length camera image, and the size of the non-shortest focal length camera image is scaled to the minimum enclosing rectangle to complete the alignment with the shortest focal length camera image.

4. The image generation method based on multi-exposure and multi-focal length fusion according to claim 1, 2 or 3, characterized in that: The specific formula for nonlinear stretching of the top image of the fused new Laplacian pyramid is: Among them, R k represents the top layer of the Laplacian pyramid, i.e., the kth layer image; L is the top layer image of the Laplacian pyramid after nonlinear enhancement; the parameter z is related to the image R k The histogram of C is related to R k The parameter value for which the cumulative distribution function is equal to 0.

1.

5. The image generation method based on multi-exposure and multi-focal length fusion according to claim 1, 2 or 3, characterized in that: The specific process of similar image block matching is as follows: Take any image block obtained from the shortest focal length camera image, and let the i-th block be y i,o ; Correspondingly, in any other zoomed image of a corresponding focal length camera, a neighboring window Ω of size m×m is set with the i-th image block as the center, and the j-th image block in the window Ω is represented by y i,j Represents, calculate its difference with the image block y in the shortest focal length camera image i,o The Euclidean distance of Select the window Ω and y i,o The t image blocks with the smallest Euclidean distance are taken as the most similar blocks, and the Euclidean distance D i,j To divide the weights, we can get the image block y in the shortest focal length image i,o The clear image estimation block i .

6. The image generation method based on multi-exposure and multi-focal length fusion according to claim 5, characterized in that: The calculation formula of Euclidean distance is: Among them, p and q represent the horizontal and vertical coordinates respectively, D i,j represents the calculated y i,o With y i,j The Euclidean distance between .

7. The image generation method based on multi-exposure and multi-focal length fusion according to claim 6, characterized in that: Clear Image Estimation Block i for: where ω i,j represents the weight, c represents the number of camera images with non-shortest focal lengths and image block y i,o After the coordinate transformation, there is a corresponding relationship, that is, the final clear image estimation block is obtained by weighted averaging of small image blocks from c cameras.

8. The image generation method based on multi-exposure and multi-focal length fusion according to claim 1, 2 or 3, characterized in that: The specific process of solving multi-focal length fusion images is as follows: For the pixel value at the coordinate (p′, q′) on the shortest focal length camera image, if the point is covered by the clear image block estimated after matching similar image blocks, the pixel value after fusion is calculated as: Where f(p′,q′) is the pixel value with coordinates (p′,q′) in the shortest focal length camera image calculated after fusion; N means that there are N clear estimation blocks covering the pixel point (p′,q′); e x (p′, q′) is the pixel value of the x-th clear estimation block at (p′, q′) covering the point; If the point at (p′, q′) is not covered by the clear image block estimated after matching similar image blocks, the original value is retained.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 8.

10. An image generation system based on multi-exposure and multi-focal length fusion, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 8.

Images