A panoramic stitching image fusion method, system and application

Abstract

The present invention belongs to the field of image processing technology and discloses a panoramic stitching image fusion method, system and application. The method uses the power average fusion of a scale pyramid to achieve a smooth transition while retaining the detailed information of the input image. The method can also choose between smoothness and details by setting the exponent size of the power function as needed. In response to the shortcomings of existing image fusion algorithms, the present invention improves the fusion method of the Laplace pyramid based on the Laplace pyramid fusion algorithm. The power average is used instead of the arithmetic average to achieve a smooth transition while retaining the detailed information of the input image. The method can also choose between smoothness and details by setting the exponent size of the power function as needed. The expected benefits and commercial value of the technical solution of the present invention after transformation are: improving the problem of people disappearing at the seams of the vehicle's 360-degree surround view image and improving vehicle driving safety.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a panoramic stitching image fusion method, system and application. Background Art

[0002] Traditional image fusion algorithms generate a fused image that smoothly transitions between the input images. However, objects that don't overlap between the two input images are very blurry, which is sufficient for everyday photography. However, in automotive and airborne optoelectronic systems, panoramic surround vision is used to detect surrounding obstacles, and objects that appear in only one input image are still crucial. Therefore, the fused image must achieve a smooth transition while preserving the clarity of objects that don't overlap between the two input images.

[0003] Through the above analysis, the problems and defects of the existing technology are: in the existing surround observation system, the target easily disappears at the seam, making it impossible to effectively retain the detailed information of multiple input images, resulting in unclear target images. Summary of the Invention

[0004] To overcome the problems existing in related technologies, the present invention discloses a method, system, and application for panoramic image fusion. Specifically, the invention relates to Laplacian pyramids and image fusion technologies. The present invention aims to achieve a smooth transition while preserving detail information between two input images through power-averaged fusion of scale pyramids. This solves the problem of objects disappearing at the seam in surround-view observation systems.

[0005] The technical solution is as follows: a panoramic stitching image fusion method, including: through scale pyramid power average fusion, the image is smoothly transitioned while retaining the detail information of the input image, and according to needs, the exponent size of the power function is set to make a trade-off between smoothness and detail.

[0006] Furthermore, in the power average fusion of the scale pyramid, the top-level expression of the fused Laplacian pyramid is:

[0007]

[0008] Among them, L′ n is the top layer of the fused Laplacian pyramid, L j，n is the top Laplacian pyramid of the jth image, w j is the weight of the jth image, m is the number of input images, and j is the number of images.

[0009] Furthermore, in the power average fusion of the scale pyramid, the i-th layer of the fused Laplacian pyramid is:

[0010]

[0011]

[0012] Among them, L′ i is the i-th layer of the fused Laplace pyramid, sign is the symbolic operation, S i It is an intermediate variable for calculation and has no practical significance. α is the exponent of the power function, m is the number of input images, and L j，i is the i-th layer Laplacian pyramid of the j-th image, w j is the weight of the j-th image;

[0013] Set the exponent size of the power function to make a trade-off between smoothness and detail. The exponent α of the power function is set to a value greater than or equal to 1 as needed; the smaller α is, the smoother the transition is, and the larger α is, the more details are retained.

[0014] Furthermore, before the power average fusion through the scale pyramid, it is necessary to generate a Laplacian pyramid of at least two input images and select a suitable number of pyramid levels n according to the size of the input images.

[0015] Furthermore, the Laplacian pyramid of at least two input images is generated, and the calculation process of each input image includes:

[0016] Generate a Gaussian pyramid. The 0th layer G0 of the Gaussian pyramid is the input image, then the i-th layer G i The recursive formula is:

[0017]

[0018] Where i>1, G i is the input image of the i-th layer of the Gaussian pyramid, G i-1 is the input image of the i-th layer of the Gaussian pyramid, is the convolution operation, is a Gaussian low-pass filter of size 5×5, and DOWN is downsampling;

[0019] The top layer L of the unfused Laplacian pyramid n is the top of Gauss's pyramid G n , the expression is:

[0020] L n =G n ;

[0021] Unfused Laplacian pyramid layer L i i for:

[0022]

[0023] Among them, UP is up sampling, i<n, L iis the i-th level Laplace pyramid, G i is the i-th level Gaussian pyramid, G i+1 is the i+1th layer of the Gaussian pyramid, is a Gaussian low-pass filter of size 5×5.

[0024] Furthermore, after the power average fusion of the scale pyramid, it is necessary to generate a Gaussian pyramid of the fused image according to the Laplacian pyramid of the fused image to obtain an output image.

[0025] Furthermore, the top layer of the Laplacian pyramid of the fusion graph is:

[0026] G′ n =L′ n ;

[0027] The Gaussian pyramid layer i of the fusion graph is:

[0028]

[0029] Among them, UP is upsampling, i<n, G′ i is the i-th level Gaussian pyramid of the fusion graph, G′ i+1 is the i+1th layer of the Gaussian pyramid of the fusion graph, is a Gaussian low-pass filter of size 5×5, L′ i is the i-th level Laplacian pyramid of the fusion graph.

[0030] The 0th level Gaussian pyramid is the output fusion graph G′0.

[0031] Another object of the present invention is to provide a panoramic stitching image fusion system for implementing the panoramic stitching image fusion method, the system comprising:

[0032] A Laplacian pyramid generation module is used to generate a Laplacian pyramid of at least two input images;

[0033] A power average processing module is used for Laplacian pyramids to generate Laplacian pyramids of fusion images using power averages;

[0034] The fused image generation module is used to generate a fused image according to the Laplacian pyramid of the fused image.

[0035] Another object of the present invention is to provide an application on a vehicle-mounted image taking device, which is equipped with the panoramic stitching image fusion system for panoramic viewing to obtain images of surrounding obstacles.

[0036] Another object of the present invention is to provide an application on an airborne optoelectronic system, which is equipped with the panoramic stitching image fusion system for panoramic viewing to obtain images of surrounding obstacles.

[0037] Combining all of the above technical solutions, the present invention offers the following advantages and positive effects: Addressing the shortcomings of existing image fusion algorithms, the present invention improves the Laplacian pyramid fusion method based on the Laplacian pyramid fusion algorithm. By using power averaging instead of arithmetic averaging, the image transition is smoothed while retaining the details of the input image. The exponent of the power function can be adjusted as needed to balance smoothness and detail.

[0038] The positive effects of this invention are also reflected in the following key aspects: The expected benefits and commercial value of this technical solution after implementation include: improving the problem of people disappearing at the seams in the vehicle's 360-degree surround view images, thereby enhancing driving safety. In transition areas, the transition is smooth while preserving the target information in any input image, making the target at the seam more prominent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure;

[0040] Figure 1 This is a flow chart of a panoramic stitching image fusion method provided by an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of a panoramic stitching image fusion system provided by an embodiment of the present invention;

[0042] Figure 3 This is a simulation scene effect diagram provided by an embodiment of the present invention, in which there is a street lamp at a 45° angle to the left front of the car, and the street lamp is located exactly at the left front seam;

[0043] Figure 4-1 This is an effect diagram taken by a front fisheye camera provided by an embodiment of the present invention;

[0044] Figure 4-2 This is a rendering taken by the left fisheye camera provided by an embodiment of the present invention;

[0045] Figure 4-3 This is an effect picture taken by the rear fisheye camera provided by an embodiment of the present invention;

[0046] Figure 4-4 This is a rendering taken by the right fisheye camera provided by an embodiment of the present invention;

[0047] Figure 5-1 The embodiment of the present invention provides a method of converting the left camera image into a top-down simulation scene effect diagram;

[0048] Figure 5-2The embodiment of the present invention provides a method of converting the right camera image into a top-down simulation scene effect diagram;

[0049] Figure 6-1 The traditional method uses a preset dividing line to transform the left camera image into a top-down simulation scene rendering;

[0050] Figure 6-2 The traditional method uses a preset dividing line to convert the right camera image into a top-down simulation scene rendering;

[0051] Figure 7-1 It is a traditional 360° surround view rendering;

[0052] Figure 7-2 This is the effect diagram provided by the average value fusion algorithm;

[0053] Figure 7-3 It is a fusion effect diagram of the algorithm provided by the present invention;

[0054] In the figure: 1. Laplacian pyramid generation module; 2. Power average processing module; 3. Fusion image generation module. DETAILED DESCRIPTION

[0055] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Example 1, as Figure 1 As shown, the panoramic stitching image fusion method provided by the embodiment of the present invention includes the following steps:

[0057] S1, generates a Laplacian pyramid of at least two input images; including:

[0058] Select an appropriate number of pyramid levels n according to the size of the input image.

[0059] First, generate a Gaussian pyramid. The 0th layer G0 of the Gaussian pyramid is the input image, and the i-th layer G i The recursive formula is:

[0060]

[0061] in, is the convolution operation, is a Gaussian low-pass filter of size 5×5, and DOWN is downsampling.

[0062] The top layer L of the fused Laplace pyramid n is the top of Gauss's pyramid G n

[0063] L n =G n ;

[0064] The i-th (i<n) layer L of the unfused Laplacian pyramid i for

[0065]

[0066] Among them, UP stands for upsampling.

[0067] S2, uses power averaging to generate the Laplacian pyramid of the fusion graph; specifically includes:

[0068] There are m input images in total.

[0069] The top layer L′ of the fused Laplacian pyramid n for:

[0070]

[0071] Among them, L′ n is the top layer of the fused Laplacian pyramid, L j，n is the top Laplacian pyramid of the jth image, w j is the weight of the jth image, m is the number of input images, and j is the number of images.

[0072] The i-th (i<n) layer L′ of the fused Laplacian pyramid i for:

[0073]

[0074]

[0075] Among them, L′ i is the i-th layer of the fused Laplace pyramid, sign is the symbolic operation, S i It is used to calculate the intermediate variables and has no practical significance. α is the exponent of the power function and can be set to a value greater than or equal to 1 as needed. The smaller α is, the smoother the transition is. The larger α is, the more details are retained. m is the number of input images, L j，i is the i-th layer Laplacian pyramid of the j-th image, w j is the weight of the j-th image.

[0076] S3, generating a fused image based on the Laplacian pyramid of the fused image; specifically, including:

[0077] The top layer of the Gaussian pyramid of the fusion graph is:

[0078] G′ n =L′ n ;

[0079] The Gaussian pyramid layer i (i<n) of the fusion graph is:

[0080]

[0081] The 0th layer G′0 Gaussian pyramid is the output fusion graph.

[0082] Example 2, as Figure 2 As shown, the panoramic stitching image fusion system provided by the embodiment of the present invention includes the following steps:

[0083] A Laplacian pyramid generation module 1 is used to generate Laplacian pyramids of two or more input images respectively;

[0084] A power average processing module 2 is used for generating a Laplacian pyramid of a fusion image using power average.

[0085] A fused image generation module 3 is used to generate a fused image according to the Laplacian pyramid of the fused image;

[0086] The Laplacian pyramid generation module 1 includes:

[0087] The pyramid layer number selection module is used to select the appropriate number of pyramid layers according to the size of the input image.

[0088] Gaussian pyramid generation module, used to generate Gaussian pyramid;

[0089] The Laplacian pyramid generation module is used to generate a Laplacian pyramid based on the Gaussian pyramid.

[0090] The power average processing module 2 includes:

[0091] The top fusion module of the Laplacian pyramid is used to fuse the top image of the Laplacian pyramid through the weighted average algorithm;

[0092] The non-top fusion module of the Laplacian pyramid is used to fuse the images except the top layer of the Laplacian pyramid using the weighted power average algorithm;

[0093] The fused image generation module 3 includes:

[0094] The Laplacian pyramid parsing module is used to generate the Gaussian pyramid of the fused graph based on the Laplacian pyramid of the fused graph.

[0095] The fused image output module uses the bottom layer of the Gaussian pyramid as the output image.

[0096] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0097] The information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment.

[0099] Based on the technical solutions described in the above embodiments of the present invention, the following application examples can be further proposed.

[0100] According to an embodiment of the present application, the present invention also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0101] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0102] An embodiment of the present invention also provides an information data processing terminal, which is used to provide a user input interface to implement the steps in the above method embodiments when executed on an electronic device. The information data processing terminal is not limited to mobile phones, computers, and switches.

[0103] An embodiment of the present invention further provides a server, which is used to provide a user input interface to implement the steps in the above method embodiments when executed on an electronic device.

[0104] An embodiment of the present invention further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0106] In order to further demonstrate the positive effects of the above embodiment, the present invention conducts the following experiments based on the above technical solution.

[0107] The algorithm effect is verified by simulation images, such as Figure 3 As shown, there is a street lamp at 45 degrees to the left front of the car, and the street lamp is located at the left front joint. Figure 4-1 is the image of the front fisheye camera, Figure 4-2 is the image from the rear fisheye camera, Figure 4-3 This is the image from the fisheye camera on the left. Figure 4-4 This is the image of the fisheye camera on the right, where the cameras in front and on the left can see the streetlights. Figure 5-1 As shown, the left camera image is transformed into a top-view image; Figure 5-2 As shown in the figure, the right camera image is converted into a bird's-eye view image. Since the street lamp is not on the ground, the images of the street lamp do not overlap.

[0108] Traditional methods use preset segmentation lines to stitch the overhead images together; Figure 6-1 The traditional method uses a preset dividing line to transform the left camera image into a top-down simulation scene rendering; Figure 6-2 The traditional method uses a preset dividing line to convert the right camera image into a top-down simulation scene rendering. Figure 7-1For the traditional 360° surround view, you can only see the base of the street lamp. In order to see the street lamp, use Figure 5-1 and Figure 5-2 After fusion is performed, although the target has a double image, it will not disappear. Figure 7-2 It is an average fusion algorithm, which can see the street lights but they are very blurry; Figure 7-3 The fused image produced by this patented algorithm shows a clearer street light. Simulation tests show that the fused image produced by this patented algorithm has a smooth transition area and clear targets.

[0109] The above description is only a preferred specific implementation method of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A panoramic stitching image fusion method, characterized in that: The method includes: using a scale pyramid power average fusion to make the image transition smooth while retaining the detail information of the input image, and according to needs, the exponent size of the power function is set to make a trade-off between smoothness and detail; In the power average fusion through the scale pyramid, the top-level expression of the fused Laplacian pyramid is: ； in, is the top layer of the fused Laplacian pyramid, For the The top Laplacian pyramid of the image, For the The weight of the image, is the number of input images, is the number of images; in the power average fusion of the scale pyramid, the number of the fused Laplacian pyramid is The layers are: ； ； in, The first layer, is a symbolic operation, To calculate the intermediate variables, it has no practical significance. is the exponent of the power function, is the number of input images, For the Image No. layer Laplacian pyramid, For the The weight of the image; Set the exponential size of the power function. In the trade-off between smoothness and detail, the exponent of the power function Set a value greater than or equal to 1 as needed; The smaller the transition, the smoother it is. The larger the size, the more details are retained; After the power average fusion through the scale pyramid, it is necessary to further perform the following steps: generating a fused image according to the Laplacian pyramid of the fused image; The top layer of the Laplacian pyramid of the fusion graph is: ； Gaussian pyramid of the fusion graph The layers are: ； in, For upsampling, , The fusion graph Layer Gaussian pyramid, The fusion graph Layer Gaussian pyramid, For size A Gaussian low-pass filter, The fusion graph Layer Laplacian pyramid; The 0th level Gaussian pyramid is the output fusion map , .

2. The panoramic image fusion method according to claim 1, wherein: Before the power average fusion through the scale pyramid, it is necessary to generate a Laplacian pyramid of at least two input images and select the appropriate number of pyramid layers according to the size of the input image. .

3. The panoramic image fusion method according to claim 2, wherein: Generating the Laplacian pyramid of at least two input images includes: Generate Gaussian pyramid, Gaussian pyramid layer 0 is the input image, then layer The recursive formula is: ； in, is the first layer input image, is the first layer input image, is the convolution operation, For size A Gaussian low-pass filter, is downsampling; Top layer of the unfused Laplacian pyramid The top of Gauss's pyramid , the expression is: ； The first one without fused Laplacian pyramid layer for: ； in, For upsampling, , For the layer Laplacian pyramid, For the Layer Gaussian pyramid, For the Layer Gaussian pyramid.

4. A panoramic stitching image fusion system, characterized in that: The panoramic stitching image fusion method according to any one of claims 1 to 3 is implemented, and the system comprises: A Laplacian pyramid generation module (1) is used to generate a Laplacian pyramid of at least two input images; A power average processing module (2) is used for generating a Laplacian pyramid of a fusion image using power average; The fusion image generation module (3) is used to generate a fusion image according to the Laplacian pyramid of the fusion image.

5. An application on a vehicle-mounted image taking device, characterized in that: The application is equipped with the panoramic stitching image fusion system as described in claim 4, which is used for panoramic viewing to obtain images of surrounding obstacles.

6. An application in an airborne optoelectronic system, characterized in that: The application is equipped with the panoramic stitching image fusion system as described in claim 4, which is used for panoramic viewing to obtain images of surrounding obstacles.

Images