A color difference correction method and system for multi-video panorama stitching, and an electronic device

Abstract

The application provides a color difference correction method, system and electronic equipment for multi-video panoramic splicing. The method of the application comprises the following steps: acquiring to-be-fused pictures in multiple videos, wherein there are overlapping parts in different to-be-fused pictures; performing mask processing on the to-be-fused pictures; then merging RGB three-color channels of the to-be-fused pictures after mask processing; and finally processing the to-be-fused pictures after merging of the RGB three-color channels by using a flip filter. The scheme uses Maks mask combined with a matrix filter algorithm, eliminates color difference and splicing gaps of the spliced pictures, solves the defects of color difference and splicing gaps in panoramic picture splicing, makes the spliced pictures more natural, and improves the splicing quality of the pictures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video stitching, in particular to a color difference correction method, system and electronic device for multi-video panoramic stitching. BACKGROUND

[0002] Panoramic video stitching is a technology that uses real images to form a panoramic space, which stitches multiple images into a large-scale image or a 360-degree panoramic image.

[0003] The existing multi-video panoramic stitching technology has color differences between the stitched images due to the exposure settings of the multi-camera, which causes color differences in the stitched image sequence, thereby affecting the quality of the panoramic image.

[0004] Therefore, it is necessary to provide a color difference correction method, system and electronic device for multi-video panoramic stitching to solve the above technical problems. SUMMARY

[0005] To solve the above technical problems, the present application provides a color difference correction method, system and electronic device for multi-video panoramic stitching, which uses Maks mask combined with matrix filtering algorithm to correct the color difference of the stitched image and eliminate the stitching gap, solves the defects of color difference and stitching gap in panoramic image stitching, and makes the stitched image more natural.

[0006] The present application provides a color difference correction method for multi-video panoramic stitching, comprising:

[0007] Obtaining a to-be-fused image in a plurality of videos, wherein there is an overlapping part between different to-be-fused images;

[0008] Masking the to-be-fused image;

[0009] Merging the RGB three-color channels of the masked to-be-fused image;

[0010] Processing the to-be-fused image after merging the RGB three-color channels using a flip filter.

[0011] Preferably, the to-be-fused image in the plurality of videos is obtained by:

[0012] Extracting feature points of the target image and the reference image using a SIFT algorithm;

[0013] Matching the extracted feature points to generate a 3x3 perspective transformation matrix M;

[0014] Generating a blank panoramic canvas according to the width and height of the target image and the reference image;

[0015] Performing perspective transformation on the target image according to the perspective transformation matrix M.

[0016] Preferably, the mask processing the to-be-fused picture comprises:

[0017] Once mask processing the target picture after perspective transformation to generate a Mask picture;

[0018] Second mask processing the Mask picture to generate a Mask1 picture;

[0019] Reverse mask processing the reference picture to generate a Mask2 picture.

[0020] Preferably, the RGB three-color channel of the to-be-fused picture after mask merging comprises:

[0021] RGB three-color channel merging the Mask1 picture and the Mask2 picture to obtain a Mask11 picture and a Mask22 picture.

[0022] Preferably, the RGB three-color channel merging the to-be-fused picture after flip filter processing comprises:

[0023] Using a flip filter to denoise the target picture and the reference picture;

[0024] Multiplying the denoised target picture and the reference picture with the corresponding Mask11 picture and Mask22 picture to obtain a to-be-fused picture set.

[0025] Preferably, after the RGB three-color channel merging the to-be-fused picture after flip filter processing, it further comprises:

[0026] Synthesizing the corresponding to-be-fused picture in the to-be-fused picture set to obtain a panoramic picture.

[0027] A color difference correction system for multi-video panoramic stitching comprises:

[0028] An acquisition module is configured to acquire to-be-fused pictures in multiple videos, wherein there is an overlapping part between different to-be-fused pictures;

[0029] A mask module is configured to mask process the to-be-fused pictures;

[0030] A merging module is configured to merge the RGB three-color channels of the to-be-fused pictures after mask processing;

[0031] A filtering module is configured to process the to-be-fused pictures after RGB three-color channel merging using a flip filter.

[0032] Preferably, the acquisition module comprises:

[0033] A feature extraction submodule is configured to extract feature points of the target picture and the reference picture by using a SIFT algorithm;

[0034] A feature matching submodule is configured to match the extracted feature points to generate a 3*3 perspective transformation matrix M;

[0035] A canvas generation module is configured to generate a blank panoramic canvas according to the width and height of the target picture and the reference picture;

[0036] A perspective transformation module is configured to perform perspective transformation on the target picture according to the perspective transformation matrix M.

[0037] Preferably, the mask module comprises:

[0038] A primary mask submodule is configured to perform primary mask processing on the target picture after perspective transformation to generate a Mask picture;

[0039] A secondary mask submodule is configured to perform secondary mask processing on the Mask picture to generate a Mask1 picture;

[0040] A reverse mask submodule is configured to perform reverse mask processing on the reference picture to generate a Mask2 picture.

[0041] An electronic device comprises:

[0042] A memory storing executable program codes;

[0043] A processor coupled with the memory;

[0044] The processor invokes the executable program codes stored in the memory to execute the color difference correction method.

[0045] Compared with the related art, the color difference correction method, system and electronic device for multi-video panoramic stitching provided by the present application have the following beneficial effects:

[0046] The method provided by the present application comprises the following steps: obtaining to-be-fused pictures in multiple videos, wherein there is an overlapping part between different to-be-fused pictures; performing mask processing on the to-be-fused pictures; merging RGB three-color channels of the to-be-fused pictures after mask processing; and finally processing the to-be-fused pictures after merging of the RGB three-color channels by using a flip filter. The present application uses a Maks mask combined with a matrix filter algorithm to eliminate color difference and a stitching gap of the stitched picture, solves the defects of color difference and a stitching gap in panoramic picture stitching, makes the stitched picture more natural, and improves the stitching quality of the picture. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 FIG. 1 is a schematic diagram of the overall process of the color difference correction method for multi-video panoramic stitching disclosed by the present application; FIG. 1 is a schematic diagram of the overall process of the color difference correction method for multi-video panoramic stitching disclosed by the present application;

[0048] Figure 2 A flowchart of a process for obtaining a to-be-fused picture of a multi-video panoramic splicing color difference correction method disclosed in the present application is shown in FIG. 1.

[0049] Figure 3 A flowchart of a mask processing process of a multi-video panoramic splicing color difference correction method disclosed in the present application is shown in FIG. 4.

[0050] Figure 4 A flowchart of a filter processing process of a multi-video panoramic splicing color difference correction method disclosed in the present application is shown in FIG. 5.

[0051] Figure 5 Another flowchart of a multi-video panoramic splicing color difference correction method disclosed in the present application is shown in FIG. 6.

[0052] Figure 6 An architecture diagram of a multi-video panoramic splicing color difference correction system disclosed in the present application is shown in FIG. 7.

[0053] Figure 7 An electronic device structure diagram disclosed in the present application is shown in FIG. 8.

[0054] Figure 8 A panoramic picture spliced by using the present application. DETAILED DESCRIPTION

[0055] The present application will be further described below in combination with the drawings and embodiments.

[0056] A multi-video panoramic splicing color difference correction method provided by the present application includes the following steps, as shown in FIG. 1. Figure 1 and Figure 5

[0057] Step S101: obtaining a to-be-fused picture in a multi-video, wherein there is an overlapping part between different to-be-fused pictures.

[0058] Specifically, as shown in FIG. 2, step S101 includes the following operation steps. Figure 2

[0059] Step 201: extracting feature points of a target picture and a reference picture by using a SIFT algorithm: through a scale space extreme value detection method, feature points existing in the same to-be-fused picture at different scales are explored, then a series of pictures at different scales are generated by continuously downsampling and smoothing the original picture, and finally feature points existing in each picture are found out.

[0060] ​​Step 202: matching the extracted feature points to generate a 3x3 perspective transformation matrix M: a set of key point descriptors is established for the target picture and the reference picture respectively, a similarity measure of the key point descriptors with 128 dimensions is implemented by using the Euclidean distance, a robust parameter estimation method is used to remove mismatched points, and finally a perspective transformation matrix M is generated according to the matched feature points.

[0061] Step 203: generating a blank panoramic canvas according to the width and height of the target picture and the reference picture.

[0062] Step 204: performing perspective transformation on the target picture according to the perspective transformation matrix M: first, performing perspective transformation on the target picture, and then performing perspective transformation on the target picture on the panoramic canvas canvas according to the perspective transformation matrix M to obtain a perspective-transformed picture.

[0063] Step S102: performing mask processing on the to-be-fused picture.

[0064] Specifically, as shown in Figure 3 Step S102 includes the following operation steps.

[0065] Step S301: performing one-time mask processing on the perspective-transformed target picture to generate a Mask picture;

[0066] More specifically, the picture generated in step S204 is masked, and all pixel points in the canvas whose B, G, and R values have one non-zero value are set to pure white, and vice versa, to generate a mask Mask, where B, G, and R represent blue, green, and red in the three primary colors.

[0067] Step S302: performing two-time mask processing on the Mask picture to generate a Mask1 picture;

[0068] More specifically, based on the Mask picture generated in step S301, the mask operation is performed again, that is, a gradient mask is generated according to the width of the Mask picture and the perspective-transformed picture, and the formula is as follows:

[0069] Mask1(c, r) = s + (c-offset) * ((e-c) / (w-o))

[0070] Where s represents the start point start of the Mask; c represents the width column of the Mask; e represents the end point end of the Mask; w represents the width of the new empty mask; and o represents offset, which is obtained according to the traversal mask.

[0071] Step S303: performing reverse mask processing on the reference picture to generate a Mask2 picture;

[0072] Specifically, the reference picture is executed step S302, the opposite starting point s and the ending point e are imported, the final mask2 picture is finally obtained, and the Mask1 picture is imported into the next step.

[0073] Step S103: merge the RGB three color channels of the mask after the to-be-fused picture;

[0074] Specifically, the RGB three color channels of the Mask1 picture and the Mask2 picture are merged to obtain the Mask11 picture and the Mask22 picture.

[0075] Step S104: processing the RGB three color channel merged to-be-fused picture by using a flip filter;

[0076] Specifically, the target picture and the reference picture are processed by using a flip filter for noise reduction, and then the noise-reduced target picture and the reference picture are multiplied by the corresponding Mask11 picture and the Mask22 picture to obtain a to-be-fused picture set.

[0077] After obtaining the to-be-fused picture set, the corresponding to-be-fused pictures in the to-be-fused picture set are synthesized to obtain a panoramic picture.

[0078] The application discloses a color difference correction method for multi-video panoramic splicing.

[0079] A color difference correction system for multi-video panoramic splicing, as shown in Figure 6 The system includes an acquisition module for acquiring to-be-fused pictures in multiple videos, wherein there are overlapping parts between different to-be-fused pictures; a mask module for mask processing the to-be-fused pictures; a merging module for merging the RGB three color channels of the mask after the to-be-fused pictures; and a filtering module for processing the RGB three color channel merged to-be-fused pictures by using a flip filter.

[0080] More specifically, the acquisition module comprises a feature extraction submodule for extracting feature points of the target picture and the reference picture by using a SIFT algorithm, a feature matching submodule for matching the extracted feature points to generate a 3x3 perspective transformation matrix M, a canvas generation module for generating a blank panorama canvas according to the width and height of the target picture and the reference picture, and a perspective transformation module for performing perspective transformation on the target picture according to the perspective transformation matrix M.

[0081] More specifically, the mask module comprises a primary mask submodule for performing primary mask processing on the perspective-transformed target picture to generate a Mask picture, a secondary mask submodule for performing secondary mask processing on the Mask picture to generate a Mask1 picture, and a reverse mask submodule for performing reverse mask processing on the reference picture to generate a Mask2 picture.

[0082] The color difference correction system for multi-video panorama stitching disclosed in the present application is implemented based on the color difference correction method for multi-video panorama stitching described above, and will not be described again here.

[0083] The color difference correction system for multi-video panorama stitching disclosed in the present application is implemented based on the color difference correction method for multi-video panorama stitching described above, and will not be described again here.

[0084] An electronic device comprising a memory storing executable program code;

[0085] A processor coupled to the memory;

[0086] The processor invokes the executable program code stored in the memory to execute the color difference correction method.

[0087] The processor is configured to acquire, by using an acquisition module, a to-be-fused picture in a plurality of videos, wherein there is an overlapping part between different to-be-fused pictures, perform, by using a mask module, mask processing on the to-be-fused picture, merge, by using a merging module, RGB three-color channels of the mask-processed to-be-fused picture, and process, by using a flip filter of a filter module, the to-be-fused picture after merging of the RGB three-color channels.

[0088] In the embodiment, the processor is further configured to acquire the feature points of the target picture and the reference picture by using a feature extraction submodule, match the extracted feature points by using a feature matching submodule, generate the perspective transformation matrix M of the target picture and the reference picture by using a canvas generation submodule, and perform perspective transformation on the target picture by using a perspective transformation submodule according to the perspective transformation matrix M.

[0089] In the embodiment, the processor is further configured to perform one-time mask processing on the target picture after perspective transformation by using a one-time mask submodule, generate a Mask picture, perform two-time mask processing on the Mask picture by using a two-time mask submodule, generate a Mask1 picture, and perform inverse mask processing on the reference picture by using an inverse mask submodule, and generate a Mask2 picture.

[0090] The electronic device disclosed in the present application is implemented based on the color difference correction method for multi-video panorama stitching disclosed in the above embodiment, and thus will not be described herein.

[0091] The electronic device disclosed in the present application is implemented based on the color difference correction method for multi-video panorama stitching disclosed in the above embodiment, and thus will not be described herein.

[0092] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0093] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation obtained by using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for color difference correction in multi-video panoramic stitching, characterized in that, include: Acquire images to be merged from multiple video segments, wherein there is a framing overlap between different images to be merged; The images to be fused are masked. The RGB three color channels of the image to be merged after merging the mask; Using a flip filter to process the image to be merged after combining the RGB three color channels; The process of obtaining the images to be merged from multiple video segments includes: Feature points of the target image and reference image are extracted using the SIFT algorithm; Match the extracted feature points to generate 3 The perspective transformation matrix M of 3; Generate a blank panoramic canvas based on the width and height of the target image and the reference image; Perform perspective transformation on the target image based on the perspective transformation matrix M; The masking process for the image to be fused includes: A mask image is generated from the target image after perspective transformation using a single masking process. The Mask image is processed by a second masking step to generate the Mask1 image; The reference image is processed using a reverse mask to generate a Mask2 image; The RGB three-color channels of the image to be merged after merging the mask include: The Mask1 and Mask2 images are merged using the RGB three-color channels to obtain the Mask11 and Mask22 images; The process of merging the RGB three-color channels into a single image using a flip filter includes: Denoising of the target image and reference image is performed using a flip filter; The denoised target image and reference image are multiplied with the corresponding Mask11 and Mask22 images to obtain the image set to be fused.

2. The color difference correction method for multi-video panoramic stitching according to claim 1, characterized in that, After processing the merged RGB three-color channel image using the flip filter, the process further includes: The images to be merged from the set of images to be merged are combined to obtain a panoramic image.

3. A color difference correction system for multi-video panoramic stitching, characterized in that, include: The acquisition module is used to acquire images to be merged from multiple video segments, wherein there is a framing overlap between different images to be merged. The mask module is used to mask the image to be fused. The merging module is used to merge the RGB three color channels of the masked images to be merged. The filtering module is used to process the image to be merged after the RGB three color channels are combined using the flip filter; The acquisition module includes: The feature extraction submodule is used to extract feature points from the target image and the reference image using the SIFT algorithm; The feature matching submodule is used to match the extracted feature points and generate 3D models. The perspective transformation matrix M of 3; The canvas generation module is used to generate a blank panoramic canvas based on the width and height of the target image and the reference image. The perspective transformation module is used to perform perspective transformation on the target image based on the perspective transformation matrix M. The mask module includes: The primary masking submodule is used to perform a primary masking process on the target image after perspective transformation, generating a mask image. The secondary masking submodule is used for secondary masking of the Mask image to generate the Mask1 image; The reverse mask submodule is used to reverse mask the reference image and generate a Mask2 image.

4. An electronic device, characterized in that, include: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the color difference correction method according to any one of claims 1-2.

Images