A fast image stitching method based on YUV color space

Abstract

The application discloses a kind of based on YUV color space's image fast splicing method, pixel mapping equation is established to the characteristic of the planar motion of driving mechanism, pixel mapping table is established in combination with the position of camera obtained by encoder and the image depth obtained by depth camera, so that the fast transformation of image pixels between different visual angles is realized;Pixel mapping equation is extended from a single pixel to matrix form, so that the whole picture or multiple pictures can be mapped in parallel, the fast splicing of large scene and multiple pictures can be realized, in the image splicing process, for the problem that there may be a seam in image fusion by directly using weighted average, while ensuring the image splicing effect, the time of image splicing is shortened.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for rapid image stitching based on the YUV color space. Background Technology

[0002] Currently, most mainstream panoramic image stitching processes use feature matching methods based on camera images. In the patent "A Method for Stitching Images of Objects on a Conveyor Belt Based on SIFT Features" (Patent No.: 202110329935.4), SIFT features are used for matching and stitching. In the patent "A Multi-Path Image Stitching Method Based on End-to-End Neural Networks" (Patent No.: 202010553739.0), a neural network is used to estimate image features and the optimal homography matrix, thereby achieving image stitching. The patent "A Real-Time Panoramic Shooting Method for Mobile Phones Based on Deep Learning" (Patent No.: 202111269335.X) also uses SIFT features for image stitching.

[0003] However, current image feature-based stitching methods have three problems: First, if the environmental texture and color information are relatively simple, it may be difficult to obtain enough good features, leading to feature matching failure; second, feature calculation and matching are also time-consuming; and third, the homography matrix calculated for feature point matching can cause cumulative errors. While cumulative errors can be eliminated through image analysis (BA) optimization, the stitching time will be very long when the number of images is large.

[0004] The patent "Panoramic Video Fast Stitching Method and System" (Patent No.: 201911001401.8) utilizes the characteristic that the camera translation distance is negligible relative to the distance of the photographed object to propose a panoramic fast stitching method. However, in the scenario of vehicle movement, the camera translation distance cannot be ignored, so this stitching method is not applicable.

[0005] Therefore, it is necessary to provide a novel image fast stitching method based on the YUV color space to overcome the above-mentioned shortcomings. Summary of the Invention

[0006] The purpose of this invention is to provide a fast image stitching method based on the YUV color space, which enables rapid transformation of image pixels between different viewpoints and parallel mapping, and can achieve fast stitching and fusion of large scenes and a large number of images.

[0007] To achieve the above objectives, the present invention provides a method for rapid image stitching based on the YUV color space, comprising the following steps:

[0008] S1: The motion mechanism that controls the car performs a global scan of the scene to be stitched, and collects the camera position t, RGB image data and depth image data at the same time. The depth image data is mapped to the RGB image through the camera extrinsic parameters and camera intrinsic parameters to obtain the depth value z of each pixel.

[0009] S2: Use the pixel mapping equation to convert the pixels of all images to the pixels of the image from the initial viewpoint;

[0010] S3: Fuse all the transformed RGB values ​​to obtain a stitched panoramic image;

[0011] Preferably, step S2 further includes step S21: using the triangulation principle to obtain the relationship between the same point at two different pixel positions:

[0012]

[0013] Where P1 is the homogeneous coordinate of the pixel of this point in the camera image at position 1, with a dimension of 3x1, and z1 is the depth of this point in the camera image at position 1.

[0014] P2 is the homogeneous coordinate of the pixel at position 2 in the camera image, with dimensions of 3x1, and z2 is the depth of the point at position 2 in the camera image.

[0015] K is the camera intrinsic parameter; R is the camera rotation vector; and t is the camera position translation vector.

[0016] The camera's rotation vector R is an identity matrix, and the camera depth z1 = z2. The pixel mapping equation can be transformed into:

[0017]

[0018] The pixel mapping equation becomes:

[0019] P 1(3，n) =P 2(3，n) -Kt (3，n) *diag(z (n，1) ) -1 ;

[0020] Through the above P 1(3，n) =P 2(3，n) -Kt (3，n) *diag(z (n，1) ) -1 The pixel mapping equation performs parallel mapping on an image, where P 1(3，n) These are the homogeneous coordinates of the mapped pixel coordinates, with dimensions 3xn, and each column represents a mapped pixel coordinate; P 2(3，n)These are the homogeneous coordinates of the pixel before mapping, with dimensions 3xn, where each column represents a pixel coordinate before mapping; z (n，1) It is a depth vector corresponding to the pixel coordinates, with a dimension of nx1, and each row is the depth value of the pixel before mapping;

[0021] diag(z (n，1) `diag(z)` is used to diagonalize the depth vector, which means mapping an n-dimensional vector onto the diagonal of an n x n matrix, with all other elements set to 0; (n，1) ) -1 It means taking the inverse of the diagonalized matrix; t (3，n) It is the camera position translation vector, with a dimension of 3xn, and each row corresponds to the camera's translational displacement on the three coordinate axes respectively;

[0022] For the same image, the values ​​in rows n are the same. After transforming the mapping equation into matrix form, the GPU is used to perform parallel mapping and transformation of image pixels.

[0023] If parallel mapping of m images is required, then the pixel mapping equation is used:

[0024] P 1(3，n*m) =P 2(3，n*m) -Kt (3，n*m) *diag(z (n*m，1) ) -1 :

[0025] The pixel mapping equation yields a mapping relationship between the image pixels at the initial viewpoint and the image pixels at other camera viewpoints. This mapping relationship is then used to transform the RGB images of the other camera viewpoints to the image from the initial viewpoint.

[0026] RGB_origin[P 1(3，n) (0:1,:)] = RGB[P 2(3，n) (0:1,:)];

[0027] Where P 1(3，n) (0:1, :) represents taking P. 1(3，n) The first two rows of elements in the homogeneous coordinates, RGB_origin represents the camera image at the initial viewpoint, and RGB represents the camera image captured during camera movement.

[0028] Preferably, step S3 further includes step S31: taking the image of the initial viewpoint image pixels after conversion as the image to be stitched, converting the obtained panoramic image and the image to be stitched into grayscale images respectively, and performing binarization, and obtaining the image of the overlapping area through bitwise AND operation;

[0029] Convert the RGB values ​​of the overlapping regions of the initial panoramic image and the overlapping regions of the image to be stitched to the YUV color space. Calculate the luminance coefficient k of the ratio of the Y components of the overlapping regions of the initial panoramic image and the image to be stitched. Luminance coefficient:

[0030]

[0031] Among them, YUV B1 (0, :) represents the Y component of all pixels in B1, YUV B2 (0,:) represents the Y component of all pixels in B2, and avg() means taking the average value of the brightness coefficient;

[0032] Convert the images to be stitched to the YUV color space, and multiply the Y component of all pixels by this k coefficient. C (0, :) = k*YUV C (0.:);

[0033] Convert the image to be stitched from the YUV color space back to the RGB color space, and then copy the image to be stitched directly to the corresponding pixels of the panoramic image to obtain the merged panoramic image.

[0034] Compared with existing technologies, the advantages are: 1) A pixel mapping equation is established for the characteristics of planar motion of the traveling mechanism, and a pixel mapping table is established by combining the camera position obtained by the encoder and the image depth obtained by the depth camera, thereby realizing the rapid transformation of image pixels between different viewpoints.

[0035] 2) This invention extends the pixel mapping equation from a single pixel to a matrix form, thereby enabling parallel mapping of the entire image or multiple images, and achieving rapid stitching of large scenes and a large number of images.

[0036] 3) This invention addresses the issue of stitching seams that may occur when directly using weighted average for image fusion during image stitching. It shortens the image stitching time while ensuring the image stitching effect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 The flowchart illustrates the rapid image stitching method based on the YUV color space provided by this invention.

[0039] Figure 2This is a schematic diagram of a scenario according to an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram of pixel projection according to an embodiment of the present invention.

[0041] Figure 4 This is a schematic diagram of image stitching and fusion according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described in this specification are merely for explaining the invention and are not intended to limit the invention.

[0043] Please see Figure 1 This invention provides a method for rapid image stitching based on the YUV color space, comprising the following steps:

[0044] S1: The motion mechanism that controls the car performs a global scan of the scene to be stitched, and collects the camera position t, RGB image data and depth image data at the same time. The depth map data can be mapped to the RGB image through the camera extrinsic and intrinsic parameters to obtain the depth value z of each pixel.

[0045] S2: Use the pixel mapping equation to convert the pixels of all images to the pixels of the image from the initial viewpoint;

[0046] like Figure 3 As shown, the relationship between the same point at two different pixel positions is obtained using the principle of triangulation:

[0047]

[0048] Where P1 is the homogeneous coordinate of the pixel of this point in the camera image at position 1, with a dimension of 3x1, and z1 is the depth of this point in the camera image at position 1.

[0049] P2 is the homogeneous coordinate of the pixel at position 2 in the camera image, with dimensions of 3x1, and z2 is the depth of the point at position 2 in the camera image.

[0050] K is the camera intrinsic parameter; R is the camera rotation vector; and t is the camera position translation vector.

[0051] Since the vehicle's motion mechanism moves within a plane, the camera's rotation vector R is an identity matrix, and the camera depth z1 = z2, the pixel mapping equation can be transformed into:

[0052]

[0053] To improve the speed of pixel mapping, the pixel mapping equation is transformed into:

[0054] P 1(3，n) =P 2(3，n) -Kt (3，n) *diag(z (n，1) ) -1 ;

[0055] Through the above P 1(3，n) =P 2(3，n) -Kt (3，n) *diag(z (n，1) ) -1 The pixel mapping equation performs parallel mapping on an image, where P 1(3，n) These are the homogeneous coordinates of the mapped pixel coordinates, with dimensions 3xn, and each column represents a mapped pixel coordinate; P 2(3，n) These are the homogeneous coordinates of the pixel before mapping, with dimensions 3xn, where each column represents a pixel coordinate before mapping; z (n，1) It is a depth vector corresponding to the pixel coordinates, with a dimension of nx1, and each row is the depth value of the pixel before mapping;

[0056] diag(z (n，1) `diag(z)` is used to diagonalize the depth vector, which means mapping an n-dimensional vector onto the diagonal of an n x n matrix, with all other elements set to 0; (n，1) ) -1 This involves inverting the diagonalized matrix (i.e., taking the reciprocals of its diagonal elements); t (3，n) It is the camera position translation vector, with a dimension of 3xn. Each row corresponds to the camera's translational displacement on the three coordinate axes.

[0057] For the same image, the values ​​in the n rows are the same. After transforming the mapping equation into a matrix form, the GPU can be used to perform parallel mapping of image pixel transformations, thereby improving the mapping speed.

[0058] If parallel mapping of m images is required, the pixel mapping equation can be used:

[0059] P 1(3，n*m) =P 2(3，n*m) -Kt (3，n*m) *diag(z (n*m，1) ) -1 ;

[0060] The pixel mapping equation yields a mapping relationship between the image pixels at the initial viewpoint and the image pixels at other camera viewpoints. This mapping relationship is used to transform the RGB images of other camera viewpoints to the camera image at the initial viewpoint.

[0061] RGB_origin[P1(3，n) (0∶1,:)]=RGB[P 2(3，n) (0:1, :)]

[0062] Where P 1(3，n) (0:1, :) represents taking P. 1(3，n) The first two rows of elements in the homogeneous coordinates, RGB_origin represents the camera image at the initial viewpoint, and RGB represents the camera image captured during camera movement.

[0063] S3: Fuse all the transformed RGB values ​​to obtain a stitched panoramic image.

[0064] S4: The image of the initial viewpoint pixels after S2 is used as the image to be stitched together.

[0065] The panoramic image and the image to be stitched obtained in S3 are converted into grayscale images and binarized. The overlapping area image is obtained by bitwise AND operation.

[0066] Convert the RGB values ​​of the overlapping regions of the initial panoramic image and the overlapping regions of the image to be stitched to the YUV color space. Calculate the luminance coefficient k of the ratio of the Y components of the overlapping regions of the initial panoramic image and the image to be stitched. Luminance coefficient:

[0067]

[0068] Among them YUV B1 (0, :) represents the Y component of all pixels in B1, YUV B2 (0,:) represents the Y component of all pixels in B2, and avg() means taking the average value of the brightness coefficient;

[0069] Convert the images to be stitched to the YUV color space, and multiply the Y component of all pixels by this k coefficient. C (0, :) = k*YUV C (0, :);

[0070] Finally, the image to be stitched is converted from the YUV color space back to the RGB color space, and the image to be stitched is directly copied to the corresponding pixels of the panoramic image to obtain the panoramic stitching and fusion result.

[0071] like Figure 2 As shown in the figure, the cart can move in the XY plane. Both the RGB camera and the depth camera are mounted on the cart; both are mounted perpendicular to the plane of motion and pointing downwards. Initially, the cart is located at the origin O, and the camera's viewpoint is at position c1 (e.g., ...). Figure 3 As shown), after moving to a position x in the X direction i and move a position y in the Y directioni Then reach position c2 (e.g.) Figure 3 As shown in the diagram, the displacement of the vehicle can be obtained through an encoder located on the vehicle. Then, the camera's displacement vector... The translation in the Z direction is 0.

[0072] To illustrate the pixel mapping process, we assume here that the image at position c2 consists of two pixels. The depth vectors of these two pixels can be obtained from a depth camera. Assuming the depth vectors... So At this point, the camera's intrinsic parameters can be obtained through checkerboard calibration.

[0073] Assume x i =0.5, y i =0.5, then

[0074] Therefore, the image from the c2 perspective can be transformed to the c1 perspective using the pixel mapping formula.

[0075]

[0076] The calculation results above show that there is a mapping relationship between the pixel position (1, 1) in the c2 view and (364.5, 364.75) in the c1 view, and a mapping relationship between the pixel position (2, 1) in the c2 view and (365.5, 364.75) in the c1 view. The image pixel values ​​in the c2 view are transformed according to this pixel mapping table to obtain the image in the c1 view. Next, we enter the image fusion stage.

[0077] like Figure 4 As shown, assuming the initial panoramic image is A and the image to be stitched is C0, after the above image perspective transformation, image C0 is transformed into image C.

[0078] Convert images A and C to grayscale images respectively, and then binarize them to obtain... and Through images and Perform a bitwise AND operation to obtain a binarized image of the overlapping region. Use on images A and C respectively Using the mask as a template for image matting, we can obtain the overlapping region of image A as B1 and the overlapping region of image C as B2.

[0079] Convert the RGB values ​​of the overlapping region B1 of the initial panoramic image and the overlapping region B2 of the image to be stitched to YUV space, respectively. Calculate the luminance coefficient k of the ratio of the Y components of the overlapping regions of the initial panoramic image and the image to be stitched. Luminance coefficient:

[0080]

[0081] For example, when YUV B1 (0, :) = [1.1 1.3], YUV B2 When (0, :) = [1.0 1.0], the coefficient calculated here is k = 1.2.

[0082] Convert the image C to be stitched to YUV space, and multiply the Y component of all pixels by this k coefficient, YUV C (0, :) = k*YUV C (0, :);

[0083] For example when Image C after brightness adjustment

[0084] Finally, the image C to be stitched is converted from YUV space back to RGB space, and image C is directly copied to the corresponding pixels of image A to obtain the fused panoramic image.

[0085] The present invention is not limited to the description in the specification and embodiments, and thus other advantages and modifications can be readily realized by those skilled in the art. Therefore, the present invention is not limited to the specific details, representative devices and examples shown and described herein without departing from the spirit and scope of the general concept as defined by the claims and their equivalents.

Claims

1. A method for rapid image stitching based on the YUV color space, characterized in that, Includes the following steps: S1: The motion mechanism that controls the car performs a global scan of the scene to be stitched, and collects the camera position, RGB image data and depth image data at the same time. The depth map data is mapped to the RGB image through the camera extrinsic parameters and camera intrinsic parameters to obtain the depth value z of each pixel. S2: Use a pixel mapping equation to transform the pixels of all images to the pixels of the initial viewpoint image. Specifically, this involves obtaining a mapping relationship between the initial viewpoint image pixels and the image pixels at other camera viewpoint positions through the pixel mapping equation. Using this mapping relationship, transform the RGB images of the images at other camera viewpoint positions to the camera image at the initial viewpoint. ； in Representative take The first two rows of elements in homogeneous coordinates This represents the camera image from the initial viewpoint. This represents the camera images captured during camera movement. Representative take The first two rows of elements in homogeneous coordinates; The pixel mapping equation is: ； Through the above The pixel mapping equation performs parallel mapping on an image, where... These are the homogeneous coordinates of the mapped pixel coordinates, with dimensions of 3xn, and each column represents a mapped pixel coordinate. These are the homogeneous coordinates of the pixel before mapping, with a dimension of 3xn, and each column represents a pixel coordinate before mapping. It's the camera's internal parameters; It is a depth vector corresponding to the pixel coordinates, with a dimension of nx1, and each row is the depth value of the pixel before mapping; It is the diagonalization of the depth vector, which is to map the n-dimensional vector onto the diagonal of an n x n square matrix, with the remaining elements being 0; It means taking the inverse of the diagonalized matrix; It is the camera position translation vector, with a dimension of 3xn, and each row corresponds to the camera's translational displacement on the three coordinate axes respectively; For the same image, the values ​​in the n rows are the same. After transforming the mapping equation into a matrix form, the GPU is used to perform parallel mapping of image pixel transformations. S3: Fuse all the transformed RGB values ​​to obtain a stitched panoramic image.

2. The image fast stitching method based on YUV color space as described in claim 1, characterized in that, Step S2 also includes step S21: using the triangulation principle to obtain the relationship between the same point at two different pixel positions: ； in, These are the homogeneous coordinates of the pixel at position 1 in the camera image, with dimensions 3x1. It is the depth of that point in the camera image at location 1; These are the homogeneous coordinates of the pixels at position 2 in the camera image, with dimensions 3x1. It is the depth of that point in the camera image at location 2; It's the camera's internal parameters; is the camera's rotation vector, and t is the camera's translation vector. Camera rotation vector It is an identity matrix, camera depth The pixel mapping equation becomes: ； The pixel mapping equation becomes: ； If parallel mapping of m images is required, then the pixel mapping equation is used: 。 3. The image fast stitching method based on YUV color space as described in claim 1, characterized in that, Step S3 also includes step S31: Assuming the panoramic image of the initial viewpoint is A, the image of the initial viewpoint image pixels after conversion is taken as the image to be stitched C. The obtained panoramic image A and the image to be stitched C are converted into grayscale images respectively and binarized. The image of the overlapping area is obtained by bitwise AND operation. Convert the RGB values ​​of the overlapping regions of the initial panoramic image A and the overlapping regions of the image to be stitched to the YUV color space, respectively. Calculate the luminance coefficient k of the ratio of the Y components of the overlapping regions of the initial panoramic image A and the image to be stitched. Luminance coefficient: ； in, Represents the Y component of all pixels in B1. Represents the Y component of all pixels in B2. This indicates that the average value is taken, B1 is the overlapping area of ​​image A, and B2 is the overlapping area of ​​image C; Convert the image C to be stitched to the YUV color space, and multiply the Y component of all pixels by this k coefficient. ; Convert the image C to be stitched from the YUV color space back to the RGB color space, and directly copy the image C to the corresponding pixels of the panoramic image A to obtain the fused panoramic image.

Images