A surround view stitching method and system for adaptive vehicle body
Through the adaptive surround view stitching method of the vehicle body, using technologies such as image acquisition, feature extraction and perspective projection transformation calculation, the image distortion and real-time problems in the existing technology are solved, and high-quality 360-degree panoramic images are generated, which improves driving safety Sex and user experience.
Patent Information
- Application Number
- CN202410296743.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-15
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-15
AI Technical Summary
The existing 360-degree surround view stitching technology cannot adapt to the body size of different models, resulting in image distortion and inability to adjust in real time, affecting the quality of the panoramic image and the driver's visual experience.
Adopting the circumferential stitching method of the adaptive vehicle body, high-quality 360-degree panoramic images are generated through the technology of image acquisition, preprocessing, feature extraction, perspective projection transformation calculation and closed shape matching, so as to achieve intelligent and adaptiveness to different models. image stitching.
Adaptive adjustments to different models are achieved, generating seamless, deformation-free 360-degree panoramic images, improving driving safety and user experience, and avoiding the complexity of manual adjustments.
Smart Images

Figure CN117893719B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image processing, and more specifically relates to a surround view stitching method and system for an adaptive vehicle body. Background Art
[0002] In recent years, with the development of automobile intelligence, on-board cameras and image processing technology have become the focus of research, among which 360-degree surround view stitching technology has received more and more attention. The 360-degree surround view system obtains image information of the vehicle's surrounding environment through multiple cameras installed around the vehicle, and then stitches these images into a 360-degree panoramic image through image processing technology. In this way, the driver can clearly see the environment around the vehicle, greatly improving driving safety.
[0003] However, due to the large differences in the size and shape of cars, the existing 360-degree surround view stitching technology is not ideal for different car models. For example, for large vehicles, due to the large length and width of the vehicle body, if a universal surround view stitching method is used, some parts of the vehicle may not be fully captured or the image may be deformed. At the same time, in order to adapt to different car models, the system often needs to be adjusted manually, which not only increases the difficulty of use, but also cannot achieve real-time automatic adjustment.
[0004] Disadvantages of existing technology:
[0005] Adaptability issue: Existing surround view stitching methods are mostly fixed modes and cannot be adaptively adjusted according to the size of the vehicle. As a result, when dealing with different vehicle models, the image may be distorted and cannot accurately reflect the actual situation around the vehicle.
[0006] Real-time problem: Existing surround view stitching methods often require manual adjustment of camera parameters to match different vehicle models. This method cannot achieve real-time image stitching and cannot meet the needs of rapid response.
[0007] Seam processing problems: When stitching images, due to the limitations of the camera's viewing angle and shooting angle, there are problems such as obvious image seams, unnatural transitions, and image deformation, which affect the effect of the panoramic image and the driver's visual experience.
[0008] Therefore, an on-vehicle surround view system device that can adaptively adjust according to the size of the vehicle and can stitch high-quality panoramic images in real time is particularly necessary. Summary of the invention
[0009] The present invention aims to solve the problems existing in the prior art, such as the inability to adaptively adjust according to the size of the vehicle body, the inability to generate high-quality panoramic images in real time, and the unnatural processing of image seams. By developing a surround view stitching method and system for adaptive vehicle bodies, intelligent and adaptive surround view image stitching for different vehicle models is achieved, the quality and real-time performance of panoramic images are improved, thereby improving driving safety and driving experience.
[0010] In order to achieve the above object, the present invention is implemented by adopting the following technical scheme: the surround view stitching method comprises:
[0011] Image acquisition: collect image data from four directions of the vehicle, namely, front, rear, left and right directions;
[0012] Image preprocessing: preprocess the collected raw image data, including denoising, normalization, and grayscale processing methods;
[0013] Feature extraction: Extract feature points from the preprocessed image for subsequent image matching.
[0014] Image matching: Use the SIFT method to find matching feature point pairs between adjacent images;
[0015] Perspective projection transformation calculation: Calculate the perspective projection transformation matrix for the matching pair to obtain the mapping relationship between each image and the panoramic coordinate system;
[0016] Closed shape matching: Use closed shape matching to solve image seams and deformation problems to generate a continuous panoramic image;
[0017] Image stitching: Based on the matching results of closed shapes, all images are stitched together according to the results of perspective projection transformation to form a panoramic image.
[0018] In one embodiment, the grayscale in the image preprocessing is expressed by the following formula:
[0019] Gray = R*0.299 + G*0.587 + B*0.114
[0020] Among them, Gray is the grayscale value, and "R", "G", and "B" represent the three primary colors of red, green, and blue respectively.
[0021] In one solution, the feature extraction uses the SIFT algorithm to extract features, and the feature point detection part of SIFT uses the Gaussian difference function:
[0022]
[0023] in, and Represent two scale space images, It’s the original picture.
[0024] In one solution, the image matching uses the Euclidean distance formula to describe the matching degree between two feature points. Assuming that the two feature descriptors are P and Q, the Euclidean distance between them can be calculated by the following formula:
[0025]
[0026] Where P and Q represent two feature points respectively, and (x1, y1) and (x2, y2) are the two-dimensional coordinates of the feature points.
[0027] In one solution, the perspective projection transformation calculation obtains the mapping relationship between images by solving the homography matrix; assuming that the point of the original image is P(x, y), and the point mapped to the new image by the homography matrix H is P'(x', y'), the formula is:
[0028]
[0029] This results in a linear equation with 8 degrees of freedom, which can be solved by calculating at least 4 pairs of matching points.
[0030] In one embodiment, the closed shape matching adopts the Poisson image editing algorithm, which combines the gradient information of the source image and the boundary information of the target image. The equation is:
[0031]
[0032] Where f* is the required result image, Δ is the Laplacian operator of the image, and g is the source image; by solving this equation, a seamlessly fused result image is obtained.
[0033] In one embodiment, the image stitching is performed based on the calculated homography matrix and the matching result of the closed shape. The image stitching process requires the homography matrix H obtained by the perspective projection transformation calculation link and the matching result of the closed shape.
[0034] First, use the homography matrix H to perform perspective transformation on each image and map each image to the panoramic coordinate system;
[0035] Then in this panoramic coordinate system, the position of each image has been determined;
[0036] In order to process the seams between images in the process of stitching a panoramic image, multi-image fusion is performed; specifically, for the overlapping areas of the images, the Alpha fusion algorithm is used to process the seams to achieve a seamless stitching effect.
[0037] On the other hand, an adaptive vehicle surround view stitching system includes an image acquisition module for acquiring image data in four directions: front, rear, left, and right of the vehicle;
[0038] The main control module includes an image preprocessing module, a feature extraction module, a picture matching module, a perspective projection transformation module, a closed shape matching module, and an image stitching module;
[0039] The display module is used to display the panoramic image captured by the camera.
[0040] In one solution, in the main control module, the image preprocessing module performs denoising, normalization, and grayscale processing on the original image data collected by the image acquisition module;
[0041] The feature extraction module extracts features from the image processed by the image preprocessing module; the image matching module uses the features extracted by the feature extraction module to match adjacent images through the SIFT method;
[0042] The perspective projection transformation module calculates the perspective projection transformation matrix for the feature point pairs obtained by the image matching module to obtain the mapping relationship between each image and the panoramic coordinate system;
[0043] A closed shape matching module performs closed shape matching on the mapping relationship obtained by the perspective projection transformation calculation module to handle image seams and deformation issues;
[0044] The image stitching module stitches the images into a panoramic image according to the matching result obtained by the closed shape matching module and the mapping relationship obtained by the perspective projection transformation calculation module.
[0045] Beneficial effects of the present invention:
[0046] The beneficial effects of the present invention are mainly reflected in that it can be adjusted adaptively to the size of the vehicle body, and accurate 360-degree panoramic images can be generated regardless of the size of the vehicle body, thereby improving the applicability of the system. At the same time, by adopting advanced image processing algorithms, image features can be extracted in real time, and feature matching and image stitching can be performed, achieving excellent real-time performance. With the help of perspective projection transformation and closed shape matching methods, the seams and deformation problems of the image are effectively handled, and the generated panoramic image has no obvious seams, high image quality, and good visual effects. This 360-degree panoramic view allows the driver to clearly understand the situation around the vehicle, greatly improving driving safety. At the same time, the present invention avoids complex manual adjustments, is easy to operate, and significantly improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 is a flow chart of the method of the present invention;
[0048] Figure 2This is a closed shape matching flow chart of the present invention;
[0049] Figure 3 This is a system block diagram of the present invention. DETAILED DESCRIPTION
[0050] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0051] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0052] like Figure 1 As shown, the surround stitching method includes:
[0053] S1 Image Acquisition: Collect image data from four directions of the vehicle, namely, front, rear, left, and right.
[0054] Design a vehicle-mounted image acquisition system consisting of four cameras, which are installed in the front, rear, left and right directions of the vehicle respectively.
[0055] Each camera’s viewing angle must cover the corresponding direction of the vehicle and obtain as much information as possible in that direction. On this basis, considering the specific shape and size of the vehicle, it may be necessary to fine-tune the camera’s viewing angle and installation position to ensure that there is enough overlap between the images collected by each camera to ensure the subsequent image stitching effect.
[0056] When the vehicle is driving, four cameras collect image data in four directions of the vehicle in real time, and these data will be sent to the main control module for subsequent processing.
[0057] S2 Image preprocessing: preprocess the collected raw image data, including denoising, normalization, and grayscale processing methods;
[0058] The grayscale in the image preprocessing is expressed by the following formula:
[0059] Gray = R*0.299 + G*0.587 + B*0.114
[0060] Among them, Gray is the grayscale value, and "R", "G", and "B" represent the three primary colors of red, green, and blue respectively.
[0061] Image denoising is an important step in image preprocessing, because the images collected by the camera usually contain some noise, such as Gaussian noise, salt and pepper noise, etc. This will cause great trouble to subsequent image processing. The median filtering method is used to sort the grayscale values in the neighborhood of the pixel point and take the median as the grayscale value of the pixel point, so as to achieve the purpose of eliminating noise.
[0062] Normalization is also an important step in image preprocessing, mainly to reduce the impact of factors such as image contrast and brightness, and limit the pixel value to a certain range, such as 0-255. The specific implementation method is: subtract the grayscale mean of the entire image from the grayscale value of each point in the image, and then divide it by the grayscale standard deviation of the entire image. In the normalized image, the mean of all pixel values is 0 and the standard deviation is 1. This not only eliminates the impact of global characteristics such as brightness and contrast between images, but also allows different images to match better when stitching.
[0063] S3 feature extraction: Extract feature points from the preprocessed image for subsequent image matching. Use the SIFT algorithm to extract features. The generation of SIFT features mainly includes the construction of scale space, the construction of DoG space, key point detection, main direction allocation and the generation of feature descriptors. The definition formula of scale space is:
[0064]
[0065] where L is the scale space function, is the pixel value of the image at position (x, y), is a 2D Gaussian function, is the scale (i.e., the standard deviation of the Gaussian kernel).
[0066] To find stable feature points of an image in scale space, we need to find extreme points as key points. This requires constructing a DoG (Difference of Gaussian) image. The calculation formula for the DoG image is:
[0067]
[0068] in, and Represent two scale space images, is the original image, k is a constant, k>1.
[0069] S4 image matching: Use SIFT method to find matching feature point pairs between adjacent images;
[0070] The image matching described above adopts SIFT feature matching. Feature matching utilizes the Euclidean distance calculation of the feature space. The similarity is measured by calculating the Euclidean distance between two feature vectors (i.e., SIFT feature descriptors). The smaller the Euclidean distance between the two feature vectors, the higher their similarity. Assuming that the two feature descriptors are P and Q, the Euclidean distance between them can be calculated by the following formula:
[0071]
[0072] Where P and Q represent two feature points respectively, and (x1, y1) and (x2, y2) are the two-dimensional coordinates of the feature points.
[0073] S5 perspective projection transformation calculation: calculate the perspective projection transformation matrix of the matching pair to obtain the mapping relationship between each image and the panoramic coordinate system;
[0074] The perspective projection transformation calculation is performed by solving the homography matrix to obtain the mapping relationship between images; assuming that the point of the original image is P(x, y), and the point mapped to the new image by the homography matrix H is P'(x', y'), the formula is:
[0075]
[0076] This results in a linear equation with 8 degrees of freedom, which can be solved by calculating at least 4 pairs of matching points.
[0077] Perspective projection transformation, the most important formula for calculating the perspective projection matrix H. Assume that between a pair of matching feature points (xi, yi) and (Xi, Yi), there is the following linear homogeneous equation:
[0078]
[0079] Here, a11, a12, ..., a33 are elements of the perspective projection matrix H, and t is a homogeneous factor. This gives us a linear system of equations consisting of 8 unknowns (ignoring the scale factor) and 2 equations for each pair of matching points, which can be solved using the least squares method to obtain the perspective projection matrix.
[0080] like Figure 2 As shown, S6 closed shape matching: using closed shape matching to solve image seams and deformation problems to generate a continuous panoramic image;
[0081] S601 First, the position of the target image and the position of the source image to be transparently merged need to be determined by selecting a path with a mouse and then forming a closed loop, that is, determining a closed area.
[0082] S602 Establishing the target equation: In this step, the target equation needs to be established. Each pixel point of the source image has a gradient value, which needs to be kept unchanged. For each pixel p in the closed loop, the goal is to find a new pixel value , so that its gradient value is equal to the gradient value of the source image, that is, ;
[0083] Where g(p) is the pixel value of the source image, Δ is the Laplace operator, and f is the pixel value in the closed loop to be solved. is the gradient of f* at pixel p.
[0084] S603 Definition of Laplace operator: In a two-dimensional discrete image, the calculation formula of the Laplace operator is: , this operator describes the relationship between pixel p and its neighboring pixels.
[0085] S604 Closed area boundary processing: Since there is a boundary between the closed area and the outside, special attention needs to be paid during the processing stage. For pixels on the closed loop boundary, This formula is no longer completely applicable. At this time, it is necessary to change the formula appropriately to consider the pixel values of the source image and the target image at the same time. The formula is On the source image, On the target image, is the gradient value of the target image at point p.
[0086] S605 solves the equation: Jacobi Iteration is used to solve the equation. This algorithm solves the equation by continuous iteration. , so that As close as possible .
[0087] S7 image stitching: Based on the matching results of closed shapes, all images are stitched according to the results of perspective projection transformation to form a panoramic image.
[0088] The image stitching is performed based on the calculated homography matrix and the matching result of the closed shape. The image stitching process requires the homography matrix H obtained from the perspective projection transformation calculation link and the matching result of the closed shape.
[0089] First, use the homography matrix H to perform perspective transformation on each image and map each image to the panoramic coordinate system;
[0090] Then in this panoramic coordinate system, the position of each image has been determined;
[0091] In order to process the seams between images in the process of stitching a panoramic image, multi-image fusion is performed; specifically, for the overlapping areas of the images, the Alpha fusion algorithm is used to process the seams to achieve a seamless stitching effect.
[0092] The image fusion process is as follows: Assume that the image and images The pixel values in a certain overlapping area are and , mix the two according to a certain weight to get the fused image pixel value , define the weight w of a pixel point. The closer the pixel point is to the center of the image, the larger its weight w is. Based on this, the original pixel mixing formula can be defined as: .
[0093] The Alpha blending algorithm achieves the fusion of two images by changing the transparency (Alpha value). and images The weights are α1 and α2 respectively. Then in the image and images In the overlapping area A, the fused image The pixel value is: ,in, The principle is that in the overlapping area A, the distance image The farther the edge is, the larger the weight α1 of the corresponding point is, and vice versa.
[0094] Embodiment 1
[0095] like Figure 3 As shown in the figure, in the surround view stitching system of the adaptive vehicle body:
[0096] Image acquisition module: The main function is to collect image data from the front, rear, left and right directions of the vehicle. Real-time image data is obtained through camera equipment and the data is transmitted to the main control module.
[0097] Main control module: It is the control center of the whole system, which consists of image preprocessing module, feature extraction module, picture matching module, perspective projection transformation module, closed shape matching module and image stitching module. The image preprocessing module is responsible for processing the original image data collected by the image acquisition module, such as denoising, normalization and grayscale; the feature extraction module is responsible for extracting feature points from the preprocessed image; the image matching module is responsible for matching adjacent images using the features extracted by the feature extraction module; the perspective projection transformation module is responsible for calculating the perspective projection transformation matrix for feature point pairs and obtaining the mapping relationship of each image to the panoramic coordinate system; the closed shape matching module is responsible for processing the results of the perspective projection transformation to solve the problems of image seams and deformation; the image stitching module is responsible for stitching each image into a panoramic image according to the results of the closed shape matching and the mapping relationship calculated by the perspective projection transformation.
[0098] Display module: responsible for displaying the panoramic image generated by the main control module. It provides a user interface through which users can see the situation around the vehicle.
[0099] The connection relationship between modules:
[0100] The image acquisition module transmits the acquired image data to the main control module, which processes the data to generate a panoramic image, which is then sent to the display module for display. Inside the main control module, the submodules have strong data dependencies and a strict sequence to complete the image stitching work together. The modules work together to complete the generation and display of panoramic images for the vehicle surround view system.
[0101] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0102] It should be understood that the detailed description of the technical solutions of the present invention by means of the preferred embodiments is illustrative rather than restrictive. A person skilled in the art may modify the technical solutions described in the embodiments, or replace some of the technical features by equivalents, based on reading the specification of the present invention; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A surround view stitching method for an adaptive vehicle body, characterized in that: The surround stitching method comprises: Image acquisition: collect image data from four directions of the vehicle, namely, images from the front, back, left and right; Image preprocessing: preprocess the collected raw image data, including denoising, normalization and grayscale processing; Feature extraction: extract feature points from the preprocessed image for subsequent image matching; Image matching: Use the SIFT method to find matching feature point pairs between adjacent images; Perspective projection transformation calculation: Calculate the perspective projection transformation matrix for the matching pair to obtain the mapping relationship between each image and the panoramic coordinate system; Closed shape matching: Use closed shape matching to solve image seams and deformation problems and generate a continuous panoramic image; Image stitching: Based on the matching results of closed shapes, all images are stitched together according to the results of perspective projection transformation to form a panoramic image; The grayscale is expressed by the following formula: Gray=R*0.299+G*0.587+B*0.114 Among them, Gray is the gray value, "R", "G", and "B" represent the three primary colors of red, green, and blue respectively; The feature extraction uses the SIFT algorithm to extract features, and the feature point detection part of SIFT uses the Gaussian difference function: D(x,y,σ)=(G(x,y,kσ)-G(x,y,σ))*I(x,y) Among them, G(x,y,kσ) and G(x,y,σ) represent two scale space images, and I(x,y) is the original image; The image matching described above uses the Euclidean distance formula to describe the matching degree between two feature points. The Euclidean distance between them is calculated by the following formula: Among them, P and Q represent two feature points respectively, (x1, y1) and (x2, y2) are the two-dimensional coordinates of the feature points; The perspective projection transformation calculation is performed by solving the homography matrix to obtain the mapping relationship between images; assuming that the point of the original image is P(x, y), and the point mapped to the new image by the homography matrix H is P'(x', y'), the formula is: [x ′ ,y ′ ,1] ′ =H*[x,y,1] ′ This results in a linear equation with 8 degrees of freedom, which can be solved by calculating at least 4 pairs of matching points; In the perspective projection transformation, when calculating the homography matrix H, it is assumed that between a pair of matching feature points (xi, yi) and (Xi, Yi), there is the following linear homogeneous equation: a11xi+a12yi+a13=Xi / t a21xi+a22yi+a23=Yi / t a31xi+a32yi+a33=1 / t Among them, a11, a12, ..., a33 are the elements of the homography matrix H, and t is a homogeneous factor. From this, we can get a linear equation system consisting of 8 unknowns and 2 equations provided by each pair of matching points. The perspective projection matrix is solved by the least squares method. The closed shape matching adopts the Poisson image editing algorithm, which combines the gradient information of the source image and the boundary information of the target image. The formula is: Δf*=Δg Where f* is the required result image, Δ is the Laplacian operator of the image, and g is the source image. By solving this equation, we can get the seamless fusion result image. The specific steps of the closed shape matching are: Determine the location of the target image and the location of the source image that needs to be transparently merged by selecting a path with the mouse and then forming a closed loop, that is, determining a closed area; Establish the target equation: Each pixel of the source image has a gradient value, which needs to be kept unchanged. For each pixel p in the closed loop, the goal is to find a new pixel value f*(p) so that its gradient value is equal to the gradient value of the source image, that is, Δf*(P)=Δg(P), where g(P) is the pixel value of the source image, f is the pixel value in the closed loop to be solved, and Δf*(P) is the gradient value of f* at pixel p; Definition of Laplace operator: In a two-dimensional discrete image, the calculation formula of the Laplace operator is: Δf(p)=f(p+1)+f(p-1)-2*f(p) This operator describes the relationship between pixel p and its neighboring pixels; Closed area boundary processing: At the pixels on the closed loop boundary, the pixel values of the source image and the target image are considered at the same time. The formula is Δf*(P)=Δg(P)if p is on the source image, Δf*(P)=Δf(P)if p is on the target image, Δf(p) is the gradient value of the target image at point p; Solve the equation: Use the Jacobi iteration method to solve f* through continuous iteration, so that Δf* is close to Δg; The image stitching is performed based on the calculated homography matrix H and the matching result of the closed shape; First, use the homography matrix H to perform perspective transformation on each image and map each image to the panoramic coordinate system; Then in this panoramic coordinate system, the position of each image has been determined; In order to process the seams between images in the process of stitching a panoramic image, multi-image fusion is performed; specifically, for the overlapping areas of the images, the Alpha fusion algorithm is used to process the seams to achieve a seamless stitching effect; The image fusion process is as follows: suppose the pixel values of image I1 and image I2 in a certain overlapping area are I1(x, y) and I2(x, y) respectively, and the two are mixed according to the weight to obtain the fused image pixel value If(x, y). Define the weight w of a pixel point. The closer the pixel point is to the center of the image, the larger the weight w is. The original pixel mixing formula is: If(x, y) = w*I1(x, y) + (1-w)*I2(x, y); Assume that the weight values of image I1 and image I2 are α1 and α2 respectively. In the overlapping area A of image I1 and image I2, the pixel value of the fused image If is: If(x, y) = α1I1(x, y) + α2I2(x, y), where α1+α2=1.
2. An adaptive vehicle surround view stitching system, applicable to the method as claimed in claim 1, characterized in that: The system includes an image acquisition module for acquiring image data in four directions: front, rear, left, and right of the vehicle; The main control module includes an image preprocessing module, a feature extraction module, a picture matching module, a perspective projection transformation module, a closed shape matching module, and an image stitching module; A display module, used to display the panoramic image captured by the camera; In the main control module, the image preprocessing module performs denoising, normalization and grayscale processing on the original image data collected by the image acquisition module; A feature extraction module is used to extract features from the image processed by the image preprocessing module; The image matching module uses the features extracted by the feature extraction module to match adjacent images through the SIFT method; The perspective projection transformation module calculates the perspective projection transformation matrix for the feature point pairs obtained by the image matching module to obtain the mapping relationship between each image and the panoramic coordinate system; A closed shape matching module performs closed shape matching on the mapping relationship obtained by the perspective projection transformation calculation module to handle image seams and deformation issues; The image stitching module stitches the images into a panoramic image according to the matching result obtained by the closed shape matching module and the mapping relationship obtained by the perspective projection transformation calculation module.
