Image stitching optimization method and device in vehicle surround view system

Abstract

The present invention provides an image stitching optimization method and device for a vehicle surround view system. The method comprises: extracting features from multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; determining a first error between the first target view and the second target view based on first feature points in a first target view among the multiple views and overlapping feature points corresponding to a target overlapping area image among the multiple overlapping area images; determining the first target view corresponding to the smallest first error as the standard view, and determining the first target view corresponding to the largest first error as the optimized view; performing perspective transformation on the optimized view into the first view based on the standard view; and stitching the first view and the standard view together to obtain a panoramic view. The method of the present invention can improve panoramic stitching effects.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching, and in particular to an image stitching optimization method and device in a vehicle surround view system. Background Art

[0002] Panoramic image stitching technology is widely used in the automotive electronics field. In related technologies, a surround-view system consists of multiple cameras surrounding the vehicle, image acquisition equipment, video synthesis / processing components, and a display device. These cameras capture image information around the vehicle and simultaneously feed it into the video synthesis / processing component. The images are then processed through distortion correction, perspective transformation, and stitching and fusion techniques to create a comprehensive panoramic image of the vehicle's surroundings. However, this method can still create misalignment between adjacent views, affecting the overall stitching effect. Summary of the Invention

[0003] The present invention provides an image stitching optimization method and device in a vehicle surround view system, which are used to solve the defect of poor panoramic stitching effect in the prior art and improve the panoramic stitching effect.

[0004] The present invention provides an image stitching optimization method in a vehicle surround view system, comprising:

[0005] Performing feature extraction on the multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; the overlapping area image is an image of the overlapping area between any view and an adjacent view in the multiple views;

[0006] determining a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and an overlapping feature point corresponding to a target overlapping area image among the multiple overlapping area images; the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view;

[0007] determining a first target view corresponding to the smallest first error as a standard view, and determining a first target view corresponding to the largest first error as an optimized view;

[0008] Based on the standard view, perspective transforming the optimized view into a first view;

[0009] The first view and the standard view are stitched together to obtain a panoramic view.

[0010] According to an image stitching optimization method in a vehicle surround view system provided by the present invention, extracting features from a plurality of overlapping area images to obtain overlapping feature points corresponding to each overlapping area image includes:

[0011] Inputting the plurality of overlapping area images into a feature point detection model respectively, and obtaining a first feature point corresponding to each of the overlapping area images output by the feature point detection model;

[0012] The first feature point is input into a feature point matching model, and the overlapping feature point output by the feature point matching model is obtained.

[0013] According to an image stitching optimization method in a vehicle surround view system provided by the present invention, determining a first error in converting the first target view to the second target view based on a first feature point in a first target view among the multiple views and overlapping feature points corresponding to target overlapping area images among the multiple overlapping area images, includes:

[0014] Obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image;

[0015] The first error is determined based on an error value between a first feature point in the first target view and the perspective transformed point.

[0016] According to an image stitching optimization method in a vehicle surround view system provided by the present invention, obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image includes:

[0017] Obtaining a homography matrix from the first feature point to the overlapping feature point;

[0018] The perspective transformation point is determined based on the homography matrix and the first feature point.

[0019] According to an image stitching optimization method in a vehicle surround view system provided by the present invention, the perspective transformation of the optimized view into a first view based on the standard view includes:

[0020] Using an iterative algorithm, determining an optimal homography matrix from the optimized view to the standard view;

[0021] The first view is determined based on the optimal homography matrix and the optimized view.

[0022] According to an image stitching optimization method in a vehicle surround view system provided by the present invention, the iterative algorithm is used to determine the optimal homography matrix from the optimized view to the standard view, including:

[0023] Acquire multiple groups of first overlapping feature points of the target quantity from the target overlapping area image corresponding to the optimized view;

[0024] Based on the standard image, determining a homography matrix corresponding to each group of first overlapping feature points mapped to the standard image;

[0025] Obtaining a second error corresponding to each homography matrix;

[0026] When the second error is less than a target threshold, the homography matrix corresponding to the second error is determined as the optimal homography matrix.

[0027] The present invention also provides an image stitching optimization device in a vehicle surround view system, comprising:

[0028] A first processing module is configured to perform feature extraction on a plurality of overlapping region images to obtain overlapping feature points corresponding to each overlapping region image; the overlapping region image is an image of an overlapping region between any view and an adjacent view among the plurality of views;

[0029] a second processing module, configured to determine a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and an overlapping feature point corresponding to a target overlapping area image among the multiple overlapping area images, wherein the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view;

[0030] a third processing module, configured to determine the first target view corresponding to the smallest first error as a standard view, and determine the first target view corresponding to the largest first error as an optimized view;

[0031] a fourth processing module, configured to perspective-transform the optimized view into a first view based on the standard view;

[0032] The fifth processing module is configured to stitch the first view and the standard view together to obtain a panoramic view.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the image stitching optimization method in the vehicle surround view system described above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image stitching optimization method in the vehicle surround view system as described above is implemented.

[0035] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described image stitching optimization methods in a vehicle surround view system.

[0036] The image stitching optimization method and device provided by the present invention in a vehicle surround view system can effectively reduce the error after mapping the feature points in the overlapping area image by extracting overlapping feature points in the overlapping area image, and determine the standard view and the optimized view based on the first error between the overlapping feature points and the first feature points, and then perform perspective transformation on the optimized view to generate the first view, so as to improve the effect of the first view. The first view and the standard view are then stitched together to obtain a panoramic view, which can significantly reduce the degree of misalignment between adjacent views, thereby improving the stitching effect of the finally generated panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is one of the flow charts of the image stitching optimization method in the vehicle surround view system provided by the present invention;

[0039] Figure 2 Schematic diagram of the principle of the image stitching optimization method in the vehicle surround view system provided by the present invention;

[0040] Figure 3 This is the second flow chart of the image stitching optimization method in the vehicle surround view system provided by the present invention;

[0041] Figure 4 2. It is a structural diagram of the image stitching optimization device in the vehicle surround view system provided by the present invention;

[0042] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] The following combination Figures 1 to 3 The present invention describes an image stitching optimization method for a vehicle surround view system.

[0045] It should be noted that the executor of the image stitching optimization method in the vehicle surround view system can be a vehicle, or a server connected to the vehicle for communication, or an image stitching optimization device in the vehicle surround view system connected to the vehicle for communication, or a user's terminal, including but not limited to mobile terminals such as mobile phones, watches, and vehicle-mounted terminals, and non-mobile terminals such as PCs.

[0046] like Figure 1 As shown, the image stitching optimization method in the vehicle surround view system includes: step 110, step 120, step 130, step 140 and step 150.

[0047] Step 110: extract features from the multiple overlapping region images to obtain overlapping feature points corresponding to each overlapping region image; the overlapping region image is an image of the overlapping region between any view and an adjacent view in the multiple views;

[0048] In this step, the view is an image captured by an image sensor, and different views correspond to different orientations.

[0049] In actual implementation, the image sensors may be cameras located in different areas of the vehicle, front, back, left, and right, to capture images in different directions; the multiple views include images in different directions captured by cameras in different areas.

[0050] Any adjacent views have an overlapping area, and the overlapping area image is the image of the area where the view overlaps with any adjacent view in the view.

[0051] like Figure 2 As shown, the multiple image sensors include: an image sensor C1 located in front of the vehicle, an image sensor C3 located at the rear of the vehicle, an image sensor C2 located on the right of the vehicle, and an image sensor C4 located on the left of the vehicle; the multiple views include: a front view A captured by the image sensor C1, a rear view B captured by the image sensor C3, a left view C captured by the image sensor C4, and a right view D captured by the image sensor C2.

[0052] The overlapping area image may include: a front left overlapping area image, a front right overlapping area, a rear left overlapping area, a rear right overlapping area, a left front overlapping area, a left rear overlapping area, a right front overlapping area, and a right rear overlapping area.

[0053] Among them, the image of the overlapping area between the front view A and the left view C is the front left overlapping area image; the image of the overlapping area between the front view A and the right view D is the front right overlapping area image; the image of the overlapping area between the left view C and the front view A is the left front overlapping area image; the image of the overlapping area between the left view C and the rear view B is the left rear overlapping area image.

[0054] Overlapping feature points are pixels at the same position in the overlapping area image, which are acquired by image sensors corresponding to adjacent views.

[0055] The overlapping feature points include: descriptions of the overlapping feature points and coordinates of the overlapping feature points.

[0056] During the actual execution process, features can be extracted from the overlapping area images, such as by using a pre-trained neural network model to extract features from the overlapping area images to obtain overlapping feature points.

[0057] The neural network model can be a model of any structure or form, and the present invention does not limit this.

[0058] Step 120: determining a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and overlapping feature points corresponding to a target overlapping area image in the multiple overlapping area images; the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view;

[0059] In this step, the first target view may be any view among the multiple views.

[0060] The second target view is any view adjacent to the first target view among the multiple views.

[0061] The target overlapping area image is an image of the area where the first target view and the second target view overlap.

[0062] The first feature point is a feature point of an area in the first target view that overlaps with the second target view.

[0063] The first error represents the difference between the mapped point obtained after mapping the first feature point in the first target view to the second target view and the overlapping feature point corresponding to the first feature point. This represents the "cost" of converting the first target view to the second target view.

[0064] It can be understood that the smaller the first error is, the more accurate the obtained mapping point is.

[0065] In actual implementation, the same method may be used to determine the first error corresponding to each of the multiple views.

[0066] Step 130: Determine the first target view corresponding to the smallest first error as the standard view, and determine the first target view corresponding to the largest first error as the optimized view;

[0067] In this step, the standard view is a view that does not need to be mapped among the multiple views.

[0068] The optimized view is a view that needs to be mapped to a standard view to obtain a mapped view.

[0069] In some embodiments, the standard view includes at least one view from the plurality of views, and the optimized view includes at least one view from the plurality of views.

[0070] This step will be explained using the four views of front, back, left and right as examples.

[0071] In some embodiments, the first errors corresponding to the views may be sorted by size. There are the following situations for different sorting orders:

[0072] 1) If the first errors corresponding to the front view A and the rear view B are both smaller than the first errors corresponding to the left view C and the right view D, the front view A and the rear view B are determined as the reference views, and the left view C and the right view D are determined as the optimized views;

[0073] For example, the first error a corresponding to the front view A, the first error b corresponding to the rear view B, the first error c corresponding to the left view C, and the first error d corresponding to the right view D are respectively obtained, where a<b<c<d; then the front view A and the rear view B can be determined as standard views, and the left view C and the right view D can be determined as optimized views.

[0074] 2) If the first errors corresponding to the left view C and the right view D are both smaller than the first errors corresponding to the front view A and the back view B, the left view C and the right view D are determined as the reference views, and the front view A and the back view B are determined as the optimized views.

[0075] 3) In cases other than the above, the view with the largest first error is used as the optimized view, and the other three views are used as the reference views.

[0076] For example, the first error a corresponding to the front view A, the first error b corresponding to the rear view B, the first error c corresponding to the left view C, and the first error d corresponding to the right view D are respectively obtained, where a>d>c<b; then the front view A can be determined as the optimized view, and the rear view B, left view C, and right view D can be determined as standard views.

[0077] In this step, by determining the view with the largest first error as the optimized view for mapping processing, misalignment can be eliminated to the greatest extent, thereby improving the effect of the panoramic image obtained by subsequent stitching.

[0078] Step 140: Based on the standard view, perspective transform the optimized view into a first view;

[0079] In this step, the first view is a view generated by mapping the optimized view to the standard view.

[0080] One optimized view corresponds to one first view.

[0081] In actual implementation, the optimized view may be perspective-transformed into the first view using a homography matrix from the optimized view to the standard view.

[0082] For example, continue to refer to Figure 2 After the front view A and the back view B are determined as standard views and the left view C and the right view D are determined as optimized views, the homography matrices of the left view C to the front view A and the back view B are respectively used to perspective transform the left view C into the first view E;

[0083] The homography matrices of the right view D to the front view A and the back view B are respectively used to perspective transform the right view D into the first view F.

[0084] Step 150: stitch the first view and the standard view together to obtain a panoramic view.

[0085] In this step, after obtaining the first view E corresponding to the perspective transformation of the left view C and the first view F corresponding to the perspective transformation of the right view D through step 140, the front view A, the rear view B, the first view E and the right view D are panoramically stitched to obtain a panoramic view.

[0086] According to the image stitching optimization method in a vehicle surround view system provided by an embodiment of the present invention, by extracting overlapping feature points in the overlapping area image, and determining a standard view and an optimized view based on a first error between the overlapping feature points and the first feature points, and then performing a perspective transformation on the optimized view to generate a first view, the error after mapping the feature points in the overlapping area can be effectively reduced to improve the effect of the first view. The first view and the standard view are then stitched together to obtain a panoramic view, which can significantly reduce the degree of misalignment between adjacent views, thereby improving the stitching effect of the finally generated panoramic image.

[0087] like Figure 3 As shown, in some embodiments, step 110 may include:

[0088] Inputting the multiple overlapping area images into the feature point detection model respectively, and obtaining the first feature point corresponding to each overlapping area image output by the feature point detection model;

[0089] The first feature point is input into the feature point matching model to obtain overlapping feature points output by the feature point matching model.

[0090] Optionally, a feature point detection model is used to extract feature points.

[0091] The feature point matching model is used to match feature points of any two images from different overlapping areas to output successfully matched feature points.

[0092] Both the feature point detection model and the feature point matching model can be neural network models, wherein the output of the feature point detection model is connected to the input of the feature point matching model.

[0093] Specifically, the feature point detection model may be a deep learning model SuperPoint model, and the feature point matching model may be a SuperGlue model.

[0094] For example, continue to refer to Figure 2 , intercepting the overlapping area images between adjacent views, and inputting them into the feature point matching model SuperPoint after image preprocessing (such as normalization, etc.). The feature point matching model obtains relevant information corresponding to the first feature point of the overlapping area image, such as the coordinates and description of the first feature point;

[0095] The description is a 256-dimensional vector, and each dimension represents some attributes of the first feature point (such as gradient, color, etc.).

[0096] The coordinates of the first feature points and the description of the first feature points are then input into the feature point matching model SuperGlue model. The feature point matching pairs the first feature points according to the similarity of the descriptions of different first feature points, and finally obtains the coordinates of the paired feature points between the two images in the overlapping area of ​​the panoramic image, that is, the overlapping feature points.

[0097] In some embodiments, the sample overlapping area image may be used as a sample, and the sample feature points corresponding to the sample overlapping area image may be used as sample labels to train the feature point detection model.

[0098] The sample feature points are actual pixel points in the sample overlapping area image.

[0099] In some embodiments, the feature point matching model may be trained using sample feature points as samples and sample overlapping feature points corresponding to the sample feature points as sample labels.

[0100] The sample overlapping feature points are sample feature points that exist in two adjacent sample overlapping area images among the sample feature points.

[0101] According to the image stitching optimization method in the vehicle surround view system provided by the embodiment of the present invention, overlapping feature points of overlapping area images of adjacent views are obtained by using a feature point matching method (SuperPoint and SuperGlue) based on deep learning, which has high precision and accuracy and high learning ability.

[0102] In some embodiments, step 120 may include:

[0103] Obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image;

[0104] A first error is determined based on an error value between the first feature point and the perspective transformed point in the first target view.

[0105] Optionally, the perspective transformation point is a mapping point where the first feature point is mapped to the second target view.

[0106] The following describes the implementation of this step by taking the first target view as the front view A as an example.

[0107] For example, the first feature point of the front view A is taken as a set of points, with P front_1 Indicates that the overlapping feature points of the overlapping area images of the left view C and the right view D with the front view A are taken as another set of points, with P left_right Indicates that at this time P front_1 to P left_right The perspective transformation point is P front_2 express;

[0108] Then calculate P front_1 and P front_2 The error value between them is the first error a corresponding to the front view A.

[0109] In the same way, the first error c corresponding to the left view converted to the front and back views, the first error b corresponding to the back view converted to the left and right views, and the first error d corresponding to the right view converted to the front and back views can be calculated.

[0110] In some embodiments, obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image may include:

[0111] Get the homography matrix from the first feature point to the overlapping feature points;

[0112] Based on the homography matrix and the first feature point, a perspective transformation point is determined.

[0113] Optionally, the homography matrix is ​​a projection mapping from one plane to another plane.

[0114] In the actual implementation process, the implementation method of this embodiment is described by continuing to take the first target view as the front view A as an example.

[0115] The first feature point of the front view A is taken as a set of points, with P front_1 Indicates that the overlapping feature points of the overlapping area images of the left view C and the right view D with the front view A are taken as another set of points, with P left_right Expressed, and then through the following formula:

[0116] P front_2 =H*P front_1

[0117] Determine P front_2 , where P front_2 P front_1 to P left_right The perspective transformation point, H is P front_1 to P left_right The homography matrix of .

[0118] The following is a detailed description of how to determine the homography matrix.

[0119] A homography is a three-dimensional matrix that, when multiplied by homogeneous coordinates, creates a perspective transformation. Perspective transformations include rotations, translations, and affine transformations, and so do the components of a homography.

[0120] The principle of perspective transformation is as follows:

[0121]

[0122] Among them, (x1, y1, 1) represents the homogeneous coordinates of the feature points in the source image (i.e., the image before mapping, such as the target view); (x2, y2, 1) represents the homogeneous coordinates of the feature points in the target image (i.e., the image obtained after mapping); h 11 、h 12 、h 21 and h 22 Indicates transformations such as rotation and scaling, h 13 and h 23 represents translation transformation, h 31 and h 32 Space transformation for perspective transformation, h 33 is 1.

[0123] It should be noted that only h 11 ~h 32 The homography matrix H can be obtained with eight unknowns. Two unknowns can be obtained for each pair of matching points, so only four pairs of matching points are needed to obtain the homography matrix.

[0124] According to an embodiment of the present invention, an image stitching optimization method in a vehicle surround view system is provided. By obtaining a perspective transformation point from a first feature point in a first target view to an overlapping feature point corresponding to an image of a target overlapping area, a first error is determined based on an error value between the first feature point and the perspective transformation point in the first target view. By subsequently determining a standard view by a user, the error of the determined standard view can be minimized, thereby helping to reduce the error of subsequent image stitching and improving the image stitching effect.

[0125] Continue to refer Figure 3 In some embodiments, step 140 may include:

[0126] An iterative algorithm is used to determine the optimal homography matrix from the optimized view to the standard view;

[0127] Based on the optimal homography matrix and the optimized view, a first view is determined.

[0128] Optionally, the optimal homography matrix is ​​a homography matrix that can minimize the error of a first view obtained by mapping the optimized view to the standard view.

[0129] In the actual implementation process, the random sample consensus algorithm (RANSAC) can be used to continuously randomly select feature points, filter out mismatched points, and then regress the optimal homography matrix to determine the optimal homography matrix.

[0130] Of course, in other embodiments, other methods may also be used, such as determining the optimal homography matrix by using a neural network model or other methods, and the present invention is not limited thereto.

[0131] Based on the optimal homography matrix and the optimized view, the first view is determined, which can be expressed as taking the product of the optimal homography matrix and the optimized view as the first view.

[0132] In some embodiments, an iterative algorithm is used to determine the optimal homography matrix from the optimized view to the standard view, which may include:

[0133] Acquire multiple groups of first overlapping feature points of the target from the target overlapping area image corresponding to the optimized view;

[0134] Based on the standard image, determining a homography matrix corresponding to each group of first overlapping feature points mapped to the standard image;

[0135] Obtaining the second error corresponding to each homography matrix;

[0136] When the second error is less than the target threshold, the homography matrix corresponding to the second error is determined as the optimal homography moment.

[0137] Optionally, the target quantity may be user-defined, which is not limited in the present invention.

[0138] The target threshold can be user-defined, which is not limited in the present invention.

[0139] It should be noted that the first overlapping feature points corresponding to different groups should not be completely the same.

[0140] The first overlapping feature points are any multiple overlapping feature points in the same overlapping region image.

[0141] The second error is used to characterize the error between the mapping point obtained by mapping based on the current homography matrix and the actual feature point.

[0142] For example, first randomly select the first overlapping feature points of the target number and calculate the homography matrix corresponding to the group of points;

[0143] Then, other first overlapping feature points are put into the model to calculate the second error, and determine whether the first overlapping feature points conform to the model;

[0144] If the second error is less than the target threshold, the first overlapping feature point is determined to be an interior point, otherwise it is an exterior point;

[0145] Iterate multiple times and select the first overlapping feature points of different groups to calculate multiple models. The model with the most inliers is the optimal model that meets all points (i.e., the optimal homography matrix).

[0146] According to the image stitching optimization method in the vehicle surround view system provided by the embodiment of the present invention, the optimal homography matrix from the optimized view to the standard view is determined through an iterative algorithm, which can filter out mismatched points, so that the determined homography matrix is ​​the optimal value, thereby further improving the stitching effect.

[0147] The image stitching optimization device in the vehicle surround view system provided by the present invention is described below. The image stitching optimization device in the vehicle surround view system described below and the image stitching optimization method in the vehicle surround view system described above can be referred to each other.

[0148] like Figure 4 As shown, the image stitching optimization device in the vehicle surround view system includes: a first processing module 410, a second processing module 420, a third processing module 430, a fourth processing module 440 and a fifth processing module 450.

[0149] The first processing module 410 is configured to perform feature extraction on the multiple overlapping region images to obtain overlapping feature points corresponding to each overlapping region image; the overlapping region image is an image of the overlapping region between any view and an adjacent view among the multiple views;

[0150] A second processing module 420 is configured to determine a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and an overlapping feature point corresponding to a target overlapping area image among the multiple overlapping area images, wherein the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view;

[0151] A third processing module 430 is configured to determine the first target view corresponding to the smallest first error as the standard view, and determine the first target view corresponding to the largest first error as the optimized view;

[0152] A fourth processing module 440 is configured to perform perspective transformation of the optimized view into a first view based on the standard view;

[0153] The fifth processing module 450 is configured to stitch the first view and the standard view together to obtain a panoramic view.

[0154] According to the image stitching optimization device in the vehicle surround view system provided by an embodiment of the present invention, by extracting overlapping feature points in the overlapping area image, and determining the standard view and the optimized view based on the first error between the overlapping feature points and the first feature points, and then performing perspective transformation on the optimized view to generate the first view, the error after mapping the feature points in the overlapping area can be effectively reduced to improve the effect of the first view. The first view and the standard view are then stitched together to obtain a panoramic view, which can significantly reduce the degree of misalignment between adjacent views, thereby improving the stitching effect of the finally generated panoramic image.

[0155] In some embodiments, the first processing module 410 may also be used to:

[0156] Inputting the multiple overlapping area images into the feature point detection model respectively, and obtaining the first feature point corresponding to each overlapping area image output by the feature point detection model;

[0157] The first feature point is input into the feature point matching model to obtain overlapping feature points output by the feature point matching model.

[0158] In some embodiments, the second processing module 420 may also be used to:

[0159] Obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image;

[0160] A first error is determined based on an error value between the first feature point and the perspective transformed point in the first target view.

[0161] In some embodiments, the second processing module 420 may also be used to:

[0162] Get the homography matrix from the first feature point to the overlapping feature points;

[0163] Based on the homography matrix and the first feature point, a perspective transformation point is determined.

[0164] In some embodiments, the fourth processing module 440 may also be used to:

[0165] An iterative algorithm is used to determine the optimal homography matrix from the optimized view to the standard view;

[0166] Based on the optimal homography matrix and the optimized view, a first view is determined.

[0167] In some embodiments, the fourth processing module 440 may also be used to:

[0168] Acquire multiple groups of first overlapping feature points of the target from the target overlapping area image corresponding to the optimized view;

[0169] Based on the standard image, determining a homography matrix corresponding to each group of first overlapping feature points mapped to the standard image;

[0170] Obtaining the second error corresponding to each homography matrix;

[0171] When the second error is less than the target threshold, the homography matrix corresponding to the second error is determined as the optimal homography moment.

[0172] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the image stitching optimization method in the vehicle surround view system, which includes: performing feature extraction on multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; the overlapping area image is an image of the overlapping area with the adjacent view on any view in the multiple views; based on the first feature point in the first target view in the multiple views, and the overlapping feature point corresponding to the target overlapping area image in the multiple overlapping area images, determine the first error of converting the first target view to the second target view; the second target view is adjacent to the first target view, and the target overlapping area image is the overlapping area between the first target view and the second target view; the first target view corresponding to the smallest first error is determined as the standard view, and the first target view corresponding to the largest first error is determined as the optimized view; based on the standard view, the optimized view is perspective transformed into the first view; the first view and the standard view are stitched together to obtain a panoramic view.

[0173] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image stitching optimization method in the vehicle surround view system provided by the above methods, the method including: performing feature extraction on multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; the overlapping area image is an image of the overlapping area with the adjacent view on any view in the multiple views; based on the first feature point in the first target view in the multiple views, and the overlapping feature point corresponding to the target overlapping area image in the multiple overlapping area images, determining the first error of converting the first target view to the second target view; the second target view is adjacent to the first target view, and the target overlapping area image is the overlapping area between the first target view and the second target view; the first target view corresponding to the smallest first error is determined as the standard view, and the first target view corresponding to the largest first error is determined as the optimized view; based on the standard view, the optimized view is perspective transformed into the first view; the first view and the standard view are stitched together to obtain a panoramic view.

[0175] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image stitching optimization method in the vehicle surround view system provided above, the method comprising: performing feature extraction on multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; the overlapping area image is an image of the overlapping area with the adjacent view on any view in the multiple views; based on the first feature point in the first target view in the multiple views, and the overlapping feature point corresponding to the target overlapping area image in the multiple overlapping area images, determining the first error of converting the first target view to the second target view; the second target view is adjacent to the first target view, and the target overlapping area image is the overlapping area between the first target view and the second target view; determining the first target view corresponding to the smallest first error as the standard view, and determining the first target view corresponding to the largest first error as the optimized view; based on the standard view, perspective transforming the optimized view into the first view; stitching the first view and the standard view to obtain a panoramic view.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0177] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An image stitching optimization method in a vehicle surround view system, characterized in that: include: Perform feature extraction on multiple overlapping area images to obtain overlapping feature points corresponding to each overlapping area image; The overlapping area image is an image of an overlapping area between any view and an adjacent view among the multiple views; determining a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and an overlapping feature point corresponding to a target overlapping area image among the multiple overlapping area images; the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view; determining a first target view corresponding to the smallest first error as a standard view, and determining a first target view corresponding to the largest first error as an optimized view; Based on the standard view, perspective transforming the optimized view into a first view; The first view and the standard view are stitched together to obtain a panoramic view.

2. The image stitching optimization method in a vehicle surround view system according to claim 1, characterized in that: The step of extracting features from the multiple overlapping area images to obtain overlapping feature points corresponding to the overlapping area images includes: Inputting the plurality of overlapping area images into a feature point detection model respectively, and obtaining a first feature point corresponding to each of the overlapping area images output by the feature point detection model; The first feature point is input into a feature point matching model, and the overlapping feature point output by the feature point matching model is obtained.

3. The image stitching optimization method in a vehicle surround view system according to claim 1, characterized in that: The determining, based on a first feature point in a first target view among the multiple views and overlapping feature points corresponding to target overlapping area images among the multiple overlapping area images, a first error in converting the first target view to the second target view includes: Obtaining a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image; The first error is determined based on an error value between a first feature point in the first target view and the perspective transformed point.

4. The image stitching optimization method in a vehicle surround view system according to claim 3, characterized in that: The obtaining of a perspective transformation point from a first feature point in the first target view to an overlapping feature point corresponding to the target overlapping area image includes: Obtaining a homography matrix from the first feature point to the overlapping feature point; The perspective transformation point is determined based on the homography matrix and the first feature point.

5. The image stitching optimization method in a vehicle surround view system according to any one of claims 1 to 4, characterized in that: The perspective transforming the optimized view into a first view based on the standard view includes: Using an iterative algorithm, determining an optimal homography matrix from the optimized view to the standard view; The first view is determined based on the optimal homography matrix and the optimized view.

6. The image stitching optimization method in a vehicle surround view system according to claim 5, characterized in that: The adopting an iterative algorithm to determine the optimal homography matrix from the optimized view to the standard view includes: Acquire multiple groups of first overlapping feature points of the target quantity from the target overlapping area image corresponding to the optimized view; Based on the standard image, determining a homography matrix corresponding to each group of first overlapping feature points mapped to the standard image; Obtaining a second error corresponding to each homography matrix; When the second error is less than a target threshold, the homography matrix corresponding to the second error is determined as the optimal homography matrix.

7. An image stitching optimization device in a vehicle surround view system, characterized in that: include: The first processing module is used to extract features from the multiple overlapping area images and obtain overlapping feature points corresponding to the overlapping area images; The overlapping area image is an image of an overlapping area between any view and an adjacent view among the multiple views; a second processing module, configured to determine a first error in converting the first target view to a second target view based on a first feature point in a first target view among the multiple views and an overlapping feature point corresponding to a target overlapping area image among the multiple overlapping area images, wherein the second target view is adjacent to the first target view, and the target overlapping area image is an overlapping area between the first target view and the second target view; a third processing module, configured to determine the first target view corresponding to the smallest first error as a standard view, and determine the first target view corresponding to the largest first error as an optimized view; a fourth processing module, configured to perspective-transform the optimized view into a first view based on the standard view; The fifth processing module is configured to stitch the first view and the standard view together to obtain a panoramic view.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image stitching optimization method in the vehicle surround view system according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image stitching optimization method in a vehicle surround view system according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image stitching optimization method in a vehicle surround view system according to any one of claims 1 to 6 is implemented.

Images