Panoramic image stitching method and system, electronic device and storage medium
By determining the mapping relationship of pixel points in the fisheye image and using the weighted fusion method, the fisheye image is converted into a bird's-eye view, the problem of poor real-time styling on low-computing power platforms is solved, and efficient image styling and frame rate stability are achieved .
Patent Information
- Application Number
- CN202011265428.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-11-12
AI Technical Summary
The existing surround view splicing method is difficult to ensure real-time performance on low-computing platforms, and the calculation complexity is high, resulting in a decrease in video frame rate.
By determining the mapping relationship between pixel points and public visible areas in the fisheye image, the fisheye image is converted into a bird's-eye view, and the distance-weighted fusion and fixed-value weighted fusion methods are used to reduce the computational complexity and improve the real-time nature of image conversion.
Under the condition of ensuring the video frame rate, the calculation complexity of the splicing process is reduced, the obvious problem of excessive splicing seams is eliminated, the gradual over-change at the image splicing is achieved, and the computing efficiency is improved.
Smart Images

Figure CN112435161B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a panoramic image stitching method and system, electronic equipment and storage medium. Background Art
[0002] When a car parks in a narrow parking lot through narrow roads or traffic, it is easy to collide due to the limited vision of the driver, causing unnecessary losses. In terms of automatic parking environment perception, the application of visual sensors in automatic parking has led to the rapid development of 360-degree panoramic view technology.
[0003] By using four fisheye cameras installed on the front, rear, left and right sides of the vehicle body to obtain real-time environmental information around the vehicle body, zero-blind-angle detection of the environment around the vehicle body is achieved, which improves the driver's ability to monitor blind spots. At the same time, it can also be combined with image processing technology to realize functions such as parking space recognition, obstacle detection and drivable area recognition. It is an important part of the automatic parking system.
[0004] Since it is necessary to consider the consistent exposure of the four cameras and the continuity of the stitching seams, the existing surround stitching method will repeatedly traverse most of the pixel values of the four images during the calculation of each frame, which will greatly affect the real-time image data processing capabilities on low-computing power platforms.
[0005] Therefore, how to reduce the computational complexity of the stitching process while ensuring the video presentation frame rate has become an urgent problem to be solved. Summary of the invention
[0006] In view of the defects in the prior art, the embodiments of the present invention provide a panoramic image stitching method and system, an electronic device and a storage medium, which at least solve the technical problems that when converting fisheye images into panoramic images, the computational complexity is high and the real-time performance of image conversion is difficult to ensure on a low computing power platform.
[0007] In a first aspect, an embodiment of the present invention provides a panoramic image stitching method, comprising:
[0008] Get fisheye image;
[0009] Determine a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image;
[0010] According to the mapping relationship, the fisheye image is converted into a bird's-eye view, and a panoramic image is obtained according to the bird's-eye view.
[0011] Optionally, in the panoramic image stitching method,
[0012] The determining of a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image includes:
[0013] For any pixel point in the fisheye image, determining whether its image coordinates are within the public visible area;
[0014] If it is determined that the pixel point belongs to the common visible area, the image coordinates of the pixel point are obtained, and the image coordinates of the pixel point corresponding to the pixel point in the image in the common visible area with the fisheye image are formed into a mapping relationship;
[0015] If it is determined that the pixel does not belong to the public visible area, it is determined that the pixel point has nothing to do with the image in the public visible area of the fisheye image.
[0016] Optionally, in the panoramic image stitching method,
[0017] The converting the fisheye image into a bird's-eye view according to the mapping relationship comprises:
[0018] For a pixel point within the public visible area, determine a pixel value corresponding to the pixel point in the bird's-eye view according to a pixel value corresponding to the image coordinate of the pixel point in the fisheye image, a pixel value corresponding to the image coordinate of the pixel point in the image where the fisheye image has the public visible area, and the mapping relationship;
[0019] For a pixel point that does not belong to the public visible area, the pixel value corresponding to the pixel point in the bird's-eye view is determined according to the pixel value of the fisheye image corresponding to the image coordinates of the pixel point.
[0020] Optionally, in the panoramic image stitching method,
[0021] Before the step of determining the pixel value corresponding to the pixel point in the bird's-eye view according to the pixel value corresponding to the image coordinate of the pixel point in the fisheye image, the pixel value corresponding to the image coordinate of the pixel point in the image where the fisheye image has the public visible area, and the mapping relationship, the step further includes:
[0022] Determine a first pixel value and a second pixel value of a first pixel;
[0023] Comparing the difference between the first pixel value and the second pixel value in the color space;
[0024] Determine whether the pixel value difference is greater than a preset threshold;
[0025] Among them, the first pixel point is located in the common visible area; the first pixel value is the pixel value corresponding to the first pixel point in the fisheye image; the second pixel point is the pixel value corresponding to the first pixel point in the image that has a common visible area with the fisheye image.
[0026] Optionally, in the panoramic image stitching method,
[0027] The step of determining the pixel value corresponding to the pixel point in the bird's-eye view according to the pixel value corresponding to the image coordinate of the pixel point in the fisheye image, the pixel value corresponding to the image coordinate of the pixel point in the image in which the fisheye image has the public visible area, and the mapping relationship for the pixel point in the public visible area specifically includes:
[0028] If it is determined that the pixel value difference is not greater than a preset threshold, a distance weighted fusion method is used to determine a pixel value corresponding to the first pixel point in the bird's-eye view;
[0029] If it is determined that the pixel value difference is greater than a preset threshold, a fixed-value weighted fusion method is used to determine the pixel value corresponding to the first pixel point in the bird's-eye view.
[0030] Optionally, in the panoramic image stitching method,
[0031] If it is determined that the pixel value difference is not greater than a preset threshold, using a distance weighted fusion method to determine a pixel value corresponding to the first pixel point in the bird's-eye view image includes:
[0032] Obtaining the distance between the image coordinates of the first pixel and the projection boundaries of the public visible area;
[0033] The distance between the boundaries on both sides of the projection of the public visible area is used as a weight value to achieve weighted fusion of the pixel values of the first pixel points, and determine the pixel value corresponding to the first pixel point in the bird's-eye view.
[0034] Optionally, in the panoramic image stitching method,
[0035] If it is determined that the pixel value difference is greater than a preset threshold, using a fixed value weighted fusion method to determine a pixel value corresponding to the first pixel point in the bird's-eye view image includes:
[0036] According to the preset weight value, weighted fusion of the pixel values of the first pixel point is achieved to determine the pixel value corresponding to the pixel point in the bird's-eye view.
[0037] In a second aspect, an embodiment of the present invention provides a panoramic image stitching system, including:
[0038] An acquisition module, used for acquiring fisheye images;
[0039] A determination module, used to determine a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image;
[0040] The conversion module is used to convert the fisheye image into a bird's-eye view according to the mapping relationship, and obtain a panoramic image according to the bird's-eye view.
[0041] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the processor and the memory communicate with each other via a bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the various steps of the panoramic image stitching method described above.
[0042] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the various steps of the panoramic image stitching method as described above.
[0043] The panoramic image stitching method and system, electronic device and storage medium provided by the embodiments of the present invention can focus the algorithm processing on the common visible area of the four fisheye cameras by determining the mapping relationship of each pixel point in the image. It is only necessary to traverse all the pixels once to obtain a complete fused bird's-eye view. The method has low computing power cost and further eliminates the problem of overly obvious stitching seams while ensuring the frame rate of video presentation. It provides a buffer for the exposure differences and perspective differences of adjacent cameras in the common area of the image, so that objects on the ground have a gradual transition trend at the stitching point, thereby achieving a gradual transition at the image stitching point. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 A flow chart of a panoramic image stitching method provided by an embodiment of the present invention;
[0046] Figure 2 A fisheye image obtained by a forward-facing camera provided in an embodiment of the present invention;
[0047] Figure 3 A fisheye image obtained by the left camera provided in an embodiment of the present invention;
[0048] Figure 4A forward bird's-eye view provided for an embodiment of the present invention;
[0049] Figure 5 A left-side bird's-eye view provided for an embodiment of the present invention;
[0050] Figure 6 A schematic diagram of the splicing of a forward bird's-eye view and a left bird's-eye view provided in an embodiment of the present invention;
[0051] Figure 7 A 360° panoramic view provided by an embodiment of the present invention;
[0052] Figure 8 A schematic diagram of local perspective imbalance of a bird's-eye view provided by an embodiment of the present invention;
[0053] Fig. 9 A schematic diagram of a fixed-value weighted fusion method provided in an embodiment of the present invention;
[0054] Fig.10 A schematic diagram of the structure of a panoramic image stitching system provided by an embodiment of the present invention;
[0055] Fig.11 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] 360-degree surround view, with four fisheye cameras, the left and right cameras are installed under the rear mirror, the front camera is installed at the car logo, and the rear camera is installed at the trunk. There is a 20%-30% common viewing area between every two adjacent cameras.
[0058] The main technologies for 360-degree surround view stitching currently include: distortion correction of the fisheye camera’s intrinsic parameters; external parameter calibration after the four fisheye cameras are fixed at four positions outside the vehicle body, and a bird’s-eye view image is obtained through perspective transformation; to ensure that the bird’s-eye view image has no sense of disobedience in the transition area, the stitching seams need to be calculated in the common visible area of adjacent cameras, and the exposure needs to be balanced at the same time, so that when the vehicle is in two scenes with different lighting at the same time, the stitching seams will have obvious light and dark transitions due to the different exposures of adjacent cameras.
[0059] Figure 2A fisheye image obtained by a forward-facing camera provided in an embodiment of the present invention, Figure 3 The fisheye image obtained by the left camera provided in the embodiment of the present invention is as follows: Figure 2 and Figure 3 shown.
[0060] Figure 4 A forward bird's-eye view provided for an embodiment of the present invention, Figure 5 The left bird's-eye view provided in the embodiment of the present invention is obtained by first performing internal reference dedistortion processing on each fisheye image according to the prior art. Figure 4 and Figure 5 Forward and left side aerial views shown.
[0061] Figure 6 A schematic diagram of splicing a forward bird's-eye view and a left bird's-eye view provided in an embodiment of the present invention, such as Figure 6 As shown in FIG. 1 , if the adjacent forward bird's-eye view and the left bird's-eye view are spliced, there is a certain overlap area between the spliced bird's-eye views, and the black line in the figure is the boundary of the overlap area. This part is the common visible area between the fisheye image (forward) and the adjacent fisheye image (left).
[0062] Based on the characteristic that there are overlapping areas when the fisheye images are converted into bird's-eye views and then stitched together, the present invention provides a panoramic view image stitching method, which enables four fisheye images to be directly converted into a panoramic view image at one time without the need for repeated conversion calculations, thus saving computing resources and improving the real-time performance of image conversion. Figure 1 A flow chart of a panoramic image stitching method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:
[0063] Step S1, obtaining a fisheye image;
[0064] Step S2, determining a mapping relationship between pixel points in the fisheye image and an image having a common visible area with the fisheye image;
[0065] Step S3: converting the fisheye image into a bird's-eye view according to the mapping relationship, and obtaining a panoramic image according to the bird's-eye view.
[0066] Specifically, in order to facilitate the explanation of the embodiments of the present invention, the stitching and fusion of the fisheye image obtained by the left camera and the front camera is taken as an example for explanation. The principle of the stitching and fusion method of the fisheye images obtained by other adjacent cameras is the same as that of the embodiments of the present invention, and no further explanation is given here.
[0067] In step S1, a fisheye image is obtained by using a fisheye camera. Taking the front and left sides of the four-way fisheye image as an example, the obtained fisheye image is as follows: Figure 2 and Figure 3 shown.
[0068] In step S2, the mapping relationship between any pixel in the fisheye image and the adjacent fisheye image is determined. It should be noted that, for the convenience of explanation, in the embodiment of the present invention, the adjacent fisheye image refers to an image that has a common visible area with the fisheye image.
[0069] In specific implementation, for each pixel point, if it is not within the overlapping area of the bird's-eye view, then it only corresponds to the mapping relationship of one fisheye image; if the pixel point is within the overlapping area of the bird's-eye view, then it corresponds to the mapping relationship of two adjacent fisheye images.
[0070] It should be noted that the size of the overlapping area is related to the size of the common visible area of two adjacent fisheye cameras that is set in advance, and can be adjusted according to actual conditions, and the embodiment of the present invention does not limit this.
[0071] In step S3, according to the mapping relationship of all pixels in the fisheye image obtained in step S2, the fisheye image is corrected, and a bird's-eye view transformation is performed on the fisheye image. Based on the pixel values of the fisheye image pixels obtained by the bird's-eye view transformation, the four fisheye images are directly spliced and fused to convert the fisheye image into a bird's-eye view, thereby obtaining a panoramic ring view.
[0072] It should be noted that, in the embodiments of the present invention, Figure 4-6 As an auxiliary explanation of this solution, in fact, this solution does not need to convert the four fisheye images into the corresponding four bird's-eye views respectively before performing image stitching. The four fisheye images can be directly converted into a complete bird's-eye view (i.e., a panoramic view) according to the pixel values of the four fisheye images corresponding to the bird's-eye view after the top-down transformation and the mapping relationship corresponding to each pixel. Figure 7 As shown, Figure 7 A 360° panoramic view provided by an embodiment of the present invention.
[0073] It should be noted that four fisheye images are used in this embodiment. In addition, the position and number of fisheye cameras can be adjusted according to actual needs, and the number of fisheye images used can be further adjusted. This embodiment does not limit this.
[0074] Furthermore, based on this embodiment, the obtained plane image can be converted into a three-dimensional image to improve the user experience and reflect the relative position relationship between the vehicle and the object more clearly.
[0075] The panoramic image stitching method provided by the embodiment of the present invention can focus the algorithm processing on the common visible area of the four fisheye cameras by determining the mapping relationship of each pixel in the image. It only needs to traverse all the pixels once to obtain the complete fused bird's-eye view. The method has low computing cost. Except for the preparation calculation required for initialization, the time complexity of the main loop calculation is controlled at ~o(ab), where a and b are the length and width of the pixels of the bird's-eye view, respectively. Under the condition of ensuring the frame rate of video presentation, the problem of overly obvious stitching seams is further eliminated, and the exposure difference and perspective difference of adjacent cameras are buffered in the common area of the image, so that the objects on the ground have a gradual transition trend at the stitching point, and a gradual transition at the image stitching point is achieved.
[0076] Based on the above embodiment, optionally, in the panoramic image stitching method, determining the mapping relationship between the pixel points in the fisheye image on the image having a common visible area with the fisheye image includes:
[0077] For any pixel point in the fisheye image, determining whether its image coordinates are within the public visible area;
[0078] If it is determined that the pixel point belongs to the common visible area, the image coordinates of the pixel point are obtained, and the image coordinates of the pixel point corresponding to the pixel point in the image in the common visible area with the fisheye image are formed into a mapping relationship;
[0079] If it is determined that the pixel does not belong to the public visible area, it is determined that the pixel point has nothing to do with the image in the public visible area of the fisheye image.
[0080] Specifically, the mapping relationship actually reflects the process of determining the pixel value of each pixel point during the process of converting the fisheye image into a bird's-eye view.
[0081] For each pixel point, if its image coordinates are not within the common visible area between the fisheye image and its adjacent fisheye image, which means that after conversion to a bird's-eye view, the pixel point is also not within the overlapping area of the bird's-eye view, then it only corresponds to the mapping relationship of one fisheye image, that is, when converting the fisheye image to a bird's-eye view, only the pixel value of the corresponding pixel point in one fisheye image is used to determine that the pixel point is independent of the adjacent fisheye image.
[0082] If the image coordinates of a pixel point are within the common visible area between the fisheye image and its adjacent fisheye image, it means that after conversion to a bird's-eye view, the pixel point is also within the overlapping area of the bird's-eye view. Then, when the fisheye coordinates are converted to a bird's-eye view, the pixel value is related to the pixel value of the corresponding pixel point in the two adjacent fisheye images. The image coordinates of the pixel point and the image coordinates of the corresponding pixel point in the adjacent fisheye image are obtained to form a mapping relationship.
[0083] For example, assuming that the bird's-eye view obtained after processing the four fisheye images should be a 280*300 image, in which the coordinates of the pixel in the upper left corner are (0, 0), then the pixel with coordinates (2, 0) should be in the overlapping area between the forward bird's-eye view and the left bird's-eye view. At this time, the pixel has a corresponding point in the left fisheye image, and there are also related pixels in the forward fisheye image, forming a mapping relationship.
[0084] It should be noted that the above method for constructing the corresponding mapping relationship is only used as a specific example to explain the embodiment of the present invention. In addition, the mapping relationship can also be reflected in other ways, which is not limited in this embodiment.
[0085] On the basis of the above-mentioned embodiment, the embodiment of the present invention determines whether the coordinates of the pixel point are related to only one fisheye image or two fisheye images when converting the fisheye image to a bird's-eye view through the relationship between the pixel point image coordinates and the consolidated visible area between the fisheye image and its adjacent fisheye image, and constructs a corresponding mapping relationship, so as to focus the algorithm processing mainly on the common visible area of the four-way fisheye camera, reduce the complexity of the calculation, and save computing resources.
[0086] Based on the above embodiment, optionally, in the panoramic image stitching method, converting the fisheye image into a bird's-eye view according to the mapping relationship includes:
[0087] For a pixel point within the public visible area, determine a pixel value corresponding to the pixel point in the bird's-eye view according to a pixel value corresponding to the image coordinate of the pixel point in the fisheye image, a pixel value corresponding to the image coordinate of the pixel point in the image where the fisheye image has the public visible area, and the mapping relationship;
[0088] For a pixel point that does not belong to the public visible area, the pixel value corresponding to the pixel point in the bird's-eye view is determined according to the pixel value of the fisheye image corresponding to the image coordinates of the pixel point.
[0089] Specifically, the mapping relationship of the pixel points is essentially aimed at realizing the transformation of the pixel values in the process of converting the fisheye image to the bird's-eye view image. In other words, when the fisheye image is converted to the bird's-eye view, the pixel values of the overlapping area are recalculated, and the conversion and optimization of the overlapping area are realized in one step.
[0090] According to the mapping relationship, the pixel value corresponding to the fisheye image to be used when converting to the bird's-eye view is determined.
[0091] For a pixel point within the public visible area, determine the pixel value corresponding to the pixel point in the bird's-eye view according to the pixel values corresponding to the image coordinates of the pixel point in the fisheye image and its adjacent fisheye image, and the mapping relationship;
[0092] For a pixel point that does not belong to the public visible area, the pixel value corresponding to the pixel point in the bird's-eye view is determined according to the pixel value of the fisheye image corresponding to the image coordinates of the pixel point.
[0093] Based on the above embodiment, optionally, in the panoramic image stitching method, before the step of determining the pixel value corresponding to the pixel point in the bird's-eye view image according to the pixel value corresponding to the image coordinate of the pixel point in the fisheye image, the pixel value corresponding to the image coordinate of the pixel point in the image where the fisheye image has the public visible area, and the mapping relationship, the method further includes:
[0094] Determine a first pixel value and a second pixel value of a first pixel;
[0095] Comparing the difference between the first pixel value and the second pixel value in the color space;
[0096] Determine whether the pixel value difference is greater than a preset threshold;
[0097] Among them, the first pixel point is located in the common visible area; the first pixel value is the pixel value corresponding to the first pixel point in the fisheye image; the second pixel point is the pixel value corresponding to the first pixel point in the image that has a common visible area with the fisheye image.
[0098] Specifically, Figure 8 A schematic diagram of the imbalance of local viewing angles in a bird's-eye view provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown in Figure 2, the perspective of the three-dimensional obstacle will be unbalanced when crossing the fusion area. Figure 8 In the figure, the cone barrel has a double image in the common visible area of the left and front cameras due to the difference in viewing angle. Therefore, it is necessary to add the judgment of overlapping pixel differences in the color grading process.
[0099] In an embodiment of the present invention, in order to further enhance the optimization of the fusion of overlapping areas, different pixel value determination methods are used when converting fisheye images into bird's-eye views, so that the image can solve the problem of double images of three-dimensional obstacles and improve the problem of unbalanced viewing angles.
[0100] Before image conversion, a pixel difference threshold is set in advance, and the difference in color space between the corresponding pixel value in the fisheye image and the corresponding pixel value in the adjacent fisheye image is compared to determine whether the pixel value difference is greater than the preset threshold. Based on the comparison results, the pixel value fusion method to be used is selected.
[0101] It should be noted that the above color spaces include RGB color space, HSI color space and HVS color space, etc., which can be selected according to actual conditions. In addition, the size of the threshold can also be set according to actual conditions, which is not limited in this embodiment.
[0102] Based on the above embodiment, optionally, in the panoramic image stitching method, for the pixel points belonging to the public visible area, according to the pixel values corresponding to the image coordinates of the pixel points in the fisheye image, the pixel values corresponding to the image coordinates of the pixel points in the image where the fisheye image has the public visible area, and the mapping relationship, determining the pixel values corresponding to the pixel points in the bird's-eye view, specifically includes:
[0103] If it is determined that the pixel value difference is not greater than a preset threshold, a distance weighted fusion method is used to determine a pixel value corresponding to the first pixel point in the bird's-eye view;
[0104] If it is determined that the pixel value difference is greater than a preset threshold, a fixed-value weighted fusion method is used to determine the pixel value corresponding to the first pixel point in the bird's-eye view.
[0105] Specifically, the pixel value fusion method to be adopted is determined according to the difference between the corresponding pixel value in the fisheye image and the corresponding pixel value in the adjacent fisheye image in the color space.
[0106] When the pixel value difference is not greater than the preset threshold, the distance weighted fusion method is used to determine the pixel value corresponding to the pixel in the bird's-eye view. In the distance weighted fusion method, the size of the weight is not fixed and is related to the position of the pixel. The distance weighted fusion method can effectively reflect the image coordinate position of the pixel to the pixel, effectively improve the situation where there is a difference in brightness between the two fisheye images during fusion, and reduce the abnormality of the transition of the fusion area of the two adjacent fisheye images, providing a buffer for the exposure difference and perspective difference, so that the objects on the ground have a gradual transition trend at the joint.
[0107] When the pixel value difference is greater than the preset threshold, the fixed value weighted fusion method is used to determine the pixel value corresponding to the pixel point in the bird's-eye view. Selecting appropriate weights for weighted average fusion can select more pixel values in the fisheye image according to the position of the pixel point. This improves the blur problem in the picture and makes the image more realistic.
[0108] Based on the above embodiment, optionally, in the panoramic image stitching method, if it is determined that the pixel value difference is not greater than a preset threshold, a distance weighted fusion method is used to determine the pixel value corresponding to the first pixel point in the bird's-eye view, including:
[0109] Obtaining the distance between the image coordinates of the first pixel and the projection boundaries of the public visible area;
[0110] The distance between the boundaries on both sides of the projection of the public visible area is used as a weight value to achieve weighted fusion of the pixel values of the first pixel points, and determine the pixel value corresponding to the first pixel point in the bird's-eye view.
[0111] Specifically, Figure 6 A schematic diagram of splicing a forward bird's-eye view and a left bird's-eye view provided in an embodiment of the present invention, such as Figure 6 As shown, the solid lines in the figure are the boundaries on both sides of the projection of the public visible area, the image coordinates of the point P in the figure are marked as (x, y), and the dotted lines in the figure are the distances from the point P to the boundaries on both sides of the projection.
[0112] After determining that the pixel value difference is not greater than a preset threshold, obtaining the distance between the image coordinates of the pixel point and the projection boundaries of the common visible area on both sides;
[0113] The distance between the boundaries on both sides of the projection of the public visible area is used as a weight value to achieve weighted fusion of the pixel value of the pixel point and determine the pixel value corresponding to the pixel point in the bird's-eye view.
[0114] The pixel value in the final 360-degree bird’s-eye view is From the overall bird's eye view, any point P (x,y) The pixel value of is the perspective transformation relationship defined by the external parameters of each camera:
[0115] P dst (x, y) = P src (map x (x,y),map y (x, y)
[0116] Find the original image P src The pixel value corresponding to the image coordinates before the pixel is mapped to the bird's-eye view.
[0117] Still taking the forward and left directions as an example, Figure 6 As shown, the upper boundary of the overlapped area is the top projection boundary l_left of the left camera, and the distance from point P to the straight line l_left is The pixel values in the fisheye image on the left are The lower boundary of the overlapped area is taken as the top-down projection boundary l_front of the front camera, and the distance from P to the straight line l_front is The pixel value in the forward fisheye image is
[0118] The pixel value of pixel point P in the final 360-degree panoramic view is
[0119]
[0120] It should be noted that in addition to using the distance from the pixel point to the boundaries on both sides of the projection as the weight of weighted fusion, it can be understood that the distance mentioned in the embodiment of the present invention can also be the distance between the intersection of two straight lines drawn from the pixel point in the horizontal and vertical directions and the boundary lines on both sides of the projection, which can be used as the weight of weighted fusion.
[0121] Based on the above embodiment, optionally, in the panoramic image stitching method, if it is determined that the pixel value difference is greater than a preset threshold, a fixed value weighted fusion method is used to determine the pixel value corresponding to the first pixel point in the bird's-eye view, including:
[0122] According to the preset weight value, weighted fusion of the pixel values of the first pixel point is achieved to determine the pixel value corresponding to the pixel point in the bird's-eye view.
[0123] Specifically, Fig. 9 A schematic diagram of a fixed-value weighted fusion method provided in an embodiment of the present invention, such as Fig. 9 As shown, the coordinates of the pixel point P in the figure are (x, y). After determining that the pixel value difference is greater than the preset threshold, the weighted fusion of the pixel value of the pixel point is realized according to the preset weight value to determine the pixel value corresponding to the pixel point in the bird's-eye view.
[0124] Since the weights are fixed, we first need to choose which fisheye image to retain more pixel values based on the coordinates of the pixel points.
[0125] For example, according to the coordinates of the corner point of the fusion area, the slope of the line connecting the corner point (m, n) of the fusion area and the boundary point is obtained, and the positional relationship between the pixel point and the slope is determined according to the comparison of the slopes.
[0126] The specific formula for determining the position relationship of points based on the slope is:
[0127]
[0128] From the above formula, if the relationship in the formula is satisfied, it means that the pixel point is above the diagonal line, that is, more pixel values of the pixel point P in the forward fisheye image are retained. Correspondingly, for the pixel points below the diagonal line, it means that more pixel values of the point P in the left fisheye image are retained. It is necessary to simply rewrite the fusion formula and assign the weight k to the pixel value of the left fisheye image.
[0129] For any point P (x,y) The pixel value in the final 360-degree bird’s-eye view is
[0130] in, is the pixel value of pixel point P in the left fisheye image, is the pixel value of pixel point P in the forward fisheye image.
[0131] The weight k is greater than 0.75 and less than 0.9. Selecting a weight within this range can blur the fused image to a certain extent, reduce the abnormal transition of three-dimensional obstacles at critical positions, and reduce the reflection effect of ground lights, which can effectively improve the image fusion effect.
[0132] It should be noted that this embodiment only focuses on the fusion of the forward fisheye image and the left fisheye image. The fusion method for the other side fisheye images is the same as the basic method of the embodiment of the present invention, and only simple adjustments are required, which will not be elaborated in this embodiment.
[0133] In addition, in the embodiment of the present invention, the distance weighted fusion method and the fixed value weighted fusion method are both implemented for pixel points in the public visible area. For pixel points that are not in the public visible area, it is only necessary to convert the pixel values in the corresponding fisheye image into the corresponding pixel values in the bird's-eye view. The specific conversion method and the corresponding formula are not limited in this embodiment.
[0134] Fig.10 A schematic diagram of the panoramic image stitching system provided by an embodiment of the present invention is shown in FIG. Fig.10 As shown, the panoramic image stitching system includes:
[0135] An acquisition module 101 is used to acquire a fisheye image;
[0136] A determination module 102, configured to determine a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image;
[0137] The conversion module 103 is used to convert the fisheye image into a bird's-eye view according to the mapping relationship, and obtain a panoramic image according to the bird's-eye view.
[0138] Specifically, in order to facilitate the explanation of the embodiments of the present invention, the stitching and fusion of the fisheye image obtained by the left camera and the front camera is taken as an example for explanation. The principle of the stitching and fusion method of the fisheye images obtained by other adjacent cameras is the same as that of the embodiments of the present invention, and no further explanation is given here.
[0139] The acquisition module 101 is used to acquire fisheye images through a fisheye camera. Taking the front and left sides of the four fisheye images as an example, the acquired fisheye images are as follows: Figure 2 and Figure 3 shown.
[0140] The determination module 102 is used to determine the mapping relationship between any pixel point in the fisheye image and an adjacent fisheye image.
[0141] For each pixel point, if it is not within the overlapping area of the bird's-eye view, then it only corresponds to the mapping relationship of one fisheye image; if the pixel point is within the overlapping area of the bird's-eye view, then it corresponds to the mapping relationship of two adjacent fisheye images.
[0142] It should be noted that the size of the overlapping area is related to the size of the common visible area of two adjacent fisheye cameras that is set in advance, and can be adjusted according to actual conditions, and the embodiment of the present invention does not limit this.
[0143] The conversion module 103 is used to convert the fisheye image into a bird's-eye view according to the mapping relationship of all pixels in the fisheye image obtained by the determination module 102.
[0144] It should be noted that, in the embodiments of the present invention, Figure 4-6 As an auxiliary explanation of this solution, in fact, this solution does not need to convert the four fisheye images into the corresponding four bird's-eye views respectively and then perform image stitching. The four fisheye images can be directly converted into a complete bird's-eye view based on the four fisheye images and the mapping relationship corresponding to each pixel point, such as Figure 7 As shown, Figure 7 A complete bird's-eye view of an embodiment of the present invention is provided.
[0145] The panoramic image stitching system provided by the embodiment of the present invention can focus the algorithm processing on the common visible area of the four fisheye cameras by determining the mapping relationship of each pixel in the image. It only needs to traverse all the pixels once to obtain the complete fused bird's-eye view. The method has low computing cost. Except for the preparatory calculation required for initialization, the time complexity of the main loop calculation is controlled at ~o(ab), where a and b are the length and width of the pixels of the bird's-eye view, respectively. Under the condition of ensuring the frame rate of video presentation, the problem of overly obvious stitching seams is further eliminated, and the exposure difference and perspective difference of adjacent cameras are buffered in the common area of the image, so that the objects on the ground have a gradual transition trend at the stitching point, and a gradual transition at the image stitching point is achieved.
[0146] It should be noted that the panoramic view image stitching system provided in the embodiment of the present invention is used to execute the above-mentioned panoramic view image stitching method, and its specific implementation method is consistent with the method implementation method, which will not be repeated here.
[0147] Fig.11 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.11As shown, the electronic device may include: a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 communicate with each other through the communication bus 114. The processor 111 may call the logic instructions in the memory 113 to execute the above-mentioned power demand load forecasting method, including: obtaining a fisheye image; determining the mapping relationship between the pixel points in the fisheye image and the image having a common visible area with the fisheye image; according to the mapping relationship, converting the fisheye image into a bird's-eye view, and obtaining a panoramic view image according to the bird's-eye view.
[0148] In addition, the logic instructions in the above-mentioned memory 113 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0149] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the panoramic image stitching method provided by the above-mentioned method embodiments, including: obtaining a fisheye image; determining a mapping relationship between pixel points in the fisheye image on an image that has a common visible area with the fisheye image; according to the mapping relationship, converting the fisheye image into a bird's-eye view, and obtaining a panoramic image based on the bird's-eye view.
[0150] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the panoramic image stitching method provided in the above embodiments, including: obtaining a fisheye image; determining a mapping relationship between pixel points in the fisheye image on an image that has a common visible area with the fisheye image; according to the mapping relationship, converting the fisheye image into a bird's-eye view, and obtaining a panoramic image based on the bird's-eye view.
[0151] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A panoramic image stitching method, It is characterized in that include: Get fisheye image; Determine a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image; According to the mapping relationship, the fisheye image is converted into a bird's-eye view, and a panoramic image is obtained according to the bird's-eye view; The determining of a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image includes: For any pixel point in the fisheye image, determining whether its image coordinates are within the public visible area; If it is determined that the pixel point belongs to the common visible area, the image coordinates of the pixel point are obtained, and the image coordinates of the pixel point corresponding to the pixel point in the image in the common visible area with the fisheye image are formed into a mapping relationship; If it is determined that the pixel does not belong to the public visible area, it is determined that the pixel point has nothing to do with the image in the public visible area of the fisheye image; The converting the fisheye image into a bird's-eye view according to the mapping relationship comprises: For a pixel point within the common visible area, determine a pixel value corresponding to the pixel point in the bird's-eye view according to a pixel value corresponding to the image coordinate of the pixel point in the fisheye image, a pixel value corresponding to the image coordinate of the pixel point in the image having a common visible area with the fisheye image, and the mapping relationship; For a pixel point that does not belong to the public visible area, the pixel value corresponding to the pixel point in the bird's-eye view is determined according to the pixel value of the fisheye image corresponding to the image coordinates of the pixel point.
2. The method according to claim 1, It is characterized in that Before the step of determining the pixel value corresponding to the pixel point in the bird's-eye view according to the pixel value corresponding to the image coordinate of the pixel point in the fisheye image, the pixel value corresponding to the image coordinate of the pixel point in the image having the common visible area with the fisheye image, and the mapping relationship for the pixel point in the common visible area, the step further includes: Determine a first pixel value and a second pixel value of a first pixel; Comparing the difference between the first pixel value and the second pixel value in the color space; Determine whether the pixel value difference is greater than a preset threshold; Among them, the first pixel point is located in the common visible area; the first pixel value is the pixel value corresponding to the first pixel point in the fisheye image; the second pixel value is the pixel value corresponding to the first pixel point in the image that has a common visible area with the fisheye image.
3. The method according to claim 2, It is characterized in that The determining, for the pixel points in the common visible area, according to the pixel values corresponding to the image coordinates of the pixel points in the fisheye image, the pixel values corresponding to the image coordinates of the pixel points in the image having the common visible area with the fisheye image, and the mapping relationship, the pixel values corresponding to the pixel points in the bird's-eye view specifically include: If it is determined that the pixel value difference is not greater than a preset threshold, a distance weighted fusion method is used to determine a pixel value corresponding to the first pixel point in the bird's-eye view; If it is determined that the pixel value difference is greater than a preset threshold, a fixed-value weighted fusion method is used to determine the pixel value corresponding to the first pixel point in the bird's-eye view.
4. The method according to claim 3, It is characterized in that If it is determined that the pixel value difference is not greater than a preset threshold, using a distance weighted fusion method to determine a pixel value corresponding to the first pixel point in the bird's-eye view image includes: Obtaining the distance between the image coordinates of the first pixel and the projection boundaries of the public visible area; The distance between the boundaries on both sides of the projection of the public visible area is used as a weight value to achieve weighted fusion of the pixel values of the first pixel points, and determine the pixel value corresponding to the first pixel point in the bird's-eye view.
5. The method according to claim 3, It is characterized in that If it is determined that the pixel value difference is greater than a preset threshold, using a fixed value weighted fusion method to determine a pixel value corresponding to the first pixel point in the bird's-eye view image includes: According to the preset weight value, weighted fusion of the pixel values of the first pixel point is achieved to determine the pixel value corresponding to the pixel point in the bird's-eye view.
6. A panoramic image stitching system, It is characterized in that include: An acquisition module, used for acquiring fisheye images; A determination module, used to determine a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image; A conversion module, used for converting the fisheye image into a bird's-eye view according to the mapping relationship, and obtaining a panoramic image according to the bird's-eye view; The determining of a mapping relationship between pixels in the fisheye image and an image having a common visible area with the fisheye image includes: For any pixel point in the fisheye image, determining whether its image coordinates are within the public visible area; If it is determined that the pixel point belongs to the common visible area, the image coordinates of the pixel point are obtained, and the image coordinates of the pixel point corresponding to the pixel point in the image in the common visible area with the fisheye image are formed into a mapping relationship; If it is determined that the pixel does not belong to the public visible area, it is determined that the pixel point has nothing to do with the image in the public visible area of the fisheye image; The converting the fisheye image into a bird's-eye view according to the mapping relationship comprises: For a pixel point within the common visible area, determine a pixel value corresponding to the pixel point in the bird's-eye view according to a pixel value corresponding to the image coordinate of the pixel point in the fisheye image, a pixel value corresponding to the image coordinate of the pixel point in the image having a common visible area with the fisheye image, and the mapping relationship; For a pixel point that does not belong to the public visible area, the pixel value corresponding to the pixel point in the bird's-eye view is determined according to the pixel value of the fisheye image corresponding to the image coordinates of the pixel point.
7. An electronic device, It is characterized in that It includes a memory and a processor, and the processor and the memory communicate with each other through a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the panoramic image stitching method as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the panoramic image stitching method as described in any one of claims 1 to 5 is implemented.
