A panoramic image generation method and device, electronic equipment and storage medium

Abstract

The application provides a panoramic image generation method and device, electronic equipment and storage medium, and relates to the technical field of visual images.The panoramic image generation method comprises: processing real-time images collected by a camera according to a preset lookup table to generate projection images corresponding to viewing angles, wherein the preset lookup table is used to represent the mapping relationship between the images collected by the camera and the projection images; extracting a feature point pair from the overlapping area of the projection images of adjacent viewing angles, and adjusting the projection images of adjacent viewing angles according to the feature point pair when the feature point pair meets a preset correction condition; and generating a panoramic image according to the adjusted projection images of adjacent viewing angles.The application adjusts the projection images of adjacent viewing angles according to feature points, realizes secondary correction, and finally generates a panoramic image according to the adjusted projection images of adjacent viewing angles.Because the advantages and disadvantages of the lookup table method and the feature point matching algorithm are comprehensively considered, the balance between the fusion effect and the computing power consumption is realized.

Description

Technical Field

[0001] This invention relates to the field of visual image technology, and more specifically, to a panoramic image generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Panoramic image generation refers to the process of stitching together images or video frames captured by multiple cameras into a complete 360° or wide-angle image, which has wide applications in fields such as panoramic parking systems for automobiles. Currently, methods such as calibration lookup table method or feature point matching algorithm are commonly used to generate panoramic images. For the calibration lookup table method, camera calibration is required in the offline stage. A lookup table (LUT) is generated based on the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The lookup table describes the mapping relationship between each pixel in the panoramic image and the pixels in the original fisheye image. Then, in the online stage, the lookup table can be used to quickly convert the real-time acquired fisheye image into a panoramic image. For the feature point matching algorithm, the fisheye image first needs to be preprocessed with distortion correction, grayscale conversion, and noise reduction. After preprocessing, feature points need to be detected and matched between adjacent images. With reliable feature point matching pairs, the transformation relationship between the images can be calculated based on these points. The calculated transformation matrix is ​​used to transform one image into the coordinate system of another image, and then the images are stitched and fused to generate a complete panoramic image.

[0003] However, while the calibration lookup table method is fast and robust, it only works on flat surfaces because the camera extrinsic parameters are obtained using a checkerboard calibration algorithm. In scenes with objects of height, the height information is often difficult to obtain directly from the 2D image, making it impossible to accurately describe the position and orientation of the objects in 3D space, thus leading to inherent stitching misalignments. While the feature point matching algorithm does not have inherent stitching misalignments compared to the calibration lookup table method, it is not robust enough. In scenes with few detectable feature points, the stitching effect will be very poor. Furthermore, since fisheye images usually have high resolution and most feature point matching algorithms are time-consuming, the overall algorithm speed is not fast enough. Summary of the Invention

[0004] The problem addressed by this invention is how to achieve a balance between fusion effect and computational power consumption when generating panoramic images.

[0005] To address the aforementioned problems, this invention provides a panoramic image generation method, apparatus, electronic device, and storage medium.

[0006] In a first aspect, the present invention provides a panoramic image generation method, comprising:

[0007] The real-time images captured by the camera are processed according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map.

[0008] Extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints; when the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0009] A panoramic image is generated based on the projection map of the adjusted adjacent viewpoints.

[0010] Optionally, the step of processing the real-time images captured by the camera according to a preset lookup table to generate a projection image of the corresponding viewpoint includes:

[0011] The real-time image is preprocessed to obtain a preprocessed image;

[0012] The preprocessed image is converted into a planar projection according to the preset lookup table to generate the projection image from the corresponding viewpoint.

[0013] Optionally, the preprocessing of the real-time image to obtain a preprocessed image includes:

[0014] The real-time image is corrected based on the distortion coefficient of the camera;

[0015] The corrected real-time image is then processed to convert it into a grayscale image.

[0016] The grayscale image is denoised to obtain the preprocessed image.

[0017] Optionally, the preset lookup table is determined in the following manner:

[0018] The camera is calibrated to determine its intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients.

[0019] The preset lookup table is generated based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients.

[0020] Optionally, adjusting the projection map of adjacent viewpoints based on the feature point pair when the feature point pair meets the preset correction conditions includes:

[0021] When the number of feature point pairs is greater than or equal to a preset number threshold, and the average distance between corresponding feature point pairs in the projection map of adjacent viewpoints is greater than or equal to a preset distance threshold, the feature point pair is determined to meet the preset correction condition.

[0022] The pixels in one of the two projection maps of adjacent viewpoints are mapped to the other adjacent projection map using the transformation matrix of the two projection maps of adjacent viewpoints, so as to adjust the projection maps of adjacent viewpoints.

[0023] Optionally, the panoramic image generation method further includes:

[0024] When the number of feature point pairs is less than the preset number threshold, and / or the average distance between corresponding feature point pairs in the projection maps of adjacent viewpoints is less than the preset distance threshold, it is determined that the feature point pairs do not meet the preset correction conditions, and the projection maps of adjacent viewpoints are directly fused to generate the panoramic image.

[0025] Optionally, generating the panoramic image based on the adjusted projection map of adjacent viewpoints includes:

[0026] The pixels in the overlapping areas of the two projection images from adjacent viewpoints are fused according to preset weights to generate the panoramic image.

[0027] In a second aspect, the present invention provides a panoramic image generation device, comprising:

[0028] The first module is used to process the real-time images captured by the camera according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map.

[0029] The second module is used to extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints, and when the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0030] The third module is used to generate panoramic images based on the adjusted projection maps of adjacent viewpoints.

[0031] Thirdly, the present invention provides an electronic device, including a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is configured to implement the panoramic image generation method as described in the first aspect when executing the computer program.

[0034] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panoramic image generation method as described in the first aspect.

[0035] The beneficial effects of the panoramic image generation method of the present invention are as follows: First, the real-time images captured by the camera are processed according to a preset lookup table to generate projection maps of the corresponding viewpoints. Then, feature point pairs are extracted from the overlapping areas of the projection maps of adjacent viewpoints. The number of feature point pairs can be used to determine whether the feature point pairs meet the preset correction conditions. When the feature point pairs meet the preset correction conditions, the projection maps of adjacent viewpoints are adjusted according to the feature point pairs to achieve secondary correction. Finally, a panoramic image is generated based on the adjusted projection maps of adjacent viewpoints. By comprehensively considering the advantages and disadvantages of the lookup table method and the feature point matching algorithm, the advantages of the lookup table method in terms of fast running speed and low computing power consumption are retained, and the advantages of the feature point matching algorithm in effectively identifying objects with height are also retained, thus achieving a balance between fusion effect and computing power consumption. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the panoramic image generation method according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram illustrating the process of generating a projection image from a corresponding viewpoint according to an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the process for preprocessing real-time images according to an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the camera calibration process according to an embodiment of the present invention;

[0040] Figure 5 This is a schematic flowchart illustrating the process of adjusting the projection diagram of adjacent viewpoints according to an embodiment of the present invention.

[0041] Figure 6 This is a system architecture diagram of the panoramic image generation device according to an embodiment of the present invention;

[0042] Figure 7 This is a system architecture diagram of an electronic device according to an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of the overlapping area in an embodiment of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0045] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0046] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0047] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0048] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0049] like Figure 1 As shown in the figure, an embodiment of the present invention provides a panoramic image generation method, including:

[0050] S100: Process the real-time image captured by the camera according to a preset lookup table to generate a projection map of the corresponding viewpoint, wherein the preset lookup table is used to represent the mapping relationship between the image captured by the camera and the projection map.

[0051] Specifically, vehicles are typically equipped with cameras to capture images of the area around the vehicle. Commonly used cameras include fisheye cameras, which are widely used because they can provide an ultra-wide-angle view and reduce the number of image stitchings. Fisheye cameras can acquire fisheye images in real time from the front, back, left, and right of the vehicle. Then, by combining a preset lookup table, the circular view of the fisheye image can be converted into a planar projection, thereby generating a projection map of the corresponding view. For example, the field of view of the fisheye camera (such as the left side of the left camera) can be converted into a projection map of the left bird's-eye view.

[0052] S200: Extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints. When the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0053] Specifically, taking the projection map of the left bird's-eye view and the projection map of the front bird's-eye view as examples, the front bird's-eye view is the adjacent view of the left bird's-eye view. The two projection maps have a partially overlapping area (the area to be fused). Since the overlapping area of ​​the two projection maps is similar in image content and has the same resolution, feature point detection can be performed from the overlapping area of ​​either projection map to determine the feature point pair of the two projection maps in the overlapping area. When the feature point pair meets the preset correction conditions, it means that the two projection maps obtained by using only the lookup table method have a certain difference. At this time, the projection maps of the adjacent view are adjusted according to the feature point pair.

[0054] In the process of feature point extraction and feature point matching, the two regions can be divided into grids, and feature points in the two regions can be detected separately. Then, each feature matching region is scored. For a true match, the surrounding small regions are more similar and have a high score, thus filtering out false matches with scores below the threshold.

[0055] Feature point pairs refer to corresponding points in two images that can represent the same physical location. They are usually regions that are easy to detect and identify, such as corners (like the corners of buildings), regions with abrupt changes in texture (like the intersections of the edges of objects), or points with significant changes in color or brightness.

[0056] S300: Generate a panoramic image based on the projection map of the adjusted adjacent viewpoints.

[0057] Specifically, a panoramic image is generated based on the adjusted projection maps of adjacent viewpoints. For scenes where there are many objects in the overlapping area of ​​the projection maps of adjacent viewpoints, this embodiment has a better stitching effect than the lookup table method, and consumes less computing resources. In addition, for scenes where there are few objects in the overlapping area of ​​the projection maps of adjacent viewpoints, this embodiment has a better stitching effect than the feature point matching algorithm. Since it only adjusts the overlapping area, the resolution of the algorithm's input image is reduced, and it consumes less computing resources than the feature point matching algorithm. In other words, this embodiment comprehensively considers the advantages and disadvantages of the lookup table method and the feature point matching algorithm, and achieves a balance between fusion effect and computing power consumption.

[0058] In this embodiment, the real-time images captured by the camera are first processed according to a preset lookup table to generate a projection map of the corresponding viewpoint. Then, feature point pairs are extracted from the overlapping area of ​​the projection maps of adjacent viewpoints. The number of feature point pairs can be used to determine whether the feature point pairs meet the preset correction conditions. When the feature point pairs meet the preset correction conditions, the projection maps of adjacent viewpoints are adjusted according to the feature point pairs to achieve secondary correction (that is, based on the calibration lookup table method, if the preset correction conditions are met, feature point matching correction is triggered to improve the effect of the calibration lookup table method). Finally, a panoramic image is generated based on the adjusted projection maps of adjacent viewpoints. By comprehensively considering the advantages and disadvantages of the lookup table method and the feature point matching algorithm, a balance between fusion effect and computing power consumption is achieved.

[0059] Optionally, the step of processing the real-time images captured by the camera according to a preset lookup table to generate a projection image of the corresponding viewpoint includes:

[0060] S110: Preprocess the real-time image to obtain a preprocessed image.

[0061] Specifically, in combination Figure 2 As shown, since fisheye images usually have obvious distortions and are affected by noise and color information, preprocessing is required before detecting feature points to ensure the stability and accuracy of subsequent algorithms.

[0062] S120: Convert the preprocessed image into a planar projection according to the preset lookup table to generate the projection image from the corresponding viewpoint.

[0063] Specifically, in combination Figure 2 As shown, in order to generate a bird's-eye view, the circular perspective of the fisheye image needs to be converted into a planar projection. A pre-defined lookup table can be used to convert the preprocessed image into a planar projection (e.g., using cylindrical projection or equidistant projection). The four images in the front, back, left, and right will generate four bird's-eye view projections respectively.

[0064] In this optional embodiment, the preprocessed image is converted into a planar projection by using a preset lookup table, which can project feature points between different images onto the same geometric plane, facilitating feature point matching and fusion.

[0065] Optionally, the preprocessing of the real-time image to obtain a preprocessed image includes:

[0066] S111: Correct the real-time image according to the distortion coefficient of the camera.

[0067] Specifically, in combination Figure 3 As shown, the distortion coefficients of a fisheye camera are used to perform distortion correction on the original image, eliminating radial and tangential distortion of the lens, thereby achieving real-time image correction.

[0068] S112: Perform grayscale processing on the corrected real-time image to convert the real-time image into a grayscale image.

[0069] Specifically, in combination Figure 3 As shown, color information is usually irrelevant in the process of feature point pair detection. Converting real-time images to grayscale images can reduce the amount of computation and enhance robustness to changes in illumination.

[0070] S113: Denoise the grayscale image to obtain the preprocessed image.

[0071] Specifically, in combination Figure 3 As shown, since noise can interfere with feature point pair detection, algorithms such as Gaussian filtering can be used to smooth the image and remove minor noise.

[0072] In this optional embodiment, by preprocessing the real-time image, the image can be used more accurately and efficiently for subsequent tasks such as feature point pair detection, stitching, and fusion.

[0073] Optionally, the preset lookup table is determined in the following manner:

[0074] S101: The camera is calibrated to determine the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the camera.

[0075] Specifically, in combination Figure 4 As shown, in order to convert a fisheye image into a normal panoramic image, it is necessary to first calibrate the fisheye camera and obtain parameters related to optical distortion, such as intrinsic matrix, extrinsic matrix, and distortion coefficients. The intrinsic matrix is ​​used to describe the camera's focal length, principal point (optical center), etc., while the extrinsic matrix is ​​used to describe the camera's position and orientation (rotation matrix and translation vector) relative to a calibration board such as a checkerboard. The distortion coefficients are used to describe the nonlinear distortion of the fisheye lens, which typically includes radial distortion (controlling barrel or pincushion distortion of the image) and tangential distortion (caused by the tilt of the lens and sensor).

[0076] S102: Generate the preset lookup table based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients.

[0077] Specifically, in combination Figure 4 As shown, after calibration, a LUT table (two-dimensional array) is generated based on the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The LUT table is used to describe the mapping relationship between each pixel in the panoramic image and the pixels in the original fisheye image.

[0078] In this optional embodiment, the processing speed during panoramic image generation is improved by calibrating the camera to determine a preset lookup table.

[0079] Optionally, adjusting the projection map of adjacent viewpoints based on the feature point pair when the feature point pair meets the preset correction conditions includes:

[0080] S210: When the number of feature point pairs is greater than or equal to a preset number threshold, and the average distance between corresponding feature point pairs in the projection map of adjacent viewpoints is greater than or equal to a preset distance threshold, it is determined that the feature point pair satisfies the preset correction condition.

[0081] Specifically, in combination Figure 5 As shown, when the number of feature point pairs is greater than or equal to the preset number threshold (an empirical value that can be determined by experiment), it indicates that there are a large number of feature point pairs, and there may be objects or ground markings in the overlapping area. Therefore, it is necessary to further determine the relationship between the average distance between the corresponding feature point pairs and the preset distance threshold. When the average distance between the corresponding feature point pairs in the projection images of adjacent viewpoints is greater than or equal to the preset distance threshold, it indicates that there may be objects with height in the overlapping area, causing a certain difference between the two images obtained by the lookup table method. At this time, the feature point pairs meet the preset correction conditions.

[0082] S220: The pixels in one of the two projection maps of adjacent viewpoints are mapped to the other adjacent projection map by the transformation matrix of the two projection maps of adjacent viewpoints, so as to adjust the projection maps of adjacent viewpoints.

[0083] Specifically, in combination Figure 5 and Figure 8 As shown, suppose there are two adjacent image regions: region A and region B, with an overlapping region C between them. First, the feature point pair between region A and region B is found through the feature point matching algorithm, and the transformation matrix T is calculated to ensure that the coordinate system of region B is aligned with that of region A. For the part that belongs to the overlapping region C, the pixels in region A and region B are merged. For the part that belongs to region B but is not in the overlapping region C, the pixel values ​​in region B are directly retained to avoid information loss.

[0084] The transformation matrix T is used to describe the geometric transformation relationship from one image viewpoint to another, such as an affine transformation matrix or a perspective transformation matrix.

[0085] In this optional embodiment, the continuity and integrity of the stitched image are ensured by adjusting the projection map of adjacent viewpoints according to the feature point pair when the feature point pair meets the preset correction conditions.

[0086] Optionally, the panoramic image generation method further includes:

[0087] When the number of feature point pairs is less than the preset number threshold, and / or the average distance between corresponding feature point pairs in the projection maps of adjacent viewpoints is less than the preset distance threshold, it is determined that the feature point pairs do not meet the preset correction conditions, and the projection maps of adjacent viewpoints are directly fused to generate the panoramic image.

[0088] Specifically, when the number of feature point pairs is less than a preset threshold, it indicates that the number of feature point pairs is low. The main reason is likely that the complexity of the image content is low, such as few objects or ground markings. In this case, no correction needs to be triggered (the feature point matching algorithm will actually perform poorly when the number of feature point pairs is low). For feature points (a1, b1, c1, ...) in region A and feature points (a2, b2, c2, ...) in region B, calculate the average of the differences (δ1, δ2, δ3, ...) of the Euclidean distances between the coordinates of the two feature point pairs:

[0089]

[0090] If the average value (avg) is less than the preset distance threshold, it means that the feature points do not come from objects with height, and no correction needs to be triggered (even if the feature point matching algorithm is used, the effect is similar to that of the lookup table method). That is, the feature point pair does not meet the preset correction conditions, and the projection images of adjacent viewpoints are directly fused to generate a panoramic image.

[0091] Since the checkerboard calibration algorithm also has certain errors, and in reality, the road surface is unlikely to be as flat as the calibration site, an error of up to 30 millimeters is allowed within two meters of the overlapping area. The preset distance threshold can be adjusted based on this error requirement.

[0092] In this optional embodiment, when the feature point pair does not meet the preset correction conditions, the projection images of adjacent viewpoints are directly fused to generate a panoramic image, which reduces the computational power consumption while satisfying the fusion effect.

[0093] Optionally, generating the panoramic image based on the adjusted projection map of adjacent viewpoints includes:

[0094] The pixels in the overlapping areas of the two projection images from adjacent viewpoints are fused according to preset weights to obtain the panoramic image.

[0095] Specifically, taking regions A and B as examples, in the process of image stitching or image fusion, especially in the fusion of overlapping regions C, in order to avoid obvious gaps or abrupt visual transitions at the image edges, it is necessary to perform weighted fusion of the pixels of the two images so that the stitched image looks natural and coherent. For example, the RGB value of a1 in image A is rgb1, and the RGB value of a2 in image B is rgb2. The final result rgb = weight * rgb1 + (1 - weight) * rgb2. Since the sum of the coefficients before rgb1 and rgb2 is equal to 1, the brightness of the fused and non-fused regions can be consistent, and the transition is natural. Here, a1 and a2 correspond to the same point on the fused image, and weight is the corresponding weight coefficient. The larger the angle between the line connecting a1 and the lower left corner of the fused region and the bottom edge of the fused region, the larger the weight coefficient, and vice versa.

[0096] Alternatively, the Laplacian pyramid fusion method can be used to decompose the image into different frequency bands and fuse them in each band, which can preserve details and reduce seams.

[0097] In this optional embodiment, the pixels in the overlapping areas of the projection map are fused by weighting, which not only improves the quality of image stitching, but also reduces visual discomfort caused by color difference and lighting difference, making the stitched image more natural.

[0098] like Figure 6 As shown, an embodiment of the present invention provides a panoramic image generation device 600, comprising:

[0099] The first module 610 is used to process the real-time images captured by the camera according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map.

[0100] The second module 620 is used to extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints, and when the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0101] The third module 630 is used to generate a panoramic image based on the projection map of the adjusted adjacent viewpoints.

[0102] like Figure 7 As shown, an electronic device 700 provided in this embodiment of the invention includes a memory 720 and a processor 710; the memory 720 is used to store a computer program; the processor 710 is used to implement the panoramic image generation method as described above when the computer program is executed.

[0103] Alternatively, an electronic device 700 includes a memory 720 and a processor 710 coupled to the memory 720; the memory 720 is configured to store a computer program; and the processor 710 is configured to perform the following operations when the computer program is executed:

[0104] The real-time images captured by the camera are processed according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map.

[0105] Extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints; when the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0106] A panoramic image is generated based on the projection map of the adjusted adjacent viewpoints.

[0107] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the panoramic image generation method described above.

[0108] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations:

[0109] The real-time images captured by the camera are processed according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map.

[0110] Extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints; when the feature point pairs meet the preset correction conditions, adjust the projection maps of adjacent viewpoints according to the feature point pairs.

[0111] A panoramic image is generated based on the projection map of the adjusted adjacent viewpoints.

[0112] The present invention will now be described an electronic device 700 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. Electronic device 700 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 700 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0113] Electronic device 700 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0114] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0115] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for generating panoramic images, characterized in that, include: The real-time images captured by the camera are processed according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map. Feature point pairs are extracted from the overlapping area of ​​the projection maps of adjacent viewpoints. When the feature point pairs meet the preset correction conditions, the projection maps of adjacent viewpoints are adjusted according to the feature point pairs. The preset correction conditions include that the number of feature point pairs is greater than or equal to a preset number threshold, and the average distance between corresponding feature point pairs in the projection maps of adjacent viewpoints is greater than or equal to a preset distance threshold. A panoramic image is generated based on the projection map of the adjusted adjacent viewpoints.

2. The panoramic image generation method according to claim 1, characterized in that, When the feature point pair meets the preset correction conditions, adjusting the projection map of adjacent viewpoints according to the feature point pair includes: When the number of feature point pairs is greater than or equal to a preset number threshold, and the average distance between corresponding feature point pairs in the projection map of adjacent viewpoints is greater than or equal to a preset distance threshold, the feature point pair is determined to meet the preset correction condition. The pixels in one of the two projection maps of adjacent viewpoints are mapped to the other adjacent projection map using the transformation matrix of the two projection maps of adjacent viewpoints, so as to adjust the projection maps of adjacent viewpoints.

3. The panoramic image generation method according to claim 2, characterized in that, Also includes: When the number of feature point pairs is less than the preset number threshold, and / or the average distance between corresponding feature point pairs in the projection maps of adjacent viewpoints is less than the preset distance threshold, it is determined that the feature point pairs do not meet the preset correction conditions, and the projection maps of adjacent viewpoints are directly fused to generate the panoramic image.

4. The panoramic image generation method according to claim 1, characterized in that, The step of processing the real-time images captured by the camera according to a preset lookup table to generate a projection image of the corresponding viewpoint includes: The real-time image is preprocessed to obtain a preprocessed image; The preprocessed image is converted into a planar projection according to the preset lookup table to generate the projection image from the corresponding viewpoint.

5. The panoramic image generation method according to claim 4, characterized in that, The preprocessing of the real-time image to obtain a preprocessed image includes: The real-time image is corrected based on the distortion coefficient of the camera; The corrected real-time image is then processed to convert it into a grayscale image. The grayscale image is denoised to obtain the preprocessed image.

6. The panoramic image generation method according to claim 4, characterized in that, The preset lookup table is determined in the following way: The camera is calibrated to determine its intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients. The preset lookup table is generated based on the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients.

7. The panoramic image generation method according to any one of claims 1 to 6, characterized in that, The process of generating a panoramic image based on the adjusted projection map of adjacent viewpoints includes: The pixels in the overlapping areas of the two projection images from adjacent viewpoints are fused according to preset weights to generate the panoramic image.

8. A panoramic image generation device, characterized in that, include: The first module is used to process the real-time images captured by the camera according to a preset lookup table to generate a projection map of the corresponding viewpoint. The preset lookup table is used to represent the mapping relationship between the images captured by the camera and the projection map. The second module is used to extract feature point pairs from the overlapping area of ​​the projection maps of adjacent viewpoints. When the feature point pairs meet the preset correction conditions, the projection maps of adjacent viewpoints are adjusted according to the feature point pairs. The preset correction conditions include that the number of feature point pairs is greater than or equal to a preset number threshold, and the average distance between corresponding feature point pairs in the projection maps of adjacent viewpoints is greater than or equal to a preset distance threshold. The third module is used to generate panoramic images based on the adjusted projection maps of adjacent viewpoints.

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the panoramic image generation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the panoramic image generation method as described in any one of claims 1 to 7.

Images