Panoramic image generation method and device, electronic equipment and storage medium

Abstract

The invention provides a panoramic image generation method and device, electronic equipment and a storage medium, and relates to the technical field of visual images. The panoramic image generation method comprises the steps that a real-time image collected by a camera is processed according to a preset lookup table to generate a projection drawing of a corresponding view angle, and the preset lookup table is used for representing the mapping relation between the image collected by the camera and the projection drawing; extracting a feature point pair from an overlapping region of the projection drawings of the adjacent visual angles, and when the feature point pair meets a preset correction condition, adjusting the projection drawings of the adjacent visual angles according to the feature point pair; and generating a panoramic image according to the adjusted projection drawings of the adjacent viewing angles. According to the method, the projection drawings of the adjacent visual angles are adjusted according to the feature points, secondary correction is achieved, finally, the panoramic image is generated according to the adjusted projection drawings of the adjacent visual angles, and due to the fact that the advantages and disadvantages of a table look-up method and a feature point matching algorithm are comprehensively considered, the fusion effect and calculation power consumption are balanced.

Description

Technical Field

[0001] The present invention relates to the field of visual image technology, and in particular to a panoramic image generation method, device, electronic device and storage medium. Background Art

[0002] The generation of panoramic images refers to the process of stitching images or video frames taken by multiple cameras into a complete 360° or wide-angle image, which is widely used in fields such as automobile panoramic parking systems. At present, panoramic images are usually generated by calibration lookup table method or feature point matching algorithm. For calibration lookup table method, camera calibration is required in the offline stage, and a lookup table (LUT) is generated according to the camera internal parameters, external parameters and distortion parameters. The mapping relationship between each pixel point of the panoramic image and the pixel point on the original fisheye image is described by the lookup table. Then, the fisheye image collected in real time can be quickly converted into a panoramic image using the lookup table in the online stage; for feature point matching algorithm, the fisheye image needs to be preprocessed by dedistortion, grayscale, noise reduction, etc. After preprocessing, feature points need to be detected and matched between adjacent images. After having reliable feature point matching pairs, the transformation relationship between images can be calculated based on these points, and one image is transformed into the coordinate system of another image according to the calculated transformation matrix, and stitching and fusion are performed to generate a complete panoramic image.

[0003] However, although the calibration lookup table method runs very fast and has high algorithm robustness, the camera extrinsic parameters are obtained based on the checkerboard calibration algorithm and can only be applied to flat ground. For scenes with objects at height, the height information is often difficult to obtain directly in the two-dimensional image, and it is impossible to accurately describe the position and posture of the object in the three-dimensional space, which leads to inherent stitching misalignment. Although the feature point matching algorithm does not have inherent stitching misalignment compared with the calibration lookup table method, the algorithm is not robust enough. For scenes with fewer feature points that can be detected, the stitching effect will be very poor. In addition, since the resolution of the fisheye image is usually relatively high, most feature point matching algorithms are relatively time-consuming, resulting in the algorithm running slowly. Summary of the invention

[0004] The problem solved by the present invention is how to achieve a balance between fusion effect and computing power consumption when generating panoramic images.

[0005] In order to solve the above problems, the present invention provides a panoramic image generation method, device, electronic device and storage medium.

[0006] In a first aspect, the present invention provides a method for generating a panoramic image, comprising:

[0007] Processing the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image captured by the camera and the projection image;

[0008] Extracting feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and adjusting the projection images of adjacent viewing angles according to the feature point pairs when the feature point pairs meet a preset correction condition;

[0009] A panoramic image is generated according to the adjusted projection images of adjacent viewing angles.

[0010] Optionally, the processing of the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle includes:

[0011] Preprocessing the real-time image to obtain a preprocessed image;

[0012] The preprocessed image is converted into a plane projection according to the preset lookup table to generate the projection image of the corresponding viewing angle.

[0013] Optionally, preprocessing the real-time image to obtain a preprocessed image includes:

[0014] Correcting the real-time image according to the distortion coefficient of the camera;

[0015] Performing grayscale processing on the corrected real-time image to convert the real-time image into a grayscale image;

[0016] De-noising the grayscale image to obtain the pre-processed image.

[0017] Optionally, the preset lookup table is determined in the following manner:

[0018] Calibrate the camera to determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient of the camera;

[0019] The preset lookup table is generated according to the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion coefficient.

[0020] Optionally, when the feature point pair satisfies a preset correction condition, adjusting the projection image of adjacent viewing angles according to the feature point pair includes:

[0021] When the number of the feature point pairs is greater than or equal to a preset number threshold, and the average of the distances between the corresponding feature point pairs in the projection images of adjacent viewing angles is greater than or equal to a preset distance threshold, it is determined that the feature point pairs meet the preset correction condition;

[0022] The pixels in one of the two projection images of adjacent viewing angles are mapped to another projection image adjacent thereto through the transformation matrix of the two projection images of adjacent viewing angles, so as to adjust the projection images of adjacent viewing angles.

[0023] Optionally, the panoramic image generation method further includes:

[0024] When the number of the feature point pairs is less than the preset number threshold, and / or the average of the distances between the corresponding feature point pairs in the projection images of adjacent perspectives is less than the preset distance threshold, it is determined that the feature point pairs do not satisfy the preset correction conditions, and the projection images of adjacent perspectives are directly fused to generate the panoramic image.

[0025] Optionally, generating a panoramic image according to the adjusted projection images of adjacent viewing angles includes:

[0026] The pixels of the overlapping area of ​​the two projection images of the adjusted adjacent viewing angles are fused according to preset weights to generate the panoramic image.

[0027] In a second aspect, the present invention provides a panoramic image generation device, comprising:

[0028] The first module is used to process the real-time image collected by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image collected by the camera and the projection image;

[0029] A second module is used to extract feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and when the feature point pairs meet a preset correction condition, adjust the projection images of adjacent viewing angles according to the feature point pairs;

[0030] The third module is used to generate a panoramic image according to the adjusted projection map of adjacent viewing angles.

[0031] In a third aspect, the present invention provides an electronic device, including a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is used to implement the panoramic image generation method as described in the first aspect when executing the computer program.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the panoramic image generation method as described in the first aspect is implemented.

[0035] The beneficial effects of the panoramic image generation method of the present invention are: first, the real-time image captured by the camera is processed according to a preset lookup table to generate a projection image of the corresponding viewing angle, and then feature point pairs are extracted from the overlapping areas of the projection images of adjacent viewing angles. The number of feature point pairs can be judged to determine whether the feature point pairs meet the preset correction conditions, and when the feature point pairs meet the preset correction conditions, the projection images of adjacent viewing angles are adjusted according to the feature point pairs to achieve secondary correction, and finally a panoramic image is generated according to the adjusted projection images of adjacent viewing angles. Due to comprehensive consideration of the advantages and disadvantages of the table lookup method and the feature point matching algorithm, the advantages of the table lookup method with fast running speed and low computing power consumption are retained, and the advantage of the feature point matching algorithm in effectively identifying objects with height is retained, thereby achieving a balance between fusion effect and computing power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of a flow chart of a panoramic image generation method according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of a process for generating a projection image corresponding to a viewing angle according to an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a process for preprocessing a real-time image according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a camera calibration process according to an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a process for adjusting projection images of adjacent viewing angles according to an embodiment of the present invention;

[0041] Figure 6 is a system architecture diagram of a panoramic image generating device according to an embodiment of the present invention;

[0042] Figure 7 A system architecture diagram of an electronic device according to an embodiment of the present invention;

[0043] Figure 8 Schematic diagram of the overlapping area of ​​an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0045] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0046] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for generating a panoramic image, comprising:

[0050] S100: Processing a real-time image captured by a camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image captured by the camera and the projection image.

[0051] Specifically, a camera is usually provided on a vehicle to collect images near the vehicle. Commonly used cameras include fisheye cameras. Fisheye cameras are widely used because they can provide an ultra-wide-angle viewing angle and reduce the number of image stitching. The fisheye camera can be used to obtain fisheye images of the front, rear, left and right of the vehicle in real time. Then, the circular viewing angle of the fisheye image can be converted into a plane projection in combination with a preset lookup table, thereby generating a projection image of the corresponding viewing angle, for example, converting the field of view of the fisheye camera (such as the left area of ​​the left camera) into a projection image of the left bird's-eye view.

[0052] S200: extracting feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and when the feature point pairs meet preset correction conditions, adjusting the projection images of adjacent viewing angles according to the feature point pairs.

[0053] Specifically, taking the projection image of the left bird's-eye view and the projection image of the front bird's-eye view as examples, the front bird's-eye view is an adjacent perspective of the left bird's-eye view, and there is a partial overlapping area (area to be fused) between the two projection images. Since the overlapping area of ​​the two projection images are similar in image content and have the same resolution, feature point detection can be performed from the overlapping area of ​​any projection image to determine the feature point pairs of the two projection images in the overlapping area; when the feature point pair meets the preset correction conditions, it means that there is a certain gap between the two projection images obtained only by using the table lookup method. At this time, the projection image of the adjacent perspective is adjusted according to the feature point pair.

[0054] Among them, when performing feature point pair extraction and feature point matching, the two areas can be gridded separately, the feature points of the two areas can be detected separately, and then each feature matching area can be scored. For a true match, multiple small areas around are more similar and have high scores, so that false matches with scores less than the threshold can be filtered out.

[0055] Among them, feature point pairs refer to corresponding points that can represent the same physical location in two images, usually some areas that are easy to detect and identify, such as corner points (such as the corners of buildings), texture mutation areas (such as the intersection of the edges of objects), or points with significant changes in color or brightness.

[0056] S300: Generate a panoramic image according to the adjusted projection images of adjacent viewing angles.

[0057] Specifically, a panoramic image is generated according to the adjusted projection images of adjacent perspectives. For scenes where there are more objects in the overlapping areas of the projection images of adjacent perspectives, the present embodiment has a better stitching effect than the table lookup method, and the increase in computing power resource consumption is less. In addition, for scenes where there are fewer objects in the overlapping areas of the projection images of adjacent perspectives, the present embodiment has a better stitching effect than the feature point matching algorithm, and since only the overlapping areas are adjusted, the resolution of the algorithm input image is reduced, and the computing power resource consumption is less than that of the feature point matching algorithm. That is, the present embodiment comprehensively considers the advantages and disadvantages of the table lookup method and the feature point matching algorithm, and achieves a balance between fusion effect and computing power consumption.

[0058] In this embodiment, the real-time image captured by the camera is first processed according to a preset lookup table to generate a projection image of the corresponding viewing angle, and then feature point pairs are extracted from the overlapping areas of the projection images of adjacent viewing angles. Whether the feature point pairs meet the preset correction conditions can be determined by judging the number of feature point pairs, and when the feature point pairs meet the preset correction conditions, the projection images of adjacent viewing angles are adjusted according to the feature point pairs to achieve secondary correction (i.e., on the basis of the calibration lookup method, if the preset correction conditions are met, the feature point matching correction is triggered to improve the effect of the calibration lookup method). Finally, a panoramic image is generated according to the adjusted projection images of adjacent viewing angles. By comprehensively considering the advantages and disadvantages of the lookup method and the feature point matching algorithm, a balance between fusion effect and computing power consumption is achieved.

[0059] Optionally, the processing of the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle includes:

[0060] S110: Preprocessing the real-time image to obtain a preprocessed image.

[0061] Specifically, combined Figure 2 As shown, since fisheye images usually have obvious distortion and are interfered by noise and color information, preprocessing is required before detecting feature point pairs to ensure the stability and accuracy of subsequent algorithms.

[0062] S120: Convert the preprocessed image into a plane projection according to the preset lookup table to generate the projection image of the corresponding viewing angle.

[0063] Specifically, combined Figure 2 As shown, in order to generate a bird's-eye view, the circular perspective of the fisheye image needs to be converted into a plane projection. The preprocessed image can be converted into a plane projection (for example, using cylindrical projection or equidistant projection) through a preset lookup table. The four images in front, back, left and right will generate four bird's-eye projection images respectively.

[0064] In this optional embodiment, the preprocessed image is converted into a plane projection through a preset lookup table, so that the feature points between different images can be projected onto the same geometric plane, which facilitates feature point matching and fusion.

[0065] Optionally, preprocessing the real-time image to obtain a preprocessed image includes:

[0066] S111: Correcting the real-time image according to the distortion coefficient of the camera.

[0067] Specifically, combined Figure 3 As shown, the distortion coefficient of the fisheye camera is used to dedistort the original image and eliminate the radial and tangential distortion of the lens, thereby realizing the correction of the real-time image.

[0068] S112: performing grayscale processing on the corrected real-time image to convert the real-time image into a grayscale image.

[0069] Specifically, combined Figure 3 As shown in the figure, in the process of feature point pair detection, color information is usually irrelevant. Converting the real-time image into a grayscale image can reduce the amount of calculation and enhance the robustness to lighting changes.

[0070] S113: De-noising the grayscale image to obtain the pre-processed image.

[0071] Specifically, combined Figure 3 As shown in the figure, since noise will interfere with the detection of feature point pairs, algorithms such as Gaussian filtering can be used to smooth the image and remove small noise.

[0072] In this optional embodiment, by preprocessing the real-time image, the image can be used more accurately and efficiently for subsequent tasks such as feature point pair detection, splicing and fusion.

[0073] Optionally, the preset lookup table is determined in the following manner:

[0074] S101: Calibrate the camera to determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient of the camera.

[0075] Specifically, combined Figure 4 As shown in the figure, in order to convert the fisheye image into a normal panoramic image, it is necessary to calibrate the fisheye camera first to obtain parameters related to optical distortion, such as the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion coefficient, where the intrinsic parameter matrix is ​​used to describe the focal length, the principal point (optical center) of the camera, etc., the extrinsic parameter matrix is ​​used to describe the position and direction (rotation matrix and translation vector) of the camera relative to the calibration plate such as the chessboard, and the distortion coefficient is used to describe the nonlinear distortion of the fisheye lens, which usually includes radial distortion (barrel or pincushion distortion of the control image) and tangential distortion (caused by the tilt of the lens and sensor).

[0076] S102: Generate the preset lookup table according to the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion coefficient.

[0077] Specifically, combined Figure 4 As shown, after the calibration is completed, a LUT table (two-dimensional array) is generated according to the camera's intrinsic parameters, extrinsic parameters and distortion parameters. The LUT table is used to describe the mapping relationship between each pixel point of the panoramic image and the pixel point on the original fisheye image.

[0078] In this optional embodiment, the camera is calibrated to determine a preset lookup table, thereby improving the processing speed when generating the panoramic image.

[0079] Optionally, when the feature point pair satisfies a preset correction condition, adjusting the projection image of adjacent viewing angles according to the feature point pair includes:

[0080] S210: When the number of the feature point pairs is greater than or equal to a preset number threshold, and the average of the distances between the corresponding feature point pairs in the projection images of adjacent viewing angles is greater than or equal to a preset distance threshold, it is determined that the feature point pairs satisfy the preset correction condition.

[0081] Specifically, combined Figure 5 As shown in the figure, when the number of feature point pairs is greater than or equal to the preset number threshold (empirical value, which can be determined based on experiments), it means that the number of feature point pairs is large, and there may be objects or ground signs in the overlapping area. Therefore, it is necessary to further judge the relationship between the mean value of the distance between the corresponding feature point pairs and the preset distance threshold. When the mean value of the distance between the corresponding feature point pairs in the projection image of adjacent perspectives is greater than or equal to the preset distance threshold, it means that there may be objects with height in the overlapping area, resulting in a certain gap between the two images obtained by the table lookup method. At this time, the feature point pairs meet the preset correction conditions.

[0082] S220: Mapping pixels in one of the two projection images of adjacent perspectives to another adjacent projection image through a transformation matrix of the two projection images of adjacent perspectives, so as to adjust the projection images of the adjacent perspectives.

[0083] Specifically, combined Figure 5 and Figure 8 As shown, suppose there are two adjacent image areas: area A and area B, and there is an overlapping area C between them. First, the feature point pairs between area A and area B are found through the feature point matching algorithm, and the transformation matrix T is calculated to ensure that the coordinate system of area B is aligned with area A. For the part belonging to the overlapping area C, the pixels in area A and area B are fused. For the part belonging to area B but not in the overlapping area C, the pixel values ​​in area B are directly retained to avoid information loss.

[0084] The transformation matrix T is used to describe the geometric transformation relationship from one image perspective to another image perspective, such as an affine transformation matrix or a perspective transformation matrix.

[0085] In this optional embodiment, when the feature point pair meets the preset correction condition, the projection image of the adjacent viewing angle is adjusted according to the feature point pair, thereby ensuring the continuity and integrity of the spliced ​​image.

[0086] Optionally, the panoramic image generation method further includes:

[0087] When the number of the feature point pairs is less than the preset number threshold, and / or the average of the distances between the corresponding feature point pairs in the projection images of adjacent perspectives is less than the preset distance threshold, it is determined that the feature point pairs do not satisfy the preset correction conditions, and the projection images of adjacent perspectives are directly fused to generate the panoramic image.

[0088] Specifically, when the number of feature point pairs is less than the preset number threshold, it means that the number of feature point pairs is small. The main possible reason is that the complexity of the image content is low, for example, there are few objects or ground signs. At this time, there is no need to trigger correction (for the case of a small number of feature point pairs, the effect of the feature point matching algorithm will be very poor); for the feature points (a1, b1, c1, ...) of area A and the feature points (a2, b2, c2, ...) of area B, calculate the average value of the difference (δ1, δ2, δ3, ...) of the Euclidean distance of the coordinates of the feature point pairs of the two regions:

[0089]

[0090] If the average value avg is less than the preset distance threshold, it means that the feature point does not come from an object with height, and there is no need to trigger correction (even if the feature point matching algorithm is used, the effect is close to the table lookup method). In other words, the feature point pair does not meet the preset correction conditions, and the projection images of adjacent perspectives are directly fused to generate a panoramic image.

[0091] Among them, since the checkerboard calibration algorithm will also have certain errors, and in actual situations, the road surface cannot be as flat as the calibration site, an error of no more than 30 millimeters is allowed within two meters in the overlapping area, and the preset distance threshold can be adjusted based on this error requirement.

[0092] In this optional embodiment, when the feature point pair does not meet the preset correction conditions, the projection images of adjacent perspectives are directly fused to generate a panoramic image, thereby reducing computing power consumption while satisfying the fusion effect.

[0093] Optionally, generating a panoramic image according to the adjusted projection images of adjacent viewing angles includes:

[0094] The pixels of the overlapping area of ​​the two projection images of the adjusted adjacent viewing angles are fused according to preset weights to obtain the panoramic image.

[0095] Specifically, taking area A and area B as examples, in the process of image stitching or image fusion, especially in the fusion of overlapping area C, in order to avoid obvious gaps at the edges of the images or abrupt visual transitions, it is necessary to fuse the pixels of the two images according to weights, so that the stitched image looks natural and coherent; for example, the RGB value of a1 on the image of area A is rgb1, and the RGB value of a2 on the image of area 2 is rgb2. The final result is rgb=weight*rgb1+(1-weight)*rgb2. Since the sum of the coefficients before rgb1 and rgb2 is equal to 1, the brightness of the fused area and the non-fused area can be made consistent, and the transition is natural; wherein a1 and a2 correspond to the same point on the fusion map, weight is the corresponding weight coefficient, and the larger the angle between a1 and the lower left corner of the fused area and the bottom edge of the fused area, the larger the weight coefficient, and vice versa.

[0096] Among them, the Laplacian pyramid fusion method can also be used for fusion, which decomposes the image into different frequency bands and fuses them in each frequency band, which can retain details and reduce seams.

[0097] In this optional embodiment, pixels in overlapping areas of the projection images are fused by weighting, which not only improves the quality of image stitching, but also reduces visual discomfort caused by color difference and lighting differences, making the stitched image more natural.

[0098] like Figure 6 As shown, a panoramic image generating device 600 provided by an embodiment of the present invention includes:

[0099] The first module 610 is used to process the real-time image collected by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image collected by the camera and the projection image;

[0100] The second module 620 is used to extract feature point pairs from the overlapping areas of the projection images of adjacent viewing angles, and when the feature point pairs meet a preset correction condition, adjust the projection images of adjacent viewing angles according to the feature point pairs;

[0101] The third module 630 is used to generate a panoramic image according to the adjusted projection images of adjacent viewing angles.

[0102] like Figure 7 As shown, an electronic device 700 provided by an embodiment of the present invention includes a memory 720 and a processor 710; the memory 720 is used to store computer programs; the processor 710 is used to implement the panoramic image generation method as described above when executing the computer program.

[0103] In other words, an electronic device 700 includes a memory 720 and a processor 710 coupled to the memory 720; the memory 720 is configured to store a computer program; and the processor 710 is configured to perform the following operations when executing the computer program:

[0104] Processing the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image captured by the camera and the projection image;

[0105] Extracting feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and adjusting the projection images of adjacent viewing angles according to the feature point pairs when the feature point pairs meet a preset correction condition;

[0106] A panoramic image is generated according to the adjusted projection images of adjacent viewing angles.

[0107] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the panoramic image generation method as described above is implemented.

[0108] In other words, a non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following operations:

[0109] Processing the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image captured by the camera and the projection image;

[0110] Extracting feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and adjusting the projection images of adjacent viewing angles according to the feature point pairs when the feature point pairs meet a preset correction condition;

[0111] A panoramic image is generated according to the adjusted projection images of adjacent viewing angles.

[0112] An electronic device 700 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device 700 is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 700 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described herein and / or required.

[0113] The electronic device 700 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0114] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0115] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A method for generating a panoramic image, characterized in that: include: Processing the real-time image captured by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image captured by the camera and the projection image; Extracting feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and adjusting the projection images of adjacent viewing angles according to the feature point pairs when the feature point pairs meet a preset correction condition; A panoramic image is generated according to the adjusted projection images of adjacent viewing angles.

2. The panoramic image generation method according to claim 1, characterized in that: When the feature point pair satisfies a preset correction condition, adjusting the projection image of adjacent viewing angles according to the feature point pair includes: When the number of the feature point pairs is greater than or equal to a preset number threshold, and the average of the distances between the corresponding feature point pairs in the projection images of adjacent viewing angles is greater than or equal to a preset distance threshold, it is determined that the feature point pairs meet the preset correction condition; The pixels in one of the two projection images of adjacent viewing angles are mapped to another projection image adjacent thereto through the transformation matrix of the two projection images of adjacent viewing angles, so as to adjust the projection images of adjacent viewing angles.

3. The panoramic image generation method according to claim 2, characterized in that: Also includes: When the number of the feature point pairs is less than the preset number threshold, and / or the average of the distances between the corresponding feature point pairs in the projection images of adjacent perspectives is less than the preset distance threshold, it is determined that the feature point pairs do not satisfy the preset correction conditions, and the projection images of adjacent perspectives are directly fused to generate the panoramic image.

4. The panoramic image generation method according to claim 1, characterized in that: The processing of the real-time image collected by the camera according to the preset lookup table to generate a projection image of the corresponding viewing angle includes: Preprocessing the real-time image to obtain a preprocessed image; The preprocessed image is converted into a plane projection according to the preset lookup table to generate the projection image of the corresponding viewing angle.

5. The panoramic image generation method according to claim 4, characterized in that: The preprocessing of the real-time image to obtain the preprocessed image comprises: Correcting the real-time image according to the distortion coefficient of the camera; Performing grayscale processing on the corrected real-time image to convert the real-time image into a grayscale image; De-noising the grayscale image to obtain the pre-processed image.

6. The panoramic image generation method according to claim 4, characterized in that: The preset lookup table is determined in the following manner: Calibrate the camera to determine the intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient of the camera; The preset lookup table is generated according to the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion coefficient.

7. The panoramic image generation method according to any one of claims 1 to 6, characterized in that: Generating a panoramic image according to the adjusted projection map of adjacent viewing angles comprises: The pixels of the overlapping area of ​​the two projection images of the adjusted adjacent viewing angles are fused according to preset weights to generate the panoramic image.

8. A panoramic image generation device, characterized in that: include: The first module is used to process the real-time image collected by the camera according to a preset lookup table to generate a projection image of a corresponding viewing angle, wherein the preset lookup table is used to represent a mapping relationship between the image collected by the camera and the projection image; A second module is used to extract feature point pairs from overlapping areas of the projection images of adjacent viewing angles, and when the feature point pairs meet a preset correction condition, adjust the projection images of adjacent viewing angles according to the feature point pairs; The third module is used to generate a panoramic image according to the adjusted projection map of adjacent viewing angles.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the panoramic image generation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the panoramic image generation method according to any one of claims 1 to 7 is implemented.

Images