Panoramic image generation method, panoramic camera system, and storage medium

Abstract

The application relates to the technical field of image processing, and discloses a panoramic image generation method, which comprises the following steps: acquiring an initial pose relationship between a laser radar and each camera, point cloud features of the laser radar, and image features of original images of each camera; matching the point cloud features and the image features to obtain feature point pairs composed of the point cloud features and the image features; optimizing a rotation matrix and a translation vector based on point cloud coordinates corresponding to the point cloud features and camera coordinates corresponding to the image features, to obtain a registration rotation matrix and a registration translation vector; determining pixel coordinates and depth of each original image according to the feature point pairs, the registration rotation matrix and the registration translation vector; determining target pixel coordinates and depth of the pixel coordinates of each original image according to the pixel coordinates and the depth; and generating a panoramic image according to the target pixel coordinates and the depth of the pixel coordinates of each original image. In the above manner, the quality of the panoramic image is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to a panoramic image generation method, a panoramic camera system and a storage medium. BACKGROUND

[0002] The panoramic camera system is an advanced multi-sensor fusion technology, aiming to generate high-precision and high-fidelity panoramic images and three-dimensional maps by combining LiDAR (Laser Radar) and multi-camera data. This system has wide application prospects in the fields of autonomous driving, intelligent transportation, robot navigation, etc.

[0003] The panoramic camera system includes a multi-camera and a LiDAR. The panoramic camera system can generate high-precision panoramic images and three-dimensional maps by fusing LiDAR point cloud data and multi-camera image data through algorithms. In the design of the panoramic camera system, multiple camera lenses are usually installed towards different directions, which results in the non-overlapping of the optical centers of multiple cameras. This not only leads to different positions of the same object on the imaging planes of different cameras, but also causes complex geometric relationships between different cameras, making the spliced images prone to obvious seams, ghosting or distortion. Therefore, in the case of non-overlapping optical centers of the multi-camera of the panoramic camera system, the quality of the panoramic image generated by the panoramic camera system may be reduced. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide a panoramic image generation method, a panoramic camera system and a storage medium, which improve the quality of the panoramic image.

[0005] According to an aspect of the embodiments of the present application, a panoramic image generation method is provided, applied to a panoramic camera system, the panoramic camera system including a laser radar and at least two cameras, the method including: obtaining an initial pose relationship between the laser radar and each camera, point cloud features scanned by the laser radar, and image features of original images captured by each camera, the initial pose relationship including a rotation matrix and a translation vector; for the laser radar and each camera, matching the point cloud features and the image features to obtain feature point pairs composed of the point cloud features and the image features; optimizing the rotation matrix and the translation vector based on point cloud coordinates corresponding to the point cloud features and camera coordinates corresponding to the image features in the feature point pairs to obtain a registration rotation matrix and a registration translation vector; determining pixel coordinates of each original image and depths corresponding to the pixel coordinates according to the point cloud coordinates, the registration rotation matrix and the registration translation vector in the feature point pairs; determining target pixel coordinates of the pixel coordinates of each original image in the panoramic image and depths corresponding to the target pixel coordinates according to the pixel coordinates and the depths corresponding to the pixel coordinates; and generating the panoramic image according to the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the depths corresponding to the target pixel coordinates.

[0006] In an optional manner, for the laser radar and each camera, the point cloud features and the image features are matched to obtain feature point pairs composed of the point cloud features and the image features, and further comprising: for the laser radar and each camera, the point cloud features and the image features are matched according to a nearest neighbor algorithm to obtain feature point pairs composed of the point cloud features and the image features; wherein the nearest neighbor algorithm is min||F_image-F_pointcloud||, wherein F_image represents the point cloud features, F_pointcloud represents the image features, and min|| || represents the minimum distance between two features.

[0007] In an optional manner, the rotation matrix and the translation vector are optimized based on the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs to obtain a registration rotation matrix and a registration translation vector, and further comprising: determining the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs; converting the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain converted point cloud coordinates; determining a registration error of all feature point pairs according to the camera coordinates and the converted point cloud coordinates; and optimizing the rotation matrix and the translation vector according to the registration error to obtain the registration rotation matrix and the registration translation vector.

[0008] In an optional manner, the rotation matrix and the translation vector are optimized according to the registration error to obtain a registration rotation matrix and a registration translation vector, and further comprising: obtaining a maximum optimization number of the rotation matrix and the translation vector; determining whether the registration error is less than a preset registration error; if the registration error is less than the preset registration error, determining the rotation matrix and the translation vector for determining the registration error as the registration rotation matrix and the registration translation vector; if the registration error is greater than or equal to the preset registration error, optimizing the rotation matrix and the translation vector, and determining the optimized rotation matrix and translation vector as the rotation matrix and the translation vector, and proceeding to the step of converting the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain converted point cloud coordinates until the number of optimized rotation matrix and translation vector reaches the maximum optimization number, and determining the rotation matrix and the translation vector obtained by the last optimization as the rotation matrix and the translation vector.

[0009] In an optional manner, the registration error of all feature point pairs is wherein, represents the i-th camera coordinate, represents the i-th point cloud coordinate, represents the rotation matrix, represents the translation vector, represents the i-th converted point cloud coordinate.

[0010] ​In an optional manner, the pixel coordinates of each original image and the depth corresponding to the pixel coordinates are determined according to the point cloud coordinates in the feature point pair, the registration rotation matrix and the registration translation vector, and the method further comprises: obtaining the intrinsic matrix of each camera; converting the point cloud coordinates in the feature point pair into the camera coordinate system according to the point cloud coordinates in the feature point pair, the registration rotation matrix and the registration translation vector to obtain converted point cloud coordinates; determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the intrinsic matrix and the converted point cloud coordinates; determining the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates, and the method further comprises: obtaining the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera and the virtual intrinsic matrix; determining the three-dimensional coordinates corresponding to the pixel coordinates according to the intrinsic matrix, the pixel coordinates and the depth corresponding to the pixel coordinates; determining the pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates, wherein the pose relationship comprises an optical center rotation matrix and an optical center translation vector; determining the camera coordinates of each original image in the panoramic camera according to the three-dimensional coordinates, the optical center rotation matrix and the optical center translation vector; determining the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the camera coordinates in the panoramic camera and the virtual intrinsic matrix.

[0011] In an optional manner, the three-dimensional coordinates corresponding to the pixel coordinates are: , wherein, represents the three-dimensional coordinates corresponding to the i-th pixel coordinates, represents the intrinsic matrix, represents the i-th pixel coordinates, represents the depth corresponding to the i-th pixel coordinates; the camera coordinates P_virtual of the panoramic camera are: P_virtual=R_virtual·P_3D,i+T_virtual, wherein R_virtual represents the optical center rotation matrix, and T_virtual represents the optical center translation vector.

[0012] In an optional manner, the panoramic image is generated according to the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates, and the method further comprises: when the pixel coordinates of at least two original images in the panoramic image are the same target pixel coordinates and the depths corresponding to the target pixel coordinates are not equal, calculating the average value of the depths corresponding to the at least two target pixel coordinates; re-determining the average value of the depths corresponding to the at least two target pixel coordinates as the depth corresponding to the target pixel coordinates in the panoramic image; and generating the panoramic image according to the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates.

[0013] According to another aspect of the embodiments of the present application, a panoramic camera system is provided, comprising a processor, a memory, a laser radar and at least two cameras; the laser radar is configured to scan to obtain point cloud features; the cameras are configured to capture original images; the memory stores a computer program; and the processor executes the computer program to implement the panoramic image generation method provided by any of the above embodiments.

[0014] According to still another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the panoramic image generation method provided by any of the above embodiments.

[0015] The embodiments of the present application match the point cloud features scanned by the laser radar and the image features of the original images captured by each camera to obtain feature point pairs composed of the point cloud features and the image features, optimize the rotation matrix and the translation vector in the initial pose relationship between the laser radar and each camera according to the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs, obtain the registration rotation matrix and the registration translation vector, so that the pixel coordinates of each original image and the corresponding depth can be determined according to the point cloud coordinates, the registration rotation matrix and the registration translation vector in the feature point pairs, then the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the corresponding depth can be determined according to the pixel coordinates of each original image and the corresponding depth, and finally the panoramic image can be generated according to the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the corresponding depth. In the above manner, the depth corresponding to the pixel coordinates of the original image captured by each camera can be determined, so that the original images captured by multiple cameras with non-overlapping optical centers can be projected to the same optical center through the depth, ensuring the consistency of the geometric structure and the color of the panoramic image generated by the panoramic camera system, and improving the quality of the panoramic image.

[0016] The above description is only a summary of the technical solutions of the embodiments of the present application, in order to more clearly understand the technical means of the embodiments of the present application, the embodiments of the present application can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are only used to show the embodiments, and are not considered as limitations of the present application. Moreover, the same reference signs are used to represent the same parts throughout the drawings. In the drawings:

[0018] Figure 1 A structure schematic diagram of the panoramic camera system provided by the embodiments of the present application is shown;

[0019] Figure 2 A flow schematic diagram of the panoramic image generation method provided by the embodiments of the present application is shown;

[0020] Figure 3 A structure diagram of a panoramic image generation device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0022] The panoramic camera system includes a multi-view camera and a LiDAR. The multi-view camera generally includes multiple camera lenses, each of which is responsible for capturing images at different angles, which can be stitched into a panoramic image. The LiDAR generates high-precision three-dimensional point cloud data by emitting laser light and receiving reflected light signals, which provides depth information of the environment and helps to construct a three-dimensional map. The panoramic camera system fuses LiDAR point cloud data and multi-view camera image data through algorithms to generate high-precision panoramic images and three-dimensional maps.

[0023] In the design of the panoramic camera system, multiple camera lenses are usually installed towards different directions to enable multiple lenses to capture images at different angles, so that the panoramic camera synthesis system can cover a wider field of view. However, this results in the optical centers of multiple cameras not overlapping, i.e., the optical centers of each camera are not at the same point, which not only causes the positions of the same object on the imaging planes of different cameras to be different, making it difficult to accurately align the images when stitching and prone to misalignment, but also causes the geometric relationship between different cameras to be complex, making the stitching algorithm need to consider more correction parameters, and if the correction is not accurate, the stitched images will have obvious seams, ghosting or distortion. In summary, in the case of non-overlapping optical centers of the multi-view camera of the panoramic camera system, the quality of the panoramic image generated by the panoramic camera system may be reduced.

[0024] It is found through research that to solve the problem of non-overlapping optical centers of multiple cameras in the panoramic camera system, the optical center points of different cameras need to be combined into a virtual optical center point of the same panoramic camera, and in this process, the pixel coordinates of images captured by different cameras need to be re-projected to the virtual optical center point of the same panoramic camera.

[0025] Based on this, this application provides a panoramic image generation method. This method determines the depth corresponding to the pixel coordinates of the images captured by each camera using depth information scanned by a LiDAR scanner. Then, based on the depth corresponding to the pixel coordinates, it determines the pixel coordinates and depth of each pixel coordinate projected onto a virtual optical center point in the same panoramic camera. Finally, the panoramic image is generated using the pixel coordinates and depth projected onto the panoramic camera. This method ensures that images captured by multiple cameras with non-overlapping optical centers are projected onto a virtual optical center, guaranteeing the consistency of the geometric structure and color of the panoramic image generated by the panoramic camera system, thus improving the quality of the panoramic image.

[0026] The panoramic image generation method provided in this application is applied to a panoramic camera system. Figure 1 The diagram illustrates the structure of a panoramic camera system provided in an embodiment of this application. Specific embodiments of this application do not limit the specific implementation of the panoramic camera system. As shown in Figure 1, the panoramic camera system 10 includes: a processor 11, a memory 12, a lidar 13, and at least two cameras.

[0027] Memory 12 is used to store computer program 121. Memory 12 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device. The computer program 121 may include computer-executable instructions.

[0028] Processor 11 is used to execute computer program 121 to implement an embodiment of the panoramic image generation method. Processor 11 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of this application. The panoramic camera system 10 includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0029] The lidar 13 is used to scan and obtain point cloud features. Specifically, the emitter of the lidar 13 emits a laser towards the target area. After the laser encounters the target object, it is reflected. The receiver of the lidar 13 captures the reflected light signal. By measuring the time difference between laser emission and reception (Time of Flight, or ToF), the target distance can be calculated. Combined with the lidar's own rotation or scanning angle (mechanical or solid-state scanning), a point cloud (containing the X / Y / Z coordinates and reflection intensity of each point) can be constructed point by point.

[0030] Regarding at least two cameras,Figure 1 The panoramic camera system 10 can also be provided with 3 or 4, or even more cameras, which are not limited in the present application. The first camera 14a and the second camera 14b can be wide-angle cameras or fisheye cameras, and the first camera 14a and the second camera 14b respectively capture the original images of the target region from different angles.

[0031] Figure 2 A flowchart of a panoramic image generation method provided by an embodiment of the present application is shown, which is executed by the processor 11 of the panoramic camera system 10 as shown. Figure 1 As shown in the flowchart, the method comprises the following steps: Figure 2

[0032] Step 100: Obtain the initial pose relationship between the laser radar and each camera, the point cloud features of the laser radar scanning, and the image features of the original images taken by each camera. The initial pose relationship includes a rotation matrix and a translation vector.

[0033] The initial pose relationship between the laser radar and each camera can be represented by formula (1):

[0034] , (1)

[0035] Wherein, P lidar represents the point cloud coordinates in the LiDAR coordinate system, P represents the converted point cloud coordinates, i.e. the three-dimensional coordinates in the camera coordinate system converted from the point cloud coordinates in the LiDAR coordinate system, R represents the rotation matrix, and t represents the translation vector.

[0036] The initial pose relationship between the laser radar and each camera can be obtained through a calibration process. Specifically, the relative physical positions between the laser radar and each camera can be measured, and then the initial pose relationship between the laser radar and each camera can be determined according to the relative physical positions. For example, as shown in Figure 1 When there is a 30° rotation between the laser radar 13 and the first camera 14a, the first camera 14a is located 10 cm in front of and 5 cm above the laser radar 13, the rotation matrix and the translation vector of the initial pose relationship between the laser radar 13 and the first camera 14a are as follows:

[0037] [ - 0 0 0 1], = [10, 5, 0].

[0038] ​Point cloud features are extracted from three-dimensional point cloud data scanned by a laser radar, and are used to describe the geometric shape and surface characteristics of objects in the point cloud. Point cloud features include geometric features, local feature descriptors, and voxel-based features; among them, geometric features include the coordinates (X, Y, Z) of points, normal vectors, curvatures, etc.; local feature descriptors such as SHOT (Signature of Histograms of OrienTations, orientation histogram feature) are used to describe the geometric shape of local regions in the point cloud. Point cloud features are usually represented in the form of three-dimensional coordinates of points and local feature descriptors.

[0039] Image features are extracted from two-dimensional original images, and are usually used to describe the shape, texture, edge, etc. of objects in the image, and generally include corner features, SIFT (Scale-Invariant Feature Transform, scale-invariant feature transform) features, SURF (Speeded-Up Robust Features, accelerated robust features) and ORB (Oriented FAST and Rotated BRIEF, oriented FAST and rotated BRIEF) features. Among them, the SIFT feature can extract the key points (positions) and their descriptors (feature vectors) in the image. Image features are usually represented in the form of key points and descriptors.

[0040] Step 200: For each camera, match the point cloud features and the image features to obtain feature point pairs composed of point cloud features and image features.

[0041] Specifically, the point cloud features and the image features can be matched according to the nearest neighbor algorithm, and the nearest neighbor algorithm is as follows:

[0042] min||F_image-F_pointcloud||, (2)

[0043] Where F_image represents the point cloud feature, F_pointcloud represents the image feature, and min|| || represents the minimum distance between the two features.

[0044] After feature matching, a plurality of feature point pairs of the laser radar and each camera can be obtained. For example, a plurality of feature point pairs of the laser radar 13 and the first camera 14a, and a plurality of feature point pairs of the laser radar 13 and the second camera 14b.

[0045] Step 300: Based on the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features, the rotation matrix and the translation vector are optimized to obtain the registration rotation matrix and the registration translation vector.

[0046] Specifically, algorithms such as KC (K-means Clustering), CPD (Coherent Point Drift), RPM (Robust Point Matching), and ICP (Iterative Closest Point) can be used to finely register the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs to optimize the rotation matrix and the translation vector.

[0047] When using the ICP algorithm to register the point cloud coordinates and the camera coordinates, step 300 specifically includes the following steps:

[0048] Step 310: Determine the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs.

[0049] For each feature point pair, the point cloud coordinates of the point cloud features and the camera coordinates corresponding to the image features can be determined.

[0050] For example, for the i-th feature point pair F_i of the lidar 13 and the first camera 14a, the i-th point cloud coordinate corresponding to the point cloud feature F_image can be determined , and the i-th camera coordinate corresponding to the image feature F_pointcloud can be determined .

[0051] Step 320: Convert the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain converted point cloud coordinates.

[0052] The point cloud coordinates can be converted into the camera coordinate system through the pose relationship to obtain the converted point cloud coordinates. Specifically, the i-th point cloud coordinate can be converted into the camera coordinate system through formula (1) to obtain the converted i-th point cloud coordinate .

[0053] In this way, the point cloud coordinates corresponding to the point cloud features in each feature point pair can be converted into the camera coordinate system to obtain the converted point cloud coordinates. In this way, for each feature point pair of the lidar 13 and the first camera 14a, the camera coordinates corresponding to each feature point pair and the converted point cloud coordinates can be obtained.

[0054] Step 330: Determine the registration error of all feature point pairs according to the camera coordinates and the converted point cloud coordinates.

[0055] By optimizing the rotation matrix R and the translation vector T to minimize the sum of the Euclidean distances between the camera coordinates and the converted point cloud coordinates, the accurate alignment of the camera coordinates and the point cloud coordinates can be achieved.

[0056] wherein the registration error of all feature point pairs is

[0057] , (3)

[0058] wherein, represents the i-th camera coordinate, represents the i-th point cloud coordinate, represents a rotation matrix, represents a translation vector, represents the i-th converted point cloud coordinate.

[0059] Step 340: optimizing the rotation matrix and the translation vector according to the registration error to obtain a registration rotation matrix and a registration translation vector.

[0060] Specifically, step 340 comprises the following steps:

[0061] Step 341: obtaining a maximum optimization number of the rotation matrix and the translation vector.

[0062] Step 342: judging whether the registration error is less than a preset registration error, if yes, executing step 343, and if no, executing step 344.

[0063] Step 343: determining the rotation matrix and the translation vector of the determined registration error as the registration rotation matrix and the registration translation vector.

[0064] Step 344: optimizing the rotation matrix and the translation vector, and determining the optimized rotation matrix and the translation vector as the rotation matrix and the translation vector, and turning to execute step 320 until the number of optimizing the rotation matrix and the translation vector reaches the maximum optimization number, and determining the rotation matrix and the translation vector obtained by the last optimization as the rotation matrix and the translation vector.

[0065] Firstly, the camera coordinates and the converted point cloud coordinates corresponding to all feature point pairs obtained in step 320, and the rotation matrix and the translation vector in the initial pose relationship obtained in step 100 are substituted into the above formula (3), and the registration error of all feature point pairs is obtained. When the registration error does not satisfy the preset registration error, the rotation matrix and the translation vector are optimized to obtain the first optimized rotation matrix and the optimized translation vector .

[0066] Then, the camera coordinates and the converted point cloud coordinates corresponding to all feature point pairs, and the optimized rotation matrix and the optimized translation vector ​Substitute the above formula (3), the registration error of all feature point pairs can be obtained When the registration error of all feature point pairs is less than the preset registration error When the registration error of all feature point pairs is less than the preset registration error , the rotation matrix and the translation vector are continued to be optimized to obtain the second optimized rotation matrix , and the translation vector , and the translation vector The registration error of all feature point pairs is less than the preset registration error, the optimization of the rotation matrix and the translation vector is stopped, and the determined registration error , and the translation vector The registration rotation matrix , and the registration translation vector ; or, when the number of times of optimizing the rotation matrix and the translation vector reaches the maximum optimization number, the optimization of the rotation matrix and the translation vector is stopped, and the optimized rotation matrix , and the translation vector The registration rotation matrix , and the registration translation vector .

[0067] By iterating the optimization of the rotation matrix and the translation vector through steps 341-343, the registration error can be minimized, and finally the accurate registration rotation matrix and the registration translation vector are obtained, and by setting the preset registration error and the maximum optimization number, the infinite loop of the optimization operation can be avoided.

[0068] Through the above steps 310-340, the rotation matrix and the translation vector can be optimized several times to make the registration error of all feature point pairs tend to be minimum, and finally the optimal registration rotation matrix and the registration translation vector are obtained, and the high-precision registration of the point cloud coordinates and the camera coordinates is realized.

[0069] Step 400: determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the point cloud coordinates in the feature point pair, the registration rotation matrix and the registration translation vector.

[0070] Specifically, step 400 specifically includes the following steps:

[0071] Step 410: obtaining the intrinsic matrix of each camera.

[0072] The intrinsic matrix of each camera can be represented as K_i.

[0073] Step 420: converting the point cloud coordinates in the feature point pair into the camera coordinate system according to the point cloud coordinates in the feature point pair, the registration rotation matrix and the registration translation vector to obtain the converted point cloud coordinates.

[0074] For the point cloud coordinates (X lidar, Y lidar, Z lidar ) in each feature point pair, the point cloud coordinates can be converted into the camera coordinate system by formula (1). Specifically, the point cloud coordinates (X lidar, Y lidar, Z lidar ), the registration rotation matrix and the registration translation vector are substituted into formula (1) to obtain the converted point cloud coordinates (X camera, Y camera, Z camera ).

[0075] Step 430: determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the intrinsic matrix and the converted point cloud coordinates.

[0076] After obtaining the converted point cloud coordinates (X camera, Y camera, Z camera ), the converted point cloud coordinates (X camera, Y camera, Z camera ) are multiplied by the intrinsic matrix K_i, and the pixel coordinates (u_i, v_i) of the original image are obtained. The depth corresponding to the pixel coordinates (u_i, v_i) is the Z camera of the converted point cloud coordinates.

[0077] In this way, by steps 410-430, the point cloud coordinates in each feature point pair can be determined one by one, and then all the pixel coordinates of the original image and the depth corresponding to the pixel coordinates can be determined.

[0078] Step 500: determining the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates.

[0079] Specifically, step 500 specifically includes the following steps:

[0080] Step 510: obtaining the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera, and the virtual intrinsic matrix.

[0081] The optical center coordinates of the camera are the coordinates of the optical center of the camera, which can be represented as C_i.

[0082] In the embodiments of the present application, the panoramic camera is a virtual camera. The virtual optical center coordinates of the panoramic camera can be represented as C virtual, and the virtual intrinsic matrix can be represented as K virtual. The virtual intrinsic matrix of the panoramic camera can be the same as or different from the intrinsic matrix of any camera in the panoramic camera system.

[0083] In some embodiments, for example, when the panoramic camera includes an odd number of cameras and the plurality of cameras are symmetrically arranged, the panoramic camera can be the camera located at the center of symmetry in the plurality of cameras, and the virtual optical center coordinates of the panoramic camera are the optical center coordinates of the camera.

[0084] Step 520: determining the three-dimensional coordinates corresponding to the pixel coordinates according to the intrinsic matrix, the pixel coordinates and the depth corresponding to the pixel coordinates.

[0085] Specifically, the three-dimensional coordinates corresponding to the pixel coordinates can be calculated by the following formula (4):

[0086] , (4)

[0087] wherein, represents the three-dimensional coordinates corresponding to the i-th pixel coordinates, represents the intrinsic matrix, represents the i-th pixel coordinates, represents the depth corresponding to the i-th pixel coordinates.

[0088] Step 530: determining the pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates, the pose relationship including the optical center rotation matrix and the optical center translation vector.

[0089] After determining the optical center coordinates C_i of each camera and the virtual optical center coordinates C_virtual of the panoramic camera, the pose relationship between the optical center coordinates C_i and the virtual optical center coordinates C_virtual can be determined by calibration. The pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates can be represented by the following formula (5):

[0090] , (5)

[0091] wherein, R_virtual represents the optical center rotation matrix, and T_virtual represents the optical center translation vector.

[0092] Step 540: determining the camera coordinates of the pixel coordinates of each original image in the panoramic camera according to the three-dimensional coordinates, the optical center rotation matrix and the optical center translation vector.

[0093] After determining the three-dimensional coordinates corresponding to the pixel coordinates of the original image, the camera coordinates P_virtual of the panoramic camera can be calculated by the following formula (6):

[0094] P_virtual=R_virtual·P_3D,i+T_virtual. (6)

[0095] Step 550: determining the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the camera coordinates in the panoramic camera and the virtual intrinsic matrix.

[0096] After obtaining the camera coordinates P_virtual in the panoramic camera, the pixel coordinates (u_virtual, v_virtual) of the original image can be determined according to formula (7). The specific formula is as follows:

[0097] .

[0098] Assuming the camera coordinates P_virtual are (X_virtual, Y_virtual, Z_virtual), then the depth corresponding to the pixel coordinates (u_virtual, v_virtual) is Z_virtual.

[0099] In this way, through steps 510-550, the three-dimensional coordinates corresponding to all pixel coordinates of each original image can be determined one by one. Then, the camera coordinates of each pixel coordinate in the panoramic camera can be determined through the three-dimensional coordinates. Thus, the target pixel coordinates and the depth corresponding to the target pixel coordinates in the panoramic image can be determined through the camera coordinates, thereby realizing the projection of the pixel coordinates of each original image onto the virtual optical center of the panoramic camera.

[0100] Step 600: Generate a panoramic image based on the target pixel coordinates and the depth corresponding to the target pixel coordinates in the panoramic image of each original image's pixel coordinates.

[0101] Taking a panoramic camera system with two cameras as an example, such as Figure 1 As shown, after determining the target pixel coordinates and depth of each pixel coordinate in the panoramic image for each original image, when the target pixel coordinates in the panoramic image are different for the original images captured by the first camera 14a and the second camera 14b, the panoramic image can be generated directly based on the target pixel coordinates and their corresponding depths in the panoramic image. When the target pixel coordinate 1 of the original image captured by the first camera 14a is (200, 300) in the panoramic image, and the depth 1 corresponding to the target pixel coordinate 1 is 5cm, and the target pixel coordinate 2 of the original image captured by the second camera 14b is (200, 300) in the panoramic image, and the depth 2 corresponding to the target pixel coordinate 2 is 5cm, then the final depth corresponding to the target pixel coordinate (200, 300) in the panoramic image is 5cm.

[0102] After determining the coordinates of all target pixels and their corresponding depths in the panoramic image, the panoramic image can be generated by combining each target pixel coordinate with its corresponding depth.

[0103] In some embodiments, step 600 specifically includes the following steps:

[0104] Step 610: When the pixel coordinates of the at least two original images are the same in the target pixel coordinates of the panoramic image, and the depths corresponding to the target pixel coordinates are not equal, calculate the average of the depths corresponding to the at least two target pixel coordinates.

[0105] For example, when the pixel coordinate 1 of the original image photographed by the first camera 14a is (200, 300) in the target pixel coordinate 1 of the panoramic image, the depth 1 corresponding to the target pixel coordinate 1 is 2 cm, the pixel coordinate 2 of the original image photographed by the second camera 14b is (200, 300) in the target pixel coordinate 2 of the panoramic image, and the depth 2 corresponding to the target pixel coordinate 2 is 5 cm, then the average of the depth 1 and the depth 2 is calculated, and the average of the depths corresponding to the target pixel coordinate 1 and the target pixel coordinate 2 is 3.5 cm.

[0106] Step 620: Redetermine the average of the depths corresponding to the at least two target pixel coordinates as the depth corresponding to the target pixel coordinate in the panoramic image.

[0107] The target pixel coordinate (200, 300) of the panoramic image finally corresponds to a depth of 3.5 cm.

[0108] Step 630: Generate a panoramic image according to the target pixel coordinates and the depths corresponding to the target pixel coordinates of each original image in the panoramic image.

[0109] By calculating the average of the depths corresponding to the at least two target pixel coordinates, the depth conflict in the panoramic image synthesis process can be solved, the accuracy and consistency of the depth information are improved, and thus the quality of the panoramic image is improved.

[0110] The embodiment of the present application matches the point cloud features scanned by the laser radar and the image features of the original images captured by each camera, obtains feature point pairs composed of the point cloud features and the image features, optimizes the rotation matrix and the translation vector in the initial pose relationship between the laser radar and each camera based on the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs, obtains a registration rotation matrix and a registration translation vector, so that the pixel coordinates of each original image and the corresponding depth can be determined through the point cloud coordinates, the registration rotation matrix and the registration translation vector in the feature point pairs, and then the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the corresponding depth can be determined through the pixel coordinates of each original image and the corresponding depth, and finally the panoramic image can be generated according to the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the corresponding depth. In the above manner, the depth corresponding to the pixel coordinates of the original image captured by each camera can be determined, so that the original images captured by the multiple cameras with non-overlapping optical centers can be projected to the same optical center through the depth, ensuring the consistency of the geometric structure and the color of the panoramic image generated by the panoramic camera system, and improving the quality of the panoramic image.

[0111] Figure 3 The structure schematic diagram of the panoramic image generation device provided by the embodiment of the present application is shown. As shown in the figure, Figure 3 The panoramic image generation device 700 includes an acquisition module 710, a matching module 720, an optimization module 730, a first determination module 740, a second determination module 750 and a generation module 760.

[0112] The acquisition module 710 is configured to acquire the initial pose relationship between the laser radar and each camera, the point cloud features scanned by the laser radar and the image features of the original images captured by each camera, and the initial pose relationship includes a rotation matrix and a translation vector;

[0113] The matching module 720 is configured to match the point cloud features and the image features for the laser radar and each camera, and obtain feature point pairs composed of the point cloud features and the image features;

[0114] The optimization module 730 is configured to optimize the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pairs, and obtain a registration rotation matrix and a registration translation vector;

[0115] The first determination module 740 is configured to determine the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the feature point pairs, the registration rotation matrix and the registration translation vector;

[0116] The second determination module 750 is configured to determine the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates;

[0117] The generation module 760 is configured to generate the panoramic image according to the target pixel coordinate of each original image in the panoramic image and the depth corresponding to the target pixel coordinate.

[0118] The panoramic image generation apparatus 700 provided by the embodiment is used to implement the technical solutions of the panoramic image generation method in the foregoing method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0119] The embodiment of the present application provides a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the panoramic image generation method embodiments.

[0120] The embodiment of the present application provides a computer program, which can be executed by a processor to implement the panoramic image generation method embodiments.

[0121] The embodiment of the present application provides a computer program product, the computer program product includes a computer program, and the computer program is executed by a processor to implement the panoramic image generation method embodiments.

[0122] In several embodiments provided in the present application, any function realized in the form of a software function module / unit and sold or used as an independent product can be stored in a computer readable storage medium. Based on this understanding, part or all of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or an electronic device) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing computer program codes.

[0123] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the teachings herein. The construction required to construct such systems as described above is apparent to those skilled in the art from the description above. In addition, the embodiments of the present application are not directed to any particular programming language. It should be understood that the embodiments of the present application described herein can be implemented using various programming languages, and the description of specific languages above is to disclose the best mode of the present application.

[0124] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage. The use of relative terms such as 'about','substantially', 'approximately' and the like in connection with an element or step provided herein will be understood by those of ordinary skill in the art to refer to a range of values that one of ordinary skill in the art would consider as the equivalent of a recited value in the context of the present application. The steps of a process as described herein can be carried out in any suitable order unless otherwise indicated above or otherwise clearly contradicted by context. The foregoing description details certain embodiments of the application. It will be appreciated, however, that no matter how detailed the above descriptions are, they are still only teaching the generic and exemplary of the application. Further, the above descriptions are not meant to limit the protective scope of this application. It is therefore important to restrict this application only according to the following claims.

[0125] The above embodiments are only some of the embodiments of the present application, which are described in a more specific and detailed manner, but should not be understood as limiting the scope of the patent of the present application. It should be noted that for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A panoramic image generation method, applied to a panoramic camera system, characterized in that, The panoramic camera system includes a lidar and at least two cameras, and the method includes: The initial pose relationship between the lidar and each of the cameras is obtained, along with the point cloud features scanned by the lidar and the image features of the original images captured by each of the cameras. The initial pose relationship includes a rotation matrix and a translation vector. For each of the lidar and each of the cameras, the point cloud features and the image features are matched to obtain feature point pairs composed of the point cloud features and the image features; Based on the point cloud coordinates corresponding to the point cloud features in the feature point pair and the camera coordinates corresponding to the image features, optimize the rotation matrix and the translation vector to obtain the registration rotation matrix and the registration translation vector. The pixel coordinates and the depth corresponding to each pixel coordinate of the original image are determined based on the point cloud coordinates in the feature point pair, the registration rotation matrix, and the registration translation vector. The target pixel coordinates and the corresponding depth of each pixel coordinate in the original image are determined in the panoramic image based on the pixel coordinates and the depth corresponding to the pixel coordinates. The panoramic image is generated based on the target pixel coordinates in the panoramic image and the depth corresponding to the target pixel coordinates of each pixel coordinate in the original image. The step of determining the pixel coordinates and the depth corresponding to each pixel coordinate of the original image based on the point cloud coordinates in the feature point pair, the registration rotation matrix, and the registration translation vector further includes: Obtain the intrinsic parameter matrix for each of the cameras; The point cloud coordinates in the feature point pair are transformed into the camera coordinate system based on the point cloud coordinates in the feature point pair, the registration rotation matrix, and the registration translation vector to obtain the transformed point cloud coordinates. The pixel coordinates and the depth corresponding to each pixel coordinate of the original image are determined based on the intrinsic parameter matrix and the transformed point cloud coordinates. The step of determining the target pixel coordinates and the corresponding depth of each pixel coordinate in the original image in the panoramic image based on the pixel coordinates and the depth corresponding to the pixel coordinates further includes: Obtain the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera, and the virtual intrinsic parameter matrix; The three-dimensional coordinates corresponding to the pixel coordinates are determined based on the intrinsic parameter matrix, the pixel coordinates, and the depth corresponding to the pixel coordinates. Determine the pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates, wherein the pose relationship includes the optical center rotation matrix and the optical center translation vector; The camera coordinates of each pixel in the original image are determined in the panoramic camera based on the three-dimensional coordinates, the optical center rotation matrix, and the optical center translation vector. The target pixel coordinates and the depth corresponding to the target pixel coordinates in the panoramic image are determined based on the camera coordinates in the panoramic camera and the virtual intrinsic parameter matrix.

2. The method according to claim 1, characterized in that, The step of matching the point cloud features and the image features for each of the lidar and each of the cameras to obtain feature point pairs composed of the point cloud features and the image features further includes: For each of the LiDAR and each of the cameras, the point cloud features and the image features are matched according to the nearest neighbor algorithm to obtain feature point pairs composed of the point cloud features and the image features; wherein, the nearest neighbor algorithm is: min||F_image-F_pointcloud||, Where F_image represents the point cloud feature, F_pointcloud represents the image feature, and min|| represents the minimum distance between two features.

3. The method according to claim 1, characterized in that, The step of optimizing the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pair and the camera coordinates corresponding to the image features to obtain the registration rotation matrix and the registration translation vector further includes: Determine the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features in the feature point pair; The point cloud coordinates are transformed into the camera coordinate system based on the rotation matrix and the translation vector to obtain the transformed point cloud coordinates; The registration error of all feature point pairs is determined based on the camera coordinates and the transformed point cloud coordinates. The rotation matrix and the translation vector are optimized based on the registration error to obtain the registration rotation matrix and the registration translation vector.

4. The method according to claim 3, characterized in that, The step of optimizing the rotation matrix and the translation vector based on the registration error to obtain the registration rotation matrix and the registration translation vector further includes: Obtain the maximum number of optimization iterations for the rotation matrix and the translation vector; Determine whether the registration error is less than the preset registration error; If the registration error is less than the preset registration error, then the rotation matrix and translation vector that determine the registration error are determined as the registration rotation matrix and the registration translation vector; If the registration error is greater than or equal to the preset registration error, then the rotation matrix and the translation vector are optimized, and the optimized rotation matrix and translation vector are determined as the rotation matrix and the translation vector. Then, the process proceeds to the step of transforming the point cloud coordinates to the camera coordinate system based on the rotation matrix and the translation vector to obtain the transformed point cloud coordinates. This process continues until the number of times the rotation matrix and the translation vector are optimized reaches the maximum number of optimizations. Finally, the rotation matrix and translation vector obtained from the last optimization are determined as the rotation matrix and the translation vector.

5. The method according to claim 3, characterized in that, Registration error of all feature point pairs for: ， in, This represents the coordinates of the i-th camera. This represents the coordinates of the i-th point cloud. Represents the rotation matrix, Denotes the translation vector. This represents the coordinates of the i-th transformed point cloud.

6. The method according to claim 1, characterized in that, The three-dimensional coordinates corresponding to the pixel coordinates are: ， in, This represents the 3D coordinates corresponding to the i-th pixel. This represents the intrinsic parameter matrix. This represents the coordinates of the i-th pixel. This represents the depth corresponding to the coordinates of the i-th pixel; The camera coordinates P_virtual of the panoramic camera are: P_virtual=R_virtual·P_3D,i+T_virtual, Where R_virtual represents the optical center rotation matrix and T_virtual represents the optical center translation vector.

7. The method according to claim 1, characterized in that, The step of generating the panoramic image based on the pixel coordinates of each original image, the target pixel coordinates of the panoramic image, and the depth corresponding to the target pixel coordinates further includes: When the pixel coordinates of at least two of the original images are the same as the target pixel coordinates of the panoramic image, and the depths corresponding to the target pixel coordinates are not equal, calculate the average value of the depths corresponding to the at least two target pixel coordinates. The average value of the depths corresponding to the at least two target pixel coordinates is redefined as the depth corresponding to the target pixel coordinates in the panoramic image; The panoramic image is generated based on the target pixel coordinates in the panoramic image and the depth corresponding to the target pixel coordinates of each pixel coordinate in the original image.

8. A panoramic camera system, characterized in that, Includes a processor, memory, LiDAR, and at least two cameras; The lidar is used to scan and obtain point cloud features; The camera is used to capture the original image; The memory stores computer programs; The processor executes the computer program to implement the panoramic image generation method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the panoramic image generation method according to any one of claims 1 to 7.

Images