Panoramic image generation method, panoramic camera system and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses a panoramic image generation method, and the method comprises the steps: obtaining an initial pose relation between a laser radar and each camera, the point cloud features of the laser radar, and the image features of an original image of each camera; matching the point cloud features with the image features to obtain feature point pairs formed by the point cloud features and the image features; optimizing the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector; determining pixel coordinates and depth of each original image according to the feature point pairs, the registration rotation matrix and the registration translation vector; determining a target pixel coordinate and a depth of the pixel coordinate of each original image according to the pixel coordinate and the depth; and generating a panoramic image according to the target pixel coordinate and the depth of the pixel coordinate of each original image. In this way, the quality of the panoramic image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and more particularly to a panoramic image generation method, a panoramic camera system, and a storage medium. Background Art

[0002] The panoramic camera system is an advanced multi-sensor fusion technology that aims to generate high-precision, high-fidelity panoramic images and three-dimensional maps by combining data from LiDAR (laser radar) and multiple cameras. This system has broad application prospects in areas such as autonomous driving, intelligent transportation, and robot navigation.

[0003] A panoramic camera system consists of multiple cameras and LiDAR. Using algorithms, panoramic camera systems fuse LiDAR point cloud data with multi-camera image data to generate high-precision panoramic images and three-dimensional maps. When designing a panoramic camera system, multiple camera lenses are typically installed facing different directions. This results in the optical centers of the multiple cameras not overlapping. This not only causes the same object to appear in different positions on the imaging planes of different cameras, but also leads to complex geometric relationships between the different cameras, making the stitched image prone to noticeable seams, ghosting, or distortion. Therefore, if the optical centers of the multiple cameras in a panoramic camera system do not overlap, the quality of the panoramic image generated by the system may be reduced. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide a panoramic image generation method, a panoramic camera system, and a storage medium, which improve the quality of panoramic images.

[0005] According to one aspect of an embodiment of the present application, a panoramic image generation method is provided, which is applied to a panoramic camera system, wherein the panoramic camera system includes a laser radar and at least two cameras, and the method includes: obtaining an initial pose relationship between the laser radar and each camera, point cloud features scanned by the laser radar, and image features of an original image captured by each camera, the initial pose relationship including a rotation matrix and a translation vector; for the laser radar and each camera, matching the point cloud features and the image features to obtain feature point pairs consisting of the point cloud features and the image features; optimizing the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pair and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector; determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the point cloud coordinates, the registration rotation matrix, and the registration translation vector in the feature point pair; determining the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates; and generating a panoramic image based on the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates.

[0006] In an optional manner, for the laser radar and each camera, point cloud features and image features are matched to obtain feature point pairs consisting of the point cloud features and the image features, further including: for the laser radar and each camera, point cloud features and image features are matched according to a nearest neighbor algorithm to obtain feature point pairs consisting of the point cloud features and the image features; wherein the nearest neighbor algorithm is: min||F_image-F_pointcloud||, wherein F_image represents point cloud features, F_pointcloud represents image features, and min|| || represents the minimum distance between the two features.

[0007] In an optional manner, the rotation matrix and translation vector are optimized based on the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to obtain the registration rotation matrix and the registration translation vector, further including: determining the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features; converting the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain the converted point cloud coordinates; determining the registration error of all feature point pairs according to the camera coordinates and the converted point cloud coordinates; optimizing the rotation matrix and the translation vector according to the registration error to obtain the registration rotation matrix and the registration translation vector.

[0008] In an optional manner, the rotation matrix and translation vector are optimized according to the registration error to obtain the registration rotation matrix and registration translation vector, further including: obtaining the maximum optimization times of the rotation matrix and translation vector; judging whether the registration error is less than the preset registration error; if the registration error is less than the preset registration error, determining the rotation matrix and translation vector that determine the registration error as the registration rotation matrix and registration translation vector; if the registration error is greater than or equal to the preset registration error, optimizing the rotation matrix and translation vector, and determining the optimized rotation matrix and translation vector as the rotation matrix and translation vector, and executing the step of converting the point cloud coordinates into the camera coordinate system according to the rotation matrix and translation vector to obtain the converted point cloud coordinates, until the number of optimization times of the rotation matrix and translation vector reaches the maximum optimization number, and determining the rotation matrix and translation vector obtained by the last optimization as the rotation matrix and translation vector.

[0009] In an optional approach, the registration error of all feature point pairs is for: ,in, represents the i-th camera coordinate, represents the coordinates of the i-th point cloud, represents the rotation matrix, represents the translation vector, Represents the coordinates of the i-th transformed point cloud.

[0010] In an optional manner, the pixel coordinates of each original image and the depth corresponding to the pixel coordinates are determined according to the point cloud coordinates, the registration rotation matrix and the registration translation vector in the feature point pair, further comprising: obtaining the intrinsic parameter matrix of each camera; converting the point cloud coordinates in the feature point pair into the camera coordinate system according to the point cloud coordinates, the registration rotation matrix and the registration translation vector in the feature point pair to obtain the converted point cloud coordinates; determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the intrinsic parameter matrix and the converted point cloud coordinates; determining the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates The method further includes: obtaining the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera and the virtual internal parameter matrix; determining the three-dimensional coordinates corresponding to the pixel coordinates according to the internal parameter matrix, the pixel coordinates and the depth corresponding to the pixel coordinates; determining the posture relationship between the optical center coordinates of each camera and the virtual optical center coordinates, the posture relationship including the optical center rotation matrix and the optical center translation vector; determining the camera coordinates of the pixel coordinates of each original image in the panoramic camera according to the three-dimensional coordinates, the optical center rotation matrix and the optical center translation vector; determining the target pixel coordinates of the pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the camera coordinates in the panoramic camera and the virtual internal parameter matrix.

[0011] In an optional manner, the three-dimensional coordinates corresponding to the pixel coordinates are: ,in, Represents the three-dimensional coordinates corresponding to the i-th pixel coordinates, represents the internal parameter matrix, represents the i-th pixel coordinate, Represents the depth corresponding to the i-th pixel coordinate; the camera coordinate P_virtual of the panoramic camera is: P_virtual=R_virtual·P_3D,i+T_virtual, where R_virtual represents the optical center rotation matrix and T_virtual represents the optical center translation vector.

[0012] In an optional manner, generating a panoramic image based on the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates further includes: when the pixel coordinates of at least two original images are the same in the target pixel coordinates of the panoramic image and the depths corresponding to the target pixel coordinates are not equal, calculating the average value of the depths corresponding to the at least two target pixel coordinates; re-determining the average value of the depths corresponding to the at least two target pixel coordinates as the depth corresponding to the target pixel coordinates in the panoramic image; and generating a panoramic image based on the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates.

[0013] According to another aspect of an embodiment of the present application, a panoramic camera system is provided, including a processor, a memory, a laser radar and at least two cameras; the laser radar is used to scan and obtain point cloud features; the camera is used to capture and obtain the original image; the memory stores a computer program; and the processor executes the computer program to implement the panoramic image generation method provided in any of the above embodiments.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the panoramic image generation method provided by any of the above embodiments is implemented.

[0015] In the embodiment of the present application, the point cloud features scanned by the laser radar are matched with the image features of the original image captured by each camera to obtain feature point pairs consisting of point cloud features and image features. The rotation matrix and translation vector in the initial pose relationship between the laser radar and each camera are optimized using the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector. The pixel coordinates, registration rotation matrix, and registration translation vector in the feature point pairs can then be used to determine the pixel coordinates of each original image and their corresponding depth. The target pixel coordinates of each original image's pixel coordinates in the panoramic image and their corresponding depth can then be determined using the pixel coordinates of each original image and their corresponding depth. Finally, a panoramic image can be generated based on the target pixel coordinates of each original image's pixel coordinates in the panoramic image and their corresponding depth. In this way, the depth corresponding to the pixel coordinates of the original image captured by each camera can be determined, so that the original images captured by multiple cameras with non-overlapping optical centers can be projected onto the same optical center based on the depth, ensuring the consistency of the geometric structure and color of the panoramic image generated by the panoramic camera system and improving the quality of the panoramic image.

[0016] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings: Figure 1 A schematic structural diagram of a panoramic camera system provided by an embodiment of the present application is shown; Figure 2 A schematic diagram showing a flow chart of a panoramic image generation method provided by an embodiment of the present application is shown; Figure 3A schematic structural diagram of a panoramic image generation device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0019] Panoramic camera systems include multi-cameras and LiDAR. Multi-cameras typically include multiple camera lenses, each capturing images from a different angle. These images are then stitched together to create a panoramic image. LiDAR emits lasers and receives reflected light signals, generating high-precision 3D point cloud data. This point cloud provides depth information about the environment and helps construct 3D maps. Panoramic camera systems use algorithms to fuse LiDAR point cloud data with multi-camera image data to generate high-precision panoramic images and 3D maps.

[0020] When designing a panoramic camera system, multiple camera lenses are usually installed in different directions so that the multiple lenses can capture images at different angles, allowing the panoramic camera synthesis system to cover a wider field of view. However, this results in the optical centers of the multiple cameras not overlapping, that is, the optical centers of the cameras are not at the same point. This not only causes the same object to be in different positions on the imaging planes of different cameras, making it difficult to accurately align the images when stitching, and prone to misalignment, but also leads to complex geometric relationships between different cameras, requiring the stitching algorithm to consider more correction parameters. If the correction is inaccurate, the stitched image will have obvious seams, ghosting, or distortion. In summary, when the optical centers of the multiple cameras in a panoramic camera system do not overlap, the quality of the panoramic image generated by the panoramic camera system may be reduced.

[0021] After research, it was found that in order to solve the problem of non-overlapping optical centers of multiple cameras in a panoramic camera system, it is necessary to merge the optical center points of different cameras into the virtual optical center point of the same panoramic camera. In this process, the pixel coordinates of the images taken by different cameras need to be reprojected to the virtual optical center point of the same panoramic camera.

[0022] Based on this, the present application provides a method for generating panoramic images. This method uses the depth information scanned by a laser radar to determine the depth corresponding to the pixel coordinates of the image captured by each camera. Then, based on the depth corresponding to the pixel coordinates, the pixel coordinates and depth of each pixel coordinate of the image captured by each camera are determined to be projected onto a virtual optical center point in the same panoramic camera. Finally, a panoramic image is generated by projecting the pixel coordinates and depth into the panoramic camera. In this way, images captured by multiple cameras with non-overlapping optical centers can be projected onto the virtual optical center, ensuring the consistency of the geometric structure and color of the panoramic image generated by the panoramic camera system and improving the quality of the panoramic image.

[0023] The panoramic image generation method provided in the embodiment of the present application is applied to a panoramic camera system. Figure 1 The following is a schematic diagram of the structure of a panoramic camera system provided in an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the panoramic camera system. As shown in FIG1 , the panoramic camera system 10 includes: a processor 11, a memory 12, a lidar 13, and at least two cameras.

[0024] The memory 12 is used to store a computer program 121. The memory 12 may include a high-speed RAM memory, or may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The computer program 121 may include computer-executable instructions.

[0025] Processor 11 is configured to execute computer program 121 to implement an embodiment of the panoramic image generation method. Processor 11 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in panoramic camera system 10 may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0026] The LiDAR 13 is used to scan and obtain point cloud features. Specifically, the LiDAR 13's transmitter emits laser light toward the target area. The laser light reflects off the target object, and the LiDAR 13's receiver captures the reflected light signal. By measuring the time difference between laser emission and reception (Time of Flight, or ToF), the target distance can be calculated. Combined with the LiDAR's own rotation or scanning angle (mechanical or solid-state scanning), a point cloud (including each point's X / Y / Z coordinates and reflection intensity) can be constructed point by point.

[0027] About at least two cameras, Figure 1 In the example, two cameras are shown, namely, first camera 14a and second camera 14b. Panoramic camera system 10 may also be provided with three, four, or even more cameras, which is not limited in this application. First camera 14a and second camera 14b may be wide-angle cameras or fisheye cameras, and each captures an original image of the target area from a different angle.

[0028] Figure 2 A flowchart of a panoramic image generation method provided by an embodiment of the present application is shown. Figure 1 The processor 11 of the panoramic camera system 10 shown in FIG. Figure 2 As shown, the method includes the following steps: Step 100: Obtain the initial pose relationship between the lidar and each camera, the point cloud features scanned by the lidar, and the image features of the original image taken by each camera. The initial pose relationship includes a rotation matrix and a translation vector.

[0029] The initial pose relationship between the lidar and each camera can be expressed by formula (1): , (1) Among them, P_lidar represents the point cloud coordinates in the LiDAR coordinate system, and P represents the converted point cloud coordinates, that is, the point cloud coordinates in the LiDAR coordinate system are converted to the three-dimensional coordinates in the camera coordinate system. represents the rotation matrix, Represents the translation vector.

[0030] The initial pose relationship between the LiDAR and each camera can be obtained through a calibration process. Specifically, the relative physical position between the LiDAR and each camera can be measured, and then the initial pose relationship between the LiDAR and each camera can be determined based on the relative physical position. For example, Figure 1 As shown, when there is a 30° rotation between the laser radar 13 and the first camera 14a, and the first camera 14a is located 10 cm in front of the laser radar 13 and 5 cm above it, the rotation matrix in the initial posture relationship between the laser radar 13 and the first camera 14a is and translation vectors for: =[ - 0 0 0 1], =[10,5,0].

[0031] Point cloud features are extracted from 3D point cloud data scanned by LiDAR and are used to describe the geometry and surface properties of objects in the point cloud. Point cloud features include geometric features, local feature descriptors, and voxel-based features. Geometric features include point coordinates (X, Y, Z), normal vectors, and curvature. Local feature descriptors, such as SHOT (Signature of Histograms of Orientations), describe the geometry of local regions in the point cloud. Point cloud features are typically represented as 3D point coordinates and local feature descriptors.

[0032] Image features are extracted from the original two-dimensional image and are typically used to describe information such as the shape, texture, and edges of objects in the image. These features typically include corner features, SIFT (Scale-Invariant Feature Transform) features, SURF (Speeded-Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF). SIFT features can be used to extract key points (locations) and their descriptors (feature vectors) in an image. Image features are typically represented as key points and descriptors.

[0033] Step 200: For the laser radar and each camera, match the point cloud features and image features to obtain feature point pairs consisting of the point cloud features and image features.

[0034] Specifically, the point cloud features and image features can be matched according to the nearest neighbor algorithm, which is as follows: min||F_image-F_pointcloud||, (2) Among them, F_image represents the point cloud feature, F_pointcloud represents the image feature, and min|| || represents the minimum distance between two features.

[0035] After feature matching, multiple feature point pairs between the laser radar and each camera can be obtained, for example, multiple feature point pairs between the laser radar 13 and the first camera 14a, and multiple feature point pairs between the laser radar 13 and the second camera 14b.

[0036] Step 300: Optimize the rotation matrix and translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pair and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector.

[0037] Specifically, algorithms such as KC (K-means Clustering), CPD (Coherent Point Drift), RPM (Robust Point Matching), and ICP (Iterative Closest Point) can be used to finely align the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to optimize the rotation matrix and translation vector.

[0038] When using the ICP algorithm to align point cloud coordinates with camera coordinates, step 300 specifically includes the following steps: Step 310: Determine the point cloud coordinates corresponding to the point cloud feature and the camera coordinates corresponding to the image feature in the feature point pair.

[0039] For each feature point pair, the point cloud coordinates of the point cloud feature and the camera coordinates corresponding to the image feature can be determined.

[0040] For example, for the i-th feature point pair F_i between the laser radar 13 and the first camera 14a, the i-th point cloud coordinate corresponding to the point cloud feature F_image can be determined. , and the i-th camera coordinates corresponding to the image feature F_pointcloud .

[0041] Step 320: transform the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain the transformed point cloud coordinates.

[0042] The point cloud coordinates can be transformed into the camera coordinate system through the pose relationship to obtain the transformed point cloud coordinates. Specifically, the i-th point cloud coordinates can be converted into the camera coordinate system through formula (1). Convert to the camera coordinate system to obtain the converted i-th point cloud coordinates .

[0043] In this way, the point cloud coordinates corresponding to the point cloud features in each pair of feature points can be converted to the camera coordinate system to obtain the converted point cloud coordinates. In this way, for each feature point pair between the laser radar 13 and the first camera 14a, the camera coordinates and the converted point cloud coordinates corresponding to each feature point pair can be obtained.

[0044] Step 330: Determine the registration errors of all feature point pairs based on the camera coordinates and the converted point cloud coordinates.

[0045] By optimizing the rotation matrix R and translation vector T to minimize the sum of the Euclidean distances between the camera coordinates and the transformed point cloud coordinates, accurate alignment of the camera coordinates and the point cloud coordinates can be achieved.

[0046] Among them, the registration error of all feature point pairs is for: , (3) in, represents the i-th camera coordinate, represents the coordinates of the i-th point cloud, represents the rotation matrix, represents the translation vector, Represents the coordinates of the i-th transformed point cloud.

[0047] Step 340: Optimize the rotation matrix and translation vector according to the registration error to obtain the registration rotation matrix and the registration translation vector.

[0048] Specifically, step 340 includes the following steps: Step 341: Obtain the maximum optimization times of the rotation matrix and translation vector.

[0049] Step 342: Determine whether the registration error is less than a preset registration error. If so, execute step 343; if not, execute step 344.

[0050] Step 343: Determine the rotation matrix and translation vector that determine the registration error as the registration rotation matrix and the registration translation vector.

[0051] Step 344: Optimize the rotation matrix and translation vector, and determine the optimized rotation matrix and translation vector as the rotation matrix and translation vector, and go to execute step 320 until the number of optimizations of the rotation matrix and translation vector reaches the maximum number of optimizations, and determine the rotation matrix and translation vector obtained by the last optimization as the rotation matrix and translation vector.

[0052] First, the camera coordinates and transformed point cloud coordinates corresponding to all feature point pairs obtained in step 320, as well as the rotation matrix in the initial pose relationship obtained in step 100, are converted into and translation vectors Substituting into the above formula (3), we can get the registration error of all feature point pairs , in the registration error When the preset registration error is not met, the rotation matrix and translation vector are optimized to obtain the optimized rotation matrix after the first optimization. and optimize the translation vector .

[0053] Then all feature points are matched to the corresponding camera coordinates and the transformed point cloud coordinates, as well as the optimized rotation matrix and optimize the translation vector Substituting into the above formula (3), we can get the registration error of all feature point pairs , in the registration error If the preset registration error is not met, continue to optimize the rotation matrix and translation vector to obtain the optimized rotation matrix after the second optimization. and optimize the translation vector ...until the registration error of all feature point pairs is When the preset registration error is met, the optimization of the rotation matrix and translation vector will be stopped, and the registration error will be determined. The corresponding optimized rotation matrix and optimize the translation vector Determined as the registration rotation matrix and quasi-translation vector ; Or, when the number of optimizations of the rotation matrix and translation vector reaches the maximum number of optimizations, stop optimizing the rotation matrix and translation vector, and use the optimized rotation matrix obtained by the last optimization and optimize the translation vector Determined as the registration rotation matrix and the registration translation vector .

[0054] By iteratively optimizing the rotation matrix and translation vector through steps 341 to 343, the registration error can be minimized, and ultimately an accurate registration rotation matrix and registration translation vector can be obtained. Moreover, by setting a preset registration error and a maximum number of optimization times, an infinite loop of the optimization operation can be avoided.

[0055] After the above steps 310 to 340, the rotation matrix and translation vector can be optimized several times to minimize the registration error of all feature point pairs, and finally the optimal registration rotation matrix and registration translation vector are obtained to achieve high-precision registration of point cloud coordinates and camera coordinates.

[0056] Step 400: Determine the pixel coordinates of each original image and the depth corresponding to the pixel coordinates based on the point cloud coordinates, the registration rotation matrix, and the registration translation vector in the feature point pair.

[0057] Specifically, step 400 includes the following steps: Step 410: Obtain the intrinsic parameter matrix of each camera.

[0058] The intrinsic parameter matrix of each camera can be expressed as K_i.

[0059] Step 420: transform the point cloud coordinates in the feature point pair into the camera coordinate system according to the point cloud coordinates, the registration rotation matrix, and the registration translation vector in the feature point pair to obtain transformed point cloud coordinates.

[0060] For each feature point pair, the point cloud coordinates (X_lidar, Y_lidar, Z_lidar) can be converted to the camera coordinate system using formula (1). Specifically, the point cloud coordinates (X_lidar, Y_lidar, Z_lidar), the registration rotation matrix and the registration translation vector Substituting into formula (1), we can get the transformed point cloud coordinates (X_camera, Y_camera, Z_camera).

[0061] Step 430: Determine the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the intrinsic parameter matrix and the transformed point cloud coordinates.

[0062] After obtaining the transformed point cloud coordinates (X_camera, Y_camera, Z_camera), multiply the transformed point cloud coordinates (X_camera, Y_camera, Z_camera) by the intrinsic parameter matrix K_i to obtain the pixel coordinates (u_i, vi_i) of the original image. The depth corresponding to the pixel coordinates (u_i, vi_i) is the Z_camera of the transformed point cloud coordinates.

[0063] In this way, through steps 410 to 430 , the point cloud coordinates of each feature point pair can be determined one by one, and then all pixel coordinates of the original image and the depths corresponding to the pixel coordinates can be determined.

[0064] Step 500: Determine the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates.

[0065] Specifically, step 500 includes the following steps: Step 510: Obtain the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera, and the virtual internal parameter matrix.

[0066] The optical center coordinates of the camera are the coordinates of the optical center of the camera, which can be expressed as C_i.

[0067] In the embodiments of the present application, the panoramic camera is a virtual camera. The virtual optical center coordinates of the panoramic camera can be represented as C_virtual, and the virtual intrinsic parameter matrix can be represented as K_virtual. The virtual intrinsic parameter matrix of the panoramic camera can be the same as or different from the intrinsic parameter matrix of any camera in the panoramic camera system.

[0068] In some embodiments, for example, when the panoramic camera includes an odd number of cameras and the multiple cameras are symmetrically arranged, the panoramic camera can be the camera located at the symmetry center among the multiple cameras, and the virtual optical center coordinates of the panoramic camera are also the optical center coordinates of the camera.

[0069] Step 520: Determine the three-dimensional coordinates corresponding to the pixel coordinates according to the intrinsic parameter matrix, the pixel coordinates, and the depth corresponding to the pixel coordinates.

[0070] Specifically, the three-dimensional coordinates corresponding to the pixel coordinates can be calculated using the following formula (4): , (4) in, Represents the three-dimensional coordinates corresponding to the i-th pixel coordinates, represents the internal parameter matrix, represents the i-th pixel coordinate, Indicates the depth corresponding to the i-th pixel coordinate; Step 530: Determine the pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates, where the pose relationship includes the optical center rotation matrix and the optical center translation vector.

[0071] After determining the optical center coordinates C_i of each camera and the virtual optical center coordinates C_virtual of the panoramic camera, the pose relationship between the optical center coordinates C_i and the virtual optical center coordinates C_virtual can be determined by calibration. The pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates can be expressed by the following formula (5): , (5) Among them, R_virtual represents the optical center rotation matrix, and T_virtual represents the optical center translation vector.

[0072] Step 540: Determine the camera coordinates of the pixel coordinates of each original image in the panoramic camera according to the three-dimensional coordinates, the optical center rotation matrix, and the optical center translation vector.

[0073] After determining the three-dimensional coordinates corresponding to the pixel coordinates of the original image, the camera coordinates P_virtual of the panoramic camera can be calculated using the following formula (6): P_virtual=R_virtual·P_3D,i+T_virtual. (6) Step 550: Determine the target pixel coordinates of each original image pixel coordinate in the panoramic image and the depth corresponding to the target pixel coordinates according to the camera coordinates in the panoramic camera and the virtual intrinsic parameter matrix.

[0074] After obtaining the camera coordinates P_virtual in the panoramic camera, the pixel coordinates (u_virtual, v_virtual) of the original image can be determined according to formula (7). The specific formula is as follows: .

[0075] Assuming that the camera coordinate P_virtual is (X_virtual, Y_virtual, Z_virtual), the depth corresponding to the pixel coordinate (u_virtual, v_virtual) is Z_virtual.

[0076] In this way, through steps 510 to 550, the three-dimensional coordinates corresponding to all pixel coordinates of each original image can be determined one by one, and then the camera coordinates of the pixel coordinates of each original image in the panoramic camera can be determined through the three-dimensional coordinates, so that the target pixel coordinates in the panoramic image and the depth corresponding to the target pixel coordinates can be determined through the camera coordinates, thereby realizing the projection of the pixel coordinates of each original image to the virtual optical center of the panoramic camera.

[0077] Step 600: Generate a panoramic image according to the target pixel coordinates of each original image pixel coordinate in the panoramic image and the depth corresponding to the target pixel coordinates.

[0078] Take the panoramic camera system with two cameras as an example. Figure 1 As shown, after determining the target pixel coordinates and depth of each pixel coordinate of each original image in the panoramic image, when the target pixel coordinates of the pixel coordinates of the original images captured by the first camera 14a and the second camera 14b are different in the panoramic image, the panoramic image can be directly generated according to the target pixel coordinates and the corresponding depths in the panoramic image; when the target pixel coordinates 1 of the original image captured by the first camera 14a are (200, 300) in the panoramic image, and the depth 1 corresponding to the target pixel coordinates 1 is 5 cm, and the target pixel coordinates 2 of the original image captured by the second camera 14b are (200, 300) in the panoramic image, and the depth 2 corresponding to the target pixel 2 is 5 cm, then the depth corresponding to the target pixel coordinates (200, 300) in the panoramic image is 5 cm.

[0079] After determining all target pixel coordinates and the depths corresponding to the target pixel coordinates in the panoramic image, each target pixel coordinate and its corresponding depth are combined to generate a panoramic image.

[0080] In some embodiments, step 600 specifically includes the following steps: Step 610: When pixel coordinates of at least two original images have the same target pixel coordinates in the panoramic image, and the depths corresponding to the target pixel coordinates are not equal, calculate an average value of the depths corresponding to the at least two target pixel coordinates.

[0081] For example, when pixel coordinate 1 of the original image captured by the first camera 14 a corresponds to target pixel coordinate 1 at (200, 300) in the panoramic image, and depth 1 corresponding to target pixel coordinate 1 is 2 cm, and pixel coordinate 2 of the original image captured by the second camera 14 b corresponds to target pixel coordinate 2 at (200, 300) in the panoramic image, and depth 2 corresponding to target pixel 2 is 5 cm, the average value of depth 1 and depth 2 is calculated, resulting in an average value of 3.5 cm for the depths corresponding to target pixel coordinate 1 and target pixel coordinate 2.

[0082] Step 620: Re-determine the average value of the depths corresponding to the at least two target pixel coordinates as the depth corresponding to the target pixel coordinate in the panoramic image.

[0083] The target pixel coordinate (200, 300) of the panoramic image ultimately corresponds to a depth of 3.5 cm.

[0084] Step 630: Generate a panoramic image according to the target pixel coordinates of each original image pixel coordinate in the panoramic image and the depth corresponding to the target pixel coordinates.

[0085] By calculating the average value of the depths corresponding to at least two target pixel coordinates, depth conflicts in the panoramic image synthesis process can be resolved, the accuracy and consistency of the depth information can be improved, and thus the quality of the panoramic image can be improved.

[0086] In the embodiment of the present application, the point cloud features scanned by the laser radar are matched with the image features of the original image captured by each camera to obtain feature point pairs consisting of point cloud features and image features. The rotation matrix and translation vector in the initial pose relationship between the laser radar and each camera are optimized using the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector. The pixel coordinates, registration rotation matrix, and registration translation vector in the feature point pairs can then be used to determine the pixel coordinates of each original image and their corresponding depth. The target pixel coordinates of each original image's pixel coordinates in the panoramic image and their corresponding depth can then be determined using the pixel coordinates of each original image and their corresponding depth. Finally, a panoramic image can be generated based on the target pixel coordinates of each original image's pixel coordinates in the panoramic image and their corresponding depth. In this way, the depth corresponding to the pixel coordinates of the original image captured by each camera can be determined, so that the original images captured by multiple cameras with non-overlapping optical centers can be projected onto the same optical center based on the depth, ensuring the consistency of the geometric structure and color of the panoramic image generated by the panoramic camera system and improving the quality of the panoramic image.

[0087] Figure 3 FIG. 1 shows a schematic diagram of the structure of a panoramic image generation device provided by an embodiment of the present application. Figure 3As shown, the panoramic image generating device 700 includes: an acquisition module 710 , a matching module 720 , an optimization module 730 , a first determination module 740 , a second determination module 750 and a generation module 760 .

[0088] The acquisition module 710 is used to obtain the initial pose relationship between the laser radar and each camera, the point cloud features scanned by the laser radar, and the image features of the original image captured by each camera. The initial pose relationship includes a rotation matrix and a translation vector. The matching module 720 is used to match the point cloud features and image features of the laser radar and each camera to obtain feature point pairs consisting of the point cloud features and the image features; The optimization module 730 is used to optimize the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pair and the camera coordinates corresponding to the image features to obtain the registration rotation matrix and the registration translation vector; The first determination module 740 is used to determine the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the feature point pairs, the registration rotation matrix and the registration translation vector; The second determining module 750 is configured to determine the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates based on the pixel coordinates and the depth corresponding to the pixel coordinates; The generating module 760 is configured to generate a panoramic image according to the target pixel coordinates of each original image in the panoramic image and the depth corresponding to the target pixel coordinates.

[0089] The panoramic image generation device 700 provided in this embodiment is used to execute the technical solution of the panoramic image generation method in the aforementioned method embodiment. Its implementation principle and technical effects are similar and will not be described in detail here.

[0090] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned panoramic image generation method embodiment is implemented.

[0091] An embodiment of the present application provides a computer program, which can be executed by a processor to implement the above-mentioned panoramic image generation method embodiment.

[0092] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned panoramic image generation method embodiment is implemented.

[0093] In the several embodiments provided in this application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, part or all of the technical solution of this application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or other electronic device) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store computer program code.

[0094] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0095] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims that list several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0096] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A panoramic image generation method, applied to a panoramic camera system, characterized in that: The panoramic camera system includes a laser radar and at least two cameras, and the method includes: Acquire an initial pose relationship between the laser radar and each of the cameras, point cloud features scanned by the laser radar, and image features of the original image captured by each of the cameras, wherein the initial pose relationship includes a rotation matrix and a translation vector; For the laser radar and each of the cameras, matching the point cloud features with the image features to obtain feature point pairs consisting of the point cloud features and the image features; Optimizing the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector; Determining the pixel coordinates of each of the original images and the depth corresponding to the pixel coordinates according to the point cloud coordinates of the feature point pairs, the registration rotation matrix, and the registration translation vector; Determining, according to the pixel coordinates and the depth corresponding to the pixel coordinates, the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates; The panoramic image is generated according to the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates.

2. The method according to claim 1, characterized in that The step of matching the point cloud features and the image features with respect to the laser radar and each of the cameras to obtain feature point pairs consisting of the point cloud features and the image features further includes: For the laser radar and each of the cameras, the point cloud features and the image features are matched according to a nearest neighbor algorithm to obtain feature point pairs consisting of the point cloud features and the image features; wherein the nearest neighbor algorithm is: min||F_image-F_pointcloud||, Wherein, F_image represents the point cloud feature, F_pointcloud represents the image feature, and min|| || represents the minimum distance between two features.

3. The method according to claim 1, characterized in that The optimizing the rotation matrix and the translation vector based on the point cloud coordinates corresponding to the point cloud features in the feature point pairs and the camera coordinates corresponding to the image features to obtain a registration rotation matrix and a registration translation vector further includes: Determine the point cloud coordinates corresponding to the point cloud feature in the feature point pair and the camera coordinates corresponding to the image feature; Convert the point cloud coordinates into a camera coordinate system according to the rotation matrix and the translation vector to obtain converted point cloud coordinates; Determining the registration errors of all feature point pairs based on the camera coordinates and the converted point cloud coordinates; The rotation matrix and the translation vector are optimized according to the registration error to obtain the registration rotation matrix and the registration translation vector.

4. The method according to claim 3, characterized in that Optimizing the rotation matrix and the translation vector according to the registration error to obtain the registration rotation matrix and the registration translation vector further includes: Obtaining a maximum number of optimizations of the rotation matrix and the translation vector; Determining whether the registration error is less than a preset registration error; If the registration error is less than the preset registration error, determining the rotation matrix and the translation vector for determining the registration error as the registration rotation matrix and the registration translation vector; If the registration error is greater than or equal to the preset registration error, the rotation matrix and the translation vector are optimized, and the optimized rotation matrix and translation vector are determined as the rotation matrix and the translation vector, and the step of converting the point cloud coordinates into the camera coordinate system according to the rotation matrix and the translation vector to obtain the converted point cloud coordinates is performed until the number of optimizations of the rotation matrix and the translation vector reaches the maximum number of optimizations, and the rotation matrix and translation vector obtained by the last optimization are determined as the rotation matrix and the translation vector.

5. The method according to claim 3, characterized in that The registration error of all feature point pairs for: ， in, represents the i-th camera coordinate, represents the coordinates of the i-th point cloud, represents the rotation matrix, represents the translation vector, Represents the coordinates of the i-th transformed point cloud.

6. The method according to claim 1, characterized in that The step of determining the pixel coordinates of each original image and the depth corresponding to the pixel coordinates according to the point cloud coordinates in the feature point pair, the registration rotation matrix, and the registration translation vector further includes: Obtaining the intrinsic parameter matrix of each camera; Converting the point cloud coordinates in the feature point pair into a camera coordinate system according to the point cloud coordinates in the feature point pair, the registration rotation matrix, and the registration translation vector to obtain converted point cloud coordinates; Determine the pixel coordinates of each of the original images and the depth corresponding to the pixel coordinates according to the intrinsic parameter matrix and the converted point cloud coordinates; The step of determining the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates according to the pixel coordinates and the depth corresponding to the pixel coordinates further includes: Obtaining the optical center coordinates of each camera, the virtual optical center coordinates of the panoramic camera, and the virtual intrinsic parameter matrix; Determine the three-dimensional coordinates corresponding to the pixel coordinates according to the intrinsic parameter matrix, the pixel coordinates, and the depth corresponding to the pixel coordinates; Determine a pose relationship between the optical center coordinates of each camera and the virtual optical center coordinates, wherein the pose relationship includes an optical center rotation matrix and an optical center translation vector; Determine the camera coordinates of the pixel coordinates of each of the original images in the panoramic camera according to the three-dimensional coordinates, the optical center rotation matrix, and the optical center translation vector; The target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates are determined according to the camera coordinates in the panoramic camera and the virtual intrinsic parameter matrix.

7. The method according to claim 6, characterized in that The three-dimensional coordinates corresponding to the pixel coordinates are: ， in, Represents the three-dimensional coordinates corresponding to the i-th pixel coordinates, represents the internal parameter matrix, represents the i-th pixel coordinate, Indicates the depth corresponding to the i-th pixel coordinate; The camera coordinates P_virtual of the panoramic camera are: P_virtual=R_virtual·P_3D,i+T_virtual, Among them, R_virtual represents the optical center rotation matrix, and T_virtual represents the optical center translation vector.

8. The method according to claim 1, characterized in that Generating the panoramic image according to the pixel coordinates of each original image at the target pixel coordinates of the panoramic image and the depth corresponding to the target pixel coordinates further includes: When pixel coordinates of at least two original images are identical in target pixel coordinates of the panoramic image, and depths corresponding to the target pixel coordinates are not equal, calculating an average value of the depths corresponding to the at least two target pixel coordinates; re-determining an average value of the depths corresponding to the at least two target pixel coordinates as the depth corresponding to the target pixel coordinate in the panoramic image; The panoramic image is generated according to the target pixel coordinates of each pixel coordinate of the original image in the panoramic image and the depth corresponding to the target pixel coordinates.

9. A panoramic camera system, characterized in that: Includes a processor, memory, lidar, and at least two cameras; The laser radar is used to scan and obtain point cloud features; The camera is used to capture and obtain the original image; The memory stores a computer program; The processor executes the computer program to implement the panoramic image generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the panoramic image generation method according to any one of claims 1 to 8 is implemented.

Images