Pinhole image stitching true value acquisition and stitching method based on heterogeneous camera combination

Abstract

The application provides a pinhole image splicing true value acquisition and splicing method based on a heterogeneous camera combination. The acquisition method first symmetrically installs two pinhole cameras on a support, and installs an fisheye camera between the two pinhole cameras. Based on a double spherical camera model, the three cameras are calibrated to obtain the internal parameters and external parameters of the three cameras. The left eye pinhole view, the right eye pinhole view and the middle eye fisheye view of the target scene are simultaneously acquired by using the three cameras. The middle eye fisheye view is subjected to equirectangular projection. According to the internal parameters and external parameters of the three cameras, the left eye pinhole view and the right eye pinhole view are projected to the equirectangular projection view plane respectively, multi-camera coordinate system one is completed, and the splicing true value is obtained. The application has the core advantages of the hardware structure design and the heterogeneous camera image registration algorithm, can stably acquire images without seams and time sequence consistency, and provides strong support for the true value reference of the precision verification and model training of subsequent panoramic video splicing.

Description

Technical Field

[0001] This invention relates to the field of panoramic video stitching technology, and in particular to a method for acquiring and stitching true values ​​of pinhole images based on a combination of heterogeneous cameras. Background Technology

[0002] The core purpose of video stitching technology is to seamlessly merge multiple independent video streams to generate a complete panoramic video. Compared to single-stream videos captured by pinhole cameras, stitched panoramic videos have a wider field of view, presenting scene information more comprehensively. Specifically, many applications have high requirements for the completeness of scene capture, such as video surveillance, virtual reality, and the automotive field.

[0003] In video surveillance, wide field-of-view, full-range images are needed to meet the monitoring requirements of broad scenes such as stadiums and intersections. In the field of virtual reality, it is necessary to accurately capture and reconstruct the user's surrounding environment and then fuse it with computer graphics generated by virtual reality applications. This process places extremely high demands on the integrity and accuracy of video stitching, and panoramic video can provide the fundamental support for this fusion process. In the automotive field, both driver assistance and autonomous driving systems need better bird's-eye views and surround-view images of the vehicle's surroundings, and the core of both requires stitching together multiple video views from different directions of the vehicle.

[0004] In recent years, numerous studies both domestically and internationally have yielded various methods for image or video stitching, including keypoint-based grid alignment, depth homography-based alignment, and depth pixel-level alignment. However, existing stitching methods still cannot completely avoid problems such as ghosting, missing regions, distortion, or significant stretching in practical applications. Especially when the overlap area between multiple views is small, stretching and distortion problems become more prominent and difficult to solve effectively. Specifically, a small overlap area means that pixels of the same object may be located inside or outside the overlap area, and pixels located outside may not have a corresponding matching pixel in the reference view. In this case, if the image or video stitching method only considers the alignment accuracy within the overlap area, objects spanning overlapping and non-overlapping areas may not be accurately processed, ultimately leading to visual problems such as object deformation and misalignment, severely affecting the stitching quality. Therefore, a key to optimizing stitching algorithms is the need for realistic, seamless, and full-scene-coverage ground truth stitched images as a reference for algorithm optimization or as a basis for evaluating stitching effects, in order to optimize the alignment accuracy of overlapping areas and the transition processing effect of non-overlapping areas. Currently, a direct method for generating ground truth for the entire scene is to manually and seamlessly stitch together the component images. While this method yields a stitched ground truth image with realistic textures, manually stitching video frame by frame while maintaining temporal consistency remains extremely challenging, even without considering the substantial human cost. Another direct approach is to utilize an environment simulator with a virtual camera to render the component views and the stitched ground truth image. However, such synthetic data typically fails to capture the rich details of real-world textures, which are crucial for learning accurate cross-view correspondences in the stitching model. Therefore, existing ground truth acquisition methods have significant limitations, and there is an urgent need for a high-quality real-world scene dataset that can support fully supervised model training and full-reference quality assessment in video stitching. Summary of the Invention

[0005] In view of the above situation, the main objective of this invention is to propose a method for acquiring and stitching true values ​​of pinhole images based on heterogeneous camera combinations, so as to solve the above-mentioned technical problems.

[0006] This invention proposes a method for acquiring ground truth of pinhole images based on heterogeneous camera combinations, the method comprising the following steps:

[0007] Step 1: Install two pinhole cameras symmetrically on the support, and install a fisheye camera between the two pinhole cameras;

[0008] Step 2: Based on the dual-spherical camera model, calibrate the three cameras to obtain their intrinsic and extrinsic parameters.

[0009] Step 3: Simultaneously capture the left eye pinhole view, right eye pinhole view, and middle eye fisheye view of the target scene using three cameras;

[0010] Step 4: Perform equidistant rectangular projection on the middle fisheye view. Based on the intrinsic and extrinsic parameters of the three cameras, project the left and right pinhole views onto the viewing plane of the equidistant rectangular projection to obtain the projected left pinhole view, right pinhole view, and middle fisheye view. Use the projected middle fisheye view as the stitching truth.

[0011] This invention proposes an image stitching method that utilizes the stitching ground truth acquisition method for pinhole image stitching based on heterogeneous camera combinations, as described above. The image stitching method specifically includes the following steps:

[0012] Select an image stitching model and use the projected left eye pinhole view and the projected right eye pinhole view as stitching inputs;

[0013] The image stitching model is trained in a fully supervised manner using the stitching ground truth as the supervision label. After training, the stitched image is output.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. This invention leverages its core advantages in hardware structure design and heterogeneous camera image registration algorithm to stably acquire seamless, temporally consistent images. Through a customized quadrangular prism structure and a specified heterogeneous camera layout, combined with intrinsic and extrinsic parameter calibration, reprojection alignment, and cropping processing, the mid-eye fisheye camera can serve as a seamless, temporally consistent stitched ground truth image for both left and right eye pinhole cameras.

[0016] 2. Based on a dual-spherical camera model, this invention accurately completes the calibration of intrinsic and extrinsic parameters, providing a precise geometric foundation for subsequent coordinate system calibration, view projection, and data fusion.

[0017] 3. Leveraging the inherently large field of view of fisheye cameras, this invention provides seamless, time-consistent, full-scene real-world labeled data, overcoming the limitations of existing methods that rely on manual stitching or virtual simulation to generate ground truth stitched images. The obtained data can support fully supervised training and full-reference quality assessment of video stitching models, solving the pain point of lacking full-scene real-world labeled data in existing video stitching technologies, and possessing extremely strong practical value.

[0018] 4. This invention first performs equidistant rectangular projection on the fisheye camera video frames, and then precisely projects the video frames from the left and right pinhole cameras onto this equidistant rectangular projection plane. This ensures that the effective imaging field of view of the fisheye camera completely covers the joint shooting range of the two pinhole cameras, serving as the ground truth for stitching the pinhole camera images. This not only achieves spatial alignment and temporal synchronization of multiple camera views on the same plane, but also leverages the large field of view advantage of the fisheye camera to achieve full coverage of the fields of view of the left and right pinhole cameras, providing strong support for subsequent accuracy verification of panoramic video stitching and ground truth reference for model training.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for acquiring the true value of pinhole images based on heterogeneous camera combinations, as proposed in this invention.

[0021] Figure 2 This is a schematic diagram of the framework of a pinhole image stitching truth acquisition method based on heterogeneous camera combination according to the present invention.

[0022] Figure 3 This is a schematic diagram of a quadrangular prism structure with a base angle of 60 degrees, which is the basis of the present invention's method for acquiring true values ​​of pinhole image stitching based on heterogeneous camera combinations.

[0023] Figure 4 This is a schematic diagram illustrating the training and verification results of the CADH method based on the data collected in this invention.

[0024] Figure 5 This is a schematic diagram illustrating the training and verification results of the WSSN method based on the data collected in this invention.

[0025] In the diagram, 1 is the mounting location of the left pinhole camera; 2 is the mounting location of the right pinhole camera; 3 is the mounting location of the fisheye camera; and 4 is the mounting hole for the bracket. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0027] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0028] Please see Figure 1 and Figure 2 This embodiment provides a method for acquiring the true value of pinhole images based on heterogeneous camera combinations. The method includes the following steps:

[0029] Step 1: Install two pinhole cameras symmetrically on the support, and install a fisheye camera between the two pinhole cameras;

[0030] In a preferred embodiment of the present invention, the installation angle of the two pinhole cameras is as follows: Furthermore, the angle between the principal optical axes of the two pinhole cameras remained stable at [value missing]. The perspective of the two pinhole cameras is x ;

[0031] A fisheye camera has a field of view of 180° or greater, and it captures a complete field of view from different angles.

[0032] As a preferred embodiment of the present invention, such as Figure 3 As shown, the support adopts a square prism structure, and the base angle of the square prism structure is... The waist of the quadrangular prism structure serves as the mounting points for the pinhole cameras, specifically mounting points 1 for the left pinhole camera and 2 for the right pinhole camera. The protruding section between these two mounting points is the fisheye camera mounting point 3, and there is also a bracket mounting hole 4 at the top. The fisheye camera is centrally positioned between the two pinhole cameras. To ensure the integrity of the fisheye camera's field of view, the quadrangular prism uses a thickened base. The fisheye camera is mounted in the center of this thickened base, and the thickened design avoids obstruction by the lenses of the pinhole cameras on both sides, thus ensuring the completeness of the fisheye camera's field of view.

[0033] Step 2: Based on the dual-spherical camera model, calibrate the three cameras to obtain their intrinsic and extrinsic parameters.

[0034] As a preferred embodiment of the present invention, based on the dual spherical camera model, the April Grid 6×6 calibration board is selected as the calibration reference. By controlling the calibration board to move in multiple angles and positions within the joint field of view of the three cameras, a sufficient number of calibration images with clear images and complete feature points are captured simultaneously.

[0035] Based on the acquired calibration images, feature points are extracted and matched using the Basalt tool, and then parameter optimization algorithms are used to solve and iteratively optimize the intrinsic and extrinsic parameters of the three cameras.

[0036] The internal parameters of each of the three cameras are expressed as follows:

[0037] ;

[0038] in, This indicates the internal parameters of the left-eye pinhole camera. This indicates the internal parameters of the right-eye pinhole camera. This refers to the internal parameters of a fisheye camera. This indicates the focal length of the left-eye pinhole camera in the x-direction. This indicates the focal length of the left-eye pinhole camera in the y-direction. This indicates the coordinates of the principal point of the left-eye pinhole camera in the x-direction. This indicates the coordinates of the principal point of the left-eye pinhole camera in the y-direction. This indicates the spherical offset parameter of the left eye pinhole camera. This indicates the offset parameter of the projection plane of the left pinhole camera. This indicates the focal length of the right-eye pinhole camera in the x-direction. This indicates the focal length of the right-eye pinhole camera in the y-direction. This indicates the coordinates of the principal point of the right-eye pinhole camera in the x-direction. This indicates the coordinates of the principal point of the right-eye pinhole camera in the y-direction. This indicates the spherical offset parameter of the right pinhole camera. This indicates the offset parameter of the projection plane of the right pinhole camera. This represents the focal length of the fisheye camera in the x-direction. This represents the focal length of the fisheye camera in the y-direction. This represents the coordinates of the principal point of the fisheye camera in the x-direction. This represents the coordinates of the principal point of the fisheye camera in the y-direction. This represents the spherical offset parameter of the fisheye camera. This represents the offset parameter of the fisheye camera's projection plane.

[0039] The extrinsic parameters with reference to the fisheye camera are: the extrinsic parameters from the source camera s to the target camera t are expressed as follows: ; where the extrinsic parameters from the left-eye pinhole camera to the fisheye camera are represented as The extrinsic parameters from the right-eye pinhole camera to the fisheye camera are expressed as follows: .

[0040] Step 3: Simultaneously capture the left eye pinhole view, right eye pinhole view, and middle eye fisheye view of the target scene using three cameras;

[0041] During the acquisition process, it is necessary to ensure that the left-eye pinhole camera, the right-eye pinhole camera, and the fisheye camera simultaneously capture the video stream of the target scene at the same frame rate, fully record the visual information of diverse scenes, and maintain the time synchronization between the video frames of the three cameras to avoid view misalignment caused by acquisition delay.

[0042] Step 4: Perform equidistant rectangular projection on the middle fisheye view. Based on the intrinsic and extrinsic parameters of the three cameras, project the left and right pinhole views onto the viewing plane of the equidistant rectangular projection to obtain the projected left pinhole view, right pinhole view, and middle fisheye view. Use the projected middle fisheye view as the stitching truth.

[0043] After data acquisition, the views from the left and right pinhole cameras and the fisheye camera need to be unified onto the same equidistant rectangular projection view plane. This means finding the corresponding pixel coordinates on the source camera image for each pixel in the target equidistant rectangular projection image. The specific steps are as follows:

[0044] Perform an isometric rectangular projection operation on the fisheye view to obtain the isometric rectangular projection;

[0045] Mapping the pixel coordinates of an equidistant rectangular projection to spherical coordinates yields the spherical coordinates. The corresponding process follows the following relationship:

[0046] ;

[0047] in, Represents spherical coordinates, These represent the height and width of the equidistant rectangular projection, respectively. Represents the pixel coordinates in an equidistant rectangular projection. It represents pi (π).

[0048] Let the depth of the sphere be d. Converting the spherical coordinates to Euclidean coordinates yields the 3D point coordinates in the fisheye camera coordinate system. The corresponding process has the following relationship:

[0049] ;

[0050] in, Represents the spherical coordinates of depth d. This represents the coordinates of a three-dimensional point in the fisheye camera coordinate system.

[0051] Based on the camera's extrinsic parameters, the coordinates of a 3D point in the fisheye camera coordinate system are transformed into the coordinate systems of the left and right pinhole cameras, respectively. This yields the homogeneous coordinates of the same 3D point in the three camera coordinate systems, ensuring that all views are in the same spatial coordinate system. The corresponding process follows the following relationship:

[0052] ;

[0053] Where s represents the source camera, which is either the left-eye pinhole camera or the right-eye pinhole camera; t represents the target camera, which is either the fisheye camera. This represents the inverse of the extrinsic parameter matrix from the source camera to the target camera. This represents the homogeneous coordinates of a 3D point in the target camera coordinate system. This represents the homogeneous coordinates of a 3D point in the source camera coordinate system. This indicates transpose.

[0054] Based on a dual-spherical camera model, the homogeneous coordinates of the same 3D point in the coordinate systems of the three cameras are transformed to the corresponding pixel coordinates in the source camera images of the three cameras using the intrinsic parameters of the source cameras. This allows the views of the three cameras to be precisely aligned onto an equidistant rectangular projection plane, resulting in the projected left-eye pinhole view, right-eye pinhole view, and middle-eye fisheye view. The corresponding process has the following relationship:

[0055] ;

[0056] in, This represents the three-dimensional Euclidean coordinates selected from homogeneous coordinates. This represents the focal length of the source camera along the x-axis. This represents the focal length of the source camera along the y-axis. This represents the pixel coordinates of the source camera's principal point on the image's x-axis. This represents the pixel coordinates of the source camera's principal point on the image's y-axis. This represents the distortion parameters of the source camera. This represents the pixel coordinates of a 3D point on the source camera image. This represents the Euclidean distance from a 3D point to the optical center of the source camera. This represents the virtual projection distance after distortion correction.

[0057] Through the above process, for each pixel point on the equidistant rectangular projection image, its corresponding pixel position can be found on the three original views (left, center, and right), thereby achieving precise alignment of the three camera views onto a unified equidistant rectangular projection viewing plane.

[0058] This embodiment also provides an image stitching method, which utilizes the stitching ground truth acquisition method for pinhole image stitching based on heterogeneous camera combination described above. The image stitching method specifically includes the following steps:

[0059] Select an image stitching model and use the projected left eye pinhole view and the projected right eye pinhole view as stitching inputs;

[0060] The image stitching model is trained in a fully supervised manner using the stitching ground truth as the supervision label. After training, the stitched image is output.

[0061] After image processing is completed using the above method, the left and right pinhole views projected in step 4 are used as the images to be stitched together, and the projected middle fisheye image is used as the supervision label. The deep learning stitching model is then trained under full supervision so that it can generate high-quality stitched images, thereby optimizing the stitching effect.

[0062] Taking the representative deep learning-based image stitching models CADH and WSSN as examples, the method for optimizing the stitching model using the data collected in this invention is as follows:

[0063] 1. CADH:

[0064] CADH is an unsupervised image stitching method designed to alleviate problems such as depth differences and moving targets in real-world applications. This method utilizes a convolutional neural network to predict the offsets of the four corner points of an image, thereby estimating the homography matrix, and performs global distortion on the image to complete the stitching task. Given that the dataset collected in this invention provides accurate ground truth values ​​for stitching, the CADH algorithm is improved to support both unsupervised and supervised training modes.

[0065] The model input consists of the left and right pinhole views, synchronously acquired in step 3 and projected in step 4, with the projected middle-eye fisheye image used as the supervision label. The output dimension of the last layer of the network is adjusted from 8 to 16 to enable it to predict the offsets of eight corner points (four points in the left image and four points in the right image), and then the two homography matrices corresponding to the left and right images are calculated respectively. Based on this, by calculating the loss between the distorted image and the supervision label, effective supervision and optimization of network training are achieved. The corresponding process has the following relationship:

[0066] ;

[0067] in, This represents the loss between the distorted image and the ground truth image. Indicates the height of the images to be stitched together. This indicates the width of the images to be stitched together. Represents the pixels of the source image The value, Represents the pixels of the target image The value of .

[0068] The final result is as follows Figure 4 As shown, Figure 4 The left image in the image shows the stitched effect of the left and right pinhole views from the model after unsupervised training. Figure 4The right image in the figure shows the stitching result of the left and right pinhole views after supervised training using the ground truth collected by this invention;

[0069] The quantitative evaluation results of the left and right pinhole view alignment effect based on CADH are shown in the table below. The evaluation of the overlapping area is obtained by calculating the overlapping area of ​​the two distorted left and right images, and the evaluation result of the whole image is calculated by using the stitched fused image and the stitched ground truth. It can be seen that, whether in the overlapping area or the whole image, the CADH model trained with fully supervised training using stitched ground truth significantly outperforms the unsupervised training model.

[0070]

[0071] 2. WSSN:

[0072] WSSN is a pixel-wise aligned video stitching method. Its default input is a fisheye image, and the model needs to learn calibration parameters to project each fisheye view onto a 360° panoramic view. This invention decouples the calibration process from the local alignment process, so the model only needs to learn the alignment relationships of local textures, thus extending to different camera structures. Secondly, the video overlap region used by WSSN is narrow and the parallax is small. Therefore, its original design only uses perceptual loss for weak supervision in the overlap region, while pixel-level supervision is only applied to the non-overlapping region. If this setting is directly applied to the data collected in this invention, since the data contains a larger overlap region and a large number of close-range, high parallax targets, using perceptual loss only in the overlap region will lead to distortion in the stitching result.

[0073] Therefore, this experiment uses the fisheye ERP image obtained by isometric rectangular projection (ERP) of the mid-eye fisheye view in step 4 as the supervision label. This image was captured and unfolded by a single fisheye camera, exhibiting perfect color consistency (no color difference, no seams) and precisely corresponding to the coverage of the left and right eye pinhole views. During training, the pixel-level L2 loss between the model output and the fisheye ERP image is directly calculated, thus achieving fully supervised training. The model input consists of the left and right eye pinhole views, which were simultaneously acquired in step 3 and projected in step 4. By using the projected mid-eye fisheye image as the supervision label, the fully supervised training model achieves the stitching of the left and right eye pinhole views. In addition, the WSSN's native monotonic quadratic curve color correction module is retained, ensuring that the color distribution of the panoramic prediction result remains consistent with the supervision label.

[0074] Because fisheye lenses introduce significant barrel distortion to achieve a large field of view, their image quality is typically lower than that of equivalent pinhole cameras. In the dataset collected in this invention, the image quality of the middle-eye fisheye view is lower than that of the left and right-eye pinhole views. If pixel-wise color correction is used, the model tends to align the detail quality of the stitched result with the fisheye ERP map, thus losing the original high-resolution details of the pinhole camera. To avoid this problem, this experiment only uses global color correction based on learned parameters to ensure that the overall color distribution of the stitched result matches the fisheye ERP. Figure 1 It achieves the same effect while preserving the texture details and clarity of a pinhole camera.

[0075] ;

[0076] in, This represents the image after correction. Represents the original image. These are learnable correction parameters.

[0077] like Figure 5 As shown, Figure 5 The middle left image shows the effect of using perceptual loss only in the overlapping area, which is obviously distorted; Figure 5 The image on the right shows the effect of using the supervision label for full supervision, with even better detail.

[0078]

[0079] The quantitative evaluation results of the alignment effect of the left and right eye pinhole views based on WSSN are shown in the table above. It can be seen that, whether in the overlapping area or the whole image, the WSSN model trained in full supervision using the fisheye ERP map obtained by isometric projection of the middle fisheye view as the supervised label is significantly better than the unsupervised training effect.

[0080] The two sets of experiments above fully demonstrate that the pinhole image stitching ground truth acquisition method based on heterogeneous camera combinations has significant practical value and research significance. This method can simultaneously acquire images from heterogeneous cameras. The images acquired by the left and right pinhole cameras can be used as input data for the image stitching algorithm, while the stitching ground truth can serve as a ground truth reference for seamless stitching and temporal consistency of the stitching result. This solves key problems in traditional stitching tasks, such as the difficulty in obtaining ground truth and the lack of unified evaluation benchmarks.

[0081] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for acquiring ground truth of pinhole images based on heterogeneous camera combinations, characterized in that, The method includes the following steps: Step 1: Install two pinhole cameras symmetrically on the support, and install a fisheye camera between the two pinhole cameras; The installation angles of the two pinhole cameras are as follows: Furthermore, the angle between the principal optical axes of the two pinhole cameras remained stable at [value missing]. The perspective of the two pinhole cameras is x ; A fisheye camera has a field of view of 180° or greater, and the field of view is complete under different angles. Step 2: Based on the dual-spherical camera model, calibrate the three cameras to obtain their intrinsic and extrinsic parameters. Step 3: Simultaneously capture the left eye pinhole view, right eye pinhole view, and middle eye fisheye view of the target scene using three cameras; Step 4: Perform equidistant rectangular projection on the fisheye view of the middle eye. Based on the intrinsic and extrinsic parameters of the three cameras, project the left eye pinhole view and the right eye pinhole view onto the viewing plane of the equidistant rectangular projection to obtain the projected left eye pinhole view, right eye pinhole view and middle eye fisheye view. Use the projected middle eye fisheye view as the stitching truth. The projected left-eye pinhole view and the projected right-eye pinhole view serve as the stitching inputs to the image stitching model, while the stitching ground truth serves as the supervision label for the image stitching model.

2. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 1, characterized in that, The support adopts a square prism structure, and the base angle of the square prism structure is... The waist of the quadrangular prism structure is the mounting point for the pinhole camera, and the fisheye camera is centrally positioned between the two pinhole cameras.

3. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 2, characterized in that, In step 3, the time synchronization between the video frames of the left eye pinhole view, the right eye pinhole view, and the middle eye fisheye view is consistent.

4. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 3, characterized in that, In step 4, the fisheye view of the middle eye is subjected to an isometric rectangular projection. Based on the intrinsic and extrinsic parameters of the three cameras, the pinhole view of the left eye and the pinhole view of the right eye are projected onto the viewing plane of the isometric rectangular projection, respectively, to obtain the projected pinhole view of the left eye, the pinhole view of the right eye, and the fisheye view of the middle eye. The projected fisheye view of the middle eye is used as the stitching truth. The specific steps include the following: Perform an isometric rectangular projection operation on the fisheye view to obtain the isometric rectangular projection; The pixel coordinates of the equidistant rectangular projection are mapped to spherical coordinates to obtain spherical coordinates; Let the depth of the sphere be d. Convert the spherical coordinates to Euclidean coordinates to obtain the three-dimensional point coordinates in the fisheye camera coordinate system. Based on the camera's external parameters, the coordinates of a 3D point in the fisheye camera coordinate system are transformed into the coordinate systems of the left and right pinhole cameras, respectively, to obtain the homogeneous coordinates of the same 3D point in the three camera coordinate systems. Based on the dual-spherical camera model, the homogeneous coordinates of the same 3D point in the coordinate systems of the three cameras are transformed to the corresponding pixel coordinates on the source camera images of the three cameras using the intrinsic parameters of the source cameras of the three cameras. This allows the views of the three cameras to be precisely aligned onto the equidistant rectangular projection plane, resulting in the projected left eye pinhole view, right eye pinhole view, and middle eye fisheye view.

5. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 4, characterized in that, Mapping pixel coordinates of an equidistant rectangular projection to spherical coordinates follows the following relationship: ； in, Represents spherical coordinates, These represent the height and width of the equidistant rectangular projection, respectively. Represents the pixel coordinates in an equidistant rectangular projection. It represents pi (π).

6. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 5, characterized in that, Let the depth of the sphere be d. Converting the spherical coordinates to Euclidean coordinates yields the 3D point coordinates in the fisheye camera coordinate system. The corresponding process has the following relationship: ； in, Represents the spherical coordinates of depth d. This represents the coordinates of a three-dimensional point in the fisheye camera coordinate system.

7. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 6, characterized in that, Based on the camera extrinsic parameters, the coordinates of a 3D point in the fisheye camera coordinate system are transformed into the coordinate systems of the left and right pinhole cameras, respectively, to obtain the homogeneous coordinates of the same 3D point in the three camera coordinate systems. The corresponding process has the following relationship: ； Where s represents the source camera, which is either the left-eye pinhole camera or the right-eye pinhole camera; t represents the target camera, which is either the fisheye camera. This represents the inverse of the extrinsic parameter matrix from the source camera to the target camera. This represents the homogeneous coordinates of a 3D point in the target camera coordinate system. This represents the homogeneous coordinates of a 3D point in the source camera coordinate system. This indicates transpose.

8. The method for acquiring true values ​​of pinhole images based on heterogeneous camera combinations according to claim 7, characterized in that, Based on the dual-spherical camera model, the homogeneous coordinates of the same 3D point in the coordinate systems of the three cameras are transformed to the corresponding pixel coordinates in the source camera images of the three cameras using the intrinsic parameters of the source cameras. The corresponding process has the following relationship: ； in, This represents the three-dimensional Euclidean coordinates selected from homogeneous coordinates. This represents the focal length of the source camera along the x-axis. This represents the focal length of the source camera along the y-axis. This represents the pixel coordinates of the source camera's principal point on the image's x-axis. This represents the pixel coordinates of the source camera's principal point on the image's y-axis. This represents the distortion parameters of the source camera. This represents the pixel coordinates of a 3D point on the source camera image. This represents the Euclidean distance from a 3D point to the optical center of the source camera. This represents the virtual projection distance after distortion correction.

9. An image stitching method, comprising a projected left-eye pinhole view, a right-eye pinhole view, and a middle-eye fisheye view acquired using the pinhole image stitching truth acquisition method based on heterogeneous camera combination as described in any one of claims 1 to 8, characterized in that, The image stitching method specifically includes the following steps: Select an image stitching model and use the projected left eye pinhole view and the projected right eye pinhole view as stitching inputs; The image stitching model is trained in a fully supervised manner using the stitching ground truth as the supervision label. After training, the stitched image is output.

Images