Panoramic stitching method and device based on multi-camera, equipment and storage medium

Abstract

This application discloses a panoramic stitching method, apparatus, device, and storage medium based on multi-camera systems, relating to the field of image processing technology. The method includes: acquiring calibration images collected by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; identifying calibration board corner features in the calibration images and obtaining calibration parameters for each camera based on these features; generating a configuration file according to the calibration parameters; and processing the video streams collected by the multiple cameras based on the configuration file to obtain a panoramic video stream. This method solves the problems of low calibration accuracy and reliance on manual experience and complex online calculations in existing technologies under long-distance, large field-of-view conditions. It achieves standardization and automation of the calibration process and enables efficient, high-precision real-time panoramic video stitching on embedded devices, reducing the system's dependence on high-performance processing platforms and overall cost.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a panoramic stitching method, apparatus, device and storage medium based on multi-view cameras. Background Technology

[0002] With the rapid development of technologies such as remote conferencing and panoramic monitoring, the market demand for high-definition, seamless panoramic video is growing, especially in real-time video stitching scenarios with long distances, large fields of view, and multi-camera systems, where the requirements for geometric alignment accuracy and visual continuity are becoming increasingly stringent.

[0003] Current mainstream technical solutions mainly include those based on online calibration and real-time stitching. The calibration process relies on operators repeatedly moving a small calibration board within the camera's field of view. The computing platform simultaneously acquires multi-view images and runs complex calibration algorithms. The process requires manual intervention and is computationally complex, making the calibration accuracy significantly affected by the size of the calibration board, motion stability, and environmental factors. Especially in long-distance scenes, the small imaging size of the small calibration board and the reduced corner detection accuracy lead to inaccurate calibration of camera external parameters, stitching misalignment, or ghosting phenomena, making it difficult to meet the requirements of high-precision, large-scale panoramic stitching.

[0004] Therefore, existing technologies have the technical problem of being unable to perform high-precision calibration of multi-camera systems under long-distance and wide-field-of-view conditions, resulting in insufficient geometric alignment accuracy of subsequent real-time panoramic stitching and affecting the stitching visual effect.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a panoramic stitching method, apparatus, device, and storage medium based on multi-view cameras, aiming to solve the technical problems corresponding to the background technology.

[0007] To achieve the above objectives, this application proposes a panoramic stitching method based on multi-view cameras, the method comprising:

[0008] Acquire calibration images from multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; Identify the corner features of the calibration board in the calibration image, and obtain the calibration parameters of each camera based on the corner features of the calibration board; Generate a configuration file based on the calibration parameters; Based on the configuration file, the video streams captured by the multiple cameras are processed to obtain a panoramic video stream.

[0009] In one embodiment, acquiring calibration images from multiple cameras includes: Multiple calibration plates are pre-arranged, wherein the calibration plates are respectively set at a preset distance from the multiple cameras; Based on the optical centers of the plurality of cameras, adjust the position of each calibration plate so that the normal direction of the surface of the calibration plate points to the optical center; After the adjustment is completed, each camera is controlled to acquire calibration images according to preset shooting parameters. The calibration images of adjacent cameras include a common calibration area formed by the same calibration plate.

[0010] In one embodiment, identifying the calibration board corner features in the calibration image and obtaining the calibration parameters of each camera based on the calibration board corner features includes: Identify the calibration board corner points within the common calibration area in the calibration image to obtain the calibration board corner point features; Between the calibration images of adjacent cameras, the corner features of the calibration board are matched to obtain matching point pairs; Based on the matching point pairs, the initial transformation matrix between adjacent cameras is obtained; The initial transformation matrix is ​​unified to the global coordinate system to obtain the external parameters of the adjacent cameras; Based on the preview requirements and the matching point pairs, the internal parameters of each camera, the corner features of the calibration board, and the external parameters are jointly optimized to obtain the calibration parameters of each camera.

[0011] In one embodiment, the joint optimization of the intrinsic parameters of each camera, the corner features of the calibration board, and the extrinsic parameters based on the preview requirements and the matching point pairs to obtain the calibration parameters of each camera includes: Based on the matching point pairs, the internal parameters of each camera, the corner features of the calibration board, and the external parameters are jointly optimized to obtain the optimized camera parameters of each camera. A panoramic stitched preview image is generated based on the optimized camera parameters; When the panoramic stitching preview does not meet the preview requirements, in response to the operator's instruction to add or delete matching points in the stitching seam area of ​​the panoramic stitching preview, the calibration board corner point features in the optimized camera parameters are updated. Based on the updated calibration board corner features, the process returns to the step of jointly optimizing the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pairs to obtain optimized camera parameters. This process continues until the panoramic stitching preview image meets the preview requirements, at which point the optimized camera parameters of each camera are used as the calibration parameters of the corresponding camera.

[0012] In one embodiment, the joint optimization of the intrinsic parameters of each camera, the corner features of the calibration board, and the extrinsic parameters based on the matching point pairs to obtain optimized camera parameters for each camera includes: The internal parameters, calibration board corner features, and external parameters of each camera are used as optimization variables for the corresponding camera. The objective function is determined based on the matching point pairs; An optimization model is established based on the objective function and the optimization variables; The optimization model is iteratively solved to obtain the optimization results; Based on the optimization results, the optimized camera parameters are obtained.

[0013] In one embodiment, generating the configuration file based on the calibration parameters includes: Extract the lens distortion coefficients, internal parameter matrix, and external attitude angles for each camera from the calibration parameters; The horizontal field of view of each camera is obtained based on the image sensor information and the internal parameter matrix. The lens distortion coefficients, the internal parameter matrix, the external attitude angle, and the horizontal field of view are organized according to a preset format to generate a configuration file.

[0014] In one embodiment, processing the video streams captured by the plurality of cameras based on the configuration file to obtain a panoramic video stream includes: Acquire the video streams captured by the multiple cameras; Distortion correction parameters and perspective transformation parameters are generated based on the calibration parameters in the configuration file; Based on the distortion correction parameters, distortion correction is performed on each frame of the video stream to obtain a distortion-free normalized image. Based on the perspective transformation parameters, the distortion-free normalized image at the same moment is mapped onto the panoramic canvas, and image fusion is performed based on the overlapping area to obtain a single-frame panoramic image. The single-frame panoramic image is encoded to obtain a panoramic video stream.

[0015] Furthermore, to achieve the above objectives, this application also proposes a panoramic stitching device based on a multi-view camera, the panoramic stitching device based on a multi-view camera comprising: An image acquisition module is used to acquire calibration images captured by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; The parameter calibration module is used to identify the corner features of the calibration board in the calibration image and obtain the calibration parameters of each camera based on the corner features of the calibration board. The configuration file module is used to generate a configuration file based on the calibration parameters; The panoramic video module is used to process the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream.

[0016] In addition, to achieve the above objectives, this application also proposes a panoramic stitching device based on a multi-view camera, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the panoramic stitching method based on a multi-view camera as described above.

[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the panoramic stitching method based on multi-view cameras as described above.

[0018] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the panoramic stitching method based on a multi-view camera as described above.

[0019] This application acquires calibration images from multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; identifies calibration board corner features in the calibration images, and obtains calibration parameters for each camera based on the calibration board corner features; generates a configuration file based on the calibration parameters; and processes the video streams acquired by the multiple cameras based on the configuration file to obtain a panoramic video stream. This solves the problems of low calibration accuracy and reliance on manual experience and complex online calculations in existing technologies under long-distance, large field-of-view conditions, achieving standardization and automation of the calibration process, and realizing efficient and high-precision real-time panoramic video stitching on embedded devices, reducing the system's dependence on high-performance processing platforms and overall cost. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1This is a flowchart illustrating an embodiment of the panoramic stitching method based on multi-view cameras provided in this application. Figure 2 This is a calibration scene diagram provided in Embodiment 1 of the panoramic stitching method based on multi-view cameras in this application; Figure 3 This is a panoramic stitching operation interface provided in Embodiment 1 of the panoramic stitching method based on multi-view cameras in this application; Figure 4 This is a side view of a six-channel camera module provided in Embodiment 1 of the panoramic stitching method based on multi-view cameras in this application; Figure 5 This is a flowchart illustrating Embodiment 2 of the panoramic stitching method based on multi-view cameras provided in this application; Figure 6 This is the second panoramic stitching operation interface provided in Embodiment 2 of the panoramic stitching method based on multi-view cameras in this application; Figure 7 This is a schematic diagram of the module structure of the panoramic stitching device based on a multi-view camera according to an embodiment of this application; Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the panoramic stitching method based on multi-view cameras in the embodiments of this application.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0025] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0026] The main solution of this application embodiment is: to acquire calibration images captured by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; to identify calibration board corner features in the calibration images, and to obtain calibration parameters of each camera based on the calibration board corner features; to generate a configuration file according to the calibration parameters; and to process the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream.

[0027] In this embodiment, for ease of description, the following description will focus on a panoramic stitching system based on a multi-view camera.

[0028] Since the current mainstream technical solutions mainly include those based on online calibration and real-time stitching, the calibration process relies on operators repeatedly moving a small calibration board within the camera's field of view. The computing platform simultaneously acquires multi-view images and runs complex calibration algorithms. The process requires manual intervention and is computationally complex, making the calibration accuracy significantly affected by the size of the calibration board, motion stability, and environmental factors. Especially in long-distance scenes, the small imaging size of the small calibration board and the reduced corner detection accuracy lead to inaccurate calibration of camera external parameters, stitching misalignment, or ghosting phenomena, making it difficult to meet the requirements of high-precision, large-scale panoramic stitching.

[0029] This application provides a solution that addresses the problems of low calibration accuracy and reliance on manual experience and complex online calculations in existing technologies under long-distance, wide-field-of-view conditions. It achieves standardization and automation of the calibration process and enables efficient and high-precision real-time panoramic video stitching on embedded devices, reducing the system's dependence on high-performance processing platforms and overall cost.

[0030] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a panoramic stitching system based on a multi-view camera. The following description uses a panoramic stitching system based on a multi-view camera as an example to illustrate this embodiment and the subsequent embodiments.

[0031] Based on this, embodiments of this application provide a panoramic stitching method based on multi-view cameras, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the panoramic stitching method based on multi-view cameras in this application.

[0032] In this embodiment, the panoramic stitching method based on multi-view cameras includes steps S10~S40: Step S10: Acquire calibration images from multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; It should be noted that calibration images are static images obtained by taking pictures of a calibration target with a known geometric structure using various cameras. Adjacent cameras refer to two or more cameras whose fields of view overlap in the spatial arrangement. The common calibration area refers to the same calibration plate area that is commonly included in the images taken by these adjacent cameras.

[0033] It is understandable that since the images acquired during the calibration phase contain calibration areas shared by adjacent cameras, performing step S10 can provide a stable and reliable image correspondence for subsequent feature matching, thereby avoiding calibration failures caused by non-overlapping fields of view or discontinuous features, and improving the robustness and accuracy of multi-camera calibration.

[0034] In one feasible implementation, step S10 may include: pre-arranging multiple calibration plates, wherein the calibration plates are respectively set at a preset distance from the multiple cameras; adjusting the position of each calibration plate according to the optical center of the multiple cameras, so that the normal direction of the plate surface points to the optical center; after the adjustment is completed, controlling each camera to acquire calibration images according to preset shooting parameters, wherein the calibration images of adjacent cameras in the calibration images include a common calibration area formed by the same calibration plate.

[0035] It should be noted that the preset distance is the geometric distance between the calibration board and the optical center of the camera, usually within the range of 3 to 5 meters, to simulate the long-distance scene in actual camera operation; the optical center refers to the intersection of the optical axis of the camera lens, which approximately represents the imaging center of the camera; the preset shooting parameters include manually set fixed exposure time, aperture value, ISO and white balance parameters, used to ensure the consistency of brightness and color of multiple images.

[0036] Specifically, by placing the calibration plate at a preset distance and aligning its normal with the optical center of the camera, imaging errors caused by perspective distortion can be reduced; by controlling each camera to use the same preset shooting parameters for image acquisition, image differences introduced by automatic exposure and white balance fluctuations can be eliminated, thereby obtaining a set of calibration images with uniform illumination, clear features, and well-defined geometric relationships.

[0037] For example, such as Figure 2 As shown, six large checkerboard calibration plates, each approximately 2 meters × 0.9 meters in size, can be set up and evenly arranged on a circle with a radius of 3.5 meters, with the optical center of the camera group as the center. Each calibration plate is directly opposite the center of the circle, ensuring that the field of view of each pair of adjacent cameras contains a complete image of the same calibration plate.

[0038] Figure 2 Camera modules in Figure 3 As shown, Figure 3 This is a side view of a six-camera module (only four cameras are shown in the figure; two more are not shown). The camera module includes six independent camera modules, each uniformly embedded in a ring-shaped bracket along the circumference. The optical axis of the lens of each camera module points outward from the ring-shaped bracket and is distributed radially. The angle between adjacent camera modules is 60 degrees, so that the horizontal field of view of each camera forms continuous coverage at the preset distance, and there is an overlapping area between the field of view boundaries of adjacent cameras. The center of the ring-shaped bracket is an optical reference point, which is used as the origin of the global coordinate system and the center reference for calibrating the scene layout.

[0039] After the precise deployment of the calibrated scene is completed, image acquisition is performed. The specific process is as follows: The first step is to uniquely number the six network cameras (1 to 6) and record their corresponding network stream addresses (URLs). Ensure all cameras and the acquisition computer are on the same local area network (LAN), with a stable network connection and sufficient bandwidth to support the simultaneous transmission of six high-definition video streams. Then, preset the camera parameters. To avoid inconsistencies in brightness and color temperature between multiple images due to algorithms such as automatic exposure and automatic white balance, set the shooting mode of all cameras to Manual Mode. Exposure settings: Manually set a fixed shutter speed, aperture value, and ISO sensitivity. The parameters must be determined by observing all six images in preview mode, ensuring all calibration board images are clear, without significant overexposure or underexposure, and that the brightness of all six images is as consistent as possible. This step is crucial, especially outdoors or in environments with changing lighting, as it completely eliminates stitching interference caused by automatic exposure.

[0040] White balance settings: Manually set a fixed color temperature value (such as 5500K) or use the same white paper for manual white balance calibration to ensure consistent color performance of the six images.

[0041] Focus settings: Fix the focus at infinity or manually adjust it to the point where the calibration board image is clearest at 3-5 meters, and lock the focus to prevent the focus from changing during subsequent acquisitions.

[0042] The second step involves ensuring that there is no movement of people or objects within the calibration scene and that lighting conditions are stable during the data acquisition process. Any changes within the overlapping field of view of the cameras will introduce feature point errors, severely impacting subsequent calibration accuracy. Dedicated acquisition software is used, and the URL stream address of each camera is input sequentially. After the video stream is clear and stable, static images or screenshots are captured and saved for each camera view. Saved files should be clearly named according to the camera number (e.g., cam01_calib.jpg, cam02_calib.jpg...). While acquiring each image, carefully check the preview screen to ensure that the calibration board within the overlapping area of ​​adjacent cameras is completely still and clearly visible, ensuring a high-quality feature matching source for the software.

[0043] The third step is to select the images after acquiring them, and then upload them to the specified location. Figure 4 The independently developed panoramic stitching software shown checks all images on a computer to ensure that all calibration boards are completely in the frame and the images are clear; the brightness and color consistency of the six images meet the requirements; and there is no motion blur or unwanted occlusion in the image.

[0044] In this embodiment, by arranging stable target objects and uniformly collecting parameters in large-scale, long-distance scenes, the problems of low calibration accuracy and poor repeatability caused by small calibration board size and inconsistent shooting conditions in the prior art are solved, providing a reliable image data foundation for the subsequent extraction of high-precision panoramic stitching parameters.

[0045] The above are merely feasible implementations of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0046] Step S20: Identify the corner features of the calibration board in the calibration image, and obtain the calibration parameters of each camera based on the corner features of the calibration board; It should be noted that calibration board corner features refer to the corner coordinates of calibration board patterns detected in calibration images that have clear pixel positions and geometric meanings, such as the intersection of black and white squares in a checkerboard pattern; calibration parameters include intrinsic parameters describing the internal imaging characteristics of the camera and extrinsic parameters describing the camera's spatial position and attitude. Intrinsic parameters include focal length, principal point coordinates, distortion coefficients, etc., while extrinsic parameters include rotation matrix and translation vector.

[0047] Specifically, corner detection is performed on each calibration image to extract the pixel coordinates of the corner points of the calibration board in each image. By using the matching relationship of the same physical corner point between adjacent images, multi-view geometric constraints are established. The intrinsic and extrinsic parameters of each camera are solved together through optimization algorithms, so that the reprojection error of all corner points on the image is minimized, thereby achieving unified calibration of the parameters of each camera.

[0048] For example, a corner detection algorithm with sub-pixel precision can be used to extract checkerboard corners. The correspondence between corners of adjacent images can be established by descriptor matching or spatial constraints. Then, the intrinsic parameters, extrinsic parameters and three-dimensional corner coordinates of all cameras can be optimized simultaneously using a bundled adjustment method, and finally a set of globally consistent high-precision calibration parameters can be obtained.

[0049] Understandably, since the stable and high-precision corner features on the calibration board are directly used for multi-view matching and joint optimization, step S20 can avoid the instability and ambiguity of relying on natural feature matching, thereby improving the overall accuracy and robustness of the calibration parameters and ensuring the geometric accuracy of subsequent image stitching.

[0050] Step S30: Generate a configuration file based on the calibration parameters; It should be noted that the configuration file is a formatted data file used to store and transmit calibration results. It contains calibration parameters for all cameras and system configuration information to facilitate loading and use by the embedded platform.

[0051] Understandably, by organizing complex calibration parameters into a structured configuration file, step S30 can avoid the complexity of directly parsing and processing raw calibration data in the embedded system, thereby improving the system's deployment efficiency, maintainability, and operational reliability.

[0052] In one feasible implementation, step S30 may include: extracting the lens distortion coefficient, internal parameter matrix, and external attitude angle of each camera from the calibration parameters; obtaining the horizontal field of view of each camera based on the image sensor information and the internal parameter matrix; and organizing the lens distortion coefficient, the internal parameter matrix, the external attitude angle, and the horizontal field of view according to a preset format to generate a configuration file.

[0053] It should be noted that lens distortion coefficients are parameters describing the degree of lens distortion, including radial and tangential distortion coefficients; the internal parameter matrix is ​​a 3×3 matrix describing the internal imaging geometry of the camera, containing focal length and principal point information; the external attitude angles are the three Euler angles describing the orientation of the camera in the global coordinate system; image sensor information includes sensor size and pixel size; the horizontal field of view is the angle of the camera's field of view in the horizontal direction; and the preset format is a specific data organization method supported by the embedded platform.

[0054] Specifically, the system directly extracts the parameters required by the RK3588 platform from the calibration parameters: 1. Lens distortion correction parameters: From the optimized camera calibration parameters, parameters conforming to the Brown-Conrady distortion model are extracted, including radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2). These parameters will be used in subsequent image distortion correction processing.

[0055] 2. Camera intrinsic parameters and field of view information: 1) Extract the equivalent focal length (in pixels) directly from the intrinsic parameter matrix K.

[0056] 2) According to the formula, the horizontal field of view (FOV) h = 2 arctan(image sensor width / (2) The focal length is used to calculate the horizontal field of view for each camera. The sensor width can be estimated from the image pixel width and pixel size, or the image pixel width can be used directly as an approximation (in which case the focal length must also be in pixels).

[0057] 3) Record the image resolution (width and height) of each camera.

[0058] 3. Camera external attitude parameters: The optimized extrinsic rotation matrix R_i is converted into a more intuitive Euler angle representation (Yaw, Pitch, Roll). These angles define the mounting orientation of each camera in a unified global coordinate system.

[0059] In this embodiment, by converting the calibration parameters into a structured configuration file that can be directly called by the embedded platform, the problems of inconsistent formats and difficulty in adaptation during the traditional calibration parameter transmission and use are solved, and efficient and accurate docking between the calibration system and the running system is achieved.

[0060] The above are merely feasible implementations of step S30 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S30.

[0061] Step S40: Process the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream.

[0062] It should be noted that a video stream refers to a continuous sequence of images captured in real time from multiple cameras; a panoramic video stream refers to a continuous panoramic image sequence formed by stitching and fusing multiple video streams.

[0063] Understandably, since high-precision calibration parameters are pre-configured and used to guide real-time image processing, step S40 can avoid the computational burden and latency caused by online real-time calculation of camera parameters, thereby improving the processing efficiency, real-time performance, and stability of the panoramic video stream.

[0064] In one feasible implementation, step S40 may include: acquiring video streams captured by the plurality of cameras; generating distortion correction parameters and perspective transformation parameters based on calibration parameters in the configuration file; performing distortion correction on each frame of the video stream based on the distortion correction parameters to obtain a distortion-free normalized image; mapping the distortion-free normalized image at the same moment to a panoramic canvas based on the perspective transformation parameters, and performing image fusion based on the overlapping area to obtain a single-frame panoramic image; and encoding the single-frame panoramic image to obtain a panoramic video stream.

[0065] It should be noted that distortion correction parameters are mapping parameters used to eliminate lens imaging distortion, usually stored in the form of lookup tables; perspective transformation parameters are geometric transformation matrices used to transform images from different perspectives to a unified panoramic plane; distortion-free normalized images are images that conform to the ideal imaging model after distortion correction; panoramic canvas is a virtual image plane used to accommodate the stitching results of all camera images; overlapping regions refer to the corresponding areas in the image where the fields of view of adjacent cameras overlap; image fusion is the process of smoothly transitioning multiple images in the overlapping regions; and a single-frame panoramic image is a complete panoramic image formed by stitching and fusing all camera images at a given moment.

[0066] Specifically, the distortion correction parameters and perspective transformation parameters of each camera in the configuration file are read to generate corresponding mapping lookup tables and transformation matrices. For each video stream, the distortion correction lookup table is applied frame-by-frame for pixel remapping to obtain a standardized image with distortion eliminated. Then, based on the perspective transformation matrices of each camera, multiple standardized images at the same moment are projected onto the corresponding positions on the panoramic canvas. A weighted fusion algorithm is used in the overlapping areas of adjacent images to achieve a smooth transition, forming a seamless single-frame panoramic image. Finally, each frame of the panoramic image is compressed and encoded to form a continuously output panoramic video stream.

[0067] For example, on the RK3588 embedded platform, the lookup tables for distortion correction and perspective transformation can be pre-computed using its hardware geometry processing unit, six video streams can be processed in parallel through multi-threading, overlapping regions can be fused using a distance-weighted feathering algorithm, and panoramic video streams in H.265 format can be output in real time through a hardware encoder.

[0068] In this embodiment, by converting the high-precision parameters calibrated offline into preprocessing parameters that can be efficiently executed by the embedded platform, the problems of unstable stitching quality and slow system response caused by complex parameter calculation and large processing delay in traditional real-time stitching systems are solved, and high-quality, low-latency real-time generation of panoramic video is realized on resource-constrained embedded platforms.

[0069] The above are merely feasible implementations of step S40 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S40.

[0070] This embodiment provides a panoramic stitching method based on multiple cameras. It acquires calibration images from multiple cameras, where each calibration image includes a common calibration region of adjacent cameras. The method identifies calibration board corner features in the calibration images and obtains calibration parameters for each camera based on these features. A configuration file is generated based on the calibration parameters. The video streams acquired by the multiple cameras are processed using the configuration file to obtain a panoramic video stream. This method solves the problems of low calibration accuracy and reliance on manual experience and complex online calculations in existing technologies under long-distance, large field-of-view conditions. It standardizes and automates the calibration process and enables efficient, high-precision real-time panoramic video stitching on embedded devices, reducing the system's dependence on high-performance processing platforms and overall cost.

[0071] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 The panoramic stitching method based on multi-view cameras further includes steps S21 to S25 in step S20: Step S21: Identify the calibration board corner points within the common calibration area in the calibration image to obtain the calibration board corner point features; It should be noted that the corner points of the calibration board are feature points with clear geometric positions in the calibration board pattern, such as the intersection of the black and white squares of a checkerboard, which are represented as pixel positions with abrupt changes in grayscale in the image.

[0072] Specifically, by analyzing the images within the common calibration area, a corner detection algorithm is used to extract the pixel coordinates of the intersection points of the checkerboard on each calibration board, and these coordinates are used as a set of feature points with sub-pixel accuracy to form calibration board corner features that can be used for geometric calculations.

[0073] For example, a corner detection method based on grayscale gradient can be used to locate the precise position of each intersection point within the calibration plate area in the image, thereby obtaining a set of corner pixel coordinates with row and column index relationships.

[0074] It is understandable that by directly using the structured corner points of the calibration board as features, step S21 can avoid the problems of unstable features and poor repeatability in natural scenes, thereby improving the accuracy and reliability of feature extraction and providing unified and high-precision input data for subsequent feature matching.

[0075] Step S22: Match the corner features of the calibration board between the calibration images of adjacent cameras to obtain matching point pairs; It should be noted that a matching point pair refers to a set of two or more pairs of pixel coordinates that correspond to the same physical corner point in images from different cameras.

[0076] Specifically, based on the row and column index relationship of the calibration board corner points and their spatial distribution consistency in adjacent images, the correspondence between corner points in adjacent camera images is established, forming a set of matching point pairs that describe the projection position of the same 3D point in multiple views.

[0077] For example, a one-to-one correspondence between corner points of adjacent images can be automatically established using the known grid structure of the calibration plate and the local neighborhood descriptors of the corner points, forming a set of matching point pairs for calculating geometric transformations.

[0078] It is understandable that by using the regular structure of the calibration plate and the known correspondence for matching, step S22 can avoid mismatches and ambiguities that may occur when matching based on appearance descriptors, thereby improving the accuracy and efficiency of matching and ensuring the stability of subsequent geometric calculations.

[0079] Step S23: Based on the matching point pairs, obtain the initial transformation matrix between adjacent cameras; It should be noted that the initial transformation matrix is ​​a mathematical matrix that describes the geometric mapping relationship between two camera views, such as the homography matrix or the fundamental matrix.

[0080] Specifically, by using a set of matching point pairs, the geometric transformation model that best describes the mapping relationship between these point pairs is calculated using the least squares method or other linear estimation methods, and the initial transformation matrix between adjacent cameras is obtained.

[0081] Understandably, since geometric model estimation is based on high-precision matching point pairs, step S23 can avoid model deviations caused by matching errors or noise, thereby providing accurate initial geometric relationships and laying a good foundation for subsequent global optimization.

[0082] Step S24: Unify the initial transformation matrix to the global coordinate system to obtain the external parameters of the adjacent cameras.

[0083] It should be noted that the global coordinate system is a unified reference coordinate system used to describe the relative position and attitude of all cameras in three-dimensional space; the external parameters of adjacent cameras refer to the rotation and translation parameters of each camera relative to the global coordinate system.

[0084] Specifically, based on the initial transformation matrix between each adjacent camera pair, the extrinsic parameters of all cameras are unified into the same global coordinate system through rotation averaging and relative translation estimation methods, thereby obtaining the spatial position and orientation parameters of each camera relative to the global coordinate system.

[0085] For example, a serial or parallel attitude map optimization method can be used to accumulate or unify the relative transformations of each camera pair into a global coordinate system to obtain the rotation matrix and translation vector of each camera.

[0086] It is understandable that by unifying all cameras to the same coordinate system for parameter representation, step S24 can avoid overall geometric contradictions caused by inconsistencies in local parameters, thereby improving the overall consistency and interpretability of the multi-camera system parameters.

[0087] Step S25: Based on the preview requirements and the matching point pairs, jointly optimize the internal parameters of each camera, the corner features of the calibration board, and the external parameters to obtain the calibration parameters of each camera.

[0088] It should be noted that preview requirements refer to the visual quality standards of the panoramic stitching effect, such as the alignment of stitching seams and the continuity of lines; joint optimization refers to the process of iteratively adjusting multiple variables simultaneously to minimize the overall error.

[0089] Specifically, by using all the camera's internal parameters, external parameters, and 3D corner coordinates as optimization variables, an optimization problem is constructed with minimizing the reprojection error as the objective function. Through iterative solution, the objective function is converged, resulting in a set of globally optimal calibration parameters.

[0090] It is understandable that since the joint optimization method is used to adjust all camera parameters and 3D point coordinates at the same time, step S25 can avoid the parameter inconsistency problem caused by optimizing each camera individually, thereby improving the overall calibration accuracy and geometric consistency of the system.

[0091] In one feasible implementation, step S25 may include: jointly optimizing the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pairs to obtain optimized camera parameters for each camera; generating a panoramic stitching preview image based on the optimized camera parameters; when the panoramic stitching preview image does not meet the preview requirements, updating the calibration board corner features in the optimized camera parameters in response to an operator's instruction to add or delete matching points in the stitching seam area of ​​the panoramic stitching preview image; based on the updated calibration board corner features, returning to the step of jointly optimizing the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pairs to obtain optimized camera parameters, until the panoramic stitching preview image meets the preview requirements, and using the optimized camera parameters of each camera as the calibration parameters of the corresponding camera.

[0092] It should be noted that optimizing camera parameters refers to a set of camera parameters obtained after joint optimization; the panoramic stitching preview image is a simulated stitching effect image generated based on the current calibration parameters; and the instructions to add or delete matching points are operations performed by the operator to interactively adjust the optimization process based on the visual inspection results.

[0093] Specifically, the operator observes the alignment of the stitching seams in the panoramic stitching preview. If a significant misalignment is found in a local area, a new matching point can be manually added or an incorrect matching point can be deleted in that area. The system updates the optimization variables and re-executes the optimization process based on these interactive instructions until a satisfactory visual stitching effect is obtained.

[0094] For example, such as Figure 4As shown, after clicking the "Manual Point Selection" button, the operator can select images 2 and 3. Within the common calibration area of ​​adjacent images 2 and 3, the same checkerboard corner point is manually selected with sub-pixel precision. The system automatically adds this corner point as a high-confidence matching point pair and updates the feature point markers in the image display area in real time. After clicking the "Image Stitching" button, the system, based on all automatically matched and manually added matching point pairs, uses a bundled adjustment method to simultaneously optimize the internal parameters, external parameters, and 3D corner point coordinates of all cameras. During the optimization process, the stitching result preview area generates a panoramic stitching effect image under the current parameters in real time. The panoramic stitching effect images under historical parameters and the panoramic stitching effect image under the current parameters are displayed in the image display area. Figure 6 If the operator observes slight misalignment at the stitching seam in the preview area of ​​the stitching result shown, new matching point pairs can be added, and the system will then iteratively perform optimization. Finally, the preview area of ​​the stitching result presents a continuous, misaligned panoramic image, and the system uses the optimized camera parameters at this time as the final calibration parameters.

[0095] Further, the step of jointly optimizing the intrinsic parameters of each camera, the corner features of the calibration board, and the extrinsic parameters based on the matching point pairs to obtain optimized camera parameters for each camera includes: using the intrinsic parameters, corner features of the calibration board, and extrinsic parameters of each camera as optimization variables for the corresponding camera; determining an objective function based on the matching point pairs; establishing an optimization model based on the objective function and the optimization variables; iteratively solving the optimization model to obtain optimization results; and obtaining optimized camera parameters based on the optimization results.

[0096] It should be noted that the optimization variables are all the parameters to be solved in the optimization process; the objective function is the loss function constructed based on the reprojection error of the matching points; the optimization model is a mathematical optimization problem composed of the objective function and the optimization variables; and the optimization result is the parameter value that minimizes the objective function.

[0097] Specifically, the optimization variables include the focal length, principal point, distortion coefficient, rotation matrix, translation vector, and coordinates of all 3D corner points of each camera; the objective function is defined as the sum of squared reprojection errors of all matching point pairs on the image plane; the optimization model is constructed using a nonlinear least squares method; the optimization variables are adjusted through an iterative algorithm to minimize the objective function, and finally the optimal set of parameters is obtained as the optimized camera parameters.

[0098] In this embodiment, by introducing an interactive optimization mechanism based on visual feedback, the problem of poor stitching effect caused by insufficient local matching or noise during the automatic optimization process is solved, thus achieving a dual guarantee of calibration accuracy and visual quality.

[0099] The above are merely feasible implementations of step S25 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S25.

[0100] This embodiment provides a panoramic stitching method based on multi-camera systems. It identifies calibration board corner points within the common calibration region of the calibration images to obtain calibration board corner point features. These features are then matched between calibration images from adjacent cameras to obtain matching point pairs. Based on these matching point pairs, an initial transformation matrix between adjacent cameras is obtained. The initial transformation matrix is ​​then unified to a global coordinate system to obtain the extrinsic parameters of the adjacent cameras. Based on preview requirements and the matching point pairs, the intrinsic parameters of each camera, the calibration board corner point features, and the extrinsic parameters are jointly optimized to obtain the calibration parameters for each camera. This method solves the problems of insufficient accuracy and poor robustness of traditional calibration methods in long-distance, large-field-of-view scenarios. It avoids the uncertainties of relying on human experience and complex online calculations, achieving automation, high precision, and strong robustness in the multi-camera system calibration process.

[0101] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the panoramic stitching method based on multi-view cameras in this application. Any simple variations based on this technical concept are within the protection scope of this application.

[0102] This application also provides a panoramic stitching device based on a multi-view camera, please refer to... Figure 7 The panoramic stitching device based on multi-view cameras includes: Image acquisition module 10 is used to acquire calibration images acquired by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; The parameter calibration module 20 is used to identify the corner features of the calibration board in the calibration image and obtain the calibration parameters of each camera based on the corner features of the calibration board. Configuration file module 30 is used to generate a configuration file based on the calibration parameters; The panoramic video module 40 is used to process the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream.

[0103] The panoramic stitching device based on a multi-view camera provided in this application employs the panoramic stitching method based on a multi-view camera in the above embodiments, and can solve the technical problems of panoramic stitching based on a multi-view camera. Compared with the prior art, the beneficial effects of the panoramic stitching device based on a multi-view camera provided in this application are the same as the beneficial effects of the panoramic stitching method based on a multi-view camera provided in the above embodiments, and other technical features in the panoramic stitching device based on a multi-view camera are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0104] The image acquisition module 10 is further configured to pre-arrange multiple calibration plates, wherein the calibration plates are respectively set at a preset distance from the multiple cameras; adjust the position of each calibration plate according to the optical center of the multiple cameras so that the normal direction of the surface of the calibration plate points to the optical center; after the adjustment is completed, control each camera to acquire calibration images according to preset shooting parameters, wherein the calibration images of adjacent cameras in the calibration images include a common calibration area formed by the same calibration plate.

[0105] The parameter calibration module 20 is further configured to identify calibration board corner points within the common calibration region of the calibration image to obtain calibration board corner point features; match the calibration board corner point features between the calibration images of adjacent cameras to obtain matching point pairs; obtain an initial transformation matrix between adjacent cameras based on the matching point pairs; unify the initial transformation matrix to the global coordinate system to obtain the extrinsic parameters of the adjacent cameras; and jointly optimize the intrinsic parameters, calibration board corner point features, and extrinsic parameters of each camera based on preview requirements and the matching point pairs to obtain the calibration parameters of each camera.

[0106] The parameter calibration module 20 is further configured to jointly optimize the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pair to obtain optimized camera parameters for each camera; generate a panoramic stitching preview image based on the optimized camera parameters; when the panoramic stitching preview image does not meet the preview requirements, in response to the operator's instruction to add or delete matching points in the stitching seam area of ​​the panoramic stitching preview image, update the calibration board corner features in the optimized camera parameters; based on the updated calibration board corner features, return to execute the step of jointly optimizing the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pair to obtain optimized camera parameters, until the panoramic stitching preview image meets the preview requirements, and then use the optimized camera parameters of each camera as the calibration parameters of the corresponding camera.

[0107] The parameter calibration module 20 is further configured to use the internal parameters, calibration board corner features, and external parameters of each camera as optimization variables for the corresponding camera; determine the objective function based on the matching point pairs; establish an optimization model based on the objective function and the optimization variables; iteratively solve the optimization model to obtain the optimization result; and obtain the optimized camera parameters based on the optimization result.

[0108] The configuration file module 30 is further configured to extract the lens distortion coefficient, internal parameter matrix, and external attitude angle of each camera from the calibration parameters; obtain the horizontal field of view of each camera based on the image sensor information and the internal parameter matrix; and organize the lens distortion coefficient, the internal parameter matrix, the external attitude angle, and the horizontal field of view according to a preset format to generate a configuration file.

[0109] The panoramic video module 40 is further configured to acquire video streams captured by the multiple cameras; generate distortion correction parameters and perspective transformation parameters based on the calibration parameters in the configuration file; perform distortion correction on each frame of the video stream based on the distortion correction parameters to obtain a distortion-free normalized image; map the distortion-free normalized image at the same moment onto the panoramic canvas based on the perspective transformation parameters, and perform image fusion based on the overlapping area to obtain a single-frame panoramic image; and encode the single-frame panoramic image to obtain a panoramic video stream.

[0110] This application provides a panoramic stitching device based on a multi-view camera. The panoramic stitching device based on a multi-view camera includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the panoramic stitching method based on a multi-view camera in the above embodiment 1.

[0111] The following is for reference. Figure 8 This document illustrates a structural schematic diagram of a panoramic stitching device based on a multi-view camera, suitable for implementing embodiments of this application. The panoramic stitching device based on a multi-view camera in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The panoramic stitching device based on multi-view cameras shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 8As shown, a multi-camera-based panoramic stitching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multi-camera-based panoramic stitching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multi-camera-based panoramic stitching device to communicate wirelessly or wiredly with other devices to exchange data. Although multi-camera-based panoramic stitching devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0113] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0114] The panoramic stitching device based on a multi-view camera provided in this application, employing the panoramic stitching method based on a multi-view camera in the above embodiments, can solve the technical problems of panoramic stitching based on a multi-view camera. Compared with the prior art, the beneficial effects of the panoramic stitching device based on a multi-view camera provided in this application are the same as the beneficial effects of the panoramic stitching method based on a multi-view camera provided in the above embodiments, and other technical features in this panoramic stitching device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0117] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the panoramic stitching method based on a multi-view camera in the above embodiments.

[0118] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable storage medium may be included in a multi-camera-based panoramic stitching device; or it may exist independently and not assembled into a multi-camera-based panoramic stitching device.

[0120] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a multi-camera-based panoramic stitching device, cause the multi-camera-based panoramic stitching device to: acquire calibration images collected by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; identify calibration board corner features in the calibration images and obtain calibration parameters for each camera based on the calibration board corner features; generate a configuration file according to the calibration parameters; and process the video streams collected by the multiple cameras based on the configuration file to obtain a panoramic video stream.

[0121] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0124] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described panoramic stitching method based on multi-view cameras, thereby solving the technical problems of panoramic stitching based on multi-view cameras. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the panoramic stitching method based on multi-view cameras provided in the above embodiments, and will not be repeated here.

[0125] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the panoramic stitching method based on multi-view cameras as described above.

[0126] The computer program product provided in this application can solve the technical problem of panoramic stitching based on multi-view cameras. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the panoramic stitching method based on multi-view cameras provided in the above embodiments, and will not be repeated here.

[0127] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A panoramic stitching method based on multi-view cameras, characterized in that, The method includes: Acquire calibration images from multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; Identify the corner features of the calibration board in the calibration image, and obtain the calibration parameters of each camera based on the corner features of the calibration board; Generate a configuration file based on the calibration parameters; Based on the configuration file, the video streams captured by the multiple cameras are processed to obtain a panoramic video stream.

2. The method as described in claim 1, characterized in that, The acquisition of calibration images from multiple cameras includes: Multiple calibration plates are pre-arranged, wherein the calibration plates are respectively set at a preset distance from the multiple cameras; Based on the optical centers of the plurality of cameras, adjust the position of each calibration plate so that the normal direction of the surface of the calibration plate points to the optical center; After the adjustment is completed, each camera is controlled to acquire calibration images according to preset shooting parameters. The calibration images of adjacent cameras include a common calibration area formed by the same calibration plate.

3. The method as described in claim 1, characterized in that, The step of identifying the corner features of the calibration board in the calibration image and obtaining the calibration parameters of each camera based on the corner features of the calibration board includes: Identify the calibration board corner points within the common calibration area in the calibration image to obtain the calibration board corner point features; Between the calibration images of adjacent cameras, the corner features of the calibration board are matched to obtain matching point pairs; Based on the matching point pairs, the initial transformation matrix between adjacent cameras is obtained; The initial transformation matrix is ​​unified to the global coordinate system to obtain the external parameters of the adjacent cameras; Based on the preview requirements and the matching point pairs, the internal parameters of each camera, the corner features of the calibration board, and the external parameters are jointly optimized to obtain the calibration parameters of each camera.

4. The method as described in claim 3, characterized in that, The process of jointly optimizing the internal parameters of each camera, the corner features of the calibration board, and the external parameters based on the preview requirements and the matching point pairs yields the calibration parameters for each camera, including: Based on the matching point pairs, the internal parameters of each camera, the corner features of the calibration board, and the external parameters are jointly optimized to obtain the optimized camera parameters of each camera. A panoramic stitched preview image is generated based on the optimized camera parameters; When the panoramic stitching preview does not meet the preview requirements, in response to the operator's instruction to add or delete matching points in the stitching seam area of ​​the panoramic stitching preview, the calibration board corner point features in the optimized camera parameters are updated. Based on the updated calibration board corner features, the process returns to the step of jointly optimizing the internal parameters of each camera, the calibration board corner features, and the external parameters based on the matching point pairs to obtain optimized camera parameters. This process continues until the panoramic stitching preview image meets the preview requirements, at which point the optimized camera parameters of each camera are used as the calibration parameters of the corresponding camera.

5. The method as described in claim 4, characterized in that, The joint optimization of the internal parameters of each camera, the corner features of the calibration board, and the external parameters based on the matching point pairs yields optimized camera parameters for each camera, including: The internal parameters, calibration board corner features, and external parameters of each camera are used as optimization variables for the corresponding camera. The objective function is determined based on the matching point pairs; An optimization model is established based on the objective function and the optimization variables; The optimization model is iteratively solved to obtain the optimization results; Based on the optimization results, the optimized camera parameters are obtained.

6. The method as described in claim 1, characterized in that, The step of generating a configuration file based on the calibration parameters includes: Extract the lens distortion coefficients, internal parameter matrix, and external attitude angles for each camera from the calibration parameters; The horizontal field of view of each camera is obtained based on the image sensor information and the internal parameter matrix. The lens distortion coefficients, the internal parameter matrix, the external attitude angle, and the horizontal field of view are organized according to a preset format to generate a configuration file.

7. The method as described in claim 1, characterized in that, The process of processing the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream includes: Acquire the video streams captured by the multiple cameras; Distortion correction parameters and perspective transformation parameters are generated based on the calibration parameters in the configuration file; Based on the distortion correction parameters, distortion correction is performed on each frame of the video stream to obtain a distortion-free normalized image. Based on the perspective transformation parameters, the distortion-free normalized image at the same moment is mapped onto the panoramic canvas, and image fusion is performed based on the overlapping area to obtain a single-frame panoramic image. The single-frame panoramic image is encoded to obtain a panoramic video stream.

8. A panoramic stitching device based on a multi-view camera, characterized in that, The device includes: An image acquisition module is used to acquire calibration images captured by multiple cameras, wherein the calibration images include common calibration areas of adjacent cameras; The parameter calibration module is used to identify the corner features of the calibration board in the calibration image and obtain the calibration parameters of each camera based on the corner features of the calibration board. The configuration file module is used to generate a configuration file based on the calibration parameters; The panoramic video module is used to process the video streams captured by the multiple cameras based on the configuration file to obtain a panoramic video stream.

9. A panoramic stitching device based on a multi-view camera, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the panoramic stitching method based on a multi-view camera as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the panoramic stitching method based on a multi-view camera as described in any one of claims 1 to 7.

Images