Deep-sea adaptive video mosaicking method based on nonlinear projection

Abstract

The application discloses a deep-sea adaptive video splicing method based on nonlinear projection, and the implementation steps comprise the following steps: on the basis of determining the circle center position of the calibration board image in the physical space and underwater two scenes, radial distortion of the underwater image is corrected by using a segmentation model; the underwater image is converted into the image in the physical space state by using perspective projection; the contour similarity of the adjacent deep-sea video image overlapping area image is calculated by using a contour similarity function; the translation parameters between the video images after the cylindrical projection are optimized online; and the splicing of the multiple deep-sea video images is realized based on image fusion. The deep-sea video image splicing method designed by the application considers the radial distortion of the underwater image, can realize the splicing of the underwater video image without depending on the underwater features, and simultaneously adaptively adjusts the underwater splicing parameters according to the contour similarity of the overlapping area to adapt to the complex and changeable environment of the deep sea.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and further relates to a deep-sea adaptive video stitching method based on nonlinear projection in the field of image stitching technology. This invention can be used to stitch together acquired video streams in real time in underwater or deep-sea environments, and the stitching results can provide a reference for the safe operation of submarines. Background Technology

[0002] Real-time wide-angle underwater monitoring helps operators observe their surroundings and avoid unexpected dangers, playing a crucial role in underwater patrols. However, real-time wide-angle video systems have limitations. Wide-angle cameras experience a significant reduction in field of view underwater due to light refraction. Using fisheye cameras with even larger fields of view is not only expensive but also results in severe distortion. Therefore, video stitching technology offers significant advantages. Video stitching combines videos from cameras positioned at different angles with overlapping fields of view to create a complete wide-angle video, overcoming the limitations of a single camera's field of view. Essentially, video stitching is image stitching. The general process of image stitching includes image preprocessing, feature point detection and matching, stitching parameter calculation, image distortion, and image fusion. In video stitching, each frame is often stitched together directly using pre-calculated stitching parameters. Compared to video stitching in physical space, video stitching in underwater or deep-sea environments is far more complex. Deep-sea images often lack features or have indistinct features. However, existing video stitching techniques rely heavily on feature point detection and matching to calculate stitching parameters, making them overly dependent on features and lacking robustness, thus unsuitable for deep-sea environments. Underwater images exhibit significant geometric distortions caused by lens distortion and light refraction, disrupting the collinearity condition in imaging. Since existing solutions almost entirely ignore geometric distortion, they suffer from significant stitching errors. Furthermore, the refractive index of seawater is affected by factors such as temperature, salinity, and pressure. A single stitching model cannot account for all conditions, and existing wide-angle camera video stitching schemes for deep-sea scenes inevitably suffer from the problem of gradually accumulating errors.

[0003] In their paper "A Video Stitching System of UnderwaterImage" (Proceedings of the Hainan University HPCC Conference, December 20, 2021), Wang Zhenyu et al. proposed an underwater video stitching system. This system uses two handheld GoPro cameras to capture high-definition video. During video stitching, the system first enhances the underwater image based on relative global histogram stretching. Then, through background recognition, the deep-sea video stitching is divided into two steps: background video stitching and foreground video stitching. This results in a more stable and natural-looking panoramic video. The system uses SSIM, UCIQE, and UIQM to quantitatively evaluate the generated panoramic video. The advantage of the algorithm employed by this system is that it avoids the negative impact of foreground matching on the stitching process, such as artifacts in overlapping areas. However, the algorithm still has the following two shortcomings: First, the algorithm uses feature point detection and matching to calculate stitching parameters, which is too dependent on feature points and unstable. Insufficient number of detected feature points or uneven distribution will lead to stitching failure. Second, in order to obtain a coherent underwater panoramic video, the algorithm performs image enhancement on all frames in the video and calculates stitching parameters for each frame, which takes too long. In engineering practice, underwater image enhancement for each 2704x1520 resolution video frame on a PC usually takes 90 seconds, and stitching one frame takes 45 seconds, which is far from meeting the real-time requirements for stitching real-time video streams acquired in underwater or deep-sea environments.

[0004] Boyagongdao (Beijing) Robotics Technology Co., Ltd. proposed an underwater panoramic video stitching method in its patent application document "An Underwater Panoramic Video Stitching System and Method" (Application No.: 201911358323.7, Publication No.: CN 111105355 A, Publication Date: 2020.05.05). The implementation steps of the underwater video stitching method are as follows: (1) After the cameras are installed, panoramic calibration is performed to obtain the intrinsic parameters, distortion coefficients and extrinsic parameters between adjacent cameras of multiple cameras. Then, the transformation matrix of multiple cameras is calculated based on these parameters, and the transformation matrix is ​​converted to obtain a preset lookup table file; (2) Homomorphic filtering, histogram equalization and least squares filtering algorithms are used to enhance the underwater image; (3) The enhanced underwater image is stitched according to the preset lookup table file obtained in step (1); (4) The stitched underwater image and the preset fusion algorithm are used to fuse the image stitching seams of adjacent cameras. The advantage of this method is that it stitches underwater images by calling a lookup table file, which improves the real-time performance of panoramic stitching compared to traditional frame matching methods. However, this method still has some shortcomings. First, it does not consider the influence of light refraction during underwater imaging. When the camera is imaging underwater, the refraction of light disrupts the collinearity condition of pinhole imaging, causing the commonly used pinhole camera model in physical space to fail. Underwater images exhibit significant geometric distortion caused by lens distortion and light refraction. Directly using the MATLAB toolbox to calibrate the underwater camera will produce large calibration errors, thus affecting the stitching effect. Second, compared with underwater environment video stitching methods, the deep-sea environment is more complex, and a single underwater video stitching model cannot handle the changes in the refractive index of seawater in the deep-sea environment. Therefore, this underwater panoramic video stitching algorithm cannot be well applied to the deep-sea environment.

[0005] In summary, existing deep-sea video stitching technologies either rely entirely on underwater features, fail to consider geometric distortions caused by light refraction leading to stitching misalignment, or use a single stitching parameter resulting in errors in subsequent video frame stitching; none of these are suitable for the deep sea. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by providing a deep-sea adaptive video stitching method based on nonlinear projection, which solves the problem of stitching failure due to insufficient feature points in deep-sea environment video stitching, as well as the impact of refraction on stitching in deep-sea environment videos.

[0007] The technical approach to achieving the objective of this invention is to convert a single-frame underwater video image into an image in physical space using nonlinear projection. This projection specifically includes radial distortion correction and perspective projection based on a division model. This model takes into account the influence of underwater light refraction on imaging, including underwater image radial distortion, principal point offset, reduced field of view, and scale scaling. The underwater video can then be stitched together using stitching parameters obtained through camera calibration in physical space. This invention does not directly use underwater features to calculate stitching parameters but instead uses stitching parameters in physical space to stitch the projected video image, thereby solving the problem of excessive dependence on underwater features in existing underwater video stitching technologies. This invention addresses the radial distortion of video images caused by light refraction in underwater imaging. Since the division model for radial distortion correction is independent of the camera model and only applies to the image itself, only one calibration board pattern needs to be captured underwater. By detecting corner points or center points on this pattern, radial distortion correction can be performed using the division model. By retaining the mapping table for this radial distortion correction, geometric distortion correction can be applied to any video image captured by the camera, thus solving the problem of video stitching errors caused by this distortion. This invention also considers the impact of seawater refractive index variations on video stitching in deep-sea environments. The refractive index of seawater is affected by factors such as seawater temperature, salinity, and pressure; a single stitching model cannot account for all these factors. Image contours are crucial information in images. Therefore, this invention uses a contour similarity function to optimize stitching parameters online. First, the overlapping area of ​​video images captured by adjacent cameras is calculated based on the initial stitching parameters. Then, an edge detection algorithm is used to detect the contours. Next, using the contour similarity function as the objective function, the Levenberg-Marquardt iterative algorithm is used to optimize the translation parameters between images projected onto the same cylindrical surface online, achieving alignment of multiple deep-sea video images. This solves the problem that existing single stitching models cannot adapt to all situations.

[0008] The implementation steps of this invention are as follows:

[0009] Step 1: Determine the center position of the calibration board image in both physical space and underwater scenarios. Use a division model to correct the radial distortion of the underwater image, and then use perspective projection to convert the underwater image into an image in physical space.

[0010] Step 2: Calculate the contour similarity of overlapping regions of adjacent deep-sea video images using the contour similarity function;

[0011] Step 3: Optimize the translation parameters between video images after cylindrical projection online;

[0012] Step 4: Stitch together multiple frames of deep-sea video images based on image fusion.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] First, the present invention restores a single-frame underwater video image into an image in physical space by nonlinear projection, and then completes the underwater video stitching by using the stitching parameters obtained in the physical space in advance. This overcomes the shortcomings of existing underwater video stitching technologies that rely too much on underwater features, and enables the present invention to achieve better video stitching without relying on underwater features.

[0015] Secondly, this invention takes into account the radial distortion of images caused by the refraction of light in underwater imaging, and uses a division distortion model to correct the geometric distortion of a single underwater image, overcoming the shortcomings of this distortion in causing errors in video stitching, thus enabling this invention to achieve better underwater video stitching.

[0016] Third, this invention takes into account the impact of seawater refractive index variation on video stitching in the deep-sea environment. It uses a contour similarity function as the objective function to optimize the relative positions of images projected onto the same cylindrical surface online, thereby achieving alignment of multiple deep-sea video images. This overcomes the shortcomings of a single stitching model that cannot adapt to all situations, enabling this invention to achieve better deep-sea video stitching. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention;

[0018] Figure 2 This is a schematic diagram of radial distortion of the image according to the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] Reference Figure 1 The implementation steps of the embodiments of the present invention will be further described below.

[0021] Step 1: Determine the center positions of the calibration board images in both physical space and underwater scenarios. Use a division model to correct the radial distortion of the underwater image, and then use perspective projection to convert the underwater image into an image in physical space.

[0022] Using the Halcon calibration board as the target, three cameras captured one image of the calibration board in physical space and one image of it in an underwater scene. The target in each image occupies more than one-third of the entire frame. The reason for using the Halcon calibration board instead of the standard checkerboard calibration board is that when using the standard checkerboard calibration board to capture images underwater, there is a relatively large defocus blur, resulting in a large detection error for corner points in the image. This blur has a smaller impact on the center point in the underwater Halcon calibration board image. In order to obtain the correspondence between the images captured in physical space and the underwater scene, the same calibration board must be used for both scenes.

[0023] Using a distance vector-based method, the center of each calibration board image is detected and its position is recorded. This center position is used to obtain the correspondence between the images captured in physical space and the underwater scene.

[0024] For each camera, the radial distortion of the underwater image is corrected. For each deformed straight line in the calibration board image of the underwater scene that is at the same position as the physical space calibration board image, circular arc fitting is performed to obtain the center and radius of the arc.

[0025] The radial distortion of the underwater image is manifested as pincushion radial distortion, which is a type of radial distortion.

[0026] Reference Figure 2 The radial distortion of the image is further described. Figure 2 (a) represents a standard checkerboard image without distortion. Figure 2 (b) indicates Figure 2 (a) Image after barrel radial distortion. After barrel radial distortion, the squares in the standard checkerboard image become barrel-shaped, and the straight sides of the squares become the curved sides of the barrel bulges. Figure 2 (c) indicates Figure 2 (a) Image after pincushion radial distortion. After pincushion radial distortion, the squares in the standard checkerboard image become pincushions, and the straight sides of the squares become the curved sides of the pincushion concavity. Both barrel radial distortion and pincushion radial distortion of the image are more distorted closer to the image distortion center and less distorted closer to the image edge.

[0027] A straight line in an undistorted image becomes an arc in a distorted image. Underwater radial distortion manifests as pincushion-shaped radial distortion. Based on this characteristic, an equation is established for each fitted arc. Three previously unselected equations are randomly chosen to form the following system of equations to solve for the distortion center and radial distortion coefficients of the calibration plate image of the underwater scene:

[0028]

[0029]

[0030]

[0031] Where A0, B0, C0, A1, B1, C1 and A2, B2, C2 represent the coefficients of the general equations of the lines at the same positions on the physical space calibration board image corresponding to the three selected circular arcs. Solving this set of equations yields a unique solution for the distortion center and radial distortion coefficients of the image.

[0032] The radial distortion coefficients are substituted into the division model to perform radial distortion correction on the calibration plate image in the underwater scene, resulting in the corresponding radially distorted underwater calibration plate image. The mapping relationship of radial distortion for each image is recorded in a lookup table. When performing radial distortion correction on the underwater image, the corresponding lookup table can be read directly, which is faster.

[0033] The reason why this invention uses a division model for radial distortion of images is that traditional image correction using polynomial models requires prior knowledge of various parameters of the camera. However, when shooting underwater, the refraction of light causes commonly used pinhole camera models to fail, making underwater camera calibration more difficult and obtaining accurate parameters of the underwater camera even more challenging.

[0034] The division model is as follows:

[0035]

[0036]

[0037]

[0038] Where, x ui y ui Let x and y represent the x-coordinate and y-coordinate of the i-th pixel in the image after radial distortion correction, respectively. di y di Let x and y represent the x and y coordinates of the i-th pixel in the image before radial distortion correction, respectively, and k represent the radial distortion coefficient to be determined based on the distorted image. di The distance from a pixel in the distorted image to the distortion center is represented by x0 and y0, respectively, which are the x-coordinate and y-coordinate values ​​of the distortion center in the distorted image. The division model for radial distortion can have multiple radial distortion coefficients. However, since one coefficient is sufficient to handle significant distortion, the division model used in this invention employs only one radial distortion coefficient.

[0039] To obtain a larger field of view for video image stitching, three cameras—left, center, and right—were placed at different angles in physical space. A standard checkerboard calibration board was used as the target. Each camera took at least 15 images as intrinsic calibration images, and each pair of cameras simultaneously took at least 5 sets of images as extrinsic calibration images.

[0040] In embodiments of the present invention, the intrinsic parameter calibration images are obtained by using a standard checkerboard calibration board as the target, with at least 15 images taken by the left camera as the left camera intrinsic parameter calibration images, at least 15 images taken by the middle camera as the middle camera intrinsic parameter calibration images, and at least 15 images taken by the right camera as the right camera intrinsic parameter calibration images.

[0041] In embodiments of the present invention, a standard checkerboard calibration board is used as the target. The left and middle cameras simultaneously capture at least five sets of images as extrinsic calibration images for the left camera. Each set of images includes two images: one captured by the left camera and one captured by the middle camera. The total number of extrinsic calibration images for the left camera is at least 10. The middle and right cameras simultaneously capture at least five sets of images as extrinsic calibration images for the right camera. Each set of images includes two images: one captured by the middle camera and one captured by the right camera. The total number of extrinsic calibration images for the right camera is at least 10. The extrinsic coefficients of the middle camera are directly set as a three-dimensional identity matrix.

[0042] Camera calibration methods can be categorized into three types: traditional camera calibration, active vision camera calibration, and camera self-calibration. Active vision camera calibration is costly and requires expensive equipment, while camera self-calibration has low accuracy and poor robustness. This invention uses a traditional camera calibration method. Zhang Zhengyou's calibration method is a simple and highly accurate traditional camera calibration method. This invention uses Zhang Zhengyou's calibration method to calibrate the intrinsic parameters of the camera for image capture, the intrinsic parameter coefficients and distortion coefficients for image capture, and the extrinsic parameters of the camera for image capture. Bundle adjustment is used to optimize all camera parameters.

[0043] Using the intrinsic parameters and distortion coefficients of their respective cameras, each calibration board image captured in physical space is subjected to distortion correction using a division model to obtain the corresponding standard image.

[0044] Using a distance vector-based method, the center of each circle in the standard image is detected and its position is recorded.

[0045] Establish the perspective projection mapping relationship between the center positions of the calibration board images in both physical space and underwater scenarios and the center positions of the corresponding standard images. Perform perspective projection on each distortion-corrected underwater image to convert the underwater image into an image in physical space. Record the perspective projection mapping relationship of each image in a lookup table.

[0046] Step 2: Calculate the contour similarity of overlapping regions of adjacent deep-sea video images using the contour similarity function.

[0047] The mapping relationship between radial distortion correction and perspective projection of each underwater scene calibration board image is used to form underwater stitching parameters, which can project the video images of the three cameras into the same coordinate system.

[0048] Based on the underwater stitching parameters, two sets of overlapping area images were determined: one of the overlapping area images captured by the left and middle cameras, and the other of the overlapping area images captured by the middle and right cameras.

[0049] In embodiments of the present invention, the two sets of overlapping region images are those where the left camera and the middle camera have overlapping fields of view, and the images captured by the two cameras have overlapping regions. The first set of overlapping region images includes the overlapping region image of the image captured by the left camera and the overlapping region image of the image captured by the middle camera. The middle camera and the right camera have overlapping fields of view, and the images captured by the two cameras have overlapping regions. The second set of overlapping region images includes the overlapping region image of the image captured by the middle camera and the overlapping region image of the image captured by the right camera.

[0050] Image edges refer to the high-frequency components in an image. Compared to the low-frequency parts of an image, the human eye is more sensitive to the high-frequency parts. Therefore, this invention uses an edge detection algorithm to assemble the contours in the overlapping areas of all detected video images into a contour image corresponding to each image.

[0051] The edge detection algorithm can be any one of the Canny algorithm, Fast edge algorithm, or HED (Holistically-Nested Edge Detection) algorithm.

[0052] Using the following contour similarity function, calculate the contour similarity for each group of overlapping region images:

[0053]

[0054]

[0055]

[0056] Among them, R a (a = 1, 2) represents the contour similarity of the contour images in group a, T a and I a T represents two overlapping region contour images obtained from different cameras in the a-th group of contour images. a ' represents T a The image after image normalization, I a 'Indicates I a The image after image normalization, w a ,ha Let m represent the width and height of the two overlapping region contour images in group a, which have the same width and height, respectively. a ,n a ) indicates that for T a Images and I a The variable for summing the images, whose value ranges from 0 to m. a <w a ,0≤n a <h a ,(m a ',n a ') indicates that for T a 'Images and I' a The variable for summing the images, whose value ranges from 0 to m. a '<w a , 0≤n a '<h a .

[0057] Step 3: Optimize the translation parameters between video images after cylindrical projection online.

[0058] Due to the poor quality of deep-sea images, the embodiments of the present invention use the Retinex algorithm to enhance deep-sea images and readjust the irradiance of the images to achieve brightness correction processing.

[0059] Since each frame of video image is captured by the camera at different angles, and all images from different shooting angles are not on the same projection plane, directly stitching the overlapping parts would destroy the visual consistency of the actual scene. Therefore, based on the underwater stitching parameters, the deep-sea video images acquired by the three cameras are converted into images in physical space and then cylindrically projected.

[0060] The cylindrical projection is accomplished by the following formula:

[0061]

[0062]

[0063] Where, x bj ,y bj (b = 1, 2, 3) represent the x-coordinate and y-coordinate of the j-th pixel in the video image captured by the b-th camera, respectively. bj ,Y bj Z bj s represents the position of the j-th pixel in the video image captured by the b-th camera in the three-dimensional world coordinate system, corresponding to the position in the two-dimensional coordinate system. b Let K represent the scale of the b-th camera, which is proportional to the focal length of the b-th camera. b R represents the intrinsic parameter coefficients of the calibrated b-th camera.b u represents the extrinsic parameter coefficients of the b-th camera after calibration. bj ,v bj This represents the x-coordinate and y-coordinate of the j-th pixel in the cylindrical projection of the video image captured by the b-th camera.

[0064] Multiple video images projected onto the same cylindrical surface only have a translation relationship. Therefore, based on the underwater stitching parameters, two sets of initial translation parameters are determined for the video images acquired by the left and middle cameras after cylindrical surface projection, and for the video images acquired by the middle and right cameras. Each set of initial translation parameters includes translation parameters in the horizontal and vertical directions.

[0065] The contour similarity function is used as the objective function, and the negative of the objective function is used as the error function.

[0066] Based on the initial translation parameters, the Levenberg-Marquardt iterative algorithm is used to optimize the two sets of translation parameters between images online. When the above error function reaches its minimum value, the two sets of translation parameters at this time are taken as the optimal translation parameters to update the current underwater stitching parameters. The updated underwater stitching parameters can align and stitch together video images acquired by three different cameras.

[0067] Step 4: Stitch together multiple deep-sea video images based on image fusion.

[0068] Based on the updated stitching parameters, two sets of overlapping area images are determined: one of the overlapping area images of the video images captured by the left and middle cameras, and the other of the overlapping area images of the video images captured by the middle and right cameras.

[0069] If images are directly overlaid, the overlapping areas will appear discontinuous. Therefore, a gradient method is used to achieve a natural transition. To balance video image stitching quality and speed, this invention uses linear blending to smoothly transition the overlapping areas.

[0070] g c (d ch ,e ch )=α ch f c0 (d ch ,e ch )+(1-α ch )f c1 (d ch ,e ch )

[0071] Among them, g c (c = 1, 2) represents the fused image, f c0 and f c1Let d represent two overlapping region images obtained by different cameras in the c-th overlapping region image group. ch ,e ch Image f represents the c-th overlapping region. c0 f c1 and the fused image g c The x-coordinate and y-coordinate of the h-th pixel, α ch This represents the weight of the h-th pixel in the two overlapping region images of the c-th group.

Claims

1. A deep-sea adaptive video stitching method based on nonlinear projection, characterized in that, This video stitching method calculates underwater video stitching parameters without relying on underwater features, performs radial distortion correction on underwater images, and optimizes deep-sea video stitching parameters online. The steps of this method are as follows: Step 1: Determine the center position of the calibration board image in both physical space and underwater scenarios; use a division model to correct the radial distortion of the underwater image; and then use perspective projection to convert the underwater image into an image in physical space. Step 2: Calculate the contour similarity of overlapping regions of adjacent deep-sea video images using the contour similarity function; Step 3: Optimize the translation parameters between video images after cylindrical projection online: Step 3.1: Based on the underwater stitching parameters, convert the deep-sea video images acquired by the three cameras into images in physical space and perform cylindrical projection. Step 3.2: Determine two sets of initial translation parameters based on the underwater stitching parameters for the video images acquired by the left and middle cameras after cylindrical projection, and for the video images acquired by the middle and right cameras. Each set of initial translation parameters includes translation parameters in the horizontal and vertical directions. Step 3.3: Using the contour similarity function as the objective function, take the negative of the objective function as the error function; Step 3.4: Based on the initial translation parameters, the Levenberg-Marquardt iterative algorithm is used to optimize the two sets of translation parameters between the images online. When the above error function reaches its minimum value, the two sets of translation parameters at this time are taken as the optimal translation parameters to update the current underwater stitching parameters. Step 4: Stitch together multiple deep-sea video images based on image fusion.

2. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The steps for determining the center positions of the calibration board images in both physical space and underwater scenarios, as described in step 1, are as follows: The first step is to use the Halcon calibration board as the target and take one image of the calibration board in physical space and one image of the calibration board in an underwater scene with three cameras. The target in the image is larger than 1 / 3 of the entire image. The second step is to use a distance vector-based method to detect the center of each calibration board image and record the center position.

3. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The division model described in step 1 is as follows: ； ； ； in, These represent the first and second images in the radial distortion corrected image. The x and y coordinates of each pixel These represent the first and second images in the image before radial distortion correction. The x and y coordinates of each pixel This represents the radial distortion coefficient to be determined based on the distorted image. This represents the pixel distance from a pixel in a distorted image to the center of the distortion. These represent the x-coordinate and y-coordinate values ​​of the distortion center in the distorted image, respectively.

4. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The correction of radial distortion in underwater images refers to the following steps being performed on each camera for correction: The first step is to perform circular arc fitting on each of the deformed straight lines in the calibration board image of the underwater scene that are at the same position as the calibration board image in the physical space, so as to obtain the center and radius of the arc. The second step is to establish a corresponding equation for each fitted circular arc, and randomly select three unselected equations to form a system of equations to solve for the distortion center and radial distortion coefficients of the calibration plate image of the underwater scene: ； in, These represent the coefficients of the general equations of the lines at the same positions on the physical space calibration board images corresponding to the three selected arcs; The third step involves substituting the radial distortion coefficients into the division model to perform radial distortion correction on the calibration plate image in the underwater scene, obtaining the corresponding radial distortion-corrected underwater calibration plate image, and recording the mapping relationship of radial distortion for each image in a lookup table.

5. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The perspective projection steps described in step 1 are as follows: The first step is to place three cameras (left, center, and right) at different angles in physical space, using a standard checkerboard calibration board as the target. Each camera takes at least 15 images as intrinsic calibration images, and each pair of cameras simultaneously takes at least 5 sets of images as extrinsic calibration images. The second step is to use Zhang Zhengyou's calibration method to calibrate the intrinsic parameters of the camera and the distortion coefficients, and to calibrate the extrinsic parameters of the camera. Finally, the bundle adjustment method is used to optimize all camera parameters. The third step is to use the intrinsic parameters and distortion coefficients of each camera to perform distortion correction on each calibration board image taken in physical space using a division model, so as to obtain the standard image corresponding to the image. The fourth step is to use a distance vector-based method to detect the center of each circle in the standard image and record its position. The fifth step is to establish the perspective projection mapping relationship between the center positions of the calibration board images in both physical space and underwater scenarios and the center positions of the corresponding standard images. Perspective projection is performed on each distortion-corrected underwater image to convert the underwater image into an image in physical space. The perspective projection mapping relationship of each image is recorded in a lookup table.

6. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The steps for calculating the contour similarity of overlapping regions between adjacent deep-sea video images in step 2 are as follows: The first step is to form underwater stitching parameters by mapping the radial distortion correction and perspective projection of each calibration board image in the underwater scene. These underwater stitching parameters can project the video images from the three cameras into the same coordinate system. The second step is to determine two sets of overlapping area images based on the underwater stitching parameters: the overlapping area images of the video images captured by the left and middle cameras, and the overlapping area images of the video images captured by the middle and right cameras. The third step is to use an edge detection algorithm to combine the contours in the overlapping areas of all the detected video images into a contour image corresponding to each image. The fourth step is to use the contour similarity function to calculate the contour similarity of each group of overlapping region images.

7. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 6, characterized in that, The contour similarity function is as follows: ； ； ； in, Indicates the first Contour similarity of a group of contour images. and Indicates the first Two overlapping region contour images obtained from different cameras in a set of contour images. express The image after image normalization, express The image after image normalization, They represent the first one with the same width and height. The width and height of the outline images of the overlapping region in the group. Indicates to Images and The variable for summing the images, the range of which is: , Indicates to Images and The variable for summing the images, the range of which is: , .

8. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The cylindrical projection mentioned in step 3 is as follows: ； ； in, They represent the first The video images captured by the camera are the first The x and y coordinates of each pixel Indicates the first The video images captured by the camera are the first The position of a pixel in a two-dimensional coordinate system corresponds to its position in three-dimensional world coordinates. Indicates the first The scale of the first camera, compared to the first The focal length of each camera is directly proportional to its focal length. Indicates the calibrated first The intrinsic parameters of each camera, Indicates the calibrated first The extrinsic parameters of a camera, Indicates the first The video images captured by the camera are projected onto the cylindrical image. The x-coordinate and y-coordinate values ​​of each pixel.

9. The deep-sea adaptive video stitching method based on nonlinear projection according to claim 1, characterized in that, The image fusion steps described in step 4 are as follows: The first step is to determine two sets of overlapping area images based on the updated stitching parameters: the overlapping area images of the video images captured by the left and middle cameras, and the overlapping area images of the video images captured by the middle and right cameras. The second step is to use linear fusion to smooth the transition between overlapping image regions: ； in, This represents the merged image. and They represent the first Two overlapping region images obtained from different cameras in a set of overlapping region images. Indicates the first Group overlapping region images , and the fused image The Middle The x and y coordinates of each pixel Indicates the first In the two overlapping region images of the group, the first The weights of each pixel.

Images