An underwater VR field of view extension method and device based on multi-camera fusion

Abstract

The application provides a kind of underwater VR field of view extension method and device based on multi-camera fusion, belongs to underwater detection technical field.The method provided by the application: based on the two binocular cameras fixedly installed on remote control submersible, the first planar image and the second planar image in front of remote control submersible are acquired;The views corresponding to the first planar image and the second planar image are merged to obtain panoramic planar image of left view and right view;The panoramic planar image of left view and right view is projected to spherical surface respectively, and panoramic spherical image is generated;The position of planar window is calculated according to the real-time pose of remote control submersible, and the size of target window is generated according to the field of view size of binocular camera;Target spherical image is intercepted with target window, and sent to VR glasses for display.The underwater VR field of view extension method and device based on multi-camera fusion provided by the application are used to increase the underwater field of view of remote control submersible.

Description

Technical Field

[0001] This application relates to the field of underwater detection technology, and in particular to an underwater VR field of view expansion method and device based on multi-camera fusion. Background Technology

[0002] With the continuous expansion of fields such as marine resource exploration, deep-sea geological research, and underwater environmental monitoring, remotely operated vehicles (ROVs) have become core equipment for human exploration of the deep sea and underwater operations. Leveraging their ability to reach depths exceeding 10,000 meters and replace manual labor in high-risk tasks, ROVs are gradually becoming key tools for research institutions, energy companies, and other organizations conducting underwater work. Meanwhile, VR technology, with its immersive experience, continues to deepen its application across various industries, providing a new direction for overcoming perceptual limitations in traditional operational scenarios. Combining VR technology with ROV vision systems to construct an underwater perception and interaction system with a sense of presence has become an important research trend for improving the efficiency and ease of operation of ROVs.

[0003] Currently, the industry has begun to explore integrating VR technology into underwater operations. In terms of the visual hardware configuration for remotely operated vehicles (ROVs), the mainstream solution is to equip them with commercially available industrial cameras. Some upgrades utilize binocular cameras to acquire basic stereoscopic visual information, providing data support for subsequent image processing and depth perception.

[0004] However, existing technologies are insufficient to meet the demands for efficient and precise underwater operations. On the one hand, the industrial cameras mounted on traditional remotely operated vehicles (ROVs) lack depth information, making it impossible for operators to accurately determine the location of targets. This forces them to rely on robotic arms for small-scale exploratory operations, significantly reducing sampling efficiency and placing extremely high demands on operator skills. Ordinary researchers are unable to participate directly due to the high operational threshold, increasing cross-role communication costs and potentially causing them to miss important marine discoveries. On the other hand, even solutions using VR glasses as the visual module are limited by the field of view of binocular cameras, preventing the achievement of wide-field observation. Furthermore, relying solely on mechanical structures to adjust camera posture to expand the field of view is prone to lag and jitter, especially in underwater operations where the horizontal field of view adjustment requirement is much greater than the vertical field of view adjustment, severely impacting the immersive experience and operational accuracy. Summary of the Invention

[0005] In view of this, this application provides an underwater VR field of view expansion method and device based on multi-camera fusion, which is used to increase the underwater field of view of remotely operated vehicles.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] The first aspect of this application provides a method for expanding the underwater VR field of view based on multi-camera fusion, the method comprising:

[0008] The first planar image and the second planar image in front of the remotely operated vehicle are obtained by two binocular cameras fixedly installed on the remotely operated vehicle. The first planar image and the second planar image are obtained from different binocular cameras and capture different areas.

[0009] The views corresponding to the first planar image and the second planar image are merged to obtain panoramic planar images of the left and right views;

[0010] The panoramic planar images of the left and right views are projected onto the sphere to generate a panoramic spherical image;

[0011] The position of the planar window is calculated based on the real-time pose of the remotely operated vehicle, and the size of the planar window is calculated based on the field of view of the binocular camera to generate the target window;

[0012] The target spherical image is captured in the panoramic spherical image using the target viewport and sent to the VR glasses for display.

[0013] A second aspect of this application provides an underwater VR field of view extension device based on multi-camera fusion, the device including a panoramic extension module, a signal transmission module, and VR glasses;

[0014] The panoramic extension module includes two binocular cameras, used to acquire a first planar image and a second planar image in front of the remotely operated vehicle, and to generate a target spherical image based on the first planar image and the second planar image.

[0015] The signal transmission module is used for signal transmission between the panoramic extension module and the VR glasses;

[0016] The VR glasses are used to receive images of the target spherical surface and display them to the user;

[0017] The VR glasses are also used to control the posture of the panoramic extension module according to the user's head display posture.

[0018] The underwater VR field of view expansion method and apparatus based on multi-camera fusion provided in this application expands the underwater horizontal field of view of the remotely operated vehicle (ROV) by utilizing two binocular cameras arranged in front of the ROV and reduces the amount of image data processing. Specifically, planar images of different areas in front of the ROV are acquired by the two binocular cameras, and the views are merged to form panoramic planar images of the left and right fields of view. Then, the panoramic planar images of the left and right fields of view are projected to generate a panoramic spherical image. The target view is determined by combining the real-time pose of the ROV with the camera field of view size. The target spherical image is then cropped from the panoramic spherical image using the target view and sent to VR glasses for display. By employing a dual-camera setup and image stitching technology, the limitations of traditional single-camera or mono- and binocular cameras are overcome, broadening the underwater observation field of view of remotely operated vehicles (ROVs) and solving the problem of incomplete observation caused by narrow field of view in existing technologies. Furthermore, by accurately generating target windows based on pose and field of view size, and displaying them with VR glasses, operators can receive more immersive visual feedback that closely matches the actual scene, helping them accurately determine the location of targets, reducing tentative operations by the robotic arm, improving work efficiency, and reducing reliance on operator skills, communication costs, and preventing the loss of important marine discoveries. Attached Figure Description

[0019] Figure 1 A flowchart of an embodiment of the underwater VR field of view expansion method based on multi-camera fusion provided in this application;

[0020] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the underwater VR field of view extension device based on multi-camera fusion provided in this application. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0024] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0025] Figure 1 This is a flowchart of an embodiment of the underwater VR field-of-view expansion method based on multi-camera fusion provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0026] S101. Based on two binocular cameras fixedly installed on the remotely operated vehicle (ROV), a first planar image and a second planar image in front of the ROV are obtained. The first planar image and the second planar image are from different binocular cameras and capture different areas.

[0027] Specifically, the two binocular cameras installed on the remotely operated vehicle (ROV) are of the same model. They are used to capture a first planar image and a second planar image in front of the ROV, respectively. The first planar image is captured by one of the two cameras, and the second planar image by the other. Because the two cameras are installed in different positions, the first and second planar images include content from different areas. It should be noted that the relative distance between the two cameras may overlap, causing some content in the first and second planar images to be the same. In other words, the content in the first and second planar images may be completely different or partially different, depending primarily on the relative distance between the two cameras. Using two binocular cameras to capture different areas breaks the limitations of the limited shooting range of traditional single and binocular cameras, providing initial data support for field-of-view expansion. Simultaneously, preserving overlapping areas creates conditions for feature point matching during subsequent view merging, ensuring the feasibility and accuracy of image stitching.

[0028] It should be noted that in actual deployment, the two binocular cameras are not limited to a single fixed installation method, but can be flexibly arranged on the end effector of the robotic arm, the main body of the remotely operated vehicle (ROV), or a separate gimbal, depending on the specific task requirements. When installed on the end effector of the robotic arm, the camera can approach the target object at close range as the robotic arm joints move, enabling high-precision observation and fine-tuning guidance of local areas, which is especially suitable for close-range operation scenarios such as grasping and sampling. When installed on the front end of the ROV, the camera is fixed to the ROV body, and a wide range of horizontal and vertical fields of view can be scanned by adjusting the ROV's own heading and attitude, which is suitable for environmental perception in wide-area search and long-distance cruise missions. When installed on the gimbal, the camera's orientation can be independently adjusted without changing the ROV's heading by tilting and yawing the gimbal. This can compensate for the line-of-sight shift caused by ROV swaying and enable rapid target tracking and multi-angle observation. The installation method can be selected individually or in combination according to the actual working conditions. For example, a pair of binocular cameras can be fixed to the submersible body to obtain stable panoramic images with a large field of view, while a second pair of binocular cameras can be added to the end of the robotic arm for close-range precision operations. This forms a multi-level, reconfigurable visual perception system that takes into account both the needs of large-scale environmental monitoring and local precision operations, significantly improving the system's adaptability and flexibility to different underwater tasks.

[0029] Optionally, the two binocular cameras are mounted on the same horizontal plane of the remotely operated vehicle (ROV) at the same height, with the optical lenses of the two binocular cameras facing the same direction of travel as the ROV. The relative distance between the two binocular cameras is determined based on the shooting space of the binocular cameras.

[0030] Specifically, according to the preset installation standards, two binocular cameras with identical parameters are fixed on the same horizontal plane as the remotely operated vehicle (ROV) in the same direction of travel. The camera positions are adjusted using a level and calibration tools to ensure that the two cameras are on the same horizontal plane, at the same height, and with their lenses facing the same direction of travel as the ROV. This ensures that the first and second planar images captured can include the content in front of the field of view, ensuring that the captured areas are both different and overlapping.

[0031] The installation method, with the camera on the same horizontal plane and at the same height, eliminates parallax interference in the vertical direction, ensuring that the left and right views captured by the two binocular cameras are geometrically aligned. The reasonable relative distance design ensures that the proportion of the overlapping area of ​​the images is appropriate, providing sufficient data support for subsequent extraction of overlapping feature points and perspective transformation stitching, avoiding stitching breaks due to insufficient overlap or limiting the field of view expansion effect due to excessive overlap. The lens orientation is consistent with the forward direction of the remotely operated vehicle, making the camera's shooting range highly matched with the operating area of ​​the remotely operated vehicle. The image observed by the operator through VR glasses can directly correspond to the working scene in front, allowing them to grasp the core working area information without additional angle adjustments.

[0032] S102. Merge the views corresponding to the first planar image and the second planar image to obtain panoramic planar images of the left and right views.

[0033] Specifically, both the first and second planar images include corresponding left and right views. The corresponding groups of left and right views in the first and second planar images are clearly defined, that is, the left view of the first and second planar images together form the left view group, and the right view of the first and second planar images together form the right view group. The two groups of corresponding views are processed separately, and the two views in each group are integrated into a complete view through image fusion technology, finally obtaining the panoramic planar image of the left field of view and the panoramic planar image of the right field of view.

[0034] Furthermore, the first and second planar images are preprocessed to remove noise, underwater light interference, and other impurities, improving image clarity. Then, two sets of corresponding views are distinguished, and the first left view, second left view, first right view, and second right view are selected. The first left view and first right view refer to the left and right views in the first planar image, and the second left view and second right view refer to the left and right views in the second planar image. For the left view group, common feature points between the first and second left views are found using image feature recognition technology. Based on these common feature points, the spatial positions of the two views are aligned, and an image stitching algorithm is used to eliminate differences in view boundaries, achieving seamless fusion and generating a left-field panoramic planar image. Using the same processing logic, feature recognition, position alignment, and stitching fusion are performed on the right view group to obtain a right-field panoramic planar image.

[0035] Further, the preprocessing steps for the first and second planar images include: performing global background light estimation and transmittance map calculation on the first and second planar images based on the dark channel prior principle; refining the transmittance using guided filtering to remove the fog-like blur caused by water scattering; collaboratively correcting the attenuation differences of the red, green, and blue channels through the gray-world algorithm and local histogram matching to compensate for color shifts caused by the different absorption degrees of different wavelengths of light by the water, restoring the natural colors of the image; employing a combination of adaptive contrast stretching and limited contrast histogram equalization to enhance edge texture and local contrast while maintaining image details and suppressing noise amplification caused by underwater suspended particles; and using bilateral filtering to smooth the enhanced image while preserving edges, removing isolated noise points while retaining the outline of the target object, thus providing clear, color-accurate, and geometrically distorted basic image data for subsequent feature point extraction and view stitching. For the specific implementation process of these steps, please refer to the description in the existing technology; it will not be repeated here.

[0036] The view merging operation integrates the scattered views captured by two binocular cameras into a complete panoramic planar image, significantly expanding the observation range of the left and right fields of view. This solves the problem of narrow field of view and inability to fully cover underwater operation scenarios by traditional single and binocular cameras. Through feature alignment and seamless stitching, the integrity and coherence of the panoramic planar image are ensured, avoiding problems such as view breaks and distortion in overlapping areas. This provides high-quality planar image data for subsequent spherical projection, ensuring the clarity and accuracy of the final VR display effect.

[0037] Furthermore, before merging the views corresponding to the first planar image and the second planar image, the method provided in this embodiment also includes:

[0038] Determine the sensor width and pixel resolution of the sub-cameras of the binocular camera; using the starting coordinates of the horizontal pixels in the first planar image as a reference, calculate the left horizontal pixel range of the imaging area of ​​the left sub-camera and the right horizontal pixel range of the imaging area of ​​the right sub-camera based on the sensor width and pixel resolution; perform horizontal pixel segmentation on the first planar image based on the left horizontal pixel range and the right horizontal pixel range; take the imaging area of ​​the left horizontal pixel range as the first left view and the imaging area of ​​the right horizontal pixel range as the first right view.

[0039] Specifically, the sensor width of the sub-camera refers to its physical width, which can be obtained from the camera's technical manual. Pixel resolution refers to the horizontal number of pixels in the sub-camera's output image, i.e., the total number of pixels in the horizontal direction. The ratio of sensor width to pixel resolution represents the physical size of a single pixel. Taking the horizontal pixel starting coordinates of the first planar image (usually 0 pixels) as the reference point, the first planar image is the original image captured synchronously by the binocular cameras, containing the complete imaging areas of the left and right sub-cameras. Since the left and right sub-cameras are devices with identical parameters, and their optical centers are aligned horizontally during installation, their imaging areas are distributed left and right in the first planar image, with equal widths. It can be understood that the number of horizontal pixels occupied by the imaging area of ​​a single sub-camera in the first planar image is equal to its pixel resolution. Therefore, the left and right regions are symmetrically divided by the resolution value.

[0040] Furthermore, the pixel value corresponding to the pixel resolution of the left sub-camera from the reference point is the left horizontal pixel range, and the pixel value corresponding to the pixel resolution of the right sub-camera to 2 × the pixel resolution of the right sub-camera is the right horizontal pixel range. Based on the calculated left and right horizontal pixel ranges, pixel-level segmentation is performed in the horizontal direction of the first planar image. For example, if the pixel resolution is 1920, then the image is segmented from pixel 1919, dividing it into two continuous regions: left (0~1919) and right (1920~3839). The imaging area corresponding to the segmented left horizontal pixel range is the first left view, corresponding to the content captured by the left sub-camera; the imaging area corresponding to the segmented right horizontal pixel range is the first right view, corresponding to the content captured by the right sub-camera. The implementation process of the second planar image segmentation is the same as that of the first planar image segmentation, and will not be repeated here.

[0041] Furthermore, the specific steps for merging the views corresponding to the first planar image and the second planar image to obtain panoramic planar images of the left and right views include:

[0042] (1) Perform internal parameter calibration on the left and right views of the first planar image and the second planar image respectively;

[0043] Specifically, the left view of the first planar image is designated as the first left view, the right view of the first planar image is designated as the first right view, and the left view of the second planar image is designated as the second left view, and the right view of the second planar image is designated as the second right view.

[0044] Furthermore, a camera calibration platform was built, and a standard checkerboard calibration board was prepared. The calibration board was placed within the common shooting range of the two binocular cameras, and the angle and distance of the calibration board were adjusted to ensure that both binocular cameras could clearly capture checkerboard images in different poses (at least 10-15 sets of calibration images at different angles and distances were collected). Using Zhang Zhengyou's camera calibration algorithm, the two binocular cameras were calibrated separately. Each set of checkerboard images was read, the corner points of the checkerboard were identified, and the world coordinates and pixel coordinates of the corner points were calculated. The camera intrinsic parameter matrix (including focal length and principal point coordinates) and distortion coefficients (radial distortion and tangential distortion) were solved using the least squares method. Finally, the intrinsic parameter data of the left and right sub-cameras of the two binocular cameras were output respectively. For the implementation steps of calibration based on Zhang Zhengyou's camera calibration algorithm, please refer to the description in the relevant technology, which will not be repeated here.

[0045] Furthermore, based on the calculated intrinsic parameter matrix and distortion coefficients, distortion correction is performed on the first left view, first right view, second left view, and second right view respectively. Using a pixel remapping algorithm, each pixel in the distorted image is mapped to a distortion-free ideal image coordinate system, generating four sets of standard views without geometric distortion, thus completing the intrinsic parameter calibration.

[0046] Intrinsic calibration enables the left and right sub-cameras of the two stereo cameras to have a unified pixel-space coordinate mapping rule, ensuring that the target pixel coordinates of the first left view and the second left view, and the first right view and the second right view at the same spatial position can be compared, providing a precise coordinate reference for the spatial alignment of subsequent view stitching.

[0047] (2) Extract the first overlapping feature points of the first left view and the second left view, and the second overlapping feature points of the first right view and the second right view;

[0048] Specifically, the views are divided into two groups according to their orientation: the first group consists of the first left view and the second left view, and the second group consists of the first right view and the second right view. Feature points that reflect the relationship between overlapping areas are extracted for each group of views, namely the first overlapping feature point and the second overlapping feature point.

[0049] Furthermore, the scale-invariant feature transformation algorithm is used to process the two sets of views separately. Taking the first set (first left view and second left view) as an example, the image scale space is constructed, extreme points are detected and the location of feature points is determined; then the gradient direction and magnitude of the feature points are calculated to generate 128-dimensional feature descriptors. The feature descriptors can uniquely identify the local image information of the feature points; similarly, feature point extraction and feature descriptor generation are completed for the first right view and the second right view.

[0050] Furthermore, using the K-nearest neighbor matching algorithm, the feature descriptors of the first left view and the second left view are compared, the Euclidean distance between the feature descriptors is calculated, and the candidate matching pair with the smallest Euclidean distance is selected. A Ratio test is introduced (usually the Ratio threshold is set to 0.8) to eliminate mismatched pairs with excessively large Euclidean distance ratios, and the matching pairs that meet the conditions are retained as the first overlapping feature points. The above operation is repeated to complete the feature point matching and selection of the first right view and the second right view, and the second overlapping feature points are obtained.

[0051] By extracting overlapping feature points, the overlapping areas of the first left view and the second left view, as well as the first right view and the second right view, were clearly defined, providing specific pixel-level references for subsequent view splicing alignment operations and avoiding splicing range deviations caused by blurred positioning of overlapping areas.

[0052] (3) Based on perspective transformation and combined with the first overlapping feature points, the first left view and the second left view are stitched together to form a panoramic planar image of the left field of view;

[0053] Specifically, using the first left view as the reference image, the perspective transformation matrix from the second left view to the coordinate system of the first left view is solved using the pixel coordinates of the first overlapping feature points. The perspective transformation matrix can realize the spatial pose adjustment of the second left view, aligning it with the first left view in the overlapping area. Based on the perspective transformation matrix, the second left view is reprojected pixel by pixel, transforming it to the coordinate system of the first left view. The aligned two views are then subjected to edge blending processing to eliminate brightness and color differences at the splicing boundary, ultimately generating a panoramic planar image of the left field of view.

[0054] Furthermore, multiple sets of non-collinear matching points are randomly selected from the first overlapping feature points. The perspective transformation matrix solution requires at least 4 sets of corresponding points. The pixel coordinates (x1, y1) of each corresponding point in the first left view and the pixel coordinates (x2, y2) in the second left view are obtained respectively. The perspective transformation equations are solved by the least squares method to obtain a 3×3 perspective transformation matrix. The perspective transformation matrix contains transformation parameters such as translation, rotation, scaling, and shearing, which can achieve accurate mapping from the second left view to the coordinate system of the first left view.

[0055] Furthermore, each pixel coordinate (u,v) of the second left view is substituted into the perspective transformation formula to calculate its corresponding coordinate (u',v') in the coordinate system of the first left view. The pixel values ​​of non-integer coordinates are supplemented by bilinear interpolation algorithm to complete the spatial transformation of the second left view, so that the overlapping areas of the two views are completely aligned.

[0056] Furthermore, a fade-in / fade-out fusion algorithm is adopted. In the stitching boundary area between the two views, the pixel brightness and color of the first left view are gradient-attenuated, while the pixels of the transformed second left view are gradient-enhanced, so that the image transition at the boundary is natural. Finally, the non-overlapping areas of the two views are merged with the fused overlapping area to generate a full-size, seamless panoramic planar image of the left field of view.

[0057] By stitching together the images, the observation range of the left field of view is expanded from the field of view of the left sub-camera of a single binocular camera to the field of view of two binocular cameras. This breaks through the limitations of the field of view of a traditional single camera, enabling the left field of view to cover a wider underwater area in front of the remotely operated vehicle and solving the problem of incomplete observation of the left field of view due to narrow field of view.

[0058] (4) Based on perspective transformation and combined with the second overlapping feature points, the first right view and the second right view are stitched together to form a panoramic planar image of the right field of view.

[0059] Specifically, the implementation process for splicing the right view is similar to the implementation steps for splicing the left view described above. In this embodiment, it is not limited to these steps.

[0060] The field of view of the right field-view panoramic planar image is consistent with that of the left field of view, so that the left and right fields of view of the entire binocular vision system can symmetrically cover the area in front of the remotely operated vehicle. This provides symmetrical image data for the subsequent binocular display of VR glasses, avoiding visual fatigue or spatial judgment errors for the operator. The high-quality right field-view panoramic planar image, combined with the left field-view panoramic planar image, forms complete binocular panoramic visual data. After being projected onto a spherical surface, it can provide VR glasses with images that conform to the characteristics of human binocular vision. This allows the operator to intuitively perceive the spatial depth and breadth of the underwater scene through VR devices, improving the sense of presence and accuracy of remote operation.

[0061] S103. Project the panoramic planar images of the left and right views onto the sphere to generate a panoramic spherical image.

[0062] Specifically, a spherical projection coordinate system adapted to underwater operation scenarios is constructed, and key parameters such as the origin and coordinate axis directions of the coordinate system are determined to provide a spatial reference for the projection of panoramic planar images. For the panoramic planar images of the left and right views, spherical projection algorithms are used to map the pixel information of the planar images to the preset spherical coordinate system to form corresponding panoramic spherical images. The projected left and right field-of-view spherical images are integrated to ensure that the two spherical images maintain a consistent positional relationship in the spatial coordinate system, and finally, the panoramic spherical images corresponding to the left and right views are obtained.

[0063] Furthermore, based on the installation positions and shooting angles of the two binocular cameras, and taking the operating center of the remotely operated vehicle (ROV) as a reference, the midpoint of the line connecting the optical centers of the two binocular cameras is used as the origin of the spherical projection coordinate system. The coordinate axis directions are set to be consistent with the spatial attitude of the ROV, ensuring that the projected spherical image accurately reflects the underwater scene. For the left-view panoramic planar image, the spatial coordinates of each pixel in the spherical coordinate system are calculated using a spherical projection algorithm, converting the planar pixel information into point cloud data in spherical space. Then, the left-view panoramic spherical image is generated by rendering based on the point cloud data. Similarly, the right-view panoramic planar image is projected to obtain the right-view panoramic spherical image.

[0064] Spherical projection transforms a planar panoramic image into a panoramic spherical image with three-dimensional spatial characteristics, simulating the visual perception of the human eye in a real underwater environment. This avoids the image stretching and distortion problems that occur when planar images are displayed in a large field of view, thus improving the spatial realism of the image.

[0065] Furthermore, the specific steps for generating a panoramic spherical image by projecting the panoramic planar images of the left and right views onto the sphere respectively include:

[0066] (1) Construct a spherical coordinate system with the midpoint of the line connecting the optical centers of the two binocular cameras as the center of the sphere;

[0067] Specifically, determine the midpoint of the line connecting the optical centers of the two binocular cameras, take the midpoint as the origin, and set the coordinate axis direction of the spherical coordinate system in combination with the remotely operated vehicle (ROV) operation scenario, so that the coordinate system can accurately reflect the underwater spatial orientation. It is necessary to ensure that the subsequent image projection has a clear correspondence with the ROV's forward direction and the camera's shooting direction.

[0068] Furthermore, with the remotely operated vehicle (ROV) in a horizontally stationary state, a fixed structural point at the front of the ROV is used as a temporary reference point. The three-dimensional coordinates (x1, y1, z1) and (x2, y2, z2) of the optical centers (O1) of the first and second binocular cameras relative to the temporary reference point are measured respectively. The coordinates of the midpoint are calculated according to the distance calculation formula of the coordinate system. With the midpoint as the origin, the Z-axis is set to be consistent with the forward direction of the ROV (i.e., parallel to the principal optical axis of the camera lens), the X-axis is horizontal and perpendicular to the Z-axis and points to the right side of the ROV, and the Y-axis is perpendicular to the XZ plane and points upward, forming a spherical coordinate system with the right-hand screw rule, thus completing the construction of the spherical coordinate system.

[0069] By taking the midpoint of the line connecting the optical centers of the two cameras as the origin, the spatial reference of the spherical coordinate system is made to coincide with the visual center of the camera, thus avoiding the inaccurate reflection of the spatial relationship of the underwater scene captured by the camera due to coordinate system offset.

[0070] (2) Obtain the planar pixel coordinates of the panoramic planar images of the left and right views;

[0071] Specifically, iterate through all pixels in the panoramic planar images of the left and right views, obtain the coordinates of each pixel in the corresponding image pixel coordinate system, and form a complete set of pixel coordinates.

[0072] Furthermore, the panoramic planar images of the left and right views are read using image processing software to obtain the pixel resolution parameters of the images and determine the range of pixel coordinate values. The position of each pixel in the left view is read sequentially in row priority order, and its coordinates are recorded. The same method is used to traverse the panoramic planar image of the right view, and the coordinates of each pixel are recorded. Pixel coordinate matrices for the left and right views are generated respectively, thus completing the acquisition of planar pixel coordinates.

[0073] A complete set of planar pixel coordinates is the basic data source for converting planar images into spherical images. This ensures that each planar pixel can find its corresponding spherical spatial position, avoiding incomplete image projection due to missing pixel coordinates. A clear pixel coordinate system ensures that each pixel in the left and right views has a unique coordinate identifier, avoiding coordinate confusion during subsequent mapping and ensuring that the generated panoramic spherical image has orderly and error-free pixel arrangement.

[0074] (3) Based on the horizontal field of view and focal length, the planar pixel coordinates are mapped into the spherical coordinate system to obtain the cylindrical pixel coordinates;

[0075] Specifically, determine the horizontal field of view and camera focal length corresponding to the panoramic planar images of the left and right views; based on the relationship between the horizontal field of view, focal length and planar pixel coordinates, establish a mathematical mapping model to convert the planar pixel coordinates (u,v) into cylindrical pixel coordinates (r,θ,φ) in the spherical coordinate system, where r is the cylindrical radius, θ is the horizontal azimuth angle and φ is the vertical pitch angle.

[0076] Further, determine the horizontal field of view α of the panoramic planar images in the left and right views; read the focal length f (unit: pixels) of the sub-camera from the binocular camera technical manual, and determine the horizontal center pixel coordinate u0 (u0=W / 2, W is the image width) and the vertical center pixel coordinate v0 (v0=H / 2, H is the image height); for the planar pixel coordinates (u,v), calculate the horizontal azimuth angle θ of the pixel in the spherical coordinate system according to the formula θ=[(u-u0) / W]×α; based on the camera's vertical field of view β (determined by camera parameters), calculate the vertical pitch angle φ of the pixel using the formula φ=[(v0-v) / H]×β; set the cylindrical radius r to a fixed value (usually a constant related to the camera focal length to ensure an appropriate image projection ratio), and combine the calculated horizontal azimuth angle and vertical pitch angle to form the cylindrical pixel coordinates (r,θ,φ) of each pixel, completing the coordinate mapping from the plane to the cylinder.

[0077] By calculating the correlation between the horizontal field of view, focal length, and pixel coordinates, the two-dimensional planar pixel coordinates are converted into cylindrical coordinates with three-dimensional spatial orientation information, providing a spatial coordinate basis for the subsequent generation of spherical images and solving the problem that planar images cannot reflect the spatial depth and breadth of underwater scenes.

[0078] (4) Generate a panoramic spherical image based on the cylindrical pixel coordinates.

[0079] Specifically, based on the spatial characteristics of the spherical coordinate system, the cylindrical pixel coordinates are converted into two-dimensional pixel coordinates of the panoramic spherical image, and the pixel position of the spherical image corresponding to each cylindrical coordinate is determined. Based on the converted pixel coordinates, the pixel information of the panoramic planar images of the left and right views is assigned to the corresponding pixel positions of the spherical images to generate the panoramic spherical images corresponding to the left and right views.

[0080] Furthermore, the resolution of the panoramic spherical image is determined, and a mapping relationship between the pixel coordinates of the spherical image and the cylindrical coordinates is established. Based on the number of pixels U_total in the horizontal direction of the spherical image and the range of horizontal azimuth angles, the horizontal pixel coordinates of the spherical image corresponding to each horizontal azimuth angle θ are calculated as U = U_total × (θ + 180°) / 360°. Similarly, based on the number of pixels V_total in the vertical direction of the spherical image and the range of vertical pitch angles, the vertical pixel coordinates are calculated as V = V_total × (φ + 90°) / 180°. For each cylindrical pixel coordinate (r, θ, φ) in the left view, the RGB color value and brightness value of the corresponding pixel in the panoramic planar image of the left view are assigned to the (U, V) pixel position in the spherical image according to the calculated (U, V). The same method is used to complete the information assignment from the cylindrical pixels of the right view to the pixels of the spherical image. The generated spherical image is then processed for edge smoothing and color equalization to eliminate pixel jaggedness and color banding problems that may occur during the coordinate transformation process, and finally, the panoramic spherical images of the left and right views are obtained.

[0081] Panoramic spherical images can simulate the visual effect of the human eye in three-dimensional space, avoiding the stretching and distortion problems of traditional planar images when displayed in a large field of view. This makes the images displayed on VR glasses more in line with the operator's visual habits and enhances the sense of presence in underwater operations. Furthermore, panoramic spherical images cover the complete field of view in front of the remotely operated vehicle (ROV) after stitching, and each pixel corresponds to a clear underwater spatial orientation. This provides a complete image carrier for generating target windows and capturing images of specific areas based on the real-time pose of the ROV, ensuring that the captured target image accurately reflects the underwater scene currently facing the ROV.

[0082] S104. Calculate the position of the planar window based on the real-time pose of the remotely operated vehicle, calculate the size of the planar window based on the field of view of the binocular camera, and generate the target window.

[0083] Specifically, the system collects real-time pose information of the remotely operated vehicle (ROV) during underwater operations, including but not limited to parameters such as heading, pitch, and roll. Based on the ROV's real-time pose data and the coordinate system parameters of the panoramic spherical image, the spatial position of the planar viewport in the panoramic spherical image is determined, ensuring that the viewport position matches the current orientation of the ROV. The field-of-view size parameters of the binocular camera are extracted, and the width, height, and other dimensional parameters of the planar viewport are calculated according to the ratio between the field-of-view size and the panoramic spherical image. The position and size parameters of the planar viewport are integrated to delineate a specific viewport area in the panoramic spherical image, forming a target viewport for subsequent image cropping.

[0084] Furthermore, the attitude sensors (such as inertial measurement units and underwater positioning systems) onboard the remotely operated vehicle (ROV) collect real-time attitude data such as heading, pitch, and roll angles. This data is transmitted to a data processing module for filtering and correction to remove interference noise and obtain accurate real-time attitude parameters. Using the coordinate system of the panoramic spherical image as a reference, the horizontal position of the planar viewport is determined based on the ROV's real-time heading angle. The vertical position of the viewport is adjusted in conjunction with the pitch angle to ensure that the orientation of the planar viewport aligns with the ROV's current direction of travel. Simultaneously, the technical parameters of the binocular camera are read to obtain its horizontal and vertical field of view. Based on the correspondence between the pixel resolution and field of view of the panoramic spherical image, the pixel width and height of the planar viewport are calculated. Based on the calculated position and size parameters, a rectangular viewport area is marked in the panoramic spherical image. This rectangular viewport area is the target viewport, defining the specific range for subsequent image cropping.

[0085] The window position is calculated based on the real-time pose of the rectangular viewport region, enabling the target window to dynamically adjust with the attitude changes of the rectangular viewport region. This ensures that the captured image always corresponds to the underwater scene currently facing the rectangular viewport region, preventing the observation scene from becoming disconnected from the actual operation scene due to changes in the attitude of the rectangular viewport region. This ensures that operators can grasp the environmental information of the operation area in real time. At the same time, the window size is calculated by combining the field of view size of the binocular camera, so that the field of view of the target window is adapted to the imaging capability of the camera. This fully utilizes the large field of view advantage of the panoramic spherical image, while avoiding the problem of the image details being blurred due to the window being too large or the observation range being limited due to the window being too small. This provides the optimal image size for VR display, improving the visual experience and operational accuracy.

[0086] Furthermore, the specific steps for calculating the position of the planar viewport based on the real-time pose of the remotely operated vehicle include:

[0087] (1) Obtain the real-time attitude parameters of the remotely operated vehicle;

[0088] Specifically, the system relies on the attitude detection system built into the remotely operated vehicle (ROV) to collect raw data. An inertial measurement unit (IMU) is used to collect the pitch and roll angles of the ROV. This unit senses the angular and linear motion of the ROV through internal accelerometers and gyroscopes and outputs raw attitude data. At the same time, it is combined with an underwater acoustic positioning system or GPS (applicable to shallow water areas) for auxiliary correction to obtain more accurate heading angle data and avoid measurement deviations caused by underwater environmental interference from a single device.

[0089] (2) Establish a spherical coordinate system with the center of the panoramic spherical image as the origin, wherein the reference heading of the spherical coordinate system is consistent with the initial heading of the remotely operated vehicle;

[0090] Specifically, the origin of the spherical coordinate system is clearly defined as the center of the panoramic spherical image. The center of the sphere is consistent with the origin of the spherical coordinate system constructed when the panoramic spherical image was generated earlier (the midpoint of the line connecting the optical centers of the two binocular cameras), ensuring the consistency of the coordinate system spatial reference. The reference heading of the spherical coordinate system (i.e., the reference direction in the coordinate system) is set to be the same as the initial heading when the remotely operated vehicle starts its operation. At the same time, the three-dimensional coordinate axes of the coordinate system are defined in combination with the underwater spatial characteristics, so that the coordinate system can accurately correspond to the operating spatial orientation of the remotely operated vehicle.

[0091] Furthermore, when generating the panoramic spherical image, the center of the sphere is determined to be the midpoint of the line connecting the optical centers of the two binocular cameras. This center is directly used as the origin of the newly constructed spherical coordinate system to avoid spatial orientation confusion caused by changes in the origin. When the remotely operated vehicle (ROV) is started and enters the operational preparation state, the initial heading of the ROV at this time (such as the initial azimuth angle determined by the acoustic positioning system) is recorded. The initial heading is set as the reference heading of the spherical coordinate system (usually defined as the positive direction of the Z-axis of the coordinate system, corresponding to the forward direction of the ROV). With the origin as the center, following the right-hand screw rule, the X-axis is set to be horizontal and perpendicular to the Z-axis and pointing to the right side of the ROV, and the Y-axis is perpendicular to the XZ plane and pointing upward, forming a complete spherical coordinate system. This ensures that each coordinate axis of the coordinate system can form a clear correspondence with the spatial movement direction of the ROV.

[0092] The reference heading is consistent with the initial heading of the remotely operated vehicle (ROV), and the coordinate axis direction matches the ROV's movement direction, enabling the spherical coordinate system to directly map the ROV's operating space. The subsequent planar viewpoint position calculated based on this coordinate system can accurately correspond to the actual underwater scene orientation. The origin of the coordinate system is consistent with the center of the panoramic spherical image, ensuring that the planar viewpoint position calculation can be based on the spatial range of the panoramic spherical image, avoiding the viewpoint position from exceeding the range of the panoramic spherical image or not matching the scene orientation in the image due to coordinate system reference deviation.

[0093] (3) Calculate the position of the planar window in the spherical coordinate system based on the real-time pose parameters.

[0094] Specifically, assuming the azimuth angle corresponding to the reference heading in the spherical coordinate system is 0°, when the real-time heading angle of the remotely operated vehicle (ROV) is θ1, the horizontal azimuth angle of the planar viewport needs to be adjusted to θ1 simultaneously to ensure that the viewport orientation is consistent with the current forward direction of the ROV. If the real-time pitch angle of the ROV is θ2 (up is positive, down is negative), then the vertical pitch angle of the planar viewport needs to be adjusted to θ2 so that the viewport can cover the scene in the current pitch direction of the ROV. Based on the above mapping relationship and combined with the parameters of the spherical coordinate system, the horizontal azimuth angle (α) and vertical pitch angle (β) of the center of the planar viewport in the coordinate system are calculated. If the real-time roll angle of the ROV is θ3, the tilt angle of the planar viewport around the Z-axis (reference heading axis) of the coordinate system is corrected according to the value of θ3 so that the viewport plane always remains consistent with the current tilt state of the ROV, avoiding the tilt of the image observed by the operator, which would affect spatial judgment.

[0095] The window position is dynamically adjusted based on real-time pose parameters, so that the planar window can change synchronously with the remotely operated vehicle's turning, pitching, and rolling, ensuring that the window is always aligned with the underwater scene currently facing the remotely operated vehicle. Operators can observe the core working area without manually adjusting the window.

[0096] Furthermore, the steps for calculating the size of the planar window based on the field of view of the binocular camera include:

[0097] (1) Calculate the horizontal and vertical field of view of the panoramic planar image based on the field of view size;

[0098] Specifically, the horizontal field of view can be calculated using the following formula:

[0099] ;

[0100] in, The sensor width of the sub-camera in a binocular camera;

[0101] The focal length of the binocular camera;

[0102] The angle of overlap between the images from the two binocular cameras;

[0103] Furthermore, the vertical field of view can be calculated using the following formula:

[0104] ;

[0105] in, The sensor height of the neutron camera in the binocular camera;

[0106] This refers to the focal length of a binocular camera.

[0107] (2) Determine the width of the planar window based on the horizontal field of view, determine the height of the planar window based on the vertical field of view, and determine the size of the planar window based on the width and height.

[0108] Specifically, the horizontal field of view of the binocular camera is used as the horizontal angular span of the planar window in the spherical coordinate system, i.e., the difference in azimuth angles between the left and right boundaries of the window. The vertical field of view of the binocular camera is used as the vertical angular span of the planar window in the spherical coordinate system, i.e., the difference in pitch angles between the upper and lower boundaries of the window. This determines the geometric range of the target window in the angular domain. Based on the mapping rules of equidistant cylindrical projection, according to the ratio of the total number of pixels in the horizontal direction to the horizontal circumference of the panoramic spherical image, and the ratio of the total number of pixels in the vertical direction to the vertical viewing angle range, the horizontal angular span and the vertical angular span are converted into the window pixel width and window pixel height in pixels, respectively. This generates a target window size with uniform pixel units, ensuring that when performing pixel-level precise cropping on the panoramic spherical image, the window range maintains a strict optical correspondence with the actual imaging field of view of the binocular camera. For the implementation process of pixel conversion, please refer to the description in the relevant technology, which will not be repeated here.

[0109] The width and height of the window directly correspond to the horizontal and vertical field of view of the panoramic planar image, enabling the window to completely capture the effective scene range in the panoramic image and avoiding problems such as image edge distortion due to an excessively large window or insufficient observation range due to an excessively small window.

[0110] S105. Capture the target spherical image from the panoramic spherical image using the target viewport and send it to the VR glasses for display.

[0111] Specifically, based on the position and size of the target window, corresponding partial images are cropped from the panoramic spherical images in the left and right views to obtain left and right target spherical images; a signal transmission channel is established between the panoramic spherical image processing module and the VR glasses to ensure stable and efficient transmission of image data; the cropped left and right target spherical images are transmitted to the VR glasses, the left target spherical image is transmitted to the left lens display in the VR glasses, and the right target spherical image is transmitted to the right lens display in the VR glasses, and presented to the operator through the VR glasses' display system to form an immersive visual effect.

[0112] Furthermore, in the data processing module, the panoramic spherical images of the left and right views are cropped according to the pixel coordinate range of the target window. Image pixels within the target window range are retained, and redundant pixels outside the range are removed to generate the left and right target spherical images. Image edge optimization technology is used during the cropping process to avoid problems such as jagged edges and distortion in the cropped images. A connection is established between the panoramic spherical image processing module and the VR glasses through a wireless communication module (such as underwater wireless transmission technology or Bluetooth) or a wired transmission link, and the left and right target spherical images are transmitted to the receiving end of the VR glasses in a low-latency and high-fidelity manner. After receiving the image data, the VR glasses send the left target spherical image to the left eye display screen and the right target spherical image to the right eye display screen through the internal display driver module, using the parallax effect of human eyes to form a three-dimensional visual effect.

[0113] By using a target viewpoint to capture images, the system ensures that the spherical image of the target object completely covers the core area of ​​the remotely operated vehicle's (ROV) current operating scene. This also removes redundant pixels, reduces data transmission, and improves transmission efficiency and the ROV's display response speed, preventing delays in operator judgment due to data latency. The immersive display effect of VR glasses allows operators to intuitively and three-dimensionally perceive the underwater scene, accurately determine the spatial position, distance, and shape of targets, reduce operational errors caused by the visual limitations of planar images, decrease the frequency of tentative operations by the robotic arm, and improve underwater work efficiency. Furthermore, it simplifies the skill requirements for operators, reduces cross-role communication costs, and provides support for the efficient completion of underwater exploration and operational tasks.

[0114] Furthermore, after being displayed in VR glasses, the method provided in this embodiment includes:

[0115] (1) Obtain the horizontal rotation angle of the VR glasses;

[0116] Specifically, the horizontal rotation angle refers to the angle by which the VR glasses rotate around their vertical axis (corresponding to the horizontal rotation direction of the operator's head). VR glasses are typically equipped with a gyroscope, accelerometer, and magnetometer (or combined with a SLAM positioning module). The gyroscope is responsible for detecting the rotational angular velocity of the glasses around the vertical axis in real time, the accelerometer helps correct for deviations in the direction of gravity, and the magnetometer is used to eliminate accumulated drift errors. The sensors output horizontal angular velocity data in real time, which is transmitted to the local data processing unit of the VR glasses.

[0117] Furthermore, the local processing unit integrates the collected angular velocity data, converting the angular velocity into a horizontal rotation angle relative to the initial position (e.g., with the VR glasses' initial orientation as 0°, leftward rotation is a negative angle, and rightward rotation is a positive angle). Simultaneously, it filters out sensor noise using a Kalman filter algorithm and combines magnetometer data to calibrate long-term drift, ensuring that the calculated horizontal rotation angle error is controlled within a preset accuracy range. The calibrated horizontal rotation angle is then transmitted in real-time to the remotely operated vehicle's field-of-view extension system control unit via a signal transmission module. For details on the noise elimination process using Kalman filtering, please refer to the descriptions in relevant technologies; these details will not be elaborated here.

[0118] (2) Control the target window to slide horizontally along the panoramic spherical image according to the horizontal rotation angle;

[0119] Specifically, the horizontal rotation angle is used as the sliding angle of the target window. The pixel coefficient is calculated based on the ratio of the horizontal pixel resolution to the horizontal field of view of the panoramic spherical image. The sliding pixel distance of the target window is calculated based on the product of the pixel coefficient and the sliding angle.

[0120] Furthermore, after receiving the horizontal rotation angle sent by the VR glasses, the sliding pixel distance is calculated based on the pixel coefficient. If the horizontal rotation angle is positive (rotating to the right), the target window is controlled to move a sliding pixel distance along the positive horizontal direction of the panoramic spherical image (corresponding to the right side of the image). If the horizontal rotation angle is negative (rotating to the left), the window is moved a sliding pixel distance along the negative horizontal direction (corresponding to the left side of the image). During the sliding process, the edge coordinates of the window are detected in real time. If the window is about to exceed the horizontal pixel range of the panoramic spherical image (e.g., the left edge of the window is about to be less than 0 pixels or the right edge is about to be greater than the horizontal pixel resolution), the boundary constraint is triggered, and the window stops sliding to ensure that the window always completely covers the effective area of ​​the image.

[0121] (3) Based on the target window after horizontal sliding, re-capture the target spherical image in the panoramic spherical image and send it to the VR glasses for updated display.

[0122] Specifically, using the pixel coordinates of the upper left and lower right corners of the target window after horizontal sliding as boundaries, the pixel area to be cropped in the panoramic spherical image is clearly defined to ensure that the cropping range is completely consistent with the sliding window; local pixel data of the panoramic spherical image is extracted according to the cropping range, and edge smoothing and color correction are performed on the cropped image to eliminate pixel jaggedness or color deviation that may occur during the cropping process.

[0123] Furthermore, after horizontal sliding, the pixel coordinates of the upper left corner of the target window are (U1', V1), and the pixel coordinates of the lower right corner are (U2', V2) (the vertical coordinates V1 and V2 remain unchanged, only the horizontal coordinates U1 and U2 are updated to U1' and U2'). Using these two coordinates as boundaries, a rectangular cropping area is defined in the panoramic spherical image, and this rectangular cropping area is the range of the re-cropped target spherical image. Pixel data is extracted according to the defined area to generate the initial cropped image. A bilinear interpolation algorithm is used to smooth the image edges to eliminate jagged edges caused by pixel cropping. A color balance algorithm is used to correct the image brightness and contrast to ensure that the color effect of the cropped image is consistent with the panoramic spherical image.

[0124] The target spherical image is re-captured and updated to perfectly match the operator's perspective after horizontal rotation. The operator can observe underwater scenes in different horizontal directions in the panoramic spherical image in real time by rotating their head, breaking through the limitations of the fixed view window and greatly improving the flexibility and immersive experience of underwater observation.

[0125] Furthermore, the method provided in this embodiment also includes:

[0126] (1) Obtain the vertical rotation angle of the VR glasses;

[0127] Specifically, the vertical rotation angle refers to the angle at which the VR glasses rotate around their own horizontal axis (parallel to the direction of the line connecting the operator's eyes), corresponding to the operator's head movements of looking up and down, and can accurately reflect the changes in the vertical perspective of the head.

[0128] Furthermore, in the inertial measurement unit (IMU) of the VR glasses, the accelerometer is responsible for detecting changes in the vertical gravity component, and the gyroscope collects the rotational angular velocity around the horizontal axis in real time. The combination of the two can initially obtain vertical motion data. At the same time, the built-in attitude calculation algorithm eliminates sensor drift error and ensures that the raw data can reflect the true vertical rotation state. The collected angular velocity data is integrated and converted into the angle difference relative to the initial vertical attitude of the VR glasses, that is, the vertical rotation angle (usually defined as 0° when the initial horizontal is forward, positive angle when the head is up, and negative angle when the head is down). The Kalman filter algorithm is used to filter out noise caused by water flow impact, equipment vibration and other interference. Then, the angle value is constrained by the vertical field of view of the panoramic spherical image to ensure that the calculated vertical rotation angle is within the effective observation range. The calibrated angle data is sent to the field of view extension system control unit of the remotely operated underwater vehicle through the signal transmission module. For the implementation process of noise elimination by Kalman filtering, please refer to the description in the relevant technology, which will not be repeated here.

[0129] (2) Control the target window to slide vertically along the panoramic spherical image according to the vertical rotation angle;

[0130] Specifically, based on the above description, the pixel coefficient is calculated, and the vertical sliding distance is calculated based on the product of the pixel coefficient and the vertical rotation angle; the target window is controlled to slide vertically along the panoramic spherical image based on the vertical sliding distance.

[0131] Furthermore, after receiving the vertical rotation angle sent by the VR glasses, the vertical sliding distance is calculated based on the pixel coefficient; if the vertical rotation angle is positive (head-up rotation), the target window is controlled to slide vertically along the positive vertical direction of the panoramic spherical image (corresponding to the top of the image) by a vertical sliding distance of pixels; if the vertical rotation angle is negative (head-down rotation), the vertical sliding distance of pixels is slid along the negative vertical direction (corresponding to the bottom of the image).

[0132] (3) Based on the target window after vertical sliding, re-capture the target spherical image in the panoramic spherical image and send it to the VR glasses for updated display.

[0133] Specifically, the cropping area in the panoramic spherical image is defined by the pixel coordinates of the top left and bottom right corners of the target window after vertical sliding (the horizontal coordinates remain unchanged while the vertical coordinates are updated), ensuring that the cropping range completely overlaps with the sliding window. Image pixel data is extracted according to the cropping range, and edge anti-aliasing and color consistency correction are performed on the cropped image to eliminate image distortion caused by cropping. At the same time, the image data format is compressed to adapt to the signal transmission bandwidth and the decoding capability of VR glasses.

[0134] Furthermore, after vertical sliding, the pixel coordinates of the top left corner of the target window are updated to (U1, V1'), and the pixel coordinates of the bottom right corner are updated to (U2, V2') (the horizontal coordinates U1 and U2 remain unchanged). Using these two coordinates as the boundary of a rectangle, the pixel area to be cropped is delineated in the panoramic spherical image, clarifying the range of the target spherical image. Pixel data is extracted according to the delineated area to generate the initial cropped image. A bilinear interpolation algorithm is used to smooth the image edges, eliminating jagged edges caused by pixel cropping. The brightness and contrast of the image are adjusted through a color balance algorithm to ensure that the color effect of the cropped image is consistent with the overall panoramic spherical image. The image is compressed into the YUV420 format supported by VR glasses to reduce the data volume and improve transmission efficiency. For the implementation process of image compression, please refer to the description in the relevant steps, which will not be repeated here.

[0135] Furthermore, after cropping the target spherical image from the panoramic spherical image using a target viewpoint and sending it to the VR glasses for display, the method provided in this embodiment also includes:

[0136] (1) Set the sampling frequency period of the two binocular cameras;

[0137] Specifically, based on the remotely operated vehicle's speed, the dynamic nature of the operational scenario, and the real-time requirements of VR display, the sampling frequency periods of the two binocular cameras are preset. The sampling frequency period is the core parameter controlling the camera's shooting interval and can be adjusted according to the actual application scenario. For example, in fast-moving or complex environments, a shorter period is used to capture more details; in static or low-speed scenarios, a longer period is used to reduce computational load. In this way, by setting the sampling period reasonably, both the timeliness of image data can be guaranteed, and the waste of resources caused by frequent camera shooting can be avoided.

[0138] (2) Based on the sampling frequency period, control the two binocular cameras to acquire the first planar image and the second planar image;

[0139] Specifically, based on the above description, within each sampling frequency period, two binocular cameras are synchronously triggered to acquire the first planar image and the second planar image respectively.

[0140] (3) Determine the stitching coefficients based on the panoramic spherical image generated in the first sampling frequency period;

[0141] Specifically, in the first sampling period, feature points are extracted from the generated left and right panoramic planar images, the matching relationship of the overlapping areas of the two stereo camera images is calculated, and the stitching coefficients are determined. The stitching coefficients may include overlapping area weights, geometric correction parameters, color balance coefficients, etc., and are used as the initial benchmark. The stitching coefficients determined in the first period provide a stable benchmark for subsequent image updates, avoiding the need to recalculate the complete stitching parameters for each shot, reducing the computational load, and ensuring the stitching accuracy and visual consistency of the panoramic images.

[0142] (4) Generate an updated panoramic spherical image based on the first planar image and the second planar image combined with the stitching coefficient.

[0143] Specifically, in subsequent sampling cycles, the newly acquired first and second planar images are directly merged using the predetermined stitching coefficients to generate a new panoramic spherical image. The initial stitching coefficients are used to skip the repeated feature point matching and parameter calculation processes, and only the new image is subjected to adaptive fusion, thus shortening the processing time. For example, the overlapping parts of the new image are directly fused using the preset overlapping area weights, and lens distortion is quickly corrected using geometric correction parameters to ensure that the updated panoramic image is spatially continuous with the historical image.

[0144] Furthermore, the target viewport slides in real time within the new panoramic spherical image, and captures corresponding target spherical images along the sliding path, sending them to the VR glasses for display. Once the two binocular cameras have completed the acquisition and stitching of a new frame, they automatically replace the panoramic spherical image, thus achieving continuous image stream updates on the VR end to ensure the real-time nature and continuity of the displayed content. Please refer to the above description for the steps involved in generating the updated panoramic spherical image; they will not be repeated here.

[0145] The underwater VR field-of-view expansion method based on multi-camera fusion provided in this embodiment expands the field of view by fixing two identical binocular cameras on a remotely operated vehicle (ROV). Combined with view merging technology, images of different areas captured by the two cameras are stitched together to form panoramic planar images of the left and right fields of view. These images are then projected onto a spherical surface to generate a panoramic spherical image, overcoming the field-of-view limitations of traditional single-camera or single / binocular cameras. The vertical field of view remains consistent with that of a single camera, ensuring comprehensive coverage of the underwater scene in front of the ROV and preventing the loss of important marine discoveries due to a narrow field of view. The planar viewpoint position is calculated based on the real-time pose (heading angle, pitch angle, roll angle) of the ROV. By combining the field of view of the binocular camera to determine the window size, the generated target window can accurately match the current operating position of the remotely operated vehicle (ROV). Combined with the immersive display of VR glasses, operators can intuitively perceive the spatial depth and distance of the underwater scene, eliminating the need for tentative operations with a robotic arm to determine the target's location, thus improving operational efficiency and reducing reliance on operator skills. Ordinary researchers can also participate in operations through VR interaction, reducing cross-role communication costs. The horizontal and vertical rotation angles of the VR glasses can be controlled in real time to slide the target window on the panoramic spherical image, achieving a seamless connection between head rotation, window following, and image updates. Furthermore, boundary constraints and image optimization processing prevent the window from exceeding the image range or exhibiting edge distortion, ensuring a complete and clear display. In addition, standardized installation of the binocular camera and optimization of algorithms such as intrinsic parameter calibration and feature point matching ensure the stability and processing accuracy of image data. Even in underwater scenarios involving water flow impact and ROV vibration, high-quality visual feedback can be maintained, providing reliable technical support for underwater operations such as deep-sea exploration and resource detection.

[0146] Corresponding to the aforementioned embodiment of an underwater VR field of view expansion method based on multi-camera fusion, this application also provides an embodiment of an underwater VR field of view expansion device based on multi-camera fusion.

[0147] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the underwater VR field-of-view extension device based on multi-camera fusion provided in this application. Please refer to... Figure 2 The device provided in this embodiment includes a panoramic extension module, a signal transmission module, and VR glasses;

[0148] The panoramic extension module includes two binocular cameras, used to acquire a first planar image and a second planar image in front of the remotely operated vehicle, and to generate a target spherical image based on the first planar image and the second planar image.

[0149] The signal transmission module is used for signal transmission between the panoramic extension module and the VR glasses;

[0150] The VR glasses are used to receive images of the target spherical surface and display them to the user;

[0151] The VR glasses are also used to control the posture of the panoramic extension module according to the user's head display posture.

[0152] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0153] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0154] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0155] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for expanding the underwater VR field of view based on multi-camera fusion, characterized in that, The method includes: The remotely operated vehicle (ROV) acquires a first planar image and a second planar image in front of the ROV using two binocular cameras fixedly mounted on the ROV. The first planar image and the second planar image are obtained from different binocular cameras and capture different areas. The two binocular cameras are mounted on the same horizontal plane of the ROV at the same height. The optical lenses of the two binocular cameras are oriented in the same direction as the ROV's movement. The relative distance between the two binocular cameras is determined based on the shooting space of the binocular cameras. The views corresponding to the first planar image and the second planar image are merged to obtain panoramic planar images of the left and right views. The left view of the first planar image and the left view of the second planar image constitute a left view group, and the right view of the first planar image and the right view of the second planar image constitute a right view group. Feature recognition, position alignment, and stitching fusion are performed on the left view group and the right view group respectively to obtain a left field-of-view panoramic planar image and a right field-of-view panoramic planar image. The field of view of the right field-of-view panoramic planar image is consistent with that of the left field-of-view panoramic planar image, so that the left and right fields of view of the two binocular cameras symmetrically cover the area in front of the remotely operated vehicle. The panoramic planar images of the left and right views are projected onto the sphere to generate a panoramic spherical image; The position of the planar window is calculated based on the real-time pose of the remotely operated vehicle, and the size of the planar window is calculated based on the field of view of the binocular camera to generate the target window; A target spherical image is captured from the panoramic spherical image using a target viewpoint and sent to the VR glasses for display. Specifically, a left target spherical image is captured from the panoramic spherical image in the left view, and a right target spherical image is captured from the panoramic spherical image in the right view. The left target spherical image is transmitted to the left lens of the VR glasses for display, and the right target spherical image is transmitted to the right lens of the VR glasses for display, thereby creating a three-dimensional stereoscopic visual effect by utilizing the parallax effect of the human eye.

2. The method according to claim 1, characterized in that, After being sent to and displayed in VR glasses, it includes: Obtain the horizontal rotation angle of the VR glasses; The target window is controlled to slide horizontally along the panoramic spherical image according to the horizontal rotation angle. Based on the target viewport after horizontal sliding, a new target spherical image is captured from the panoramic spherical image and sent to the VR glasses for updated display.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the vertical rotation angle of the VR glasses; The target window is controlled to slide vertically along the panoramic spherical image according to the vertical rotation angle. The target spherical image is re-captured from the panoramic spherical image based on the target viewport after vertical sliding, and then sent to the VR glasses for updated display.

4. The method according to claim 1, characterized in that, The step of calculating the position of the planar view window based on the real-time pose of the remotely operated vehicle includes: Obtain the real-time pose parameters of the remotely operated underwater vehicle; A spherical coordinate system is established with the center of the panoramic spherical image as the origin, wherein the reference heading of the spherical coordinate system is consistent with the initial heading of the remotely operated vehicle; The position of the planar window in the spherical coordinate system is calculated based on the real-time pose parameters.

5. The method according to claim 1, characterized in that, The step of calculating the size of the planar window based on the field of view of the binocular camera includes: Calculate the horizontal and vertical field of view angles of the panoramic planar image based on the field of view size; The width of the planar window is determined based on the horizontal field of view, the height of the planar window is determined based on the vertical field of view, and the size of the planar window is determined based on the width and height.

6. The method according to claim 1, characterized in that, After capturing a target spherical image from the panoramic spherical image using a target viewpoint and sending it to the VR glasses for display, the process includes: Set the sampling frequency period of the two binocular cameras; Based on the sampling frequency period, the two binocular cameras are controlled to acquire a first planar image and a second planar image; The stitching coefficients are determined based on the panoramic spherical image generated during the first sampling frequency cycle. An updated panoramic spherical image is generated based on the first and second planar images combined with the stitching coefficients.

7. The method according to claim 1, characterized in that, The step of merging the views corresponding to the first planar image and the second planar image to obtain panoramic planar images of the left and right views includes: Intrinsic parameter calibration is performed on the left and right views of the first planar image and the second planar image, respectively; Extract the first overlapping feature points of the first left view and the second left view, and the second overlapping feature points of the first right view and the second right view; Based on perspective transformation and combined with the first overlapping feature points, the first left view and the second left view are stitched together to form a panoramic planar image of the left field of view; Based on perspective transformation and combined with second overlapping feature points, the first right view and the second right view are stitched together to form a panoramic planar image of the right field of view.

8. The method according to claim 1, characterized in that, The step of projecting panoramic planar images of the left and right views onto a sphere to generate a panoramic spherical image includes: Construct a spherical coordinate system with the midpoint of the line connecting the optical centers of the two binocular cameras as the center of the sphere; Obtain the planar pixel coordinates of the panoramic planar images of the left and right views; Based on the horizontal field of view and focal length, the planar pixel coordinates are mapped into the spherical coordinate system to obtain the cylindrical pixel coordinates; A panoramic spherical image is generated based on the cylindrical pixel coordinates.

9. An underwater VR field-of-view extension device based on multi-camera fusion, characterized in that, The device is implemented based on the method of any one of claims 1-8, and the device includes a panoramic extension module, a signal transmission module, and VR glasses; The panoramic extension module includes two binocular cameras, used to acquire a first planar image and a second planar image in front of the remotely operated vehicle, and to generate a target spherical image based on the first planar image and the second planar image. The signal transmission module is used for signal transmission between the panoramic extension module and the VR glasses; The VR glasses are used to receive images of the target spherical surface and display them to the user; The VR glasses are also used to control the posture of the panoramic extension module according to the user's head display posture.

Images