Underwater fish school detection preview method, device and equipment based on picture locking function

Abstract

The invention relates to an underwater fish school detection preview method, device and equipment based on a picture locking function, and the method comprises the steps: obtaining the attitude data of an inertial measurement unit arranged at an underwater binocular camera when the underwater binocular camera collects an underwater image, and adding a synchronization timestamp to the underwater image and the attitude data at the same collection moment; calibrating the underwater image through a pre-calibrated target calibration parameter to obtain a target underwater image; according to a preset fusion percentage and a preset reference radius, mirror image linear splicing fusion is carried out on the target underwater images of the two lenses at the same acquisition moment through the gradient weight, and a panoramic underwater image is obtained; establishing an association index for each frame of panoramic underwater image and attitude data under the same synchronization timestamp to form a structured data unit for storage; and according to the association index, calling the attitude data of the panoramic underwater image at the corresponding acquisition moment to lock, preview and display the image of the underwater binocular camera.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and device for underwater fish detection and preview based on image locking function. Background Technology

[0002] In underwater fish observation scenarios, binocular cameras are prone to shaking due to water flow, causing spatiotemporal misalignment in the image sequence. This results in significant errors in tracking the fish's location and a limited field of view that fails to cover the fish's activity range. Existing technologies cannot correct attitude shifts in real time, and the calibration method relying on planar calibration plates is ill-suited to the complex lighting and refraction environments underwater, leading to noticeable distortion in the stitched images. Summary of the Invention

[0003] The purpose of this invention is to provide an underwater fish school detection and preview method, device, and equipment based on image locking function, which aims to improve the stability of the field of view of underwater fish school image preview and enhance the user experience of fish school detection image preview display.

[0004] To achieve the above objectives, a first aspect of this disclosure provides an underwater fish school detection and preview method based on a screen locking function, the method comprising: When an underwater binocular camera acquires an underwater image, the attitude data of the inertial measurement unit of the underwater binocular camera is set, and a synchronization timestamp is added to the underwater image and the attitude data at the same acquisition time. The underwater image is calibrated using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. Based on a preset fusion percentage and a preset reference radius, the target underwater images of the two lenses at the same acquisition time are mirrored and linearly stitched together using a gradient weighting method to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined based on the field of view of the lens. Each frame of the panoramic underwater image is associated with the attitude data under the same synchronization timestamp, and a structured data unit is formed for storage. Based on the associated index, the attitude data corresponding to the acquisition time of the panoramic underwater image is invoked to lock and preview the image from the underwater binocular camera.

[0005] Optionally, the attitude data includes angular velocity data, acceleration data, and environmental magnetic field data. The step of calling the attitude data corresponding to the acquisition time of the panoramic underwater image based on the associated index to lock and preview the image from the underwater binocular camera includes: An attitude kinematic model of an underwater binocular camera is established. Based on the associated index, the angular velocity data corresponding to each frame of the panoramic underwater image is called as a basis. The attitude angle change of the underwater binocular camera is initially calculated through integral operation to obtain the predicted attitude. Using the acceleration data and environmental magnetic field data corresponding to the angular velocity data as observations, the predicted attitude is corrected by the Kalman filter algorithm to eliminate accumulated errors and noise interference, and the target attitude angle is obtained. Based on the target attitude angle, the image from the underwater binocular camera is subjected to reverse distortion correction and viewpoint compensation to achieve image lock preview display.

[0006] Optionally, the step of performing reverse distortion correction and viewpoint compensation on the image from the underwater binocular camera based on the target attitude angle to achieve image lock preview display includes: In response to the initial position coordinates, a locking reference point is determined, wherein the initial position coordinates include the locking display coordinates of the panoramic view specified by the user and received by the human-computer interaction interface; Based on the target attitude angle, determine the instantaneous change in the field of view angle of the underwater binocular camera relative to the locking reference point at the attitude angle; Based on the instantaneous change in the attitude angle, an image transformation model is established, and the image corresponding to the panoramic underwater image of the corresponding frame is subjected to reverse rotation and translation compensation, so that the field of view of the underwater binocular camera is kept at the initial position coordinates, thereby achieving image lock preview display.

[0007] Optionally, the step of establishing a scene transformation model based on the instantaneous change in the attitude angle, and performing reverse rotation and translation compensation on the scene corresponding to the panoramic underwater image of the corresponding frame, so that the field of view of the underwater binocular camera is maintained at the initial position coordinates, and the scene is locked for preview display, includes: In automatic lock playback mode, the panoramic underwater image corresponding to the instantaneous change of the attitude angle is retrieved according to the associated index, and the attitude calculation, reverse rotation and translation compensation are performed sequentially on the screen corresponding to each frame of the panoramic underwater image, and the locked preview screen is output in real time to keep the field of view of the underwater binocular camera at the initial position coordinates. In manual lock playback mode, in response to the user's interactive operation of dragging the timeline to locate the image at any time, the image of the field of view of the underwater binocular camera corresponding to the interactive operation is used as the new initial position coordinates. The image after the panoramic underwater image corresponding to the interactive operation is sequentially subjected to attitude calculation, reverse rotation and translation compensation to achieve image lock preview display.

[0008] Optionally, the target calibration parameters are obtained by calibration in the following manner: The geometric center of the lens is determined by drawing horizontal and vertical crosshairs on the original imaging image captured by the lens. Using the reference calibration center formed by the intersection of auxiliary lines on the inner wall of the cube calibration device as a reference, adjust the coordinates of the original image acquired by the lens until the position of the geometric center and the reference calibration center meets the preset position requirements, and use the translation amount of the coordinates as the corresponding center offset parameter of the lens. Given the center offset parameter, the original image is scaled to determine the effective radius of the lens's field of view. Given a fixed effective field of view radius, the original images captured by the two lenses are rendered onto a spherical circle model for Euler angle calibration to obtain the Euler angle parameters of the lenses. The target calibration parameters include the center offset parameter, the effective field of view radius, and the Euler angle parameter.

[0009] Optionally, calibrating the underwater image using pre-defined target calibration parameters to obtain the target underwater image at the corresponding acquisition time includes: Based on the center offset parameter, the underwater images corresponding to the two lenses are translated respectively so that the geometric center of the lens is aligned with the center of the preset canvas; Based on the effective field of view alarm, the translated underwater image is scaled, and invalid images that exceed the preset canvas are cropped out to obtain the effective area image; The effective area image is mapped onto the 3D semicircular model textures corresponding to the two lenses, and the attitude of the 3D semicircular model is adjusted according to the Euler angle parameters to obtain the target underwater image at the corresponding acquisition time.

[0010] Optionally, the step of acquiring attitude data set in the inertial measurement unit of the underwater binocular camera when it captures underwater images, and adding a synchronization timestamp to the underwater images and attitude data at the same acquisition time, includes: Obtain the image timestamp of each frame of the underwater image when the underwater binocular camera captures the underwater image; Obtain the data timestamp for each attitude data generated by the inertial measurement unit; If it is determined that there is no attitude data corresponding to the same acquisition time in any frame of the underwater image, based on the image timestamp of the underwater image in that frame, the first data timestamp that is adjacent to the image timestamp and the second data timestamp that is adjacent to the image timestamp are searched in the attitude data. Based on the time interval ratio, a linear interpolation algorithm is used to interpolate between the first data timestamp and the second data timestamp to obtain a target data timestamp that is synchronized with the image timestamp of the underwater image in that frame; Based on the attitude data corresponding to the first data timestamp and the attitude data corresponding to the second data timestamp, the target attitude data corresponding to the target data timestamp is determined. Add a synchronization timestamp to the target attitude data and the corresponding frame of the underwater image; or, If it is determined that there is attitude data corresponding to the same acquisition time in any frame of the underwater image, a synchronization timestamp of the underwater image in that frame and the attitude data corresponding to the same acquisition time are added according to the data timestamp of the underwater image in that frame and the data timestamp of the attitude data corresponding to the same acquisition time.

[0011] A second aspect of this disclosure provides an underwater fish school detection and preview device based on a screen locking function, the device comprising: The acquisition module is configured to acquire the attitude data set in the inertial measurement unit of the underwater binocular camera when the underwater binocular camera acquires an underwater image, and to add a synchronization timestamp to the underwater image and the attitude data at the same acquisition time. The calibration module is configured to calibrate the underwater image using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. The stitching and fusion module is configured to perform mirror linear stitching and fusion of the target underwater images of the two lenses at the same acquisition time according to a preset fusion percentage and a preset reference radius, through gradual weighting, to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined according to the field of view of the lens. The storage module is configured to establish an association index between each frame of the panoramic underwater image and the attitude data under the same synchronization timestamp, forming a structured data unit for storage; The preview module is configured to lock and preview the image from the underwater binocular camera by calling the attitude data at the time of acquisition corresponding to the panoramic underwater image based on the associated index.

[0012] A third aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects.

[0013] A fourth aspect of this disclosure provides an electronic device, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method of any one of the first aspects.

[0014] This invention provides a method, apparatus, and device for underwater fish detection and preview based on image locking functionality. Compared with existing technologies, it has the following advantages: By acquiring the attitude data of the inertial measurement unit of the underwater binocular camera when it captures underwater images, and adding a synchronization timestamp, the precise correspondence between the image and motion data in the time dimension is ensured. In the image calibration stage, the target calibration parameters obtained by calibrating the auxiliary lines on the inner wall of the cube calibration device can more accurately calibrate the underwater image, effectively eliminating image distortion caused by the complexity of the underwater environment, improving image quality, and making the target underwater image more realistically reflect the underwater scene. Furthermore, in image stitching and fusion, a preset fusion percentage is determined based on the lens field of view, and mirror linear stitching and fusion are performed in combination with a preset reference radius and gradient weights, reducing stitching artifacts and improving the realism of the generated panoramic underwater image, providing users with a more complete and clear underwater panoramic view. Each frame of the panoramic underwater image is associated with the attitude data under the same synchronization timestamp, forming a structured data unit for storage. Based on the association index, the attitude data corresponding to the acquisition time of the panoramic underwater image is called to lock and preview the image of the underwater binocular camera. This allows for continuous previewing of the fish detection image in a stable image, avoiding image jitter and enhancing the user experience when the user cannot stably observe the fish underwater.

[0015] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an underwater fish detection and preview method based on screen locking functionality, as shown in the embodiments of the instruction manual.

[0017] Figure 2 A block diagram of an underwater fish detection and preview device based on screen locking function, as shown in the embodiment of the specification.

[0018] Figure 3 This is a block diagram of another underwater fish detection and preview device based on screen locking function, as shown in the embodiment of the specification. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] In casual scenarios, underwater fish finders are typically deployed in the water in a suspended manner. Their posture is easily affected by environmental factors such as water flow and water disturbance, causing irregular movements such as rotation, tilting, and shaking. Secondly, the panoramic view formed by stitching together two large fisheye lenses has a fixed perspective. When the fish finder changes its posture, the position of the hook in the image will shift accordingly, making it impossible to keep it within a fixed field of view. Finally, the playback function of existing fish finders only allows users to manually drag and adjust the image angle. It cannot adaptively compensate for the movement of the fish finder. Even if the hook image is manually located during playback, the image will continue to drift due to the posture fluctuations during the original shooting process, making it impossible to achieve stable locking of the hook image. Consequently, it is impossible to clearly and completely record the key process of fish feeding and being hooked.

[0021] In view of this, the present disclosure provides an underwater fish school detection preview method based on image locking function, which aims to improve the stability of image preview, allow users to smoothly and clearly preview fish school detection images, and enhance user experience and underwater detection effect. Figure 1 This is a flowchart illustrating an underwater fish school detection and preview method based on a screen-locking function, according to one embodiment. The method includes: In step S11, the attitude data of the underwater binocular camera set in the inertial measurement unit of the underwater binocular camera is obtained when the underwater image is acquired, and a synchronization timestamp is added to the underwater image and the attitude data at the same acquisition time. The synchronization timestamp is a unified time identifier added to underwater images, angular velocity data, and acceleration data acquired at the same time. The time deviation between IMU data and video frames is controlled within milliseconds to avoid image stabilization failure or image distortion due to data misalignment. In this embodiment of the disclosure, when the underwater binocular camera acquires images, the IMU simultaneously measures angular velocity and acceleration. Adding a synchronization timestamp establishes a temporal correlation between the underwater images, angular velocity data, and acceleration data. Through the synchronization timestamp, the camera motion state at the corresponding moment in each underwater image can be associated.

[0022] This disclosure constructs a closed-loop control system of "attitude perception - data processing - image compensation" by adding a 9-axis inertial measurement unit (IMU), an attitude calculation module, and an image compensation module to an existing dual-fisheye lens panoramic underwater fish finder, thereby achieving stable locking of the fishhook image. The overall technical architecture consists of two parts: a hardware layer and a software layer. The hardware layer comprises a dual-fisheye lens imaging module, a 9-axis IMU module, and a data storage module, responsible for underwater image capture, fish finder attitude data acquisition, and data storage. The software layer comprises a timestamp synchronization module, a Kalman filter attitude calculation module, an image compensation module, and a playback control module, responsible for data synchronization processing, attitude calculation, adaptive image compensation, and locked playback control. These modules work collaboratively to achieve real-time perception of the fish finder's motion state and dynamic image compensation, ensuring that the fishhook image remains within a fixed field of view.

[0023] The 9-axis IMU module integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. This module is rigidly connected to the dual large fisheye lens imaging module on the same housing structure of the fish finder, ensuring that the attitude data acquired by the IMU accurately reflects the motion state of the imaging module. The IMU module acquires the motion parameters of the fish finder at a sampling frequency of no less than 100Hz. Specifically: the three-axis accelerometer acquires linear acceleration data of the fish finder in the X, Y, and Z directions, reflecting the translation and tilt of the fish finder; the three-axis gyroscope acquires angular velocity data of the fish finder around the X, Y, and Z axes, reflecting the rotation and jitter of the fish finder; and the three-axis magnetometer acquires environmental magnetic field data, providing an orientation reference for attitude calculation and correcting the drift error of the gyroscope. The acquired 9-axis raw data, along with the acquisition timestamp, is transmitted to the timestamp synchronization module.

[0024] The underwater binocular camera employs two symmetrically arranged high-resolution large fisheye lenses as image acquisition units. Both lenses have a field of view of at least 180°, and the angle between their optical axes is set to 120°-150°, ensuring that the shooting range of both lenses completely covers the area where the fishhook is located, forming a seamless panoramic image. The shooting module captures underwater images at a preset frame rate (at least 30fps), simultaneously generating a timestamp for each frame. The timestamp accuracy is down to the millisecond level, providing a time reference for subsequent synchronization with IMU data. The captured panoramic image data is transmitted in real-time to the data storage module for temporary storage.

[0025] In step S12, the underwater image is calibrated using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. In this embodiment, the auxiliary lines on the inner wall of the cube calibration device are used as a geometric reference to calibrate core parameters such as lens distortion and camera intrinsic parameters to obtain target calibration parameters. Using these parameters to correct underwater images can eliminate the effects of lens distortion and other factors, making the images more accurately reflect the real scene. Targeted geometric transformation calibration can be performed on the acquired raw images to eliminate image distortion and positional deviations caused by errors.

[0026] The target calibration parameters may include: left camera parameters: left camera center x-coordinate, left camera center y-coordinate, left camera effective radius, left camera roll angle, left camera yaw angle, and left camera pitch angle; and right camera parameters: right camera center x-coordinate, right camera center y-coordinate, right camera effective radius, right camera roll angle, right camera yaw angle, and right camera pitch angle.

[0027] In step S13, based on the preset fusion percentage and the preset reference radius, the target underwater images of the two lenses at the same acquisition time are mirrored and linearly stitched together using a gradient weighting method to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined based on the field of view of the lens. In this embodiment, pixels in the target underwater image corresponding to either of the two lenses that fall between the preset fusion percentage and the preset reference radius are mirrored and linearly stitched together with pixels in the overlapping area of ​​the target underwater image corresponding to the other lens, using a gradient weighting method to obtain a panoramic underwater image at that acquisition time. Thus, the preset fusion percentage is determined based on the lens's field of view, and combined with the preset reference radius, a gradient weighting method is used to mirror and linearly stitch together the target underwater images of the two lenses at the same acquisition time. The gradient weighting ensures a natural transition at the stitching point, avoiding obvious stitching marks, resulting in a panoramic underwater image and expanding the observation range.

[0028] The weight of the pixel corresponding to any one of the lenses gradually decreases from the side of the overlapping region closer to the lens to the side farther away from the lens, while the weight of the pixel corresponding to the other lens gradually increases as the weight of the pixel in any one of the lenses gradually decreases.

[0029] In this embodiment, since the field of view (FOV) of the lens used is approximately 200° (greater than 180°), there is a fixed overlapping area between the left and right lens images. This area is the core object of the fusion processing. In actual operation, the fusion percentage can be defined first based on the field of view. The fusion percentage can be determined based on the empirical value of the lens field of view. For example, the empirical value of the fusion percentage corresponding to a field of view of 200° is 0.9754, and the reference radius is set to 1.0.

[0030] When the fusion algorithm is executed, the fusion region is divided based on the radius range. For example, pixels in the left shot image with a radius of 0.9754 to 1.0 are mirrored and linearly fused with the corresponding overlapping area pixels in the right shot image. Linear fusion is achieved through gradual weight allocation, that is, the weight of the left shot pixels gradually decreases from the inside to the outside of the fusion region, while the weight of the right shot pixels gradually increases synchronously, ultimately making the pixel transition at the stitching point smooth and eliminating visual discontinuities.

[0031] In step S14, each frame of the panoramic underwater image is associated with the attitude data under the same synchronization timestamp to form a structured data unit for storage. The association index is an identifier that establishes the correspondence between panoramic underwater images and attitude data at the same synchronization timestamp. The structured data unit is a unit formed by organizing and storing panoramic underwater images and attitude data corresponding to the same synchronization timestamp according to a certain structure.

[0032] In this embodiment, each frame of panoramic underwater image is associated with attitude data at the same synchronization timestamp through a correlation index, such as using a database indexing mechanism or file naming rules. The associated data is stored in a structured format, such as storing image data and attitude data in specific fields respectively. For example, a frame of panoramic underwater image is stored as "image_20240101120000001.jpg", and the corresponding attitude data is stored as "attitude_20240101120000001.txt". An index is created in the database so that the corresponding attitude data can be quickly retrieved by the image file name, forming a structured data unit.

[0033] High-speed flash memory chips are used as the storage medium to store timestamped panoramic image data and IMU attitude data. Data storage employs a "frame-attitude" correspondence model, meaning that each frame of panoramic image data is associated with IMU attitude data at the same timestamp, forming structured data units for storage. This storage model ensures that during playback, the corresponding IMU attitude data can be quickly retrieved based on the currently playing image frame.

[0034] Because the dual fisheye lens imaging module and the 9-axis IMU module have different sampling frequencies (30fps for imaging, 100Hz for IMU sampling), data alignment is required through a timestamp synchronization module to achieve accurate matching of image data and attitude data. The synchronization process adopts a strategy of "image frame timestamp as the reference, IMU data interpolation matching": First, the timestamp T_img(i) (i=1,2,...,n) of each frame generated by the imaging module and the timestamp T_imu(j) (j=1,2,...,m) of each data point generated by the IMU module are obtained respectively; Second, for each image frame timestamp T_img(i), the two adjacent data point timestamps T_imu(j) and T_imu(j+1) before and after it are found in the IMU data sequence, and the IMU attitude data corresponding to T_img(i) is calculated by linear interpolation algorithm according to the time interval ratio; Finally, the interpolated IMU data is bound to the corresponding image frame to form an "image-attitude" data pair to ensure that each frame has a unique corresponding attitude data support.

[0035] In step S15, based on the associated index, the attitude data corresponding to the acquisition time of the panoramic underwater image is called to lock and preview the image of the underwater binocular camera.

[0036] Among them, the locked preview display is a method of fixing the camera image based on the posture data to make the display image stable.

[0037] In this embodiment of the disclosure, when retrieving a panoramic underwater image based on the associated index, the attitude data at the corresponding acquisition time is simultaneously obtained. The attitude data is used to transform and correct the camera footage, eliminating image jitter caused by camera movement or attitude changes, achieving locked preview display, and enabling users to stably observe the underwater scene.

[0038] For example, when a user requests a panoramic underwater image at a specific moment, the corresponding attitude data is retrieved based on the associated index, such as a pitch angle of 10° and a roll angle of 5°. The image is then transformed using this data to ensure the displayed view matches the actual underwater scene, eliminating image jitter caused by robot movement and achieving a stable preview.

[0039] The aforementioned technical solution acquires the attitude data of the inertial measurement unit of the underwater binocular camera when it captures underwater images, and adds a synchronization timestamp to ensure accurate temporal correspondence between the image and motion data. In the image calibration stage, the target calibration parameters obtained by using auxiliary lines on the inner wall of a cube calibration device can more accurately calibrate the underwater image, effectively eliminating image distortion caused by the complexity of the underwater environment, improving image quality, and making the target underwater image more realistically reflect the underwater scene. Furthermore, in image stitching and fusion, a preset fusion percentage is determined based on the lens's field of view, and mirror linear stitching and fusion are performed in combination with a preset reference radius and gradient weights, reducing stitching artifacts and improving the realism of the generated panoramic underwater image, providing users with a more complete and clear underwater panoramic view. Each frame of the panoramic underwater image is associated with the attitude data under the same synchronization timestamp, forming a structured data unit for storage. Based on the association index, the attitude data corresponding to the acquisition time of the panoramic underwater image is called to lock and preview the image of the underwater binocular camera. This allows for continuous previewing of the fish detection image in a stable image, avoiding image jitter and enhancing the user experience when the user cannot stably observe the fish underwater.

[0040] Optionally, the attitude data includes angular velocity data, acceleration data, and environmental magnetic field data. In step S15, the step of calling the attitude data corresponding to the acquisition time of the panoramic underwater image to lock and preview the image of the underwater binocular camera according to the associated index includes: In step S151, an attitude kinematic model of the underwater binocular camera is established. Based on the associated index, the angular velocity data corresponding to each frame of the panoramic underwater image is called as the basis, and the attitude angle change of the underwater binocular camera is initially calculated through integral operation to obtain the predicted attitude. Among them, the attitude kinematic model is a mathematical model that describes the law of change of an object's attitude over time, and is used to calculate the attitude angle based on data such as angular velocity.

[0041] In this embodiment, the attitude kinematics model is constructed based on the principles of rigid body kinematics, which clearly defines the relationship between angular velocity and attitude angle changes. The angular velocity data corresponding to each frame of the panoramic underwater image is retrieved using an associated index. Treating this as a time function, the angular velocity is accumulated and summed along the time axis through integration, thereby initially calculating the attitude angle change of the underwater binocular camera within that time period, and obtaining the predicted attitude. For example, the attitude kinematics model reveals the relationship between angular velocity and rotation angle. If the angular velocity data retrieved by the associated index is 5 degrees per second within a certain time period, after 2 seconds of integration, the initial calculation of the attitude angle change is 10 degrees, resulting in a predicted attitude of a 10-degree rotation.

[0042] In step S152, the acceleration data and the environmental magnetic field data corresponding to the angular velocity data are used as observations, and the predicted attitude is corrected by the Kalman filter algorithm to eliminate accumulated errors and noise interference, thereby obtaining the target attitude angle. In this embodiment of the disclosure, during the prediction phase, the predicted attitude obtained by integrating the angular velocity data is used as a priori estimate, and the corresponding acceleration data and environmental magnetic field data are used as observations. During the update phase, the priori estimate is corrected using the observations. By calculating the Kalman gain, the weights of the predicted attitude are adjusted to eliminate accumulated errors from the integration operation and noise interference in the data, resulting in a more accurate target attitude angle.

[0043] For example, based on the predicted attitude, simultaneous acceleration data and environmental magnetic field data are used as observations. If the acceleration data shows slight fluctuations, while the environmental magnetic field data is relatively stable, the Kalman filter algorithm reduces the impact of integration errors through calculation and uses the stable magnetic field data to correct the predicted attitude, resulting in a more accurate target attitude angle.

[0044] For example, the real-time attitude angles (including pitch, roll, and yaw angles) of the fish finder are calculated using 9-axis IMU data, providing a kinematic basis for image compensation. Because the raw IMU data contains issues such as gyroscope drift, accelerometer noise, and environmental interference with the magnetometer, directly using the raw data for calculation would result in insufficient accuracy in attitude resolution. Therefore, this invention employs a Kalman filter algorithm to fuse the IMU data. The specific calculation process consists of two steps: First, a kinematic model of the fish finder's attitude is established. Based on angular velocity data collected by the gyroscope, the change in the fish finder's attitude angle is initially calculated through integration to obtain the predicted attitude. Second, using linear acceleration data collected by the accelerometer (inferring attitude from the gravitational acceleration component) and magnetic field data collected by the magnetometer (correcting the heading angle by the geomagnetic north pole direction) as observations, the predicted attitude is corrected using a Kalman filter algorithm to eliminate accumulated errors and noise interference, resulting in high-precision attitude angle data. The calculated attitude angle data is transmitted to the image compensation module in real time. Simultaneously, in playback scenarios, the attitude angle of the corresponding image frame can be calculated offline based on stored IMU data.

[0045] In step S153, based on the target attitude angle, the image from the underwater binocular camera is subjected to reverse distortion correction and viewpoint compensation to achieve image lock preview display.

[0046] Inverse distortion correction is used to correct image distortion caused by factors such as lens, restoring the image to its true shape. Viewpoint compensation adjusts the viewing angle based on the target's attitude angle, eliminating image shift caused by changes in camera attitude.

[0047] In this embodiment, the degree and direction of image distortion are determined based on the target attitude angle, and an inverse distortion correction algorithm is used to correct the image, eliminating image distortion caused by lens distortion. Simultaneously, viewing angle compensation is performed based on the target attitude angle to adjust the display viewing angle of the image, ensuring that the image matches the actual underwater scene's viewing angle, thus achieving locked preview display.

[0048] For example, if the target attitude angle shows a 10-degree pitch change in the camera, the inverse distortion correction algorithm will correct the distortion in the image caused by the pitch based on this angle. Viewpoint compensation adjusts the image display to bring objects that were originally offset by the pitch back to the center of the image, achieving a stable preview.

[0049] In this embodiment of the disclosure, the core function of the image compensation module is to perform reverse distortion correction and viewing angle compensation on the panoramic image captured by the dual large fisheye lenses based on the attitude angle change output by the attitude calculation module, thereby locking the fishhook image. The compensation process consists of three steps: "initial positioning of the fishhook - attitude change calculation - reverse image compensation". First, when the user activates the fishhook locking function, the system receives the user-specified initial position coordinates (x0, y0) of the fishhook in the panoramic image through the human-computer interaction interface and sets these coordinates as the locking reference point. Second, during shooting (real-time preview mode) or playback, the attitude calculation module obtains the attitude angle changes of the fish finder relative to the initial locking moment (Δθ: pitch angle change, Δφ: roll angle change, Δψ: heading angle change). Finally, based on the attitude angle changes, an image transformation model is established to perform reverse rotation and translation compensation on the current frame of the panoramic image. That is, the angle and offset of the image to be reverse rotated are calculated based on Δθ, Δφ, and Δψ, and the image is processed using an image geometric transformation algorithm (such as bilinear interpolation) to ensure that the position of the fishhook is always maintained at the initial locking coordinates (x0, y0). For panoramic images stitched together from two large fisheye lenses, the stitching area of ​​the two lenses also needs to be adaptively adjusted simultaneously during the compensation process to ensure a smooth transition at the stitching point and avoid stitching misalignment due to compensation operations.

[0050] The aforementioned technical solution establishes a kinematic model of the posture and calculates the predicted posture through integration, providing a foundation for subsequent processing. The Kalman filter algorithm utilizes multiple types of data to correct the predicted posture, eliminating accumulated errors and noise, and obtaining an accurate target posture angle. Based on the target posture angle, inverse distortion correction and viewpoint compensation can eliminate the influence of lens distortion and camera posture changes on the image, improving the stability and accuracy of the underwater binocular camera's preview, making the preview image highly consistent with the actual underwater scene. In underwater operations such as fish finding, operators can observe the underwater situation more clearly and stably.

[0051] Optionally, in step S153, the step of performing reverse distortion correction and viewing angle compensation on the image of the underwater binocular camera according to the target attitude angle to achieve image lock preview display includes: In step S1531, in response to the initial position coordinates, a locking reference point is determined, wherein the initial position coordinates include the locking display coordinates of the panoramic view specified by the user and received by the human-computer interaction interface; The initial position coordinates can be coordinate information received by the human-computer interaction interface from the user to determine the lock display position of the panoramic view, serving as a reference benchmark for image locking. The lock benchmark point is determined based on the initial position coordinates and acts as a core reference point during the image locking process, ensuring that the image is stably displayed around it.

[0052] In this embodiment of the disclosure, in response to the initial position coordinates transmitted from the human-computer interaction interface, which are the lock display positions specified by the user on the panoramic view based on their needs, a point is determined in the image using these coordinates as the center as the lock reference point. All subsequent image adjustments are made around this point to ensure the accuracy of the locked display position.

[0053] For example, when a user clicks on a location on a panoramic underwater image through the human-computer interaction interface, the system uses the coordinates of that clicked location as the initial position coordinates. The point on the screen corresponding to these coordinates is then used as the center to determine the locking reference point. For instance, if the user clicks on a point that appears to be a gathering of fish slightly to the lower right of the image center, this point becomes the core for subsequent image locking.

[0054] In step S1532, the instantaneous change in the field of view of the underwater binocular camera relative to the locking reference point is determined based on the target attitude angle. The instantaneous change in attitude angle refers to the change in the field of view relative to the locked reference point caused by the change in the attitude of the underwater binocular camera at a given instant. The instantaneous change in attitude angle includes one or more of the following: the instantaneous change in pitch angle, the instantaneous change in roll angle, and the instantaneous change in yaw angle.

[0055] In this embodiment of the disclosure, the change in the field of view of the underwater binocular camera relative to the determined locking reference point is analyzed based on the acquired target attitude angle. By calculating the difference between the target attitude angle and the initial attitude angle, the instantaneous change in the field of view in the attitude angle dimension is obtained, thus clarifying the direction and degree of image adjustment required.

[0056] For example, if an underwater binocular camera experiences a pitch change due to water flow, the target attitude angle shows a 5-degree increase in the pitch angle. Calculations show that the field of view angle relative to the locked reference point has undergone an instantaneous downward shift in the vertical direction, indicating that the image needs to be adjusted upward to compensate for this pitch angle change.

[0057] In step S1533, a screen transformation model is established based on the instantaneous change in the attitude angle, and the screen corresponding to the panoramic underwater image of the corresponding frame is subjected to reverse rotation and translation compensation, so that the field of view of the underwater binocular camera is kept at the initial position coordinates, thereby achieving screen lock preview display.

[0058] Among them, the image transformation model is a mathematical model based on the instantaneous change in attitude angle, which is used to perform reverse rotation and translation compensation operations on panoramic underwater images.

[0059] In this embodiment, a scene transformation model is established based on the instantaneous change in attitude angle. This model includes mathematical relationships of reverse rotation and translation. Using this model, the panoramic underwater image of the corresponding frame is reversed to counteract the scene shift caused by changes in camera attitude, ensuring the field of view remains at its initial coordinates, thus achieving locked preview display.

[0060] For example, a scene transformation model is established based on the aforementioned instantaneous changes. The model stipulates that the scene needs to be rotated 5 degrees in the opposite direction and shifted upwards by a certain distance. After processing the current frame of the panoramic underwater image according to this model, the suspected fish gathering point, which was originally shifted due to the camera's pitch, returns to its initially specified position, achieving a locked preview.

[0061] The above technical solution provides users with the ability to customize the image locking position by determining the initial position coordinates and locking the reference point, meeting different observation needs. Based on the target attitude angle, it determines the instantaneous change in the field of view, accurately sensing the impact of camera attitude changes on the image. By establishing an image transformation model and performing reverse rotation and translation compensation, it can eliminate image offset in real time, keeping the field of view at its initial position, improving the stability and accuracy of the underwater binocular camera's image preview, and achieving stable image locking preview display. This allows users to clearly and stably observe the target area.

[0062] Optionally, in step S1533, the step of establishing a screen transformation model based on the instantaneous change in the attitude angle, and performing reverse rotation and translation compensation on the screen corresponding to the panoramic underwater image of the corresponding frame, so that the field of view of the underwater binocular camera is maintained at the initial position coordinates, and the screen is locked for preview display, includes: In automatic lock playback mode, the panoramic underwater image corresponding to the instantaneous change of the attitude angle is retrieved according to the associated index, and the attitude calculation, reverse rotation and translation compensation are performed sequentially on the screen corresponding to each frame of the panoramic underwater image, and the locked preview screen is output in real time to keep the field of view of the underwater binocular camera at the initial position coordinates. Among them, the automatic lock playback mode is a working mode in which the system automatically locks and previews the panoramic underwater image according to preset rules, without requiring the user to manually intervene in the positioning of specific images.

[0063] In this embodiment of the disclosure, in automatic lock playback mode, the system uses an associated index to quickly find the panoramic underwater image corresponding to the instantaneous change in attitude angle. For each frame, attitude calculation is first performed to analyze the impact of attitude angle changes on the image and determine the reverse rotation angle and translation distance. Then, based on the calculation results, reverse rotation and translation compensation operations are performed on the image to offset the image shift caused by changes in camera attitude, and the image is output in real time to ensure that the field of view of the underwater binocular camera is always at the initial position coordinates.

[0064] For example, during fish detection playback, the system is in automatic lock playback mode. The associated index maps instantaneous changes in attitude angle over a certain period to panoramic underwater images. For instance, if a frame shifts downwards due to camera pitch, the attitude angle change determines that it needs to be rotated 5 degrees backwards and translated 10 pixels upwards. The system processes the image accordingly, outputting the corrected image in real time, keeping the field of view at its initial position for easy observation of fish movement.

[0065] In automatic lock playback mode, after the user starts the playback function, the system automatically retrieves the stored "image-attitude" data pairs, performs attitude calculation and reverse compensation on each frame in sequence, and outputs the image after locking the hook in real time. The user can watch the stable hook area image without manual intervention.

[0066] In manual lock playback mode, in response to the user's interactive operation of dragging the timeline to locate the image at any time, the image of the field of view of the underwater binocular camera corresponding to the interactive operation is used as the new initial position coordinates. The image after the panoramic underwater image corresponding to the interactive operation is sequentially subjected to attitude calculation, reverse rotation and translation compensation to achieve image lock preview display.

[0067] The manual lock playback mode allows users to interactively specify the starting position for locking the image, such as dragging the timeline, and the system will then lock and preview the image accordingly. The timeline is a visual tool used in image playback to represent chronological order and locate specific moments in the image.

[0068] In this embodiment of the disclosure, in the manual lock playback mode, when the user drags the timeline to any point in the video, the system sets the field of view of the underwater binocular camera at that point as the new initial position coordinates. For the subsequent video following the panoramic underwater image corresponding to this interactive operation, attitude calculation is performed first to obtain the image adjustment parameters, and then reverse rotation and translation compensation are performed to ensure that the subsequent video is stably displayed around the new initial position coordinates, thus achieving a locked preview.

[0069] For example, during manual playback, the user drags the timeline to locate the moment a suspected large school of fish is spotted. The system sets the field of view of this scene as the new initial position coordinates. In subsequent shots, if the camera swings left or right, the attitude calculation determines the reverse rotation angle and translation distance. After image compensation, the school of fish remains near the center of the frame, allowing the user to carefully examine its features.

[0070] In manual lock playback mode, users can drag the timeline to any point in the frame and click the "Lock" button. The system will automatically use the position of the hook in the current frame as a reference to compensate for the posture of subsequent playback frames, achieving a locking effect. The module also supports adjusting the size of the locked area, allowing users to expand or shrink the locked field of view as needed, balancing the stability of the locked image with the need to observe the surrounding environment.

[0071] The above technical solution is based on two image-locking playback modes: automatic and manual, to meet the needs of different scenarios. In automatic mode, the system quickly and accurately retrieves and processes images, outputting stable images in real time, improving observation efficiency. Manual mode gives users the right to choose the starting point for image locking, enhancing operational flexibility and targeting. Through attitude calculation, reverse rotation, and translation compensation, the influence of camera attitude changes on the image is effectively eliminated, ensuring that the field of view of the underwater binocular camera remains at the initial position coordinates, guaranteeing stable and clear images.

[0072] Optionally, the target calibration parameters are obtained by calibration in the following manner: The geometric center of the lens is determined by drawing horizontal and vertical crosshairs on the original imaging image captured by the lens. The raw image is the unprocessed image directly captured by the lens, containing the original information of the scene as captured. Crosshairs are two perpendicular lines drawn on the image to aid in positioning and measurement. The geometric center is the symmetrical center point in the image; for regularly shaped image areas, it reflects the symmetrical characteristics of the lens's imaging.

[0073] In this embodiment, horizontal and vertical crosshair auxiliary lines are drawn on the original image captured by the lens. These two lines intersect each other perpendicularly. Since the image is ideally symmetrical, the intersection of the crosshair auxiliary lines is the geometric center of the image. The geometric center of the lens imaging can be accurately located, and the calibration operation is performed based on the geometric center for coordinate adjustment and parameter calculation.

[0074] For example, when photographing a regular square object, the original image captured by the lens may have some deviation. By drawing horizontal and vertical crosshairs on the image, and assuming the square image is basically symmetrical in the horizontal and vertical directions, the intersection of these crosshairs is the geometric center of the square image. This center point reflects the position of the center of symmetry on the image plane when the lens is forming the image.

[0075] Using the reference calibration center formed by the intersection of auxiliary lines on the inner wall of the cube calibration device as a reference, adjust the coordinates of the original image acquired by the lens until the position of the geometric center and the reference calibration center meets the preset position requirements, and use the translation amount of the coordinates as the corresponding center offset parameter of the lens. The cube calibration device is a specific device used for lens calibration. Its inner wall has auxiliary lines that intersect to form a reference calibration center, providing a standard reference for lens calibration. The reference calibration center is the point formed by the intersection of the auxiliary lines on the inner wall of the cube calibration device, serving as the benchmark for adjusting the lens image coordinates and ensuring that the lens image is aligned with the standard position. The center offset parameter is the numerical value of the coordinate translation when the position of the lens's geometric center and the reference calibration center does not meet the preset requirements; it describes the offset of the lens center relative to the standard position.

[0076] In this embodiment, the coordinates of the original image acquired by the lens are adjusted based on the reference calibration center formed by the intersection of auxiliary lines on the inner wall of the cube calibration device. By continuously changing the coordinate position of the image, the geometric center determined by the lens is gradually moved closer to the reference calibration center until the positions of the two meet the preset position requirements. During this process, the translation amount of the coordinates is recorded. The translation amount reflects the degree of offset of the lens center relative to the reference calibration center and is used as the center offset parameter of the corresponding lens for correcting the lens imaging position.

[0077] For example, within a cube-shaped calibration device, the intersecting auxiliary lines on its inner walls form a clearly defined reference calibration center. The geometric center of the image captured by the lens deviates slightly from this reference calibration center; suppose it's offset by 5 pixels horizontally and 3 pixels vertically. By adjusting the image coordinates to make the geometric center coincide with the reference calibration center, the image is translated by -5 pixels horizontally and -3 pixels vertically. These two translation amounts are the center offset parameters.

[0078] For example, the two center points are aligned by translation adjustment: using the real center of the scene formed by the intersection of the auxiliary lines on the inner wall of the cube as a reference, the x-axis and y-axis translation of the original image are gradually adjusted until the intersection of the crosshair auxiliary lines of the original image completely coincides with the real center of the scene. The coordinates corresponding to the x and y translation adjustments at this time are the real center (x, y) coordinates of the shot.

[0079] Given the center offset parameter, the original image is scaled to determine the effective radius of the lens's field of view. Scaling refers to the operation of enlarging or reducing the size of an image according to a certain ratio to change the image size and adapt to different calibration requirements and display scenarios. The effective field of view radius is the radius length corresponding to the area from the lens's geometric center to the image edge that can be effectively imaged, reflecting the effective range of the lens's imaging.

[0080] In this embodiment, the original image is scaled. By gradually changing the image size, the imaging performance at different sizes is observed. When the image size is adjusted to a suitable level, the radius corresponding to the area from the lens's geometric center to the image edge that can be effectively imaged in sharp focus is measured, using the lens's geometric center as the center. This radius determines the effective field of view radius of the lens. The effective field of view radius determines the range that the lens can effectively cover in actual imaging.

[0081] For example, since there is an offset between the true center of the lens and the center of the original image, in order to ensure the effective overlap area matching during subsequent binocular image stitching, it is necessary to define the effective field of view range of the lens, i.e., the "effective radius". This can be based on the wide field of view characteristic of binocular lenses—the field of view (FOV) of the selected lens is greater than 180°, and the actual field of view is about 200°. Therefore, there is a natural overlap between the right side of the left lens image and the left side of the right lens image, and this overlapping area corresponds to the same physical scene of the cube.

[0082] During calibration, after completing the first step of true center calibration, the imaging of the overlapping area is observed, and the overlapping area of ​​one lens (left or right lens) is selected as the reference. Then, the original image of the other lens is scaled and adjusted so that the size of the imaging content in the overlapping area of ​​the two lenses is exactly the same. The range parameter calculated by the image scaling ratio and the original image size is the effective image radius of each of the left and right lenses.

[0083] For example, when an image is scaled down, the edges begin to blur or become distorted as the image shrinks to a certain size, while the image from the geometric center to a certain point remains clear. Measuring the distance from this point to the geometric center, let's say 100 pixels, gives us the effective radius of the lens's field of view.

[0084] Given a fixed effective field of view radius, the original images captured by the two lenses are rendered onto a spherical circle model for Euler angle calibration to obtain the Euler angle parameters of the lenses. The target calibration parameters include the center offset parameter, the effective field of view radius, and the Euler angle parameter.

[0085] The spherical circle model is a mathematical model used to describe image rendering in 3D space. It renders the image onto a sphere to simulate human visual perception and present the scene more realistically. Euler angle calibration determines the rotation angles of an object around three coordinate axes in 3D space (Euler angles). It describes the object's spatial attitude and orientation, and is used to accurately determine the spatial position and angle of the lens. Euler angle parameters are the numerical values ​​of the angles by which the lens rotates around three coordinate axes in 3D space, used to describe the lens's spatial attitude and orientation.

[0086] In this embodiment, the original images captured by the two lenses are rendered into a spherical circular model. Within the spherical model, Euler angles are used to describe the lens's attitude in three-dimensional space by analyzing the relative position and orientation of the images. The images from the two lenses are matched and adjusted within the spherical model, and the rotation angle of each lens around three coordinate axes (typically roll, pitch, and yaw axes) is calculated. Euler angle parameters accurately describe the lens's attitude in space.

[0087] For example, after rendering the images onto a spherical model, differences were found in the positions and angles of the two images on the sphere. Through calculation and analysis, it was determined that one lens rotated 10 degrees around the roll axis, 5 degrees around the pitch axis, and -3 degrees around the yaw axis; these angles are the Euler angle parameters of that lens. The other lens also has corresponding Euler angle parameters. These parameters can be used to adjust the lens attitude, ensuring that the two images are correctly matched on the sphere.

[0088] The target calibration parameters obtained from the above technical solution, including center offset parameters, effective field of view radius, and Euler angle parameters, provide a comprehensive and precise calibration for lens imaging. The center offset parameter corrects the geometric center position of the lens imaging, making the image more accurate in coordinates; the effective field of view radius determines the effective imaging range of the lens, avoiding interference from invalid information; and the Euler angle parameter accurately describes the lens's attitude in three-dimensional space, ensuring the synergy and consistency of images acquired by the binocular lenses. This improves the quality and accuracy of lens imaging, enabling the acquired images to more realistically and accurately reflect the actual scene, providing reliable basic data for subsequent image processing, 3D reconstruction, and other applications, and enhancing the performance and reliability of the entire system.

[0089] Optionally, in step S12, calibrating the underwater image using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time includes: In step S121, the underwater images corresponding to the two lenses are translated according to the center offset parameter so that the geometric center of the lens is aligned with the center of the preset canvas. In this embodiment, a pre-calibrated center offset parameter is used to define the offset between the lens's geometric center and the center of a preset canvas in both the horizontal and vertical directions. For underwater images corresponding to two lenses, a translation operation is performed according to this offset. In the horizontal direction, if the center offset parameter indicates that the image center has shifted to the right by a certain number of pixels, the image is moved to the left by the corresponding number of pixels; the same applies to the vertical direction. This translation aligns the lens's geometric center with the center of the preset canvas, eliminating image position deviations caused by lens mounting or imaging characteristics. This provides an accurate positional basis for subsequent image processing, ensuring that the image is correctly displayed within the preset frame.

[0090] For example, when shooting underwater scenes, due to lens mounting issues, the center of the image captured by one lens is offset 10 pixels horizontally to the right and 5 pixels vertically downwards relative to the center of the preset canvas. Based on the center offset parameters, a translation operation is performed on the image, moving it 10 pixels to the left and 5 pixels upwards. This aligns the geometric center of the image with the center of the preset canvas, ensuring the image is in the correct position and preventing positional deviations from affecting the overall effect.

[0091] In step S122, based on the effective field of view alarm, the translated underwater image is scaled, and the invalid image that exceeds the preset canvas is cropped out to obtain the effective area image; In this embodiment, the translated underwater image is scaled according to the effective radius of the field of view. If the image size corresponding to the effective radius of the field of view is smaller than the preset canvas size, the image is enlarged; if it is larger than the preset canvas size, the image is reduced so that the effective portion of the image can fit the preset canvas. Then, it is checked whether the image exceeds the preset canvas range, and the excess portion is cropped out. Because the excess portion cannot be effectively displayed within the preset canvas and may contain interference information, the cropped effective area image meets the size requirements of the preset canvas while ensuring image quality.

[0092] For example, if the effective field of view radius of the translated underwater image is large, exceeding the size of the preset canvas, the image is scaled down proportionally based on the effective field of view radius to make it roughly fit the preset canvas. However, after scaling down, it is found that parts of the four corners of the image still extend beyond the preset canvas; these excess, invalid image portions are cropped out. The final effective area image is exactly within the preset canvas, retaining the image's effective information and clearly displaying the main content of the underwater scene.

[0093] In step S123, the effective area image is mapped onto the 3D semicircular model textures corresponding to the two lenses, and the attitude of the 3D semicircular model is adjusted according to the Euler angle parameter to obtain the target underwater image at the corresponding acquisition time.

[0094] Among them, the 3D semi-circular model texture is the texture information used to display images on the surface of the 3D semi-circular model. It maps the image onto the model surface, so that the model presents the corresponding visual effect.

[0095] In this embodiment, the effective area image is mapped onto the textures of the 3D semicircular models corresponding to the two lenses, allowing the image to fit the surface of the 3D semicircular models. Then, the pose of the 3D semicircular models is adjusted according to the Euler angle parameters. The Euler angle parameters define the rotation angle of the lens in three-dimensional space. By applying these angles to the 3D semicircular models, the rotation angle of the models is made consistent with the actual pose of the lenses. Thus, after mapping and pose adjustment, the image presented by the 3D semicircular models can accurately reflect the actual underwater scene at the corresponding acquisition time, obtaining the target underwater image.

[0096] In this embodiment, to eliminate lens center offset and uncertainty in the effective image range based on calibration parameters, the original images of the left and right lenses undergo the same operational process. First, based on the true center (x, y) coordinates of the left / right lenses, the corresponding original images are translated to ensure that the true center of the lenses is perfectly aligned with the center of the preset canvas, thus ensuring that the imaging reference of the two lenses remains consistent. Subsequently, based on the effective radius parameters of the calibrated left / right lenses, the translated images are scaled and adjusted. Since the effective radius defines the effective range of lens imaging, invalid image content exceeding the canvas after scaling is naturally discarded, retaining only the effective imaging area.

[0097] Furthermore, even after translation and scaling, images still exhibit geometric distortion due to lens pose deviations. This distortion is corrected through 3D rendering using calibrated pose parameters, restoring the image to a spatial pose consistent with the real scene. For example, a semi-circular 3D model is first constructed as the rendering medium to simulate the imaging projection characteristics of a binocular panoramic lens, matching the lens's wide field-of-view imaging effect. Then, the left and right camera images are mapped onto their respective 3D semi-circular model textures. The calibrated pose parameters (yaw, pitch, and roll) are then used to adjust the model's pose: adjusting the yaw angle corrects the horizontal offset, adjusting the pitch angle corrects the vertical offset, and adjusting the roll angle corrects the rotational offset. Ultimately, the rendered left and right camera images perfectly match the spatial geometry of the real scene, resulting in a corrected standard image.

[0098] The above technical solution translates the image based on the center offset parameter, eliminating image position deviations caused by lens installation or imaging characteristics, ensuring accurate image display within the preset canvas. Secondly, by scaling and cropping the image according to the effective field of view radius, the resulting effective area image meets processing size requirements while maintaining image quality and removing interference from invalid information. Finally, the effective area image is mapped onto a 3D semi-circular model and its attitude is adjusted. Combined with Euler angle parameters, the image presented by the model accurately reflects the attitude and content of the actual underwater scene, improving image realism and accuracy, and enhancing the performance and reliability of the entire underwater image processing system.

[0099] Optionally, in step S11, acquiring the attitude data of the inertial measurement unit of the underwater binocular camera when it acquires an underwater image, and adding a synchronization timestamp to the underwater image and the attitude data at the same acquisition time, includes: In step S111, the image timestamp of each frame of the underwater image is obtained when the underwater binocular camera captures the underwater image; In this embodiment of the disclosure, when the underwater binocular camera acquires underwater images, the system generates an image timestamp for each frame. This is done by using the camera's internal clock system to record the current time at the moment of image acquisition, accurate to a specific time unit, such as milliseconds, thereby clearly defining the acquisition time of each frame.

[0100] In step S112, the data timestamp of each attitude data generated by the inertial measurement unit is obtained; In this embodiment of the disclosure, the inertial measurement unit continuously generates attitude data during operation. Simultaneously, its internal clock system generates a timestamp for each attitude data point, recording the specific time the attitude data was generated.

[0101] In step S113, if it is determined that there is no attitude data corresponding to the same acquisition time in any frame of the underwater image, the first data timestamp that is earlier and the second data timestamp that is later are found in the attitude data according to the image timestamp of the underwater image of that frame. In this embodiment of the disclosure, when it is found that there is no attitude data corresponding to the same acquisition time in a certain frame of underwater image, the system will search for the first data timestamp that is earlier and the second data timestamp that is later in the attitude data based on the image timestamp of the frame image to determine the time range.

[0102] In step S114, according to the time interval ratio, a linear interpolation algorithm is used to interpolate between the first data timestamp and the second data timestamp to obtain a target data timestamp that is synchronized with the image timestamp of the underwater image of the frame. In this embodiment, a linear interpolation algorithm is used based on the time interval ratio between the image timestamp and the first and second data timestamps. Assuming that the pose data changes linearly over time, a target data timestamp corresponding to the image timestamp is inserted between the first and second data timestamps.

[0103] In step S115, the target attitude data corresponding to the target data timestamp is determined based on the attitude data corresponding to the first data timestamp and the attitude data corresponding to the second data timestamp. In this embodiment of the disclosure, based on the principle of linear interpolation, the target attitude data corresponding to the target data timestamp is calculated according to the time interval ratio by combining the attitude data corresponding to the first data timestamp and the attitude data corresponding to the second data timestamp.

[0104] In step S116, a synchronization timestamp is added to the target attitude data and the corresponding frame of the underwater image; or, In this embodiment of the disclosure, if the target attitude data is obtained by interpolation, a synchronization timestamp is added to it and the corresponding frame of underwater image.

[0105] In step S117, if it is determined that there is attitude data corresponding to the same acquisition time in the underwater image of any frame, a synchronization timestamp of the underwater image of that frame and the attitude data corresponding to the same acquisition time are added according to the data timestamp of the underwater image of that frame and the data timestamp of the attitude data corresponding to the same acquisition time.

[0106] In this embodiment of the disclosure, if attitude data corresponding to the same acquisition time exists, a synchronization timestamp is directly added based on the timestamps of the two data.

[0107] The above technical solution adds synchronized timestamps to image and attitude data, enabling accurate correlation between the two, whether through direct mapping or linear interpolation. This provides a reliable time reference for underwater scene analysis and motion estimation based on image and attitude data, improving data accuracy and usability.

[0108] The technical solution of this disclosure combines 9-axis IMU attitude sensing with Kalman filtering to achieve high-precision real-time sensing of the fish finder's motion state. The calculated attitude angle error can be controlled within ±0.5°, ensuring the accuracy of image compensation. A data processing strategy of "timestamp synchronization + interpolation matching" solves the data alignment problem between modules with different sampling frequencies, with a synchronization error of less than 10ms, ensuring accurate correspondence between "image-attitude" data. Through an image compensation algorithm based on inverse geometric transformation, it can quickly respond to changes in the fish finder's attitude, with a compensation delay of less than 33ms (corresponding to a 30fps frame rate), achieving stable, lag-free image locking. Compared to existing technologies, this invention does not require changes to the deployment method or lens structure of the fish finder; it only requires adding a low-cost IMU module and optimizing the software algorithm to achieve the fishhook image locking function. It has strong compatibility, controllable cost, and stable and reliable compensation effect, effectively capturing the critical moments of fish feeding and hooking.

[0109] This disclosure also provides an underwater fish school detection and preview device based on image locking function. See [link to relevant documentation]. Figure 2 As shown, the device includes: The acquisition module 210 is configured to acquire the attitude data set in the inertial measurement unit of the underwater binocular camera when the underwater binocular camera acquires an underwater image, and to add a synchronization timestamp to the underwater image and the attitude data at the same acquisition time. The calibration module 220 is configured to calibrate the underwater image using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. The stitching and fusion module 230 is configured to perform mirror linear stitching and fusion of the target underwater images of the two lenses at the same acquisition time according to a preset fusion percentage and a preset reference radius, through gradual weighting, to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined according to the field of view of the lens. Storage module 240 is configured to establish an association index between each frame of the panoramic underwater image and the attitude data under the same synchronization timestamp, forming a structured data unit for storage; The preview module 250 is configured to lock and preview the image of the underwater binocular camera by calling the attitude data of the corresponding acquisition time of the panoramic underwater image according to the associated index.

[0110] Optionally, the attitude data includes angular velocity data, acceleration data, and ambient magnetic field data, and the preview module 250 is configured to: An attitude kinematic model of an underwater binocular camera is established. Based on the associated index, the angular velocity data corresponding to each frame of the panoramic underwater image is called as a basis. The attitude angle change of the underwater binocular camera is initially calculated through integral operation to obtain the predicted attitude. Using the acceleration data and environmental magnetic field data corresponding to the angular velocity data as observations, the predicted attitude is corrected by the Kalman filter algorithm to eliminate accumulated errors and noise interference, and the target attitude angle is obtained. Based on the target attitude angle, the image from the underwater binocular camera is subjected to reverse distortion correction and viewpoint compensation to achieve image lock preview display.

[0111] Optionally, the preview module 250 is configured to: determine a locking reference point in response to initial position coordinates, wherein the initial position coordinates include the user-specified locking display coordinates of the panoramic view received by the human-computer interaction interface; Based on the target attitude angle, determine the instantaneous change in the field of view angle of the underwater binocular camera relative to the locking reference point at the attitude angle; Based on the instantaneous change in the attitude angle, an image transformation model is established, and the image corresponding to the panoramic underwater image of the corresponding frame is subjected to reverse rotation and translation compensation, so that the field of view of the underwater binocular camera is kept at the initial position coordinates, thereby achieving image lock preview display.

[0112] Optionally, the preview module 250 is configured to: in automatic lock playback mode, retrieve the panoramic underwater image corresponding to the instantaneous change of the attitude angle according to the associated index, and sequentially perform attitude calculation, reverse rotation and translation compensation on the screen corresponding to each frame of the panoramic underwater image, and output the locked preview display screen in real time, so that the field of view of the underwater binocular camera is kept at the initial position coordinates. In manual lock playback mode, in response to the user's interactive operation of dragging the timeline to locate the image at any time, the image of the field of view of the underwater binocular camera corresponding to the interactive operation is used as the new initial position coordinates. The image after the panoramic underwater image corresponding to the interactive operation is sequentially subjected to attitude calculation, reverse rotation and translation compensation to achieve image lock preview display.

[0113] Optionally, the device includes a calibration module configured to calibrate the target calibration parameters by determining the geometric center of the lens by drawing horizontal and vertical crosshairs on the raw imaging image acquired by the lens. Using the reference calibration center formed by the intersection of auxiliary lines on the inner wall of the cube calibration device as a reference, adjust the coordinates of the original image acquired by the lens until the position of the geometric center and the reference calibration center meets the preset position requirements, and use the translation amount of the coordinates as the corresponding center offset parameter of the lens. Given the center offset parameter, the original image is scaled to determine the effective radius of the lens's field of view. Given a fixed effective field of view radius, the original images captured by the two lenses are rendered onto a spherical circle model for Euler angle calibration to obtain the Euler angle parameters of the lenses. The target calibration parameters include the center offset parameter, the effective field of view radius, and the Euler angle parameter.

[0114] Optionally, the calibration module 220 is configured to: translate the underwater images corresponding to the two lenses according to the center offset parameter, so that the geometric center of the lens is aligned with the center of the preset canvas; scale the translated underwater image according to the effective field of view alarm, and crop out the invalid image of the underwater image that exceeds the preset canvas to obtain an effective area image; map the effective area image onto the 3D semicircular model texture corresponding to the two lenses respectively, and adjust the attitude of the 3D semicircular model according to the Euler angle parameter to obtain the target underwater image at the corresponding acquisition time.

[0115] Optionally, the acquisition module 210 is configured to: acquire the image timestamp of each frame of the underwater image when the underwater binocular camera captures the underwater image; Obtain the data timestamp for each attitude data generated by the inertial measurement unit; If it is determined that there is no attitude data corresponding to the same acquisition time in any frame of the underwater image, based on the image timestamp of the underwater image in that frame, the first data timestamp that is adjacent to the image timestamp and the second data timestamp that is adjacent to the image timestamp are searched in the attitude data. Based on the time interval ratio, a linear interpolation algorithm is used to interpolate between the first data timestamp and the second data timestamp to obtain a target data timestamp that is synchronized with the image timestamp of the underwater image in that frame; Based on the attitude data corresponding to the first data timestamp and the attitude data corresponding to the second data timestamp, the target attitude data corresponding to the target data timestamp is determined. Add a synchronization timestamp to the target attitude data and the corresponding frame of the underwater image; or, If it is determined that there is attitude data corresponding to the same acquisition time in any frame of the underwater image, a synchronization timestamp of the underwater image in that frame and the attitude data corresponding to the same acquisition time are added according to the data timestamp of the underwater image in that frame and the data timestamp of the attitude data corresponding to the same acquisition time.

[0116] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in the foregoing embodiments.

[0117] This disclosure also provides an electronic device, including: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods described in the foregoing embodiments.

[0118] Figure 3 The underwater fish school detection and preview device 100 based on image locking function, as shown, includes a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, for example, via a bus 1002. Optionally, the underwater fish school detection and preview device 100 based on image locking function may further include a communication component, which can be used for data interaction between the device 100 and other devices, such as sending or receiving data. It should be noted that in actual scheduling, the communication component is not limited to one, and the structure of this underwater fish school detection and preview device 100 based on image locking function does not constitute a limitation on the embodiments of this application.

[0119] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0120] Bus 1002 may include a pathway for transmitting information between the aforementioned components. Bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 1002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0121] The memory 1003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing program code and capable of being read by a computer, without limitation herein.

[0122] The memory 1003 is used to store program code for executing embodiments of the present disclosure, and its execution is controlled by the processor 1001. The processor 1001 is used to execute the program code stored in the memory 1003 to implement the steps shown in the aforementioned embodiment of the underwater fish school detection and preview method based on the screen locking function.

[0123] This disclosure also provides a computer-readable storage medium storing program code. When the program code is executed by a processor, it can implement the steps and corresponding content of the aforementioned underwater fish school detection and preview method embodiment based on screen locking function.

[0124] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present disclosure, various changes, modifications, substitutions and variations can be made to these embodiments, and all such changes, modifications, substitutions and variations fall within the protection scope of the present disclosure.

[0125] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction, and such combinations should also be considered as part of this disclosure. To avoid unnecessary repetition, this disclosure will not further describe the various possible combinations. The technical scope of this application is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for detecting and previewing underwater fish schools based on image locking function, characterized in that, The method includes: When an underwater binocular camera acquires an underwater image, the attitude data of the inertial measurement unit of the underwater binocular camera is set, and a synchronization timestamp is added to the underwater image and the attitude data at the same acquisition time. The underwater image is calibrated using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. Based on a preset fusion percentage and a preset reference radius, the target underwater images of the two lenses at the same acquisition time are mirrored and linearly stitched together using a gradient weighting method to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined based on the field of view of the lens. Each frame of the panoramic underwater image is associated with the attitude data under the same synchronization timestamp, and a structured data unit is formed for storage. Based on the associated index, the attitude data corresponding to the acquisition time of the panoramic underwater image is invoked to lock and preview the image from the underwater binocular camera.

2. The method according to claim 1, characterized in that, The attitude data includes angular velocity data, acceleration data, and environmental magnetic field data. The step of locking and previewing the underwater binocular camera's image by calling the attitude data corresponding to the acquisition time of the panoramic underwater image based on the associated index includes: An attitude kinematic model of an underwater binocular camera is established. Based on the associated index, the angular velocity data corresponding to each frame of the panoramic underwater image is called as a basis. The attitude angle change of the underwater binocular camera is initially calculated through integral operation to obtain the predicted attitude. Using the acceleration data and environmental magnetic field data corresponding to the angular velocity data as observations, the predicted attitude is corrected by the Kalman filter algorithm to eliminate accumulated errors and noise interference, and the target attitude angle is obtained. Based on the target attitude angle, the image from the underwater binocular camera is subjected to reverse distortion correction and viewpoint compensation to achieve image lock preview display.

3. The method according to claim 2, characterized in that, The step of performing reverse distortion correction and viewpoint compensation on the image from the underwater binocular camera based on the target attitude angle to achieve image lock preview display includes: In response to the initial position coordinates, a locking reference point is determined, wherein the initial position coordinates include the locking display coordinates of the panoramic view specified by the user and received by the human-computer interaction interface; Based on the target attitude angle, determine the instantaneous change in the field of view angle of the underwater binocular camera relative to the locking reference point at the attitude angle; Based on the instantaneous change in the attitude angle, an image transformation model is established, and the image corresponding to the panoramic underwater image of the corresponding frame is subjected to reverse rotation and translation compensation, so that the field of view of the underwater binocular camera is kept at the initial position coordinates, thereby achieving image lock preview display.

4. The method according to claim 3, characterized in that, The step of establishing a screen transformation model based on the instantaneous change in the attitude angle, and performing reverse rotation and translation compensation on the corresponding frame of the panoramic underwater image to keep the field of view of the underwater binocular camera at the initial position coordinates, thereby achieving screen lock preview display, includes: In automatic lock playback mode, the panoramic underwater image corresponding to the instantaneous change of the attitude angle is retrieved according to the associated index, and the attitude calculation, reverse rotation and translation compensation are performed on the screen corresponding to each frame of the panoramic underwater image in sequence, and the locked preview screen is output in real time to keep the field of view of the underwater binocular camera at the initial position coordinates. In manual lock playback mode, in response to the user's interactive operation of dragging the timeline to locate the image at any time, the image of the field of view of the underwater binocular camera corresponding to the interactive operation is used as the new initial position coordinates. The image after the panoramic underwater image corresponding to the interactive operation is sequentially subjected to attitude calculation, reverse rotation and translation compensation to achieve image lock preview display.

5. The method according to any one of claims 1-4, characterized in that, The target calibration parameters were obtained by calibration in the following manner: The geometric center of the lens is determined by drawing horizontal and vertical crosshairs on the original imaging image captured by the lens. Using the reference calibration center formed by the intersection of auxiliary lines on the inner wall of the cube calibration device as a reference, adjust the coordinates of the original image acquired by the lens until the position of the geometric center and the reference calibration center meets the preset position requirements, and use the translation amount of the coordinates as the corresponding center offset parameter of the lens. Given the center offset parameter, the original image is scaled to determine the effective radius of the lens's field of view. Given a fixed effective field of view radius, the original images captured by the two lenses are rendered onto a spherical circle model for Euler angle calibration to obtain the Euler angle parameters of the lenses. The target calibration parameters include the center offset parameter, the effective field of view radius, and the Euler angle parameter.

6. The method according to claim 5, characterized in that, The process of calibrating the underwater image using pre-defined target calibration parameters to obtain the target underwater image at the corresponding acquisition time includes: Based on the center offset parameter, the underwater images corresponding to the two lenses are translated respectively so that the geometric center of the lens is aligned with the center of the preset canvas; Based on the effective field of view alarm, the translated underwater image is scaled, and invalid images that exceed the preset canvas are cropped out to obtain the effective area image; The effective area image is mapped onto the 3D semicircular model textures corresponding to the two lenses, and the attitude of the 3D semicircular model is adjusted according to the Euler angle parameters to obtain the target underwater image at the corresponding acquisition time.

7. The method according to any one of claims 1-4, characterized in that, The process of acquiring attitude data from the inertial measurement unit of the underwater binocular camera when it captures underwater images, and adding synchronization timestamps to the underwater images and attitude data at the same acquisition time, includes: Obtain the image timestamp of each frame of the underwater image when the underwater binocular camera captures the underwater image; Obtain the data timestamp for each attitude data generated by the inertial measurement unit; If it is determined that there is no attitude data corresponding to the same acquisition time in any frame of the underwater image, based on the image timestamp of the underwater image in that frame, the first data timestamp that is adjacent to the image timestamp and the second data timestamp that is adjacent to the image timestamp are searched in the attitude data. Based on the time interval ratio, a linear interpolation algorithm is used to interpolate between the first data timestamp and the second data timestamp to obtain a target data timestamp that is synchronized with the image timestamp of the underwater image in that frame; Based on the attitude data corresponding to the first data timestamp and the attitude data corresponding to the second data timestamp, the target attitude data corresponding to the target data timestamp is determined. Add a synchronization timestamp to the target attitude data and the corresponding frame of the underwater image; or, If it is determined that there is attitude data corresponding to the same acquisition time in any frame of the underwater image, a synchronization timestamp of the underwater image in that frame and the attitude data corresponding to the same acquisition time are added according to the data timestamp of the underwater image in that frame and the data timestamp of the attitude data corresponding to the same acquisition time.

8. An underwater fish school detection and preview device based on image locking function, characterized in that, The device includes: The acquisition module is configured to acquire the attitude data set in the inertial measurement unit of the underwater binocular camera when the underwater binocular camera acquires an underwater image, and to add a synchronization timestamp to the underwater image and the attitude data at the same acquisition time. The calibration module is configured to calibrate the underwater image using pre-calibrated target calibration parameters to obtain the target underwater image at the corresponding acquisition time. The target calibration parameters are obtained by calibrating multiple core parameters sequentially based on multiple sets of auxiliary lines drawn on the inner wall of the cube calibration device as geometric references. The stitching and fusion module is configured to perform mirror linear stitching and fusion of the target underwater images of the two lenses at the same acquisition time according to a preset fusion percentage and a preset reference radius, through gradual weighting, to obtain a panoramic underwater image at that acquisition time. The preset fusion percentage is determined according to the field of view of the lens. The storage module is configured to establish an association index between each frame of the panoramic underwater image and the attitude data under the same synchronization timestamp, forming a structured data unit for storage; The preview module is configured to lock and preview the image from the underwater binocular camera by calling the attitude data at the time of acquisition corresponding to the panoramic underwater image based on the associated index.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. A binocular camera, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images