A tuna underwater non-contact three-dimensional reconstruction system and method

Abstract

The application discloses a tuna underwater non-contact three-dimensional reconstruction system and method, and relates to the technical field of intelligent aquaculture.The system comprises an underwater image acquisition device and a data processing terminal for receiving image data and executing a three-dimensional reconstruction algorithm.The method comprises the following steps: acquiring multi-view RGB images of tuna; using a depth estimation network to infer initial point clouds and a depth map, thereby solving the problem of underwater weak texture matching; constructing a radiation field representation based on a plane Gaussian, using the initial depth map as a geometric prior constraint, optimizing Gaussian parameters through unbiased depth rendering and geometric regularization, and reconstructing a high-fidelity surface; and using the physical size of a semicircular frame to calculate a scale factor and extract a tuna three-dimensional grid model with a real physical scale.The application uses the above-mentioned tuna underwater non-contact three-dimensional reconstruction system and method to solve the problem that tuna cannot be measured by contact due to the need for continuous swimming, and realizes high-precision and real-scale fish body monitoring.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and smart aquaculture technology, and in particular to a non-contact three-dimensional reconstruction system and method for tuna underwater. Background Technology

[0002] As a high-value fish species, monitoring the growth and development of tuna is crucial for precision aquaculture. However, tuna have a unique physiological structure; their gill muscles are degenerate, requiring them to constantly swim rapidly to force fresh water through their gills to obtain oxygen. Once they stop swimming, tuna are highly susceptible to suffocation and even death due to lack of oxygen. This makes traditional post-catch contact measurement methods prone to triggering stress responses in tuna, potentially leading to injury or death, and thus failing to meet the needs of long-term, high-frequency monitoring.

[0003] In recent years, non-contact measurement methods based on computer vision have gradually become a research hotspot. However, in underwater aquaculture environments, there are problems such as light refraction, water turbidity, and high reflectivity of tuna scales. Traditional multi-view geometric algorithms often struggle to extract sufficient feature points in areas with weak texture or high reflectivity, resulting in sparse and broken point clouds that cannot accurately fit the fish surface.

[0004] While the recently emerging 3D Gaussian Splatting (3DGS) technology excels in rendering speed and quality, it is essentially a discrete, unordered collection of point clouds, lacking explicit geometric surface constraints. Directly applying 3DGS for reconstruction often results in models with numerous buoyancy artifacts and fails to guarantee surface geometric smoothness, making it unsuitable for accurate volume calculations. Furthermore, 3DGS typically relies on SfM algorithms such as COLMAP for initialization, and SfM is highly prone to failure in weakly textured underwater regions.

[0005] Therefore, there is an urgent need for a non-contact 3D reconstruction method that can adapt to the continuous swimming characteristics of tuna, overcome underwater optical interference, and generate high-fidelity geometric surfaces. Summary of the Invention

[0006] The purpose of this invention is to provide a non-contact three-dimensional reconstruction system and method for underwater tuna to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a non-contact three-dimensional reconstruction system for underwater tuna, comprising: An underwater image acquisition device is used to acquire multi-view image data of tuna while it is swimming; The data processing terminal is connected to the underwater image acquisition device to receive image data and execute 3D reconstruction algorithms. The data processing terminal is equipped with a depth estimation network with a feature fusion architecture, which includes a monocular feature extraction branch and a multi-view feature matching branch. The monocular feature extraction branch uses a pre-trained Transformer backbone network to extract monocular semantic and geometric features of the image, which are used to process reflective and weakly textured areas on the tuna's body surface. The multi-view feature matching branch constructs the disparity cost volume through planar scanning.

[0008] Preferably, the underwater image acquisition device includes a rigid semi-circular frame with a radius of 2 meters. The semi-circular frame has three layers of mounting positions (upper, middle, and lower) on which 12 synchronously triggered motion cameras are evenly deployed. The optical axes of all cameras point to the center area of ​​the semi-circular frame.

[0009] Preferably, the three-dimensional reconstruction algorithm built into the data processing terminal executes the following steps: S1. Control 12 action cameras to simultaneously acquire multi-view RGB image sequences of tuna in swimming state, and perform noise reduction and color balance preprocessing on the multi-view RGB images. S2. To address the lack of texture and high reflectivity in underwater environments, a depth estimation network consisting of a monocular semantic branch and a multi-view matching branch is constructed. Multi-view RGB images are input into the pre-trained depth estimation network. The monocular semantic branch is used to extract global semantic features of the images, and the multi-view matching branch is used to construct the disparity cost volume. The features of the two are fused together to infer the predicted depth map and confidence map from multiple views, and then back-projected to generate the initial sparse point cloud of the scene. S3. Initialize the 3D Gaussian model based on the initial sparse point cloud, transform the 3D Gaussian model into a plane Gaussian model, introduce plane geometric constraints, and use the predicted depth map obtained in S2 as geometric prior constraints to train and optimize the plane Gaussian model. S4. Calculate the absolute scale factor using the known physical dimensions of the semi-circular frame, and convert the optimized planar Gaussian model into a three-dimensional mesh model with real physical scale.

[0010] Preferably, in step S3, the specific method for converting the three-dimensional Gaussian model into a planar Gaussian model is as follows: Scaling matrix for each Gaussian element Constraints are applied to compress the scaling factor along the normal direction to below a preset threshold, making the Gaussian primitive geometrically approximate a two-dimensional disk; and the rendering depth map is calculated using the unbiased depth rendering formula. The rendering depth map is determined by the physical intersection of the view ray and the two-dimensional disk, rather than the mixture of Gaussian center points.

[0011] Preferably, step S3 further includes constructing a joint objective function. Joint objective function From the photometric loss function Geometric regularization loss function and normal regularization function composition.

[0012] Preferably, in the optimization process of step S3, a geometric regularization loss function is introduced. , It includes single-view geometric consistency loss and multi-view geometric consistency loss; the predicted depth map generated in step S2 is used as a pseudo-true value to supervise the depth map and normal vector map rendered by the Gaussian plane, which is used to constrain the Gaussian plane to fit the main geometric surface of the fish body and prevent geometric collapse.

[0013] Preferably, the method for calculating the absolute scale factor in step S4 includes: identifying feature points of the semi-circular frame in the image, and calculating the Euclidean distance between the frame feature points in the reconstructed model. Combined with the actual physical radius or height of the frame Calculate the scaling factor and utilize Scale the coordinates of all vertices of the tuna 3D model.

[0014] A non-contact 3D reconstruction method for underwater tuna includes the following steps: Step 1: Construct an underwater filming environment. Set up a three-layer semi-circular frame with a radius of 2 meters in the aquaculture area, and deploy 12 action cameras on the semi-circular frame. Step 2: When the tuna swims through the central area of ​​the frame, 12 motion cameras are simultaneously triggered to capture multi-view RGB images; Step 3: Use a depth estimation network that integrates monocular feature extraction and multi-view feature matching branches to perform inference on multi-view RGB images, generating a predicted depth map and an initial sparse point cloud. Step 4: Initialize a planar Gaussian field based on the initial sparse point cloud, define the Gaussian primitive as flat, and use the depth map generated in Step 3 as a supervision signal to jointly optimize the position, rotation, opacity and spherical harmonic coefficients of the Gaussian primitive. Step 5: During the optimization process, a normal smoothing constraint based on image edges is used to suppress noise caused by the underwater environment. Step 6: Extract isosurfaces from the optimized planar Gaussian field to generate a triangular mesh, and correct the model scale according to the physical dimensions of the semi-circular frame to output the body length, body width and volume data of the tuna.

[0015] Therefore, the present invention employs the above-mentioned non-contact three-dimensional reconstruction system and method for tuna underwater, which has the following beneficial effects: (1) By constructing an open semi-circular frame for shooting, combined with high frame rate synchronous acquisition and fast reconstruction algorithm, it fully conforms to the physiological characteristics of tuna that must swim continuously; compared with traditional fishing contact measurement, this invention avoids the stress response, mechanical damage and hypoxia death risk of fish, and realizes safe monitoring under natural breeding conditions. (2) To address the failure of traditional multi-view geometric matching caused by light refraction and scattering in water and high reflectivity of tuna scales, a depth-prior-based initialization mechanism was introduced. By fusing monocular semantic features with multi-view geometric features, robust initial geometric structures can be inferred even in areas with weak texture on the belly of the fish or high reflectivity on the back of the fish, significantly improving the integrity and success rate of 3D reconstruction. (3) A planar Gaussian sphere is used instead of the traditional anisotropic three-dimensional Gaussian sphere. By forcibly compressing the normal dimension of the Gaussian sphere and introducing unbiased depth rendering, the surface noise and floating artifacts common in traditional methods are effectively suppressed, and a smooth, continuous and realistic three-dimensional mesh that fits the surface of the fish is reconstructed, providing a reliable data foundation for subsequent high-precision calculation of body length and volume; (4) By combining the pre-set rigid semi-circular frame as the absolute scale benchmark, the inherent scale ambiguity problem in pure visual reconstruction is overcome. By calculating the ratio factor between the reconstruction model and the physical frame, biological parameters with practical measurement significance can be directly output and can be directly applied to feeding decisions and growth assessment in precision aquaculture. (5) Based on attitude normalization, four-sided volume division method and dual-path weighted fusion of volume-density and body length-weight empirical formulas, end-to-end output of four biological parameters, body length, body width, volume and weight, from three-dimensional mesh is realized, providing a reliable data foundation for precision aquaculture.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is an overall hardware architecture diagram of an embodiment of the underwater non-contact 3D reconstruction system for tuna according to the present invention; Figure 2 This is a block diagram illustrating the principle of the depth estimation network in an embodiment of the present invention. Figure 3 This is a logic block diagram of the planar Gaussian optimization according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the semi-circular frame and the dimensional correction principle according to an embodiment of the present invention; Figure 5 This is a flowchart illustrating an embodiment of a non-contact three-dimensional reconstruction method for underwater tuna according to the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0020] Example Please see Figures 1-5 This invention provides a non-contact three-dimensional reconstruction system for underwater tuna, comprising: An underwater image acquisition device is used to acquire multi-view image data of tuna while it is swimming. The device consists of a rigid semi-circular frame made of stainless steel with a radius of 2 meters. The frame has three mounting positions (upper, middle, and lower) and is semi-submersibly fixed to the side wall of the aquaculture cage channel. Twelve synchronously triggered motion cameras are evenly deployed, with the optical axes of all cameras pointing towards the center of the semi-circular frame, forming a surrounding field of view.

[0021] The data processing terminal, connected to the underwater image acquisition device, receives image data and executes 3D reconstruction algorithms. The terminal includes a depth estimation network employing a feature fusion architecture. This network comprises a shared feature encoder, a monocular feature extraction branch, a multi-view feature matching branch, a depth fusion fine-tuning module, and a confidence regression branch. The shared feature encoder, based on a convolutional neural network, extracts shared feature maps from the input image and, after pre-training, is shared by both the monocular feature extraction and multi-view feature matching branches. The monocular feature extraction branch utilizes the shared feature maps to extract monocular semantic and geometric prior features, enhancing the robustness of depth inference in reflective and weakly textured areas on the tuna's surface. The multi-view feature matching branch constructs a multi-view matching cost volume based on planar scanning between a reference viewpoint and adjacent viewpoints, and then aggregates this cost using 3D convolutional cost to obtain a multi-view depth map. In this embodiment, the data processing terminal is connected to a motion camera via a gigabit Ethernet cable to receive synchronously triggered multi-view RGB image sequences. ,in .

[0022] The execution steps of the 3D reconstruction algorithm built into the data processing terminal are as follows: S1. Control 12 action cameras to simultaneously acquire multi-view RGB image sequences of tuna in swimming state, and perform noise reduction and color balance preprocessing on the multi-view RGB images.

[0023] S2. To address the issues of texture loss, uneven lighting, and the high reflectivity of tuna scales in underwater environments, a depth estimation network is constructed. This network is an end-to-end convolutional neural network, consisting of five cascaded / parallel modules: a shared feature encoder, a monocular feature extraction branch, a multi-view feature matching branch, a depth fusion fine-tuning module, and a confidence regression branch. The network input consists of multi-view RGB images simultaneously captured by 12 motion cameras. The optimal resolution for a single input image is 384×384 pixels, and the pixel values ​​are normalized according to the channel mean and standard deviation before being fed into the network.

[0024] The shared feature encoder is used by both the monocular semantic branch and the multi-view matching branch. It consists of 8 two-dimensional convolutional layers (Conv2d) and 2 downsampling layers. The parameter configurations of each convolutional layer are as follows: Conv1: number of channels 3→64, convolutional kernel 3×3, stride 2, output 64×192×192; Conv2: number of channels 64→64, convolutional kernel 3×3, stride 1, output 64×192×192; Conv3: number of channels 64→128, convolutional kernel 3×3, stride 2, output 128×96×96; Conv4: number of channels 1 Conv1: Channels 128→128, convolution kernel 3×3, stride 1, output 128×96×96; Conv2: Channels 128→256, convolution kernel 3×3, stride 1, output 256×96×96; Conv3: Channels 256→256, convolution kernel 3×3, stride 1, output 256×96×96; Conv4: Channels 256→256, convolution kernel 3×3, stride 1, output 256×96×96; Conv5: Channels 256→320, convolution kernel 3×3, stride 1, output 320×96×96; Conv6: Channels 356→256, convolution kernel 3×3, stride 1, output 320×96×96; Conv7: Channels 256→320, convolution kernel 3×3, stride 1, output 320×96×96. Each convolutional layer is followed by a BatchNorm normalization layer and a ReLU activation function. The shared feature encoder can be pre-trained using historical underwater tuna image datasets and publicly available underwater image datasets, and then the parameters can be frozen or fine-tuned with a small learning rate.

[0025] Monocular feature extraction branch receives shared feature map It outputs a semantic probability map and a monocular initial depth map through two parallel convolutional heads: (a) Semantic segmentation head: Consists of 5 two-dimensional convolutional layers: Conv9: 320→160, 3×3, stride 1; Conv10: 160→160, 3×3, stride 1; Conv11: 160→128, 3×3, stride 1; Conv12: 128→64, 3×3, stride 1; Conv13: 64→ 1×1, step size 1; followed by softmax activation, output size is The semantic probability graph of ×96×96 . Take 3, corresponding to the three categories of fish body, semi-circular frame and background water; (b) Monocular Depth Regression Head: Consists of 4 two-dimensional convolutional layers: Conv14: 320→160, 3×3, stride 1; Conv15: 160→128, 3×3, stride 1; Conv16: 128→64, 3×3, stride 1; Conv17: 64→1, 1×1, stride 1, outputting a monocular initial depth map with a size of 1×96×96. It is used to provide global geometric priors.

[0026] The multi-view feature matching branch constructs a multi-view cost volume based on plane sweep and obtains a multi-view depth map through 3D convolution regularization.

[0027] Rules for selecting reference viewpoints and adjacent viewpoints: Each image captured by 12 cameras Using this as a reference perspective, we proceed with depth reasoning. For reference perspective i, we select K=4 adjacent perspectives from the remaining 11 perspectives, following the selection rule: (1) Calculate the angle between the optical axis direction of each viewing angle and the optical axis direction of the reference viewing angle based on the external parameters of each camera. ; (2) Calculate the baseline length based on the world coordinates of the camera's optical center. ; (3) Prioritize selecting options that simultaneously satisfy the criteria. and From the perspective; (4) If there are more than 4 viewpoints that meet the conditions, the top 4 will be selected from the baseline-angle composite score in descending order. (5) If there are fewer than 4, then supplement them to 4 from the remaining perspectives according to the overall score.

[0028] Let the set of adjacent viewpoints be . .

[0029] Planar scanning and parallax cost volume construction: For reference viewpoint i, within the preset depth range Internally, a reverse depth uniform sampling structure is used. A depth plane; preferred The depth value of the nth depth plane. Satisfy the following formula: ; in ; The closer to the camera, the higher the depth resolution, which corresponds to the main imaging range when the tuna swims through the central area of ​​the frame.

[0030] For each depth plane Based on the camera intrinsic parameters between reference viewpoint i and adjacent viewpoint j and relative extrinsic parameters (rotation matrix) Translation vector (representing the rigid body transformation from reference viewpoint i to adjacent viewpoint j), the planar induced remapping relationship of the feature map is established according to the homography matrix method: ; in The unit normal vector of the depth plane in the reference viewpoint coordinate system is taken in this embodiment as the optical axis direction of the reference viewpoint. As a plane normal This represents the transpose of a matrix or vector. For the reference feature map... any pixel coordinate (homogeneous coordinates), whose feature map pixel coordinates are those of the adjacent viewpoint j corresponding to the nth depth plane: ; Then perform homogeneous normalization to obtain ; Bilinear interpolation is used to extract feature maps from adjacent viewpoints. Mid-sampling features yield aligned feature maps. Where c=1,...,320 are channel indices.

[0031] Calculation of scalar matching cost and multi-view aggregation: For any pixel (u, v) and any depth plane First, calculate the channel-level matching cost between the reference feature and the aligned adjacent view features for each adjacent viewpoint j: ; Then, the average cost of K=4 adjacent viewpoints is taken to obtain the scalar form of the multi-view cost volume. : ; therefore It is The three-dimensional tensor (96×96×96 in this embodiment) has each element as a scalar representing the average matching error under the assumption of that pixel and that depth.

[0032] Cost volume aggregation and multi-view depth regression: Multi-perspective cost body Treated as a single-channel input (i.e.) The input is fed into a cost aggregation subnetwork consisting of 6 3D convolutional layers for regularization: 3D Conv1: 1→16, 3×3×3, stride 1; 3D Conv2: 16→32, 3×3×3, stride 1; 3D Conv3: 32→32, 3×3×3, stride 1; 3D Conv4: 32→16, 3×3×3, stride 1; 3D Conv5: 16→8, 3×3×3, stride 1; 3D Conv6: 8→1, 3×3×3, stride 1. Each 3D Conv layer is followed by BatchNorm and ReLU. The output dimension of 3D Conv6 is... The cost tensor is then subjected to softmax normalization over the depth dimension to obtain the probability volume. The multi-view depth map was obtained using the expected value regression (soft-argmin) method. ; The output size is 1×96×96.

[0033] Deep fusion fine-tuning module: The deep fusion fine-tuning module will adjust the initial depth map of the monocular camera. Multi-view depth map and semantic probability graph By concatenating the channels according to their dimensions, the number of channels is: The tensor with size is This module contains 5 two-dimensional convolutional layers, 1 transposed convolutional layer, and 1 1×1 convolutional thinning layer: Conv18: (2+ Conv10: 128→128, 3×3, stride 1; Conv21: 64→32, 3×3, stride 1; Conv22: 32→16, 3×3, stride 1; Transposed convolutional layer: 16→16, 4×4, stride 4, upsampling the spatial resolution from 96×96 to 384×384; Conv23 (1×1 thinning): 16→1, 1×1, stride 1. The final output is a predicted depth map of size 1×384×384. It has the same resolution as the original input image.

[0034] Confidence Regression Branch: The confidence regression branch and the deep fusion fine-tuning module share the first three convolutional features from Conv18 to Conv20, and then cascade two more convolutional layers: : 64→16, 3×3, step size 1; : 16→1, 1×1, step size 1; followed by Sigmoid activation, resulting in a confidence map normalized to [0, 1]. Size and Similarly, the confidence level represents the depth prediction confidence for each pixel. During training, the supervision signal for the confidence map is constructed using the following formula: ; in The reference depth is obtained by a structured light scanner or active light 3D measurement device in the pre-collected calibration data set. σ is the normalized scale, preferably σ=0.05m.

[0035] Initial sparse point cloud Using the intrinsic parameter matrix K of the motion camera and the predicted depth map semantic probability map in image Pixels identified as "fish bodies" are back-projected onto the world coordinate system to obtain an initial sparse point cloud. Pixels identified as "background water" by the semantic probability map are not included in point cloud generation, in order to suppress the interference of suspended particles in the water on the initialization.

[0036] S3, Based on the initial sparse point cloud Initialize the planar Gaussian model and perform geometric constraint optimization on it, such as... Figure 3 As shown.

[0037] Planar Gaussian model initialization Based on the initial sparse point cloud Generate Gaussian set Each Gaussian unit Includes the following learnable parameters: (a) Central location Initialize to the coordinates of the corresponding point in the point cloud; (b) Rotation Quaternion Initialized as a random unit quaternion, it can be rotated using a matrix. Characterization; (c) Local scale vector Initialized to the k-nearest neighbor-based point spacing ,Right now ; (d) Opacity It is initialized to 0.1 and parameterized by the Sigmoid function; (e) Spherical harmonic coefficients The order is 3, and there are a total of 48 parameters used to model view-related colors.

[0038] The initial total number of Gaussian elements is preferably 30,000 to 50,000, and is increased or decreased during training according to an adaptive densification strategy.

[0039] Planarization constraints: To degenerate the Gaussian elements into two-dimensional disks attached to the surface of the fish, for each Gaussian element... The three scale components Apply a minimum compression constraint. Let the normal direction of the k-th Gaussian element be the direction corresponding to the smallest of its three principal axes, i.e. Corresponding to The other two principal axes are tangential. A planarization regularization term is introduced. The sum of the L1 norms of all Gaussian elements at the minimum scale: ; During the optimization process, This continuously forces the minimum scale component of each Gaussian element to approach 0; preferably, after more than 15,000 training iterations, a harder constraint is applied. The planarization threshold ε is preferably... That is, 0.1mm. If the threshold is exceeded, the gradient is directly truncated and the corresponding gradient is set to zero.

[0040] Unbiased depth rendering: It employs unbiased depth rendering induced by a plane, unlike traditional alpha-weighted depth. For pixel u, let the position of the view optical center in the world coordinate system be... The camera rotates to (From the world to the camera), for each Gaussian element Its smallest scale direction corresponds to the normal vector in the world coordinate system. Define the distance from the plane to the camera's optical center. for: ; The normal vector map and distance map are rendered using α-blending respectively: ; ; in To accumulate transparency, N(u) is the set of Gaussian elements traversed by the view ray u. The final unbiased rendered depth map is calculated using the following formula: ; Where ũ is the homogeneous coordinate of pixel u. This formula represents the physical intersection depth of the view ray and the rendering plane; by dividing the distance map by the dot product of the normal and back-projected rays, the geometric bias introduced by alpha weighting is eliminated. Simultaneously, according to... The neighborhood difference can analytically compute the rendering normal map. .

[0041] Joint objective function: Constructing a joint objective function It consists of a weighted average of five losses: ; in For adjustable weights, the preferred values ​​are: The optimal number of training iterations is 30,000 steps. The optimizer used is Adam, and the learning rates for each parameter are as follows: initial learning rate at the center position. Opacity ,scale Rotation spherical harmonic coefficient .

[0042] Photometric loss : Rendering RGB images Compared with actual photos The loss between them is calculated using a weighted sum of the L1 norm and SSIM: ; Choose 0.2.

[0043] Single-view geometric consistency loss Single-view geometric consistency loss To predict depth maps The geometric pseudo-true value consists of two parts: depth consistency and normal consistency. (a) Deep consistency item: ; (b) Normal consistency term: derived from prediction depth The pseudo-normal map is calculated by backprojection from the adjacent 4-neighborhood. For pixels Take the four neighboring points , , , ,in To pass And the back projection operator of the camera intrinsic parameter K. Then: ; Accordingly: ; Where 1[·] is the indicator function, Edge-aware weights based on image gradients: The larger the gradient magnitude of the image, the smaller the weight, to avoid excessive smoothing of the fish's outline.

[0044] (c) Composite form: ; Preferred , .

[0045] How to determine the confidence threshold T: The threshold T was calibrated offline using a pre-collected underwater calibration dataset of tuna. The calibration steps are as follows: (1) For each frame on the calibration dataset, run the trained depth estimation network to obtain... and ; (2) For each candidate threshold Statistical satisfaction pixel set ; (3) In The average depth error (MAE(T)) and effective pixel percentage are calculated above. ; (4) Select the threshold that makes Cov(T)≥40% and minimizes MAE(T) as the final value.

[0046] According to the calibration results of this embodiment, the preferred range of T is 0.6 to 0.8, and more preferably 0.7. In this embodiment, T=0.7 is taken, that is, the geometric consistency loss is calculated only on pixels with a confidence level greater than 0.7, so as to effectively suppress the interference of suspended particles in water, local reflections and boundary occlusion on geometric optimization.

[0047] Multi-view geometric consistency loss : To further ensure geometric consistency across multiple viewpoints, a multi-view forward and backward projection error is introduced. For any pixel in the reference viewpoint r... Using the normal vector from this perspective Distance from the plane It is mapped to the adjacent viewpoint n using a homography matrix: ,in ; Then, from the perspective of adjacent nodes, n uses its own... , Reverse mapping back to the reference viewpoint yields The forward and backward projection errors are: ; ; in for Less than the preset occlusion threshold The set of pixels of pixels, for Pixels that are not occluded or geometrically incorrect are considered as occlusions and are not included in the multi-view loss.

[0048] Normal smoothing loss : An edge-aware normal smoothing loss is employed. First, Sobel edge detection is performed on the input image to obtain the pixel gradient magnitude G(u). In non-edge regions (i.e., G(u) is less than a preset edge threshold), the gradient magnitude is determined. (Position), apply a smoothing constraint to the rendering normal vectors of the 4 neighboring pixels: ; in For edge decay weights, A value of 0.1 is preferred, used to reduce the smoothing weight at the edges of the image to avoid over-smoothing the fish body outline and fin edges.

[0049] S4. Scale recovery is performed using the known physical dimensions of the semi-circular frame, and a 3D mesh model with true physical scale is extracted from the optimized planar Gaussian model, such as... Figure 4 As shown.

[0050] Scale factor calculation: From multi-view images captured by 12 action cameras, the edge pixels of the semi-circular frame in at least three view images were extracted using the YOLOv8 algorithm. Combined with camera extrinsic parameters, the three-dimensional point set of the frame edge was reconstructed through triangulation. .

[0051] right Optimize the number of iterations when running the RANSAC circular fitting algorithm. Interior point tolerance optimization In each iteration, three points are randomly selected to determine a candidate arc, and the results are statistically analyzed. The distance from the center to the arc is less than The number of interior points is used to determine the final fitting result, and the candidate arc with the highest number of interior points is selected. If the final proportion of interior points is less than 60%, the result for this frame is discarded.

[0052] Let the reconstructed radius obtained by fitting be . The known physical radius of the semi-circular frame Then the absolute scale factor is: ; The optimized center coordinates of all Gaussian elements Multiply by the scale factor λ and simultaneously scale the scale vector. To restore the true physical scale of the scene.

[0053] TSDF fusion extraction of 3D mesh: Voxel space partitioning: Based on the minimum bounding box (AABB) of the scale-corrected planar Gaussian field, the voxel space is expanded by 20% along each axis. The voxels are then uniformly discretized with a fixed side length Δ, preferably 5 mm, to obtain the voxel set. and its center point .

[0054] For each voxel center point The pixel coordinates are calculated by camera projection from all 12 viewpoints i. and depth in camera coordinate system : ; ; in From the world to the camera The extrinsic transformation matrix.

[0055] Single-view symbol distance: in viewpoint Next, retrieve the depth value of the pixel location in the rendered depth map. Define the voxel in the field of view The signed distance below is: ; when , indicating that the voxel is in front of the visible surface (camera side); when , indicating that the voxel is behind the surface (the occluded side).

[0056] Truncated symbol distance: Set the truncation distance Then the truncated TSDF value for a single viewpoint is: ; Multi-view weighted fusion: assuming voxels in viewpoints The weights below are Defined as: ; in Whether voxels are projected onto the viewpoint The indicative function within the image (zero when it is out of bounds or occluded); The confidence level of S2 output; The angle between the line of sight and the local surface normal (through) calculate); Invalid observations far from the surface are removed. The TSDF value after multi-view fusion is: ; ; Triangular mesh extraction: The Marching Cubes algorithm is run on the fused TSDF voxel field to extract isosurfaces. The initial triangular mesh on Then proceed in turn to The following post-processing steps are performed to obtain the final 3D mesh model M of the tuna: (1) Hole filling: Filling based on Poisson surface reconstruction The area is less than 100×Δ 2 The opening; (2) Taubin smoothing: Set the smoothing factor Anti-smoothing factor Iterate 10 times to suppress high-frequency noise on the surface; (3) Normal unification: Unify the normal of the patch along the normal direction with the most consistency along the connected component direction; (4) Pruning: Based on semantic probability graph The fish mask is cut to retain only the fish body part, and the semi-circular frame and background water fragments are removed.

[0057] Calculation of body length, width, volume, and weight: The body length of the tuna was calculated based on the scale-corrected 3D mesh model M. L Body width ,volume V and weight .

[0058] Posture normalization: For the set of all vertices of mesh M Perform principal component analysis (PCA). First, calculate the vertex centroids: ; Next, calculate the covariance matrix of the centered vertex: ; right Perform singular value decomposition to obtain , where the eigenvalues Corresponding feature vector .definition: (a) First principal axis The direction is from head to tail of the tuna; (b) Second principal axis In the dorsoventral direction; (c) Third principal axis The direction is the side of the body.

[0059] Constructing rigid body transformation matrix Transform the vertices from the world coordinate system to the principal axis coordinate system: ,in ; After this transformation, the M principal axis is aligned with the xyz coordinate system.

[0060] Head and tail identification: In the normalized principal axis coordinate system, the mesh is divided into 10 equal segments along the first principal axis, and the distance from each vertex in each segment to the grid is calculated. Average distance of axis Scanning from both ends toward the middle, the end with the larger change in cross-sectional radius and the one that converges rapidly is identified as the fish tail end, and the end with the smaller change in cross-sectional radius is identified as the fish head end.

[0061] Body length L Calculation (unit: mm): Body length L Take the farthest point at the head end along the first principal axis direction farthest point from the tail Euclidean distance: ; Body width Calculation (unit: mm): Along the first principal axis (x-axis) by step size Divide the mesh into vertical sections to obtain There are several cross sections. For each cross section s, a cutting plane is used. Intersecting with mesh M yields the polygonal profile of the cross-section. The cross-sectional profile is then calculated in... Maximum inscribed chord length in a plane The width is taken as the maximum value of all cross sections. ; volume V Calculation (unit: mm) 3 ): The tetrahedral volume integral method based on the divergence theorem is employed. For each triangular facet in mesh M... (vertex , , Calculate the directed volume of the tetrahedron formed by the facet and the origin: ; The sum of the absolute values ​​of the directed volumes of all facets equals the volume of the closed mesh. ; Subsequently, the volume unit was changed from mm. 3 Convert to For use in weight calculation.

[0062] weight calculate: weight It was obtained by using a dual-path cross-correction method based on empirical body length-weight relationship and volume-density relationship.

[0063] (a) Path 1: Volume-Density Method.

[0064] ; The unit is cm 3 ρ is in g / cm³ 3 Finally multiplied by kg; ρ is an empirical value for the average body density corresponding to a tuna species, with the preferred value being: Pacific bluefin tuna: ρ = 1.08 g / cm³ 3 ; Atlantic bluefin tuna: ρ = 1.06 g / cm³ 3 ; Yellowfin tuna: ρ = 1.05 g / cm³ 3 .

[0065] The ρ values ​​of different varieties were measured during the pre-calibration stage and stored in the data processing terminal.

[0066] (b) Path 2: Empirical relationship between body length and weight.

[0067] The publicly available fitting formula for Mediterranean bluefin tuna was used: ; in Indicates body length (unit: cm). The unit is kg. The optimal selection factor is: Atlantic bluefin tuna: ; Pacific bluefin tuna: ; The coefficients a and b for other varieties are fitted offline based on samples of the same variety and stored during the pre-calibration stage.

[0068] (c) Final weight-weighted fusion ; Where η∈[0,1] is the weighting coefficient, reflecting the confidence level of volume measurement; a larger value is taken when the grid is closed and there are no obvious voids, and a smaller value is taken otherwise; preferably η=0.5.

[0069] Output result: The final output is the body length with true physical scale. L (mm), body width (mm), volume V (cm 3 ) and weight Four biological parameters (kg) are used for feeding decisions, growth assessment, and market exit prediction in precision farming.

[0070] Therefore, the present invention adopts the above-mentioned underwater non-contact three-dimensional reconstruction system and method for tuna, which solves the problem that tuna cannot be measured due to its continuous swimming, and realizes high-precision, real-scale fish body monitoring.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A non-contact three-dimensional reconstruction system for underwater tuna, characterized in that, include: An underwater image acquisition device is used to acquire multi-view image data of tuna while it is swimming; The data processing terminal is connected to the underwater image acquisition device and is used to receive image data and execute 3D reconstruction algorithms. The data processing terminal is equipped with a depth estimation network with a feature fusion architecture. The depth estimation network includes a shared feature encoder, a monocular feature extraction branch, a multi-view feature matching branch, a depth fusion fine-tuning module, and a confidence regression branch. The shared feature encoder is based on a convolutional neural network and is used to extract shared feature maps from the input image. After pre-training, it is shared by both the monocular feature extraction branch and the multi-view feature matching branch. The monocular feature extraction branch extracts monocular semantic features and geometric prior features of the image through the shared feature map output by the shared feature encoder, which is used to enhance the robustness of deep inference in reflective and weakly textured areas on the tuna body surface. The multi-view feature matching branch constructs a multi-view matching cost volume based on planar scanning between the reference view and adjacent view, and obtains a multi-view depth map through 3D convolutional cost aggregation.

2. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 1, characterized in that: The underwater image acquisition device includes a rigid semi-circular frame with a radius of 2 meters. The semi-circular frame has three layers of mounting positions (upper, middle, and lower) on which 12 synchronously triggered motion cameras are evenly deployed. The optical axes of all cameras point to the center area of ​​the semi-circular frame.

3. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 2, characterized in that, The execution steps of the 3D reconstruction algorithm built into the data processing terminal are as follows: S1. Control 12 action cameras to simultaneously acquire multi-view RGB image sequences of tuna in swimming state, and perform noise reduction and color balance preprocessing on the multi-view RGB images. S2. Construct a deep estimation network that includes a shared feature encoder, a monocular feature extraction branch, a multi-view feature matching branch, a deep fusion fine-tuning module, and a confidence regression branch; For each reference viewpoint, a predetermined number of adjacent viewpoints are selected from the remaining viewpoints according to the angle between the baseline length and the optical axis, and uniform sampling is performed inversely within a predetermined depth range. A multi-view matching cost volume is constructed based on a homography matrix in a depth plane. The cost volume is then aggregated by 3D convolution to obtain a multi-view depth map, which is then fused with the monocular depth map and the semantic probability map to output a predicted depth map. and confidence plot And generate an initial sparse point cloud by back-projecting the pixels of the fish body region. ; S3, Based on the initial sparse point cloud The planar Gaussian model is initialized, and the minimum scale component of the Gaussian primitive is compressed by the planarization regularization term to degenerate it into a two-dimensional disk. Unbiased depth rendering is used to calculate and render the depth map, and the predicted depth map obtained in S2 is used as the geometric prior constraint. The planar Gaussian model is trained and optimized by combining photometric loss, single-view geometric consistency loss, multi-view geometric consistency loss, normal regularization term and planarization regularization term. S4. Calculate the absolute scale factor using the known physical dimensions of the semi-circular frame, perform scale correction on the optimized planar Gaussian model, and extract a three-dimensional mesh model with real physical scale through TSDF fusion and Marching Cubes algorithm, and further calculate and output the body length, body width, volume and weight of the tuna.

4. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 3, characterized in that: In step S2, the number of depth planes The value ranges from 64 to 128, and the depth range is specified. The range is set from 0.5m to 4.0m; the number of adjacent viewing angles K is 2 to 6.

5. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 4, characterized in that, In step S3, the specific method for converting the 3D Gaussian model into a planar Gaussian model is as follows: For each Gaussian element scale vector By using flattening regularization The term applies an L1 compression constraint to the smallest scale component; After the optimization iterations exceed a preset number of steps, further hard constraints are applied. The planarization threshold ε is a preset value; and the rendering depth map is calculated using the unbiased depth rendering formula. The rendering depth map is determined by the physical intersection of the line-of-sight ray and the two-dimensional disk, rather than the mixture of Gaussian center points.

6. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 5, characterized in that: Step S3 also includes constructing a joint objective function, which is composed of the photometric loss function. Single-view geometric consistency loss function Multi-view geometric consistency loss function Normal regularization function and flattening regularization terms composition.

7. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 6, characterized in that: In the optimization process of step S3, a geometric regularization loss function is introduced. , Including deep consistency loss and normal consistency loss ; Using the predicted depth map generated in step S2 As a geometric false truth value, it is only present in the confidence plot. Calculation of regions where the confidence level of a mid-pixel is greater than a preset threshold T and The threshold T is determined offline by pre-collected calibration dataset, and its value ranges from 0.6 to 0.8, preferably 0.

7.

8. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 7, characterized in that, The method for calculating the absolute scale factor and outputting biological parameters in step S4 includes: extracting multi-view two-dimensional feature points of the semi-circular frame, reconstructing the three-dimensional point set based on triangulation, and then running RANSAC circular fitting to obtain the reconstructed radius. Based on the actual physical radius of the frame Calculate the scale factor The planar Gaussian model or its extracted mesh model is scaled using λ; based on the scale-corrected mesh model, the body length is determined by aligning the principal axes using principal component analysis (PCA). L The body width is determined by dividing the section along the main axis at fixed step lengths. The volume is determined by the method of volume integrals of four sides. V The weight was calculated using a weighted fusion method based on the preset variety density ρ and the pre-calibrated body length-weight fitting coefficients (a, b). .

9. The underwater non-contact three-dimensional reconstruction system for tuna according to claim 8, characterized in that: In step S4, the voxel side length Δ of the TSDF fusion is taken as 2mm~10mm, and the cutoff distance is... Use a voxel side length of 2 to 5 times; the varietal density ρ should be within the range of 1.05 g / cm³. 3 ~1.10g / cm 3 The weighted fusion coefficient η ranges from 0.4 to 0.

6.

10. A non-contact underwater 3D reconstruction method for tuna, using the non-contact underwater 3D reconstruction system for tuna as described in any one of claims 1-9, characterized in that, Includes the following steps: Step 1: Construct an underwater filming environment. Set up a three-layer semi-circular frame with a radius of 2 meters in the aquaculture area, and deploy 12 action cameras on the semi-circular frame. Step 2: When the tuna swims through the central area of ​​the frame, 12 motion cameras are simultaneously triggered to capture multi-view RGB images; Step 3: Using a depth estimation network that integrates a shared feature encoder, a monocular feature extraction branch, a multi-view feature matching branch, a deep fusion fine-tuning module, and a confidence regression branch, inference is performed on multi-view RGB images to generate predicted depth maps. Confidence plot and initial sparse point cloud ; Step 4: Based on the initial sparse point cloud Initialize a planar Gaussian field, degenerate Gaussian elements into two-dimensional disks through L1 norm minimum scale regularization terms, use the depth map generated in step 3 as a supervision signal, and jointly optimize the position, rotation, opacity and spherical harmonic coefficients of Gaussian elements. Step 5: During the optimization process, normal smoothing constraints based on image edges and multi-view forward and backward projection consistency loss are used to suppress noise caused by the underwater environment and ensure multi-view geometric consistency. Step 6: Generate a triangular mesh from the optimized planar Gaussian field by fusing with TSDF and extracting isosurfaces using Marching Cubes, and correct the model scale according to the physical dimensions of the semi-circular frame to output the tuna's body length, body width, volume, and weight data.

Images