Fish morphological phenotype acquisition method and system based on 3D point cloud segmentation network of multi-module cascade

Abstract

The application discloses a fish morphological phenotype acquisition method based on a 3D point cloud segmentation network of a multi-module cascade, and comprises the following steps: extracting key frame pictures of a multi-view video of a target fish based on a video processing tool; reconstructing a three-dimensional model of the target fish by using the key frame pictures of the target fish based on a multi-view stereo vision algorithm; and inputting the three-dimensional model of the target fish and the key frame pictures into a trained fish morphological phenotype segmentation network to obtain a phenotype segmentation result of the target fish.

Description

Technical Field

[0001] This invention relates to the field of agricultural information technology, specifically to a method and system for acquiring fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network. Background Technology

[0002] 3D point cloud information can capture the overall appearance and details of a fish, providing higher accuracy and more morphological information. The 3D point cloud data of the fish body can measure complex indicators such as volume, surface area, and 3D morphological features. Generally, after acquiring the 3D data of the fish, it is necessary to process the noise points present. Currently, many laser scanning devices or depth cameras can be used to obtain the 3D appearance of an object, thus obtaining the object's point cloud morphology, i.e., the object's 3D coordinate information. Since laser scanning for point cloud acquisition requires a long processing time, this leads to excessive contact time with the fish, affecting its condition, and purchasing the equipment incurs a certain cost. The PointNet++ model based on deep learning suffers from oversegmentation or blurred segmentation boundaries when segmenting different phenotypes of fish. Summary of the Invention

[0003] To overcome the technical problem of oversegmentation or blurred segmentation boundaries between multiple phenotypes in the PointNet++ model when segmenting different phenotypes of fish, this invention provides a method and system for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network. The network segmentation performance is improved by designing a densely connected MLP, improving the three-branch attention mechanism, and re-extracting additional information.

[0004] According to one aspect of the present invention, a method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D (three-dimensional) point cloud segmentation network is provided, comprising:

[0005] Keyframe images of the target fish were extracted from a multi-view video using video processing tools.

[0006] Based on multi-view stereo vision algorithm, the 3D model of the target fish is reconstructed using keyframe images of the target fish;

[0007] The 3D model of the target fish and keyframe images are input into the trained fish morphology phenotypic segmentation network to obtain the phenotypic segmentation results of the target fish.

[0008] The training of the fish morphological phenotype segmentation network specifically involves:

[0009] Construct training and testing sets for 3D point cloud data samples of the target fish species;

[0010] An improved PointNet++ network is used to obtain a fish morphological phenotype segmentation network, specifically the DPS-PointNet++ network, which includes: a hierarchical abstraction module based on multi-scale learning and a feature propagation module based on a three-branch attention mechanism. The hierarchical abstraction module comprises upper and lower layers. The upper layer takes a 3D model as input and introduces a densely connected multilayer perceptron (DenseNet-MLP) structure to extract enhanced point cloud features. The lower layer processes point cloud fusion features and keyframe images, and extracts multi-scale fusion features based on the multi-scale learning module. The feature propagation module based on the three-branch attention mechanism (PTA) outputs the segmentation results of the fish phenotype based on the results of the hierarchical abstraction module.

[0011] The network was trained using the dataset, and the fish morphological phenotypic segmentation network was preserved after training.

[0012] As a further implementation scheme, the reconstruction of the 3D model of the target fish includes sparse reconstruction and dense reconstruction;

[0013] The sparse reconstruction process includes: first, obtaining the file paths of the multi-view image set; then, using the corresponding point search method, performing feature extraction and feature matching; followed by matching optimization based on geometric conditions and establishing a scene graph; then, entering the incremental SfM algorithm for reconstruction, first initializing the algorithm, then using triangulation and pose estimation to obtain the overall shape of the fish, and finally using bundle adjustment to optimize and obtain sparse point clouds.

[0014] The dense reconstruction process includes: obtaining the Colmap interface and extracting sparse point clouds; obtaining the most complete point cloud possible through dense reconstruction, finding points with consistent light intensity in space under known camera parameters of the image, and then performing scene stereo matching by finding matching corresponding points from images from different perspectives to recover 3D information from 2D images; performing mesh surface reconstruction by estimating the optimal receiving input point cloud surface and performing surface reconstruction to obtain a 3D model of the croaker's mesh data; refining the mesh; processing the texture; and coloring the mesh. The mesh coloring process involves mapping the texture information obtained in the previous step onto the fish's mesh data using an algorithm, so that the model viewed using 3D model visualization tools has clear texture and color.

[0015] As a further implementation plan, the construction of the dataset includes: extracting keyframe images from multi-view videos of the target fish group using video processing tools, reconstructing a 3D model of the target fish group, matching the keyframe images with the 3D model, adding labels, and then combining them to construct the dataset.

[0016] As a further implementation scheme, the hierarchical abstraction module based on multi-scale learning improvement includes an upper-level hierarchical abstraction module, a lower-level hierarchical abstraction module, and a first feature fusion layer, specifically:

[0017] The upper-level hierarchical abstraction module consists of a multi-scale grouping module, a densely connected multilayer perceptron structure, a max pooling layer, and a second feature fusion layer. The lower-level hierarchical abstraction module consists of a multi-scale grouping module, a multi-scale learning module, and a third feature fusion layer.

[0018] The upper-level hierarchical abstraction module processes point cloud data in the 3D fish model through a multi-scale grouping module, acquiring multiple sets of local regions in the point cloud domain and the coordinates of points in the local coordinate system relative to the centroid within these regions. After structural feature extraction using a densely connected multilayer perceptron, the multi-scale enhanced geometric features of the local point clouds in each group are processed by max pooling block downsampling and then input into the second feature fusion layer to process the downsampling results of each group, resulting in point cloud geometric fusion features. The lower-level hierarchical abstraction module receives the point cloud geometric fusion features and keyframe images and groups them based on the multi-scale grouping module. Within each group, the geometric fusion features of the point cloud are matched with the color information in the keyframe images, and then input into the multi-scale learning module for multi-scale modeling. Finally, the processing results of each group of multi-scale learning modules are input into the second feature fusion layer to obtain the global fusion features of the point cloud. The point cloud geometric fusion features and the global fusion features of the point cloud are processed by the third feature fusion layer to obtain the global fusion features of the target fish.

[0019] As a further implementation scheme, the multi-scale grouping module includes a sampling layer, a grouping layer, and a point cloud layer. The sampling layer adopts iterative farthest point sampling, and the grouping layer adopts a radius sphere query strategy to search for the neighborhood of points within different radii to obtain multiple sets of local point cloud regions. After sampling and grouping processing, the 3D model obtains multiple sets of local point cloud regions. The coordinates of points within the region are converted into local coordinate system coordinates relative to the centroid point in the point cloud layer.

[0020] As a further implementation plan, the multi-scale learning module is divided into multi-scale receptive field and feature fusion, specifically:

[0021] In the multi-scale receptive field, the input features are processed in parallel through multiple branches. Each branch consists of three 1×1 convolutional layers, which extract the low-level, mid-level and high-level semantic information of the features respectively.

[0022] In feature fusion, the outputs of each branch of the multi-scale receptive field are used with channel splicing and further fused with 1×1 convolution to generate a unified enhanced fusion feature;

[0023] As a further implementation scheme, the densely connected multilayer perceptron structure includes six feature extraction layers, a fourth feature fusion layer, a convolutional layer, and a permutation layer, wherein the feature extraction layer consists of a convolutional layer and BN and ReLU layers;

[0024] The first, second, third, fourth, and fifth feature extraction layers are connected sequentially. After the fifth feature extraction layer, a convolutional layer and a permutation layer are connected to output the bottom permutation feature. The bottom permutation feature and the results of the second and fourth feature extraction layers are input into the fourth feature fusion layer for processing to obtain the preliminary fusion feature. The preliminary fusion feature is input into the sixth feature extraction layer to output the multi-scale fusion feature.

[0025] As a further implementation scheme, the feature propagation module based on the improved three-branch attention mechanism adopts a hierarchical propagation strategy of distance-based interpolation and cross-level skip links. Then, the three-branch attention mechanism network is used to perform average weight allocation on the features of the interpolated and fused point cloud, which enhances the expression effect of the features from both channel and spatial dimensions. The feature connection adopts a unit point network, and the feature vector of the points is updated by a shared fully connected layer and a ReLU activation function layer. The process is repeated, and finally the score of each point in the category is output to obtain the fish phenotypic segmentation result.

[0026] As a further implementation scheme, the three-branch attention mechanism network includes three branches: channel attention branch, first space attention branch, and second space attention branch, which perform attention modeling in different dimensions respectively;

[0027] The channel attention branch has the same structure as the second spatial attention branch, consisting of Z-pooling layer, convolutional layer, batch normalization layer, sigmoid function and multiplication layer. The structure of the first spatial attention branch is obtained by adding a permutation layer to the beginning and end of the structure of the second spatial attention branch.

[0028] The channel attention branch feeds the input features into the Z-pooling layer, pools them along the channel dimension to generate feature representations that contain cross-channel information, and then processes them through convolutional layers, batch normalization layers and activation functions to generate attention weights. The attention weights are multiplied point by point with the original input features to generate enhanced channel features.

[0029] The second spatial attention branch inputs the input features into the Z-pooling layer, generating feature representations across batch and channel information along the dimension of point number. Then, it generates attention weights through convolutional layers, batch normalization layers, and activation functions. The weights are multiplied point by point with the input features to generate the enhanced first spatial features.

[0030] The first spatial attention branch first performs dimensional permutation on the input features through a permutation layer, then inputs them into a Z-pooling layer to generate feature representations along the batch dimension, including the number of cross points and channel information. Subsequently, attention weights are generated through convolutional layers, batch normalization layers, and activation functions. The weights are then multiplied point-by-point with the input features to generate the enhanced second spatial features.

[0031] As a further implementation method, the approach also includes:

[0032] The lengths of the real reference object and the reference object in the 3D model are obtained respectively, and the proportional relationship between the real reference object and the reference object in the 3D model is obtained. According to the phenotypic shape of different parts of the fish, the phenotypic data of different parts of the fish are obtained, including but not limited to: volume and surface area.

[0033] The bottom point cloud of the fish is generated by planar coordinate projection transformation. The coefficients of the bottom equation are obtained according to the RANSAC algorithm. The coefficients are used to rotate the fish in three-dimensional coordinate space so that the projection of the fish in the XOY plane conforms to the bottom structure of the fish. The point cloud is stitched together using a density self-registration algorithm to complete the bottom of the fish.

[0034] According to another aspect of this specification, a fish morphological phenotypic acquisition system based on a multi-module cascaded 3D point cloud segmentation network is provided, comprising:

[0035] The keyframe image extraction module extracts keyframe images from the multi-view video of the target fish.

[0036] The 3D model building module uses keyframe images of the target fish to build a 3D model of the target fish based on a multi-view stereo vision algorithm.

[0037] The part segmentation module takes a 3D model of the target fish as input and outputs the phenotypic segmentation results of the target fish.

[0038] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to execute a method for acquiring fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention improves the model's ability to express geometric features by introducing a densely connected MLP on the basis of the original PointNet++, designs a hierarchical abstraction module based on multi-scale learning improvement, and enhances the model's extraction of additional information through collaborative processing while ensuring semantic consistency with geometric information. It also designs a feature propagation module based on a three-branch attention mechanism to further improve the feature expression ability. These three aspects enhance the network's feature extraction ability and achieve accurate segmentation of fish phenotypes. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram illustrating an example of 3D reconstruction effects in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a densely connected multilayer sensor in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the three-branch attention mechanism network in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram comparing the segmentation accuracy of different traits of croakers in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram comparing the segmentation results of the original model and the improved model in an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram illustrating component segmentation and bottom filling in an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of the fish morphological phenotype acquisition system based on a multi-module cascaded 3D point cloud segmentation network provided in an embodiment of the present invention.

[0049] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0050] It should be noted that:

[0051] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0054] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network in an embodiment of the present invention. The method includes:

[0055] Step 1: Extract keyframe images from the multi-view video of the target fish using video processing tools;

[0056] Step 2: Reconstruct the 3D model of the target fish using keyframe images based on a multi-view stereo vision algorithm;

[0057] Step 3: Input the 3D model of the target fish and keyframe images into the trained fish morphology phenotype segmentation network to obtain the phenotype segmentation results of the target fish.

[0058] The training of the fish morphological phenotype segmentation network specifically involves:

[0059] Construct a dataset of 3D point cloud data samples of the target fish species;

[0060] An improved PointNet++ network is used to obtain a fish morphological phenotype segmentation network, specifically the DPS-PointNet++ network, which includes: a hierarchical abstraction module based on multi-scale learning and a feature propagation module based on a three-branch attention mechanism. The hierarchical abstraction module comprises upper and lower layers. The upper layer takes a 3D model as input and introduces a densely connected multilayer perceptron structure to extract enhanced point cloud features. The lower layer processes point cloud fusion features and keyframe images, and extracts multi-scale fusion features based on the multi-scale learning module. The feature propagation module based on the three-branch attention mechanism outputs the segmentation results of the fish phenotype based on the results of the hierarchical abstraction module.

[0061] The network was trained using the dataset, and the fish morphological phenotypic segmentation network was preserved after training.

[0062] Furthermore, the reconstruction of the 3D model of the target fish includes sparse reconstruction and dense reconstruction;

[0063] The sparse reconstruction process includes: first, obtaining the file paths of the multi-view image set; then, using the corresponding point search method, performing feature extraction and feature matching; followed by matching optimization based on geometric conditions and establishing a scene graph; then, entering the incremental SfM algorithm for reconstruction, first initializing the algorithm, then using triangulation and pose estimation to obtain the overall shape of the fish, and finally using bundle adjustment to optimize and obtain sparse point clouds.

[0064] The dense reconstruction process includes: obtaining the Colmap interface and extracting sparse point clouds; obtaining the most complete point cloud possible through dense reconstruction, finding points with consistent light intensity in space under known camera parameters of the image, and then performing scene stereo matching by finding matching corresponding points from images from different perspectives to recover 3D information from 2D images; performing mesh surface reconstruction by estimating the optimal receiving input point cloud surface and performing surface reconstruction to obtain a 3D model of the croaker's mesh data; refining the mesh; processing the texture; and coloring the mesh. The mesh coloring process involves mapping the texture information obtained in the previous step onto the fish's mesh data using an algorithm, so that the model viewed using 3D model visualization tools has clear texture and color.

[0065] Furthermore, the construction of the dataset includes: extracting keyframe images from multi-view videos of the target fish group using video processing tools, reconstructing a 3D model of the target fish group, matching the keyframe images with the 3D model, adding labels, and then combining them to construct the dataset.

[0066] Specifically, the dataset is labeled based on the classification of point clouds. The labels are divided into 7 categories: head, body, tail, pectoral fin, pelvic fin, dorsal fin, and adipose fin. After that, data augmentation strategies are used to enrich the data volume, that is, taking advantage of the spatial rotation and translation characteristics of point clouds, the point clouds are randomly translated and rotated, and point cloud downsampling is performed.

[0067] Furthermore, the hierarchical abstraction module improved based on multi-scale learning includes an upper-level hierarchical abstraction module, a lower-level hierarchical abstraction module, and a first feature fusion layer, specifically:

[0068] The upper-level hierarchical abstraction module consists of a multi-scale grouping module, a densely connected multilayer perceptron structure, a max pooling layer, and a second feature fusion layer. The lower-level hierarchical abstraction module consists of a multi-scale grouping module, a multi-scale learning module, and a third feature fusion layer.

[0069] The upper-level hierarchical abstraction module processes point cloud data in the 3D fish model through a multi-scale grouping module, acquiring multiple sets of local regions in the point cloud domain and the coordinates of points in the local coordinate system relative to the centroid within these regions. After structural feature extraction using a densely connected multilayer perceptron, the multi-scale enhanced geometric features of the local point clouds in each group are processed by max pooling block downsampling and then input into the second feature fusion layer to process the downsampling results of each group, resulting in point cloud geometric fusion features. The lower-level hierarchical abstraction module receives the point cloud geometric fusion features and keyframe images and groups them based on the multi-scale grouping module. Within each group, the geometric fusion features of the point cloud are matched with the color information in the keyframe images, and then input into the multi-scale learning module for multi-scale modeling. Finally, the processing results of each group of multi-scale learning modules are input into the second feature fusion layer to obtain the global fusion features of the point cloud. The point cloud geometric fusion features and the global fusion features of the point cloud are processed by the third feature fusion layer to obtain the global fusion features of the target fish.

[0070] Furthermore, the multi-scale grouping module includes a sampling layer, a grouping layer, and a point cloud layer. The sampling layer adopts iterative farthest point sampling, and the grouping layer adopts a radius sphere query strategy to search for the neighborhood of points within different radii, thereby obtaining multiple sets of local regions of point clouds. After sampling and grouping processing, the 3D model obtains multiple sets of local regions of point cloud neighborhoods. The coordinates of points within the region are converted into local coordinate system coordinates relative to the centroid point in the point cloud layer.

[0071] Furthermore, the multi-scale learning module is divided into multi-scale receptive field and feature fusion, specifically:

[0072] In the multi-scale receptive field, the input features are processed in parallel through multiple branches. Each branch consists of three 1×1 convolutional layers, which extract the low-level, mid-level and high-level semantic information of the features respectively.

[0073] In feature fusion, the outputs of each branch of the multi-scale receptive field are used to perform channel splicing and then further fused using 1×1 convolution to generate a unified enhanced fusion feature.

[0074] Furthermore, the densely connected multilayer perceptron structure includes six feature extraction layers, a fourth feature fusion layer, a convolutional layer, and a permutation layer, wherein the feature extraction layer consists of a convolutional layer and BN and ReLU layers;

[0075] The first, second, third, fourth, and fifth feature extraction layers are connected sequentially. After the fifth feature extraction layer, a convolutional layer and a permutation layer are connected to output the bottom permutation feature. The bottom permutation feature and the results of the second and fourth feature extraction layers are input into the fourth feature fusion layer for processing to obtain the preliminary fusion feature. The preliminary fusion feature is input into the sixth feature extraction layer to output the multi-scale fusion feature.

[0076] Furthermore, the feature propagation module based on the improved three-branch attention mechanism adopts a hierarchical propagation strategy of distance-based interpolation and cross-level skip links. Then, the three-branch attention mechanism network is used to perform average weight allocation on the features of the interpolated and fused point cloud, which enhances the expression effect of the features from both channel and spatial dimensions. The feature connection adopts a unit point network, and the feature vector of the point is updated by a shared fully connected layer and a ReLU activation function layer. The process is repeated, and finally the score of the category of each point is output to obtain the fish phenotypic segmentation result.

[0077] Furthermore, the three-branch attention mechanism network comprises three branches: channel attention branch, first space attention branch, and second space attention branch, which perform attention modeling in different dimensions respectively;

[0078] The channel attention branch has the same structure as the second spatial attention branch, consisting of Z-pooling layer, convolutional layer, batch normalization layer, sigmoid function and multiplication layer. The structure of the first spatial attention branch is obtained by adding a permutation layer to the beginning and end of the structure of the second spatial attention branch.

[0079] The channel attention branch feeds the input features into the Z-pooling layer, pools them along the channel dimension to generate feature representations that contain cross-channel information, and then processes them through convolutional layers, batch normalization layers and activation functions to generate attention weights. The attention weights are multiplied point by point with the original input features to generate enhanced channel features.

[0080] The second spatial attention branch inputs the input features into the Z-pooling layer, generating feature representations across batch and channel information along the dimension of point number. Then, it generates attention weights through convolutional layers, batch normalization layers, and activation functions. The weights are multiplied point by point with the input features to generate the enhanced first spatial features.

[0081] The first spatial attention branch first performs dimensional permutation on the input features through a permutation layer, then inputs them into a Z-pooling layer to generate feature representations along the batch dimension, including the number of cross points and channel information. Subsequently, attention weights are generated through convolutional layers, batch normalization layers, and activation functions. The weights are then multiplied point-by-point with the input features to generate the enhanced second spatial features.

[0082] Furthermore, the method also includes:

[0083] The lengths of the real reference object and the reference object in the 3D model are obtained respectively, and the proportional relationship between the real reference object and the reference object in the 3D model is obtained. According to the phenotypic shape of different parts of the fish, the phenotypic data of different parts of the fish are obtained, including but not limited to: volume and surface area.

[0084] The bottom point cloud of the fish is generated by planar coordinate projection transformation. The coefficients of the bottom equation are obtained according to the RANSAC algorithm. The coefficients are used to rotate the fish in three-dimensional coordinate space so that the projection of the fish in the XOY plane conforms to the bottom structure of the fish. The point cloud is stitched together using a density self-registration algorithm to complete the bottom of the fish.

[0085] Specifically, in the 3D reconstruction of the fish, the RANSAC algorithm is used to obtain the equation of the bottom plane in the reconstructed scene, and the 3D point cloud is projected onto the XOY plane. Using the results of PCA, the eigenvector with the smallest eigenvalue is selected for projection. The minimum bounding rectangle is used to calculate the length of the segmented point cloud. The minimum bounding rectangle is a matrix found in the projected point cloud that minimizes the entire region containing the projected points and is aligned with the XOY plane. Its boundaries are aligned with the principal axes of the data, and the measured values ​​can be determined by calculating the extrema of the projected point cloud.

[0086] In the 3D reconstruction technology, a 360-degree rotation was used to obtain a 360-degree view of the fish, and SfM and MVS algorithms were used to obtain multi-view RGB point cloud data of the croaker. The bottom point cloud was generated using planar coordinate projection transformation. The coefficients of the bottom equation were obtained according to the RANSAC algorithm, and the 3D coordinate space was rotated using these coefficients to make the projection in the XOY plane conform to the bottom structure of the target fish. A density self-registration algorithm was used to stitch the point cloud together, further improving the integrity of the point cloud. The fused model introduced boundary redundant point clouds (mainly distributed in the fins, abdomen, and tail). A morphological filtering process was established in CloudCompare software to effectively remove the redundant parts. This step can reduce the error in the calculation of the fish's surface area and volume. This method is used for the relatively flat abdomen of the croaker. When there is a height difference between the tail and the body, separate fusion and stitching are required.

[0087] In a preferred implementation, step 1 uses ffmpeg (video processing software) to obtain images of each fish from different angles. A parameter setting of capturing one image per second is used to obtain an image dataset. Taking a 100-second video as an example, a dataset of 100 images is ultimately obtained. This image dataset contains the camera's intrinsic and extrinsic parameters for each image.

[0088] As a preferred implementation method, the point cloud data of the 3D model in step 2 is often noisy due to the influence of instruments, environment, and the characteristics of the scanning target, such as floating points and isolated points. Manually removing noise is labor-intensive, so filtering algorithms are often used in the experiment to remove noisy points while preserving the structure and geometric characteristics of the original data as much as possible, thereby improving the point cloud quality and facilitating subsequent feature extraction and analysis.

[0089] Statistical analysis-based methods rely on the local statistical characteristics of point clouds for denoising. The main algorithms are statistical filtering and radius filtering. Statistical Outlier Removal (SOR) removes outliers by analyzing the neighborhood distances of each point in the point cloud. It then determines whether a point is an outlier based on the average distance from each point to its neighborhood points. The main process is as follows: First, find the k nearest neighbors for each point; second, calculate the average distance from each point to its k nearest neighbors; third, calculate the global distribution of the average distances of all points and remove points that deviate from the mean by more than a certain standard range. The filtered point cloud is then output. Its advantages are significant removal of isolated and outlier points and easy parameter adjustment. Its disadvantages are poor filtering performance in point clouds with uneven density and high computational cost. Radius Outlier Removal (ROR) sets a radius range and counts the number of neighboring points within that range for each point. If the number of neighboring points is less than a set threshold, the point is considered an outlier (LI et al 2018). The steps are as follows: Set the search radius r and the neighbor point count threshold n. For each point, count the number of neighbor points within the radius r, remove points with fewer than n neighbor points, and finally output the filtered point cloud. The advantages of the ROR algorithm are its simplicity and effectiveness, and its good effect on removing isolated noise points. The disadvantages are its limited filtering effect on point clouds with uneven density, and the selection of radius and threshold parameters depends on the data distribution.

[0090] Signal processing-based methods can reduce noise while preserving geometric characteristics through techniques such as smoothing and filtering. The main algorithms include Gaussian smoothing, bilateral filtering, and median filtering.

[0091] Gaussian smoothing works by using a Gaussian kernel to perform weighted smoothing on a point cloud. It updates the point's position using the weighted average of neighborhood points, thus smoothing the geometry by reducing noise in the point cloud plane. The main steps are: first, matching each spatial point to its corresponding neighborhood points; second, calculating the weighted average of the neighborhood points using Gaussian weights, and then replacing the current point's position with this weighted average; finally, outputting the smoothed point cloud. Its advantage is its ability to effectively smooth curved point clouds, but it can lead to the loss of geometric details and blurring at edges.

[0092] Bilateral filtering combines spatial distance weights and point attribute difference weights, enabling smoothing while preserving geometric details and avoiding the edge blurring problem of Gaussian smoothing. Its disadvantage is its high computational complexity.

[0093] Median filtering is based on the statistical concept of the median, using the median value of points in the neighborhood to replace the coordinates of the current point. It is mainly used to reduce local noise. Its advantages are that it has a good suppression effect on local noise and small-scale outliers, but it can lead to blurred point cloud edges and is not suitable for processing large-scale isolated noise.

[0094] Noise removal based on robustness algorithms: This algorithm can find points (called inliers) that conform to the geometric model in data containing a large number of outliers, and points that do not conform to the geometric model requirements are called outliers. Robust algorithms can handle large amounts of noise in models of varying complexity. RANSAC is a robust algorithm suitable for handling non-uniformly distributed point cloud data. RANSAC can randomly sample subsets of data and fit a data model that conforms to maximum consistency, regardless of the uniformity of the data distribution. Its basic steps are as follows:

[0095] (1) Initialize parameters and set consistency threshold, set number of iterations. , The maximum number of random samples determines the probability of finding the optimal model. The calculation is usually based on the proportion of outliers. and success probability (Typically set to 0.99), set the consistency threshold. , It is the distance threshold for determining whether a point belongs to the model.

[0096] (2) Random sampling and model fitting. Randomly sample data... The number of points is used as the number of sample points in the current iteration. This is the minimum number of points required to fit the current model (2 points for linear fitting and 3 points for planar fitting). Model fitting involves calculating model parameters using sampling points, and generally includes linear models, planar models, etc.

[0097] (3) Consistency verification: Calculate the distance between each point and the model, such as the distance from a point to a plane, then determine whether the point is an interior point, and then count the number of interior points as a consistency measure of the model.

[0098] (4) Iterative optimization: Repeated random sampling is used to fit the model and verify consistency. In each iteration, the model with the most interior points and its parameters are recorded. The maximum number of iterations is reached when the set limit is reached. Or The iteration ends when the number of interior points of a model reaches the expected proportion of the total number of points during the iteration process.

[0099] (5) Output the optimal model, determine the final model, select the model with the most interior points as the final fitting result, and output the model parameters and the set of interior points.

[0100] As a preferred implementation, in step 2, when obtaining the 3D model of the fish, it is necessary to segment the background and foreground for reconstruction. The foreground includes the fish point cloud, while the background mostly consists of the rotated platform surface and reference objects. The reference objects are then processed separately, with specific steps including:

[0101] (1) Using texture mapping, the mesh data obtained in the above reconstruction is mapped to the texture image into a mesh data containing RGB information. The data file saved in the file is processed using Python programming and converted into PCD format (containing information such as x, y, z, r, g, b).

[0102] (2) The plane equation is obtained by using the RANSAC algorithm (random sample consensus algorithm), and then the foreground and background are segmented using the plane equation. The foreground and background are saved as pcd files respectively, and the plane equation information is recorded.

[0103] (3) Since the RGB information of the three-dimensional coordinate points obtained in step (1) is used, the HSV space color segmentation algorithm is used to process the foreground and background point clouds. By setting appropriate H, S, V parameter thresholds, the background point cloud is removed. In step (1), we obtain the RGB information of each point and convert the RGB to obtain H, S, V information, namely hue value, saturation value, and hue value. Only points that meet the set threshold can be retained, and all other points that do not meet the threshold are removed.

[0104] (4) Statistical filtering is used to remove the remaining noise. In step (3), the point clouds of objects in the foreground and the point clouds of objects in the background need to be merged to obtain the complete three-dimensional point cloud data of the fish. Statistical filtering algorithm is used to remove background noise points that are close to the fish.

[0105] (5) Using the Euclidean clustering segmentation algorithm, two point clouds that are far apart in three-dimensional space can be separated, thereby separating the fish from the reference point and obtaining the final three-dimensional point cloud of the fish.

[0106] (6) The background plane point cloud after removal contains a 3D model of the reference object. The reference object is extracted using a color segmentation method. Its height and diameter are measured using a tool. The actual height and diameter of the reference object are measured using a vernier caliper to obtain the ratio between the actual length and the length in the 3D model after 3D reconstruction.

[0107] An example of fish morphological phenotype acquisition based on a multi-module cascaded 3D point cloud segmentation network, including: model training and comparative experiments.

[0108] The training process includes:

[0109] Step 1 uses croaker as the research object. After ensuring that the influence of the shooting environment on outdoor light is negligible, video data of about 300 fish are obtained using a camera and a rotating circular platform. Each video is about 100 seconds long, obtained by rotating the platform 360 degrees. Keyframes are extracted at 1-second intervals using a video extraction tool, and keyframe images with clear and unobstructed fish bodies are selected.

[0110] Step 2: After selecting suitable keyframe images, the sparse reconstruction process in Colmap begins. This process uses an incremental SfM (Structure of Motion) algorithm to obtain the sparse point cloud of the fish and the camera pose information. The dense point cloud of the fish is then obtained using the MVS (Multi-view Stereo Vision) algorithm.

[0111] First, obtain the Colmap interface and extract sparse point cloud and camera parameters, etc.

[0112] Dense reconstruction obtains the most complete point cloud possible. Under the known camera parameters of the image, points with photometric consistency in space are found. Then, scene stereo matching is performed by finding matching corresponding points from images from different perspectives and recovering three-dimensional information from two-dimensional images.

[0113] The mesh surface is reconstructed by estimating the optimal interpretation of the input point cloud surface and then performing surface reconstruction to obtain a 3D model of the fish mesh data class.

[0114] The mesh is refined because the model in the previous step had a relatively coarse mesh on the surface, which still differed from the real 3D structure. The mesh refinement process was then used to obtain a model that was consistent with the real fish in the shooting scene in terms of details.

[0115] To calculate clear and accurate textures and obtain color information from the fish's surface, texture processing is required. Finally, the mesh is colored by mapping the color onto the fish's mesh data using an algorithm, giving the model clear texture and color, thus achieving an end-to-end reconstruction process.

[0116] Step 3: Noise Removal Process for Fish. Upon obtaining the 3D model of the fish, it is necessary to segment the background and foreground for reconstruction. The foreground includes the fish point cloud, while the background mostly consists of the rotated platform surface and reference objects. The reference objects are then processed separately, such as... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating an example of three-dimensional reconstruction effects in an embodiment of the present invention.

[0117] Step 4: Dataset Construction.

[0118] The dataset is divided into a training set and a validation set, based on the allocation principles of datasets for each model in deep learning algorithms.

[0119] Before labeling, point cloud downsampling was performed on the datasets, followed by point cloud classification labeling on both datasets. The labels were divided into seven categories: head, body, tail, pectoral fins, pelvic fins, dorsal fins, and adipose fins. Afterward, data augmentation strategies were employed to enrich the data volume, utilizing the spatial rotation and translation properties of point clouds to randomly translate and rotate the point clouds, followed by point cloud downsampling. The datasets were then partitioned, following the allocation principles for model datasets in deep learning algorithms, into a training set and a training / validation set.

[0120] After classifying and labeling the fish point cloud using the annotation tool CloudCompare, the data format was checked to ensure that the point data could be obtained through this file format during training. The DPS-PointNet++ model was then trained, and the weight file (best.pth) was obtained after training.

[0121] Step 5: Improve the PointNet++ network to obtain a fish morphological phenotype segmentation network.

[0122] The PointNet++ model suffers from issues such as unclear boundary segmentation between parts of the croaker point cloud and poor classification of individual object components. This is mainly due to the similarity in structure between adjacent boundaries and the close spatial positions of boundary points. This study employs a densely connected MLP module for feature reuse and enhancement during network training. By improving the model's ability to extract features from additional information (RGB), higher accuracy point cloud segmentation is achieved. To enable the model to learn the spatial structure of the croaker point cloud, an improved three-branch attention mechanism module is used for the point cloud segmentation task.

[0123] The overall network structure includes an input module, an SA module, a feature propagation (FP) module, and classification and segmentation heads. The input module receives point cloud data of type [B, C, N], where B is the batch size, C is the number of channels (e.g., XYZ or XYZ+RGB), and N is the number of points.

[0124] Firstly, the original MLP structure is improved by using a dense connection method, such as... Figure 3 As shown, the input is [B, Cin, K, S], where B is the batch size, Cin is the number of input channels, K represents the neighborhood size of each point, and S is the number of sampling points. In dense connections, the output of each layer is not only passed to the next layer but also connected to the input of all subsequent layers. The DenseNet-MLP architecture consists of six feature extraction layers, feature fusion layers, convolutional layers, and permutation layers. The feature extraction layers consist of 1... 1 convolutional layer (1 It consists of 1Cnov) and BN and ReLU (batch normalization and ReLU activation function) layers;

[0125] The first, second, third, fourth, and fifth feature extraction layers are connected sequentially. After the fifth feature extraction layer, a convolutional layer and a permutation layer are connected to output the bottom permutation feature. The bottom permutation feature and the results of the second and fourth feature extraction layers are input into the feature fusion layer for processing to obtain the preliminary fused feature. The preliminary fused feature is input into the sixth feature extraction layer to output the multi-scale fused feature.

[0126] Compared to the feature extraction structure based on MLP (Feature-Aware Layer) in PointNet++, this structure has the following two characteristics:

[0127] (1) Feature reuse: Dense connections allow features from each layer to be directly passed to subsequent layers, thus enabling feature reuse. Each layer does not need to relearn all basic features, and subsequent layers can perform higher-level feature extraction based on existing features, avoiding redundant information extraction.

[0128] (2) Feature enhancement: By connecting the features of the previous layers, each layer can obtain information from the previous layers, forming a multi-scale, multi-level feature representation. This layer-by-layer accumulation of features helps to enrich the representation of point clouds, making the model more discriminative, especially in the extraction of local geometric information.

[0129] Secondly, a hierarchical abstraction module based on multi-scale learning is introduced. The original PointNet++ network directly concatenates additional information with geometric information and passes it all as input to the feature extraction module. This simple concatenation method ignores the uniqueness of the additional information, and the original network lacks a dedicated processing module for additional information; all information is processed in the same channel, limiting the network's ability to extract features from additional information. Although the original PointNet++ feature extraction module introduced multi-scale grouping (MSG), it lacks detailed local information modeling capabilities when processing multimodal features and is insufficient for handling complex distributions of additional information in point clouds. Therefore, a multi-scale learning module (Multi-Scale Bottleneck, MSB) is introduced to improve the feature extraction capability for local information.

[0130] The MSB module is designed from two perspectives: multi-scale receptive field and feature fusion.

[0131] In the multi-scale receptive field, the MSB module processes the input features in parallel through multiple branches. Each branch consists of three 1×1 convolutional layers, which extract low-level, mid-level, and high-level semantic information of the features respectively. The multi-branch architecture can capture local and global features simultaneously, thereby better extracting color information from the point cloud.

[0132] In terms of feature fusion, the outputs of each branch are used to perform channel concatenation and then further fused using 1×1 convolution to generate unified enhanced features. This mechanism preserves multi-scale information while compressing redundant features.

[0133] The upper-level hierarchical abstraction module (Set Abstraction, SA) processes geometric information, primarily encoding it. It extracts multi-scale geometric features from the point cloud using the MSG method and incorporates the DenseNet-MLP module, enhancing feature extraction capabilities through close connections. The upper-level SA module outputs sampled point cloud geometric information to guide the lower-level SA module's neighborhood search and point sampling for RGB features, ensuring spatial consistency of the RGB features.

[0134] The lower-level SA module employs an improved design, primarily focusing on feature modeling for RGB information. This module uses an MSB structure to model both local and global features of the RGB information. In local modeling, multi-branch convolutions capture local features from different receptive fields, enabling the network to adapt to the complex RGB information distribution in the point cloud. In global modeling, adaptive pooling extracts global RGB patterns, further enhancing RGB feature representation. Local and global features are fused through channel concatenation, effectively improving the diversity and robustness of RGB features. Finally, the local and global features of the RGB information are fused to generate enhanced RGB features for subsequent feature propagation.

[0135] Thirdly, regarding the processing of RGB information. First, RGB information is optimized. The improved lower-layer SA module features a specially designed MSB structure, significantly enhancing the modeling ability of RGB features. By introducing local and global modeling, the network can better adapt to the complex variations of RGB information. Second, geometric feature extraction capabilities are enhanced. The upper-layer SA module uses MSG and DenseNet-MLP modules to perform multi-scale modeling and dense enhancement of geometric features, capturing richer geometric patterns. Finally, collaborative modeling of multi-module information: the collaborative design of the upper and lower-layer SA modules ensures the spatial consistency of geometric and RGB information. The lower-layer module enhances the complementarity of RGB and geometric information through multi-feature fusion, providing a more comprehensive feature representation for subsequent segmentation tasks. Finally, stronger local and global expressive capabilities: the combination of MSB and DenseNet-MLP enables joint modeling of multi-scale and global features, resulting in stronger generalization. In the first SA module, feature extraction is not performed on the additional information. Instead, in the second set abstraction layer, the additional information (RGB) of the points obtained by iterative sampling of the farthest point is acquired. Then, feature downsampling and local feature extraction are performed on the point coordinates and additional information respectively, followed by global feature fusion. The SA module achieves hierarchical processing of the point cloud through layer-by-layer sampling and feature extraction, and uses the extracted geometric information for RGB feature downsampling, avoiding the computational overhead caused by repeated sampling.

[0136] The FP module is responsible for feature upsampling, which propagates high-level abstract features back to the initial point cloud to support point prediction.

[0137] Firstly, an improved feature propagation module based on a three-branch attention mechanism. To enable more adaptive selection and enhancement of point cloud features during feature propagation, allowing the model to better focus on key local information, an improved three-branch attention mechanism suitable for point features, called Point Triplet Attention (PTA), is added before MLP. This mechanism helps to more accurately capture the spatial distribution and local structural information of points, improving the model's part classification ability.

[0138] The purpose of the PTA module is to enhance the expressive power of features by capturing dependencies across different dimensions (channel and spatial dimensions) to improve the model's local feature aggregation performance. For example... Figure 4 As shown, the channel attention branch has the same structure as the second spatial attention branch, including a Z-pooling layer and a 1 1 convolutional layer (1 The module consists of a Cnov layer, a batch normalization layer (BN), a sigmoid function, and a multiplication layer. The structure of the first spatial attention branch is obtained by adding a permutation layer to the beginning and end of the structure of the second spatial attention branch. Here, B is the batch size, C is the number of channels, and N is the number of points. The module contains three branches: Channel Attention and two spatial attention branches (Spatial Attention 1 and Spatial Attention 2), which model attention in different dimensions.

[0139] Channel attention branch: In both B and N dimensions, the input features are compressed along the channel dimensions to generate a cross-channel global representation. The main process involves first using... (Z-pooling block) Simultaneously performs MaxPooling and AvgPooling on the channels of the input features, and concatenates the results to generate a feature representation containing cross-channel information with a shape of [B, 2, N]. Next, the compressed features are processed through 1×1 convolution and batch normalization (BN). Then, a sigmoid function is used as the activation function for the attention weights, resulting in a tensor of [B, 1, N]. Finally, the attention weights are multiplied point-by-point with the original input features to generate the enhanced channel features.

[0140] The first spatial attention branch, in the two dimensions B and C, first transforms the input features from [B,C,N] to [B, N, C] through dimensional permutation in order to capture the spatial dimensional dependencies. Then, one-dimensional convolution, batch normalization, and activation functions generate attention weights. After multiplying the weights with the input features point by point, the original shape is restored through dimensional permutation, and the enhanced first spatial features are output.

[0141] The second spatial attention branch directly models the spatial attention of the input features in both dimensions B and C, using... MaxPooling and AvgPooling are applied to the number of points, and the results are concatenated to generate feature representations across height and channels. One-dimensional convolution, batch normalization, and activation functions are used to generate attention weights. The weights are multiplied point by point with the input features to output the enhanced second spatial features.

[0142] In feature propagation, the multi-channel attention mechanism of PTA can enhance different dimensions of the feature map. Especially during upsampling, it can better preserve and highlight key local features while recovering the features of the initial point, thereby improving the classification accuracy in point cloud segmentation tasks. After PTA processing, the new features are further extracted by MLP. The features processed by the PTA mechanism are richer and more information-dense, making it easier for MLP to extract effective local and global features, thus improving the model's performance.

[0143] The advantages of the DPS-PointNet++ framework improvements presented in this paper are as follows: First, information separation processing separates geometric and color information, allowing the network to optimize independently for these two different features, thereby reducing information interference. Second, feature fusion: after the multi-layer SA module, the extracted XYZ and RGB features are fused by channel concatenation, and the fused features are used as input to the next layer, enhancing the semantic consistency of point cloud feature representation. Third, improved geometric feature extraction: the geometric feature extraction in the SA module adopts a densely connected MLP module (DenseNet-MLP), which preserves the features of all intermediate layers through dense connections, significantly improving the expressive power of the features.

[0144] Fourth, RGB feature extraction is optimized. During RGB feature extraction, the feature point information set obtained through the Farthest Point Sampling (FPS) algorithm during geometric information extraction is directly reused, maintaining consistency between XYZ and RGB features. Furthermore, a multi-scale learning module is introduced into RGB feature extraction, capturing local and global features of the point cloud through multi-scale receptive fields, further improving feature expressiveness. Finally, regarding improvements to feature propagation, an improved three-branch attention mechanism is introduced into the FP module, applying average weights to the features of the interpolated and fused point cloud, enhancing feature expressiveness from both channel and spatial dimensions.

[0145] The second module is the feature propagation of points, i.e., the decoding stage. The input is the number of channels and multiple multilayer perceptron structures. It adopts a hierarchical propagation strategy with distance-based interpolation and cross-level skip connections. Then, a three-branch attention mechanism is used to distribute average weights on the features of the interpolated and fused point cloud, enhancing the feature representation effect from both channel and spatial dimensions. The connection features are handled by Unit Pointnet, which mainly consists of Conv1d (1×1 convolution), BatchNorm (batch normalization), and Dropout layers. These layers are used to update the feature vectors of points in the shared fully connected layers and ReLU (activation function) layers. The above process is repeated, and the final output is the score of each point's category.

[0146] In the component segmentation experiment, this paper uses mean accuracy (mAcc), overall accuracy (OA), and mean intersection over union (mIoU) as segmentation evaluation metrics. The accuracy of the point cloud segmentation task is calculated by comparing the predicted classification values ​​with the ground truth labels point by point. value, value and The accuracy improvement of the model stabilized after 220 training epochs. With increasing training epochs, the improved PointNet++ model showed significant improvements in evaluation metrics on the test set, among which... More than 95%, It reached 88.3%; while the baseline model... It is 94%. It is 85.5%.

[0147] As shown in Table 1, Table 1 presents the performance evaluation metrics of three different iteratively improved models in the point cloud segmentation task. By comparing the performance of each improved module on the basis of the original network... value, Value and The value is used to evaluate whether the improved module effectively improves segmentation accuracy and its generalization ability for segmentation tasks.

[0148] Table 1 compares the performance of the PointNet++-based point cloud segmentation model after optimization at each stage.

[0149]

[0150] Analysis of the model iteration process shows that the performance metrics of the baseline model segmentation prediction for croaker parts based on the PointNet++ network are as follows: The value is 94.1%. The value is 85.4%. The value was 92.2%. Using this model as a baseline, the Multilayer Perceptron (MLP) in the SA module was first improved. The improved model showed improved performance on the prediction set. The value increased by 0.4%. The value improved by 0.9%, indicating that the optimization of the MLP played a positive role in improving segmentation accuracy. Building on this, an improved three-branch attention mechanism was added, further enhancing the model's segmentation accuracy and robustness. Through this improvement, the model's... and Significant improvements of 2.0% and 1.2% were achieved respectively, demonstrating a stronger ability to segment complex parts compared to the baseline model. Next, further optimizations were made to the feature extraction of geometric and additional information, resulting in the final point cloud segmentation network model. This model achieved a new peak in overall performance. The value increased by 0.7%. An increase of 1.5%. The 0.4% improvement indicates that feature enhancement and information fusion played a crucial role in improving model performance. Finally, the comprehensively optimized model (including multi-module fusion, feature enhancement, and the introduction of attention mechanisms) showed a significant improvement in overall performance compared to the baseline model. , and The improvements were 1.4%, 3.5%, and 1.6%, respectively. The results demonstrate the effectiveness of the proposed improvement scheme in enhancing model segmentation accuracy, improving model generalization ability, and refining component recognition capabilities.

[0151] The results of different experiments show that, through effective improvement and optimization of the modules, the model exhibits higher accuracy in point cloud segmentation tasks, especially in the segmentation of multiple parts, where it demonstrates stronger robustness and better generalization performance.

[0152] In summary, the improved DSP-PointNet++ model not only surpasses the publicly disclosed segmentation models in the table in overall accuracy, but also further enhances its segmentation capabilities in multi-part and complex point cloud scenarios through multi-module optimization strategies and feature enhancement methods, demonstrating the significant effect of the improved scheme in improving segmentation accuracy and robustness. On a relatively small dataset, the improved model can be trained to obtain better weight values ​​for part segmentation inference in croakers.

[0153] Model generalization ability verification

[0154] The improved model still outperforms the original PointNet++ model in generalization ability in prediction. Prediction inference was performed on the validation set, and the evaluation results were recorded. Table 2 shows the component segmentation evaluation results of nine fish specimens with different poses. The experimental data shows that the improved network model has a significant improvement in segmentation ability compared to the baseline, and also indicates that the improved model still achieves excellent segmentation ability in actual prediction.

[0155] Table 2 Evaluation results of actual predictions from the iteratively improved network model

[0156]

[0157] Data shows that the final improved model achieved improvements in all three metrics. However, the network module based on DenseNet-MLP did not show a significant improvement in prediction evaluation compared to PointNet++; therefore, its metric values ​​were not analyzed in the experiment, as this process analysis does not affect the experimental comparison of the final model's segmentation performance. In terms of prediction values, the segmentation model based on the improved DenseNet-MLP with PTA attention mechanism... The indicator improved by 2.5%. The value increased by 0.9%; in the DPS-PointNet++ model, among the 9 fish specimens The maximum increase in value is 7.4%. It increased by 4.6%. It increased by 1.7%.

[0158] To demonstrate that the improved model still demonstrates enhanced segmentation performance even at low point cloud densities. Figure 5 This demonstrates the performance of the network before and after improvements in a 3D point cloud-based segmentation task for 15 shad (weighing 3.5-6.5g) fish parts. Comparison. Experimental data shows significant differences in segmentation performance among different morphological types. This is mainly due to the gradient distribution characteristics of the reconstructed point cloud density in smaller samples, and the point cloud density decreases significantly relative to the main body in regions with drastic curvature changes. The improved network demonstrates improved segmentation accuracy across multiple morphological types. In particular, the head region exhibits a stable geometric topology. The accuracy remained around 87.0%, with only a slight improvement in head segmentation achieved through network improvements. Segmentation accuracy for other components improved, with relatively significant improvements observed in the segmentation of the dorsal and adipose fins. These figures represent increases of 3.3% and 4.7%, respectively.

[0159] like Figure 6As shown, this paper compares the problems of the original model in predictive segmentation with those of the improved network model. The segmentation problems of the original model are mainly manifested in unclear boundary distinctions and oversegmentation, which are not conducive to researchers' research and analysis of the morphological phenotypes of croaker fish.

[0160] exist Figure 6 In (a1), the boundary segmentation of the head region is blurry and the segmentation boundary is messy and irregular; Figure 6 In (a1, c1, e1, f1, i1), there is an issue of over-segmentation of the pectoral or pelvic fins; while... Figure 6 In (d1, g1, h1), there are instances where the dorsal fin and adipose fin are not properly segmented. In contrast, Figure 6 (a2-e2) shows the significant improvement in the improved model's ability to segment different phenotypic boundaries. The improved model demonstrates clearer and neater boundary distinctions, reduces oversegmentation and undersegmentation, and can segment each phenotypic feature relatively completely. This improvement is of great significance for subsequent accurate measurement and analysis, ensuring the accuracy and reliability of the model in detail segmentation.

[0161] like Figure 7 As shown, Figure 7 This is a schematic diagram of component segmentation and bottom padding in an embodiment of the present invention. In the 3D reconstruction of the fish, the RANSAC algorithm is used to obtain the equation of the bottom plane in the reconstructed scene, and the 3D point cloud is projected onto the XOY plane. Using the results of PCA, the eigenvector with the smallest eigenvalue is selected for projection. The minimum bounding rectangle is used to calculate the length of the segmented point cloud. The minimum bounding rectangle is a matrix found in the projected point cloud that minimizes the entire region containing the projected points and is aligned with the XOY plane. Its boundaries are aligned with the principal axes of the data, and the measured values ​​can be determined by calculating the extrema of the projected point cloud.

[0162] In the 3D reconstruction technique, a 360-degree rotation was used to obtain a 360-degree panoramic view of the fish. Multi-view RGB point cloud data of the croaker was obtained using SfM and MVS algorithms, while the bottom point cloud was generated using planar coordinate projection transformation. The coefficients of the bottom surface equation were obtained according to the RANSAC algorithm, and this coefficient was used for 3D coordinate space rotation to ensure that its projection in the XOY plane conforms to the bottom structure of the croaker. A density-based registration algorithm was used for point cloud stitching, further improving the integrity of the point cloud. The fused model introduced redundant point clouds at the boundaries (mainly distributed in the fins, abdomen, and tail). A morphological filtering process was established in CloudCompare software to effectively remove the redundant parts, reducing errors in the calculation of the fish's surface area and volume. This method was used for the relatively flat abdomen of the croaker, while when there is a height difference between the tail and the body, separate fusion stitching was required.

[0163] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a fish morphological phenotypic acquisition system using a multi-module cascaded 3D point cloud segmentation network. This system is used to execute the fish morphological phenotypic acquisition method using a multi-module cascaded 3D point cloud segmentation network described in the above method embodiments.

[0164] See Figure 8 The system includes:

[0165] The keyframe image extraction module extracts keyframe images from the multi-view video of the target fish.

[0166] The 3D model building module uses keyframe images of the target fish to build a 3D model of the target fish based on a multi-view stereo vision algorithm.

[0167] The part segmentation module takes a 3D model of the target fish as input and outputs the phenotypic segmentation results of the target fish.

[0168] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the system in the above system embodiments by referring to the specific technical solutions in other method embodiments, combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, and ensuring the practicality of the technical solutions, they can obtain corresponding system-class embodiments for implementing the methods in other method-class embodiments.

[0169] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0170] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 9 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method for obtaining fish morphological phenotypes based on a 3D point cloud segmentation network of a multi-module cascade, characterized in that, include: Keyframe images of the target fish were extracted from a multi-view video using video processing tools. Based on multi-view stereo vision algorithm, the 3D model of the target fish is reconstructed using keyframe images of the target fish; The 3D model of the target fish and keyframe images are input into the trained fish morphology phenotypic segmentation network to obtain the phenotypic segmentation results of the target fish. The training of the fish morphological phenotype segmentation network specifically involves: Construct training and testing sets for 3D point cloud data samples of the target fish species; An improved PointNet++ network is used to obtain a fish morphological phenotype segmentation network, comprising: a hierarchical abstraction module based on multi-scale learning and a feature propagation module based on a three-branch attention mechanism. The hierarchical abstraction module includes upper and lower layers. The upper layer processes the input 3D fish model to obtain point cloud geometric fusion features, and introduces a densely connected multilayer perceptron structure for extracting enhanced point cloud features. The lower layer processes the point cloud fusion features and keyframe images to output global fusion features of the fish. Within the lower layer, a multi-scale learning module performs multi-scale modeling on the grouped point cloud geometric fusion features and keyframe images to obtain multi-scale fusion features. The feature propagation module based on the three-branch attention mechanism outputs the fish phenotype segmentation results based on the results of the hierarchical abstraction module. The network was trained using the dataset, and the fish morphological phenotypic segmentation network was preserved after training.

2. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 1, characterized in that... The hierarchical abstraction module based on multi-scale learning improvement includes an upper-level hierarchical abstraction module, a lower-level hierarchical abstraction module, and a first feature fusion layer, specifically: The upper-level hierarchical abstraction module consists of a multi-scale grouping module, a densely connected multilayer perceptron structure, a max pooling layer, and a second feature fusion layer. The lower-level hierarchical abstraction module consists of a multi-scale grouping module, a multi-scale learning module, and a third feature fusion layer. The upper-level hierarchical abstraction module processes point cloud data in the three-dimensional fish model through multi-scale grouping modules, obtains local regions of multiple point cloud domains and local coordinate system point coordinates relative to the centroid point within the regions, extracts structural features through densely connected multilayer perceptron, and inputs the multi-scale enhanced geometric features of the local point cloud of each group into the second feature fusion layer after downsampling processing by max pooling block to process the downsampling results of each group, thus obtaining point cloud geometric fusion features. The lower-level hierarchical abstraction module receives point cloud geometric fusion features and keyframe images, and groups them based on multi-scale grouping modules. Within each group, the geometric fusion features of the point cloud are matched with the color information in the keyframe images. Then, the data is input into the multi-scale learning module for multi-scale modeling. Finally, the processing results of each group of multi-scale learning modules are input into the second feature fusion layer to obtain the global fusion features of the point cloud. The point cloud geometric fusion features and the global fusion features of the point cloud are processed by the third feature fusion layer to obtain the global fusion features of the target fish.

3. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 2, characterized in that... The multi-scale grouping module includes a sampling layer, a grouping layer, and a point cloud layer. The sampling layer adopts iterative farthest point sampling, and the grouping layer adopts a radius sphere query strategy to search for the neighborhood of points within different radii to obtain multiple sets of local point cloud regions. After sampling and grouping processing, the 3D model obtains multiple sets of local point cloud regions. The coordinates of points within the region are converted into local coordinate system coordinates relative to the centroid point in the point cloud layer.

4. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 2, characterized in that... The multi-scale learning module is divided into multi-scale receptive field and feature fusion, specifically: In the multi-scale receptive field, the input features are processed in parallel through multiple branches. Each branch consists of three 1×1 convolutional layers, which extract the low-level, mid-level and high-level semantic information of the features respectively. In feature fusion, the outputs of each branch of the multi-scale receptive field are used with channel splicing and further fused with 1×1 convolution to generate a unified enhanced fusion feature.

5. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 2, characterized in that... The densely connected multilayer perceptron structure includes six feature extraction layers, a fourth feature fusion layer, a convolutional layer, and a permutation layer. The feature extraction layer consists of a convolutional layer and BN and ReLU layers. The first, second, third, fourth, and fifth feature extraction layers are connected sequentially. After the fifth feature extraction layer, a convolutional layer and a permutation layer are connected to output the bottom permutation feature. The bottom permutation feature and the results of the second and fourth feature extraction layers are input into the fourth feature fusion layer for processing to obtain the preliminary fusion feature. The preliminary fusion feature is input into the sixth feature extraction layer to output the multi-scale fusion feature.

6. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 1, characterized in that... The feature propagation module based on the three-branch attention mechanism adopts a hierarchical propagation strategy of distance-based interpolation and cross-level skip links. Then, the three-branch attention mechanism network is used to perform average weight allocation on the features of the interpolated and fused point cloud, which enhances the expression effect of the features from both channel and spatial dimensions. The feature connection adopts a unit point network, and the feature vector of the point is updated by a shared fully connected layer and a ReLU activation function layer. The process is repeated, and finally the score of each point in the category is output to obtain the fish phenotypic segmentation result.

7. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 6, characterized in that... The three-branch attention mechanism network comprises three branches: channel attention branch, first spatial attention branch, and second spatial attention branch, which perform attention modeling in different dimensions. The channel attention branch has the same structure as the second spatial attention branch, consisting of Z-pooling layer, convolutional layer, batch normalization layer, sigmoid function and multiplication layer. The structure of the first spatial attention branch is obtained by adding a permutation layer to the beginning and end of the structure of the second spatial attention branch. The channel attention branch feeds the input features into the Z-pooling layer, pools them along the channel dimension to generate feature representations that contain cross-channel information, and then processes them through convolutional layers, batch normalization layers and activation functions to generate attention weights. The attention weights are multiplied point by point with the original input features to generate enhanced channel features. The second spatial attention branch inputs the input features into the Z-pooling layer, generating feature representations across batch and channel information along the dimension of point number. Then, it generates attention weights through convolutional layers, batch normalization layers, and activation functions. The weights are multiplied point by point with the input features to generate the enhanced first spatial features. The first spatial attention branch first performs dimensional permutation on the input features through a permutation layer, then inputs them into a Z-pooling layer to generate feature representations along the batch dimension, including the number of cross points and channel information. Subsequently, attention weights are generated through convolutional layers, batch normalization layers, and activation functions. The weights are then multiplied point-by-point with the input features to generate the enhanced second spatial features.

8. The method for obtaining fish morphological phenotypes based on a multi-module cascaded 3D point cloud segmentation network as described in claim 1, characterized in that, The method further includes: The lengths of the real reference object and the reference object in the 3D model are obtained respectively, and the proportional relationship between the real reference object and the reference object in the 3D model is obtained. According to the phenotypic shape of different parts of the fish, the phenotypic data of different parts of the fish are obtained, including but not limited to: volume and surface area. The bottom point cloud of the fish is generated by planar coordinate projection transformation. The coefficients of the bottom equation are obtained according to the RANSAC algorithm. The coefficients are used to rotate the fish in three-dimensional coordinate space so that the projection of the fish in the XOY plane conforms to the bottom structure of the fish. The point cloud is stitched together using a density self-registration algorithm to complete the bottom of the fish.

9. A fish morphological phenotypic acquisition system based on a multi-module cascaded 3D point cloud segmentation network, used to implement the method described in any one of claims 1 to 8, characterized in that, include: The keyframe image extraction module extracts keyframe images from the multi-view video of the target fish. The 3D model building module uses keyframe images of the target fish to build a 3D model of the target fish based on a multi-view stereo vision algorithm. The part segmentation module takes a 3D model of the target fish as input and outputs the phenotypic segmentation results of the target fish.

10. An electronic device, characterized in that, The method includes a memory and a processor, the memory storing program instructions that are executed by the processor, the processor invoking the program instructions to perform the method according to any one of claims 1 to 8.

Images