Underwater in-situ sea cucumber volume measurement method based on binocular vision and electronic equipment

Abstract

An underwater in-situ sea cucumber volume measurement method based on binocular vision and electronic equipment belong to the field of image processing, are used for solving the problem of keeping in-situ sea cucumber volume measurement, and are technically characterized in that a sea cucumber image is detected in underwater images collected by a binocular camera to obtain a sea cucumber image detection frame; segmenting a sea cucumber image detected in the image according to the sea cucumber image detection frame, and calculating mask information of the sea cucumber on an image plane; mapping the mask information to a global point cloud constructed by binocular information, and obtaining point cloud data of the sea cucumber; performing grid reconstruction on the point cloud data of the sea cucumber to obtain a continuous triangular grid; according to the method, the continuous triangular grids are converted into voxel representation, the number of non-zero elements in the voxel grids and the volume of each element are counted, the volume of the grids is estimated, and the volume of the sea cucumbers is obtained, so that the volume of the in-situ sea cucumbers can be accurately evaluated, and more accurate data support is provided for scientific cultivation of the sea cucumbers.

Description

Technical Field

[0001] The invention belongs to the field of image processing and relates to an underwater in-situ sea cucumber volume measurement method based on binocular vision and electronic equipment. Background Art

[0002] With the general improvement of public health awareness and the continuous improvement of living standards, sea cucumbers, as a marine treasure with high nutritional value, have gained wide attention and importance in the current market. In the process of marine treasure breeding, farmers need all kinds of important information to evaluate the growth status of marine treasures and their adaptability to the breeding environment. Therefore, it is extremely important to conduct phased data observation. However, traditional observation methods have many limitations. Taking sea cucumbers as an example, farmers usually need to send divers to dive underwater to observe and contact sea cucumbers at close range to approximate the size, volume and other information of sea cucumbers. First of all, this method has obvious problems of labor-intensiveness and high time cost, and the measurement accuracy cannot be guaranteed, which is inefficient. Secondly, human intervention will cause a certain degree of interference and damage to sea cucumbers in the natural growth environment. Therefore, in order to improve the efficiency and accuracy of growth status observation during sea cucumber breeding, it is necessary to explore and introduce more advanced and convenient measurement technologies. The continuous development of machine vision has brought new possibilities for agricultural observation and evaluation tasks. Compared with traditional manual measurement methods, machine vision measurement has higher efficiency and accuracy. By using high-resolution cameras and advanced image processing algorithms, non-contact measurement can be achieved without human intervention, avoiding measurement errors and interference that may exist in traditional methods, thereby providing more reliable and timely production data support for agricultural assessment.

[0003] Scholars at home and abroad have explored the use of machine vision to assist agricultural production, and research on two-dimensional images has been fruitful. MIRBOD et al. developed a machine vision system for measuring apple size. They trained two deep neural network models to detect the size of candidate apples, and then extrapolated the occluded fruit area to improve the size estimation measurement, effectively measuring the outer diameter of the apple. However, due to the influence of image pixels and lens distortion, the diameter measurement of fruits near the edge of the image still has great inaccuracy. Dong Peng et al. used the underwater binocular positioning method to eliminate the image distortion caused by underwater light refraction, and on this basis, used the convex hull and rotating caliper algorithm to propose an underwater sea cucumber automatic detection and size measurement method, realizing the measurement of sea cucumber length in an underwater environment. However, this measurement method that relies on feature point selection still remains at the two-dimensional image level. Not only does it have certain measurement errors, but more importantly, it is impossible to measure the volume of sea cucumbers, and the single length information cannot meet the needs of farmers for the evaluation of various growth data of sea cucumbers.

[0004] Although the two-dimensional features of the image can provide some valuable data, the organism is essentially three-dimensional, and phenotypic information such as volume and body shape is difficult to capture in the process of two-dimensional imaging. With the continuous development of computer vision technology, related methods of three-dimensional reconstruction are increasingly used in agriculture. GEN′E-MOLA et al. proposed a point cloud generation technology based on motion structure and multi-view stereo, and used point cloud information to estimate the size of the fruit. They used the least squares method and the estimated sample consistency algorithm to fit the spherical model to the feature points of the apple, and finally applied template matching to fit the three-dimensional model of the apple points to accurately measure the volume information of the spherical fruit. Although this method can cope with the challenge of target object occlusion to a certain extent, further research is still needed in solving the feature points and data collection and processing to improve the efficiency and practicality of the method. LIAO et al. developed a 3DPhenoFish software to extract fish morphological phenotypes from three-dimensional point cloud data, providing information features for distinguishing samples of different species. However, such a method relies on high-precision industrial-grade point cloud scanners. When collecting data, the fish need to be caught and anesthetized for work. Data acquisition is difficult and will cause irreparable damage to the organism. Therefore, a volume measurement method that keeps the sea cucumber in situ is needed to estimate the volume with high accuracy without damaging the sea cucumber. However, there are still certain challenges in measuring the volume of underwater sea cucumbers. First, the underwater light is changeable and the environment is complex, so it is difficult to obtain effective and accurate three-dimensional data. Second, the acquired point cloud data is uneven and noisy, which has a great impact on the subsequent volume calculation. Summary of the invention

[0005] The object of the present invention is to solve the problem of volume measurement keeping sea cucumbers in situ.

[0006] In a first aspect, according to some embodiments of the present application, the underwater in-situ sea cucumber volume measurement method based on binocular vision comprises:

[0007] Detecting a sea cucumber image in an underwater image captured by a binocular camera to obtain a sea cucumber image detection frame;

[0008] According to the sea cucumber image detection frame, the sea cucumber image detected in the image is segmented, and the mask information of the sea cucumber on the image plane is calculated;

[0009] Mapping the mask information to a global point cloud constructed by binocular information to obtain point cloud data of the sea cucumber;

[0010] Reconstructing the point cloud data of the sea cucumber into a grid to obtain a continuous triangular grid;

[0011] The continuous triangular mesh is converted into a voxel representation, the number of non-zero elements in the voxel mesh and the volume of each element are counted, the volume of the mesh is estimated, and the volume of the sea cucumber is obtained.

[0012] According to some embodiments of the present application, an underwater in-situ sea cucumber volume measurement method based on binocular vision is used to detect a sea cucumber image in an underwater image captured by a binocular camera by integrating a YOLOv8 target detector with a multi-head self-attention mechanism, wherein the multi-head attention mechanism introduces multiple parallel attention mechanisms;

[0013] The object detector uses a dot product attention mechanism to calculate the similarity between each pair of elements;

[0014] The object detector calculates the attention weight of each element according to the similarity between each pair of elements;

[0015] The object detector uses attention weights to compute a contextual representation of each element.

[0016] According to the underwater in-situ sea cucumber volume measurement method based on binocular vision in some embodiments of the present application, the target detector uses a dot product attention mechanism to calculate the similarity between each pair of elements. i and X j , the similarity between them S(X i ,X j ) is shown in formula (1):

[0017]

[0018] Where: d is the dimension of the vector, X i ·X j is the element vector X i and X j The dot product of

[0019] The target detector uses the Softmax function to take the similarity as an index to obtain the weight of each element. For element X i and X j , the weight between them is shown in formula (2):

[0020]

[0021] Where: n is the length of the input sequence, W(X i ,X j ) is the weight of the ith element to the jth element.

[0022] According to some embodiments of the present application, an underwater in-situ sea cucumber volume measurement method based on binocular vision, wherein the multi-head attention mechanism transforms the input data through multiple linear transformations, and then divides the output of the linear transformation into multiple heads, each head independently calculates the attention weight, and uses the attention weight to perform weighted summation on the input, specifically expressed as formula (3):

[0023] M h (Q,K,V)=Concat(h1,h2,h3......h h )W0 (3)

[0024] Where Q, K and V are the input query, key and value, subscript h is the number of heads, h h is the attention output of the h-th head, W0 is a linear transformation, and Concat means connecting multiple heads.

[0025] According to the underwater in-situ sea cucumber volume measurement method based on binocular vision in some embodiments of the present application, the sea cucumber image detected in the image is segmented by adding a SAM segmentation model with an adapter mechanism, and the mask information of the sea cucumber on the image plane is calculated; wherein the SAM segmentation model with an adapter mechanism includes an image encoder, a prompt encoder and a mask decoder; wherein:

[0026] Image encoder, used to convert the input image into a multi-scale feature vector representation;

[0027] A hint encoder is used to convert the detection box of the object detector as a box hint into a vector representation of the hint feature;

[0028] A mask decoder, used for inputting the multi-scale feature vector representation of the image encoder and the vector representation of the prompt feature of the prompt encoder, and outputting the mask information of the sea cucumber on the image plane;

[0029] The image encoder includes multiple layers of Transformer layers, each of which contains a normalization layer, a multi-head self-attention mechanism, an adapter, and a feedforward neural network.

[0030] Among them, the adapter includes a low-order fully connected layer Down, a ReLU activation function and a high-order fully connected layer Up.

[0031] According to the underwater in-situ sea cucumber volume measurement method based on binocular vision in some embodiments of the present application, in the Transformer layer of the image encoder, the embedding vector of the image patch is normalized by the first normalization layer and input into the multi-head self-attention mechanism, and the correlation between each image patch and other image patches is calculated by the multi-head self-attention mechanism;

[0032] Feed the output of the self-attention mechanism into the first adapter;

[0033] Perform a residual connection between the embedding vector of the image patch and the output of the first adapter, and output an updated embedding vector of the image patch, which contains global information between the image patches;

[0034] The updated image patch embedding vector is normalized by a second normalization layer and input into a feed-forward neural network, local features are enhanced by the feed-forward neural network, and input into a second adapter;

[0035] The updated image patch embedding vector, the output of the feedforward neural network, and the second adapter are residually connected, and the output serves as the input of the next Transformer layer.

[0036] According to some embodiments of the present application, an underwater in-situ sea cucumber volume measurement method based on binocular vision, wherein the point cloud data of the sea cucumber is grid-reconstructed to obtain a continuous triangular grid, comprising:

[0037] Divide the point cloud data of the sea cucumber into spatial regions using an octree to obtain point cloud data under an octree structure;

[0038] Perform a KD tree search on the local neighborhood of each point cloud under the octree structure and calculate the normal vector representing the local surface direction near each point;

[0039] By implicitly fitting an indicator function derived from point cloud data with normal vectors;

[0040] Smoothing the indicator function, and forming a Poisson equation by approximating the gradient field of the smoothed indicator function;

[0041] Compute approximate solutions to Poisson's equation;

[0042] The marching cubes algorithm is used to extract the isosurfaces in the approximate solution of Poisson's equation, and the continuous triangular mesh of the sea cucumber is reconstructed from the point cloud data.

[0043] According to the underwater in-situ sea cucumber volume measurement method based on binocular vision in some embodiments of the present application, the YOLOv8 target detector integrating the multi-head self-attention mechanism includes a backbone network for extracting features and a neck network and a head network for feature fusion, wherein the backbone network includes an MSA module, and the MSA module includes a multi-head attention mechanism; wherein the backbone network includes

[0044] A first convolution module, which takes the underwater image as input, performs preliminary feature extraction on the underwater image, and converts the number of channels of the input image to 64;

[0045] The second convolution module takes the output of the first convolution module as input and performs further convolution operation;

[0046] The first C2f module takes the output of the second convolutional module as input and performs feature extraction through a series of convolutions, batch normalization, and activation functions.

[0047] The third convolution module takes as input the output of the first C2f module and converts the number of feature map channels to 256;

[0048] The second C2f module takes the output of the third convolutional module as input and performs feature extraction through further convolution, batch normalization and activation function processing;

[0049] The fourth convolution module takes as input the output of the second C2f module and converts the number of channels of the feature map to 512;

[0050] The third C2f module takes the output of the fourth convolutional module as input and performs feature extraction through a combination of convolution, batch normalization, and activation function;

[0051] The fifth convolution module takes as input the output of the third C2f module and converts the number of channels of the feature map to 1024;

[0052] The input of the MSA module is the output of the fifth convolutional module. Multiple attention heads are used to process the input features in parallel. Each head independently learns the feature interaction information of different dimensions, and the outputs of each head are concatenated into a concatenated vector.

[0053] The fourth C2f module takes the output of the MSA module as input and generates a feature map through feature extraction;

[0054] The SPPF module takes as input the output of the fourth C2f module and performs a spatial pyramid pooling operation to capture multi-scale information and enhance spatial resolution.

[0055] According to the underwater in-situ sea cucumber volume measurement method based on binocular vision in some embodiments of the present application, the neck network and the head network include

[0056] The first upsampling module takes the output of the SPPF module as input, performs upsampling, increases the spatial resolution of the feature map, and inputs its output and the output of the third C2f module to the first connection module in parallel;

[0057] The first connection module takes as input the output of the first upsampling module and the output of the third C2f module, concatenates them in the channel dimension to form a fused feature map;

[0058] The fifth C2f module takes the output of the first connection module as input and performs feature extraction;

[0059] The second upsampling module takes as input the output of the fifth C2f module, performs upsampling, increases the spatial resolution of the feature map, and inputs its output and the output of the second C2f module to the second connection module in parallel;

[0060] The second connection module, whose input is the output of the second upsampling module and the output of the second C2f module, is concatenated in the channel dimension;

[0061] a sixth C2f module, whose input is the output of the second connection module;

[0062] The detection head, whose input is the feature map output by the sixth C2f module, includes multiple convolutional layers and detection layers, and is used to perform target detection and classification from the processed feature map, and generate detection results according to the number of categories;

[0063] When the target detector is trained:

[0064] The image size is adaptive 640*640;

[0065] The initial learning rate is 0.01;

[0066] Momentum is 0.937;

[0067] The weight decay is 0.0005;

[0068] The batch size is 32;

[0069] The number of training rounds is 150;

[0070] Each training round is followed by 1 testing round.

[0071] In a second aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the first aspect and any possible technical solution of the first aspect.

[0072] Beneficial effects: The present invention proposes an underwater in-situ sea cucumber volume measurement method based on binocular vision. Under the premise of not disturbing the sea cucumber and obtaining the most realistic data, it maintains high detection accuracy and good segmentation effect, and does not rely on the selection of feature points, but uses three-dimensional point cloud information to measure the overall volume of the sea cucumber. This method takes the in-situ sea cucumber in the real breeding scene as the research object. First, with the help of a binocular camera carrying an unmanned boat in the sea cucumber breeding circle, a complete data video is taken; secondly, the sea cucumber is detected by the improved YOLOv8 algorithm to enhance the detection effect of the underwater environment; then, the sea cucumber is segmented with the adjusted SAM (segment anything model) segmentation model; finally, the Poisson reconstruction and voxelization method are used to measure the volume of the sea cucumber in combination with the mapping results of the segmentation mask and three-dimensional information. In order to meet the evaluation needs of farmers and provide more accurate data support for the scientific breeding of sea cucumbers. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is the overall framework for sea cucumber volume measurement.

[0074] Figure 2 This is a schematic diagram of the YOLOv8-MSA network structure.

[0075] Figure 3 It is the loss function and test matrix diagram during training.

[0076] Figure 4 This is the SAM-Adapter model structure diagram.

[0077] Figure 5 It is the volume calculation flow chart of PRV algorithm.

[0078] Figure 6 This is an example diagram of the sea cucumber volume measurement program. (a) is the result diagram of the detection, and (b) is the point cloud view and result display after the single target object is segmented.

[0079] Figure 7 These are the results of sea cucumber surface reconstruction. (a) is the Alpha shapes method, (b) is the Ball pivoting method, and (c) is the Poisson reconstruction. DETAILED DESCRIPTION

[0080] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples. It should be pointed out that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention. The present application provides a method, an electronic device, and a computer-readable storage medium. Among them, the method, the device, and the computer-readable storage medium are based on the same technical concept. Since the principles of solving problems by the method, the device, and the computer-readable storage medium are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0081] The underwater in-situ sea cucumber volume measurement method based on binocular vision of the present invention constructs a sea cucumber data set based on the following method: the present invention uses a binocular camera device to collect sea cucumber videos in the sea cucumber breeding circle, and produces a data set containing 3192 pictures, covering various growth forms of sea cucumbers in different seasons. The video shot by the present invention is 30 frames / s, and 3192 images are extracted by using a frame extraction script at 30 frames per second, and the labelimg annotation tool is used to annotate in yolo format. The data set does not use offline data enhancement, and online data enhancement is used during the training process. The data set is randomly divided into a training set (2554 pictures), a test set (319 pictures), and a verification set (319 pictures) in a ratio of 8:1:1, thereby ensuring the authenticity and validity of the test data.

[0082] Overall framework of sea cucumber volume measurement method: Binocular vision is an important form of machine vision. It is a method based on the principle of parallax and uses imaging equipment to obtain two images of the object to be measured from different positions, and obtains the three-dimensional geometric information of the object by calculating the position deviation between corresponding points of the images. Figure 1 As shown, the method mainly includes the following steps: sea cucumber target detection, sea cucumber target segmentation, and sea cucumber point cloud volume measurement. The present invention firstly shoots relevant videos of sea cucumber breeding circles through an underwater image acquisition device equipped with a binocular camera, and then uses the YOLOv8-MSA (YOLOv8 multi-head self-attention) algorithm to detect the sea cucumbers in the image, and then uses the SAM-AD (SAM-adapter) algorithm to obtain the mask information belonging only to the sea cucumber with the help of the detection frame information, and uses the global point cloud information obtained by the binocular technology and the mask information of the sea cucumber for mapping and projection, so as to obtain the point cloud information belonging only to the sea cucumber, and finally uses the Poisson reconstruction and voxelization processing algorithm PRV (Poisson reconstruction and voxelization) to solve the volume of the sea cucumber.

[0083] Design of sea cucumber detector integrating multi-head self-attention: The YOLOv8 detection model achieves an excellent balance between speed and accuracy, has efficient feature extraction and lightweight design, and is suitable for real-time detection applications. It uses advanced loss functions to improve detection accuracy and robustness. Combining the considerations of speed and accuracy, the present invention selects YOLOv8s as the basic detector of the present invention. However, the underwater environment has its own uniqueness. Because the light conditions underwater are very different from those on land, the absorption and scattering of water will cause the light to weaken and the color temperature to deviate. YOLOv8s sometimes has difficulty adapting to such changes in light conditions, which affects the detection accuracy. In addition, underwater images often suffer from reduced image quality due to water particles and motion blur. The performance of YOLOv8s will decline when dealing with these noises and blurs, especially in small targets and complex backgrounds.

[0084] In this regard, the self-attention mechanism can provide a good solution. The self-attention mechanism can dynamically assign weights on the feature map, so that the model can focus on important areas and features and ignore irrelevant background noise. This is especially important for underwater environments with complex backgrounds and significant noise interference. By capturing fine-grained features, the detection capabilities of small targets and detailed features can be improved, and the complex forms of underwater targets can be adapted. In addition, the self-attention mechanism can automatically adjust the attention weights according to the different input images, enhancing the adaptability of the model under different lighting, color distortion and noise conditions. By focusing on important feature areas, the self-attention mechanism can reduce the impact of suspended particles and dynamic interference on the model detection performance.

[0085] The multi-head self-attention mechanism is an effective attention mechanism that extracts richer feature representations by splitting the input sequence into multiple heads and processing the information of each head in parallel. The multi-head self-attention mechanism allows the model to learn different feature representations in different subspaces. Each attention head can focus on different parts of the input sequence or different feature dimensions to better capture the details of the input data. Underwater environments usually have complex lighting conditions, high noise levels, and occlusions. The multi-head self-attention mechanism can analyze and understand the characteristics of underwater targets from multiple perspectives, helping the model to more accurately identify and detect sea cucumbers under these complex conditions. In addition, through the parallel calculation of multiple heads, the multi-head self-attention mechanism can produce more diverse representations between different heads, making the model more robust to noise and input changes. Noise and interference are very common in underwater environments. The multi-head self-attention mechanism can filter out irrelevant or misleading information through different attention heads, improve the model's recognition and robustness to targets, and thus reduce false detections and missed detections. Finally, the multi-head self-attention mechanism can effectively fuse diverse information from different attention heads. This information fusion helps improve the model's ability to understand complex scenes. Underwater target detection usually requires processing a large amount of perceptual information, including color, shape, texture, etc. The multi-head self-attention mechanism can effectively integrate these multi-dimensional information to more accurately identify and locate targets, especially in complex backgrounds or when the contrast between the target and the background is low. In summary, the multi-head self-attention mechanism has significant advantages in underwater target detection tasks, mainly manifested in better adaptability to complex scenes, stronger noise resistance, and comprehensive capture of target features.

[0086] The present invention introduces a multi-head self-attention mechanism into the backbone network and constructs a YOLOv8-MSA algorithm model. The model structure of YOLOv8-MSA is as follows: Figure 2 As shown. The present invention introduces a multi-head self-attention mechanism in the last layer of the convolutional neural network of Backbone, so as to better capture the details in the image, improve the computational efficiency, enhance the robustness of the model, and improve the ability to recognize targets in complex scenes. Through this attention mechanism, the model can better understand the image content and extract richer feature representations from it, thereby more accurately identifying the target.

[0087] The model structure of YOLOv8-MSA is as follows Figure 2 As shown, it includes a backbone network, a neck network and a head network, where:

[0088] The backbone network is responsible for feature extraction: the original image input is subjected to preliminary feature extraction by the first convolution module, with a convolution kernel size of 3 and a stride of 2, which converts the number of channels of the input image to 64. The output feature map of this module is passed to the second convolution module as input for subsequent processing. The second convolution module receives the output of the first convolution module and further performs convolution operations with a convolution kernel size of 3 and a stride of 2, increasing the number of feature map channels to 128. Its output is used as the input of the first C2f module. The input of the first C2f module is the output of the second convolution module, which is subjected to a series of convolutions, batch normalization, and activation functions for feature extraction. The resulting feature map is passed to the third convolution module. The third convolution module processes the output of the first C2f module, further increasing the number of feature map channels to 256, with a convolution kernel size of 3 and a stride of 2. Its output is used as the input of the second C2f module. The second C2f module extracts features from the output of the third convolution module through further convolutions, batch normalization, and activation functions. The generated feature map is passed to the fourth convolution module. The fourth convolution module receives the output of the second C2f module, with a convolution kernel size of 3 and a stride of 2, increasing the number of feature map channels to 512. The output of this module is passed to the third C2f module. The third C2f module further processes the output of the fourth convolution module, and the generated feature map is passed to the fifth convolution module through a combination of convolution, batch normalization and activation function. The fifth convolution module performs a convolution operation on the output of the third C2f module, with a convolution kernel size of 3 and a stride of 2, increasing the number of feature map channels to 1024. Its output is passed to the MSA module. The MSA module receives the output of the fifth convolution module, processes the input features in parallel through multiple attention heads, and each head independently learns the feature interaction information of different dimensions. Finally, the outputs of each head are spliced ​​and passed to the fourth C2f module. The fourth C2f module continues to process the output of the MSA module, generates a feature map through feature extraction, and the feature map is passed to the SPPF module. The SPPF module performs a spatial pyramid pooling operation on the output of the fourth C2f module to capture multi-scale information and enhance spatial resolution. The output is passed to the first upsampling module.

[0089] Among them: the neck network and the head network are responsible for feature fusion: the first upsampling module upsamples the output of the SPPF module to double the spatial resolution of the feature map. Its output is parallelly input to the first connection module with the output of the third C2f module. The first connection module splices the output of the first upsampling module and the output of the third C2f module in the channel dimension to form a fused feature map. The feature map is passed to the fifth C2f module. The fifth C2f module further processes the feature map output by the first connection module, extracts features and outputs it to the second upsampling module. The second upsampling module upsamples the output of the fifth C2f module to further increase the spatial resolution of the feature map. Its output is parallelly input to the second connection module with the output of the second C2f module. The second connection module splices the output of the second upsampling module and the output of the second C2f module in the channel dimension, and the generated feature map is passed to the sixth C2f module. The sixth C2f module further processes the output feature map of the second connection module and sends the processed feature map to the detection head module. The detection head module accepts the feature map output by the sixth C2f module. The module includes multiple convolutional layers and detection layers, which are used to perform target detection and classification from the processed feature map and generate the final detection result according to the number of categories.

[0090] The self-attention mechanism is one of the core components in the Transformer neural network, which is used to weight the elements in the input sequence to capture the correlation between elements. The self-attention mechanism finds relevant information by calculating the similarity between elements. The similarity between each element and other elements is calculated to find elements related to the current element. To this end, the present invention uses the dot-product attention mechanism to calculate the similarity between each pair of elements. For element X i and X j , the similarity between them is shown in formula (1)

[0091]

[0092] Where: d is the dimension of the vector, X i ·X j is the vector X i and X j The dot product of .

[0093] Next, we convert the similarities into weights for use in computing context representations. To do this, we normalize the similarities so that they add up to 1. Specifically, we use the Softmax function to exponentiate the similarities to get the weight of each element. i and X j , the weight between them is shown in formula (2)

[0094]

[0095] Where: n is the length of the input sequence, W(X i ,X j ) is the weight of the i-th element to the j-th element. At this time, for each element X i , we get a weight vector, and finally use the weight vector W(X i ) to compute the contextual representation of each element.

[0096] The multi-head attention mechanism allows the network to perform multiple different SAs simultaneously in different spatial locations or feature representations. In traditional SA, the neural network obtains the weighted sum of attention by weighted averaging the input, but this method only considers one feature representation and ignores other features that may be relevant to the current task. To solve this problem, the multi-head attention mechanism introduces multiple parallel attention mechanisms to observe the input data from different angles and then merge their results. The basic idea of ​​the multi-head attention mechanism is to transform the input data through multiple linear transformations and then divide the results of these linear transformations into multiple heads. Each head independently calculates the attention weights and uses them to weighted sum the input. The specific representation of this process is shown in formula (3):

[0097] M h (Q,K,V)=Concat(h1,h2,h3......h h )W0 (3)

[0098] Where: Q, K and V are the input query, key and value, h is the number of heads, W0 is a linear transformation, and Concat connects multiple heads. The calculation method of the attention weight of each head is similar to the traditional attention mechanism, except that in the multi-head attention mechanism, the attention weight of each head is calculated independently. The main advantage of the multi-head attention mechanism is that it can pay attention to different feature representations at the same time, thereby improving the generalization and performance of the model.

[0099] When training the sea cucumber detector, the image size is adaptive 640*640, the initial learning rate is 0.01, the momentum is 0.937, the weight decay is 0.0005, the batch size is 32, the number of training rounds is 150, and each training round has 1 test round. The operating system used for training is Windows 11, the deep learning framework is PyTorch, the processor used for training is 16-core i7-13700KF, and the graphics card is NVIDIA GeForce RTX 2080ti. Figure 3 The changing trends of the loss function and test indicators of the improved sea cucumber detector during training are shown. Figure 3 As can be seen, whether in the training set or the validation set, the classification loss (cls_loss) is steadily decreasing and approaches zero in the later stage of training. The loss curve of the validation set is also consistent with that of the training set. This shows that there is no significant deviation between the validation set loss and the training set loss during the training process, that is, there is no obvious overfitting phenomenon.

[0100] In terms of test indicators, we can see that as the number of training rounds increases, both precision and mAP@0.5 quickly reach high levels of 0.947 and 0.830 respectively around round 20. Subsequently, these indicators continue to steadily improve with the increase in training rounds and eventually converge, indicating that the model has good stability and generalization ability.

[0101] Figure 3 The data show that the improved sea cucumber detector not only performs well during the training process, but also has high stability and accuracy during the testing phase without overfitting problems. The model can quickly achieve high detection performance in a short training time, proving its effectiveness in specific application scenarios.

[0102] Improvement of the sea cucumber segmentation model integrating the adapter mechanism: The purpose of segmentation is to segment the sea cucumber from the image and give its mask information on the image plane. The present invention selects SAM as the image segmentation model. SAM is highly versatile and flexible and can handle different types of segmentation tasks. It can perform zero-sample generalization on unfamiliar objects and images without additional training, thereby "cutting" any object in any image. SAM uses a convolutional neural network to extract features from the image and a lightweight encoder to convert any hints into an embedding vector. These two information sources are then combined in a lightweight decoder to predict the segmentation mask. However, the underwater environment is complex, and sea cucumbers often have colors and textures that are highly similar to the surrounding environment, making it difficult to accurately distinguish and segment. Due to the camouflage characteristics of sea cucumbers, SAM is easily disturbed in segmentation tasks, resulting in a decrease in segmentation accuracy, limiting the effectiveness of the model in underwater tasks.

[0103] On this basis, the present invention proposes the SAMAD model. In order to improve the segmentation performance of SAM in underwater scenes and enhance its performance in downstream tasks, the present invention introduces an adapter mechanism into the SAMAD model. The adjusted model framework is as follows Figure 4 The core architecture of SAM-AD consists of three parts: Image Encoder, Prompt Encoder, and Mask Decoder.

[0104] The image encoder is responsible for converting the input image into a multi-scale feature vector representation, providing rich visual information for subsequent segmentation tasks. This module takes the initial RGB image as input and outputs multi-scale image features. This feature is represented as a high-dimensional feature map, which is an important basis for the subsequent decoder to obtain the mask. Specifically, the original image is passed into the image embedding module as input and is divided into multiple small fixed-size image patches, each of which will be regarded as a separate "token". Each image patch will be flattened, converted into a vector, and then embedded into a high-dimensional space through a linear projection layer to form a fixed-dimensional embedding representation. This allows the embedding vector of each patch to be used as input to the Transformer layer.

[0105] The multi-layer transformer is the core part of the encoder. It takes the output of the image embedding module as input and is usually composed of n stacked transformer layers. Each transformer layer contains two key components: a multi-head self-attention mechanism and a feed-forward neural network. The embedding vector of each image patch is normalized by the layer and then interacted by the multi-head self-attention mechanism to calculate the relevance of each patch to other patches. The original input patch embedding and the output of the self-attention mechanism are connected with an adapter to make a residual connection. The output of this module is the updated patch embedding representation, which contains the global information between patches. The updated patch embedding is further processed by a feed-forward neural network (MLP layer) after the layer normalization operation to enhance the local features. The feed-forward network introduces nonlinear transformations, allowing the model to capture complex patterns. The input patch embedding and the output of the feed-forward network are connected with a new adapter to make a residual connection, and the output is used as the input of the next transformer. The output of this module is the updated features after nonlinear activation, which further enriches the representation ability of the image patch. The prompt encoder is responsible for converting the prompt information we provide into a vector representation that the model can understand. This module takes the detection box of YOLOv8 as the box hint as input and outputs the vector representation of the hint feature. These hint information will be combined into the decoder to guide the model to generate a specific segmentation mask. The mask decoder combines the features from the image encoder and the hint information from the hint encoder to generate the final segmentation mask. This module takes the multi-scale image features output by the image encoder and the hint features from the hint encoder as input and outputs the final segmentation mask, usually a binary image.

[0106] The adapter mechanism is a lightweight module that can be inserted into the middle layer of the pre-trained model to achieve rapid adaptation to new tasks or new fields. Specifically, the architecture of all adapters is relatively unified: a low-order fully connected layer Down, a ReLU activation function and a high-order fully connected layer Up, which can keep the parameters of the pre-trained model unchanged, and only a small number of parameters need to be trained to achieve good results, while reducing the demand for computing resources and storage space. Each Transformer block consists of two sub-blocks, namely the multi-head self-attention layer and the MLP layer. The self-attention layer of each block will be matched with an adapter adapter, and a residual connection will be performed before the second MLP sub-block. In the second MLP layer, another new adapter will be parallel to the original MLP layer to generate the output features of the current block. The present invention uses more diverse and high-quality underwater images to pre-train the SAM model, thereby enhancing its generalization ability and accuracy in underwater images, so that it can adapt to the complexity and diversity of underwater environments.

[0107] In addition, the present invention uses the recognition box of YOLOv8 as a prompt to guide SAM to perform segmentation in view of the special environment of underwater tasks. The improved YOLOv8 can quickly and accurately locate the position and size of objects in underwater images, thereby providing effective prior information for SAMAD. The present invention uses the bounding box of the marine treasures detected by YOLOv8 as input and passes it to the SAM model. The SAM model uses it as a prompt to generate a segmentation mask belonging to the target object according to the position and size of the bounding box, thereby realizing the segmentation of marine treasures. Such supplementary operations effectively reduce the errors and noise in the segmentation process and improve the accuracy and efficiency of segmentation.

[0108] Point cloud volume calculation method based on Poisson reconstruction and voxelization processing: Map the mask to the global point cloud constructed by binocular information, and then you can get the point cloud information that only belongs to the target object. But there is a problem with such sea cucumber point cloud data. The sea cucumber point cloud obtained after segmentation is not completely closed. The complexity of the environment and the accuracy of the camera may bring some point cloud noise. In addition, the lower part of the point cloud model is hollow, and most of the data is missing. At the same time, the volume calculation may also involve the determination of the height value, which will affect the volume calculation. Therefore, before performing the volume calculation, it is first necessary to reconstruct the surface of the point cloud data to achieve the effect of a closed figure.

[0109] Traditional surface reconstruction methods have their own advantages and disadvantages. The Alpha shapes algorithm is a very flexible surface reconstruction method. It controls the detail level of the reconstruction result by adjusting the parameter α, but it will produce overly smooth or incomplete surfaces, and it is also prone to topological errors, resulting in disconnected surfaces or holes; the Ball pivoting method calculates the volume by simulating a ball rolling on the surface of a point cloud, gradually connecting the point cloud to form a surface model. Although this method has certain advantages in processing point cloud data, when there are many noise points in the point cloud data, the rolling ball is easily disturbed, generating an inaccurate surface model, thereby affecting the volume calculation result and producing a large fluctuation deviation; the Poisson reconstruction method generates a continuous and leak-free mesh model by densely sampling the point cloud data and using normal information to infer surface continuity. In summary, the present invention proposes a PRV point cloud volume calculation method based on Poisson reconstruction and voxelization processing. The overall process is as follows: Figure 5 shown.

[0110] The present invention uses point cloud data for mesh reconstruction. A mesh is a collection of many triangles that connect the sampling points in the point cloud data. Poisson reconstruction infers the continuity of the surface by densely sampling the point cloud and using the normal information of the point cloud, and finally generates a mesh model representing the surface of the original object. The algorithm is as follows:

[0111] 1. Data acquisition

[0112] Input: original sea cucumber 3D point cloud data, including the 3D coordinates (x, y, z) of each point

[0113] Output: 3D point cloud data, excluding normal vector information

[0114] 2. Octree construction

[0115] Input: 3D point cloud data

[0116] Output: point cloud data under octree structure

[0117] Description: Use octree to divide the space of point cloud data. According to the set threshold, the spatial area is gradually subdivided, and the data points are placed in the corresponding small cube area until the number of points contained in each small cube is less than or equal to the threshold. Each small cube is a leaf node, and the depth D determines the fineness of the division.

[0118] 3. Normal vector calculation

[0119] Input: Point cloud data in octree structure

[0120] Output: Point cloud data with normal vectors

[0121] Description: Perform a KD tree (K-dimension tree) search on the local neighborhood of each point cloud to calculate the normal vector of each point. These normal vectors represent the local surface direction near each point.

[0122] 4. Construct indicator function

[0123] Input: Point cloud data with normal vectors

[0124] Output: indicator function

[0125] Description: By implicitly fitting an indicator function derived from point cloud data, an estimate of the object surface is given. The indicator function defines that the value inside the model is greater than 0, and the value outside is less than 0. The part with a value of 0 is the isosurface, which is extracted to be the surface of the geometric target model.

[0126] 5. Smooth indicator function constitutes Poisson equation

[0127] Input: indicator function

[0128] Output: Smoothed indicator function

[0129] Note: Since the indicator function is piecewise and discrete, a filter is used to smooth it. The gradient field of the indicator function after smoothing is equal to the smoothed result of the surface normal field, and the approximation of the gradient field constitutes the Poisson equation. Among them, is the Laplace operator, is the divergence operator, and the solution of the equation can be obtained by using the basic solution of the Laplace equation and the function product to find the indicator function. The indicator function represents the indicator function after smoothing, and the Poisson equation is established through the gradient field of the indicator function after smoothing.

[0130] 6. Solving Poisson's equation

[0131] Input: The gradient field and normal field of the smoothed indicator function and the constructed Poisson equation

[0132] Output: Approximate solution of Poisson's equation and reconstructed object surface

[0133] Description: Based on Poisson's equation, the vector field is constructed using the integral relationship and gradient field, and then the approximate solution of the indicator function is solved. The solution of Poisson's equation reflects the surface structure of the object, and the position of the isosurface is solved by the matrix iteration method.

[0134] 7. Isosurface extraction

[0135] Input: Approximate solution to Poisson's equation

[0136] Output: Reconstructed triangle mesh model of the object

[0137] Description: The marching cube algorithm is used to extract the isosurfaces in the solution of the Poisson equation and reconstruct the triangular mesh model of the object from the point cloud data. The isosurface represents the outer surface of the object, and the marching cube algorithm generates a continuous triangular mesh based on the Poisson solution.

[0138] 8. Voxelization and volume calculation

[0139] Input: Triangle mesh model

[0140] Output: An estimate of the object’s volume

[0141] Description: Convert a continuous triangular mesh to a discrete voxel representation. Voxelization is the process of dividing a geometric surface into a regular cubic grid by specifying the side length of the voxel. The size of the voxel determines the density of the voxel and the accuracy of the final volume estimate. Count the number of all non-zero elements in the voxel grid, which represent the volume units occupied by the measured object. By calculating the number of these non-zero voxels and multiplying it by the volume of each voxel (determined by the voxel side length), the volume estimate of the original mesh is obtained.

[0142] Specifically, the present invention divides the object point cloud data into spatial regions using an octree, places each point in a square space, sets a certain threshold, and if the data points in the square exceed the threshold, divides the cube again until the number of points contained in each small cube is less than or equal to the threshold. Each small cube is a leaf node with a depth of D. Adjusting the parameter size of D can change the accuracy of reconstruction. The Poisson reconstruction process depends largely on the determination of the normal vector. Therefore, the present invention needs to estimate the normal vector of each point in the spatial region in order to understand the direction information of the surface. The normal vector is a vector perpendicular to the surface, which can indicate the direction and curvature of the surface. The present invention adopts a method based on the nearest neighbor search to estimate the normal vector of each point. Specifically, the present invention uses a KD tree search method to estimate the direction of the normal by searching the nearest neighbor points around each point. Based on this, by implicitly fitting an indicator function derived from the object, an estimate of a smooth object surface can be given. Given a region S and its boundary Indicator function X s The definition is shown in formula (4):

[0143]

[0144] This indicator function defines that the value inside the model is greater than 0, the value outside is less than 0, and the part with 0 is the isosurface What is extracted is the surface of the geometric target model.

[0145] As a piecewise function, the indicator function is not continuous, so a filter is needed to smooth the convolution indicator function, and then solve the gradient field of the smoothed function. According to the Gaussian divergence theory, the gradient of the smoothed indicator function is equal to the vector field obtained by smoothing the surface normal field. For any point definition represents the inward surface normal vector, F(q) is a smoothing filter, F p (q) = F(qp) is the translation of F along the p direction. s It is difficult to derive in general, so here we use X S ×F is approximated by the derivative of F, as shown in formula (5):

[0146]

[0147] In the formula, F represents the smoothing filter function, × is the convolution symbol, which represents the smoothing operation. represents the gradient field of the smoothed indicator function, represents the smoothing of F along the p direction.

[0148] Since the point cloud data of the object surface is discrete, The distribution of each point q on the surface is unknown, which requires observing P = (p i ,n i ) to approximate, p i Indicates the spatial position coordinates of point P in the point cloud data, n i Indicates the direction of the normal line of the object surface at point P.

[0149] The present invention denotes the initial discrete sample point set as T, t is a point in T (t∈T), t contains position information (tp) and normal vector information Will Divide the space into different surface areas That is, t∈T, Convert the above formula into an integral sum, where each small integral can be approximated as a constant function, and the function and corresponding to (tp) can be used. The integral of the area is replaced by equation (6):

[0150]

[0151] In this way, the integral relationship between the sampling point and the indicator function is obtained from the gradient relationship. According to the integral relationship, the vector field of the point set is obtained by using the block division method, and the approximation of the gradient field of the indicator function is calculated to form the Poisson equation. and X S Satisfying formula (7):

[0152]

[0153] Taking the derivative of both sides at the same time, we get formula (8):

[0154]

[0155] Where Δ is the Laplace operator, is a divergence operator. The solution of the equation can be obtained by convolving the basic solution of Laplace's equation with the function to find the indicator function.

[0156] According to the Poisson's equation, the approximate solution of the isosurface is obtained by matrix iteration, and the isosurface is extracted by the marching cube algorithm. The model of the measured object is reconstructed for the measured data point set. The error of the Poisson's equation at the boundary is zero, so the obtained model does not have a false surface frame.

[0157] Once the present invention obtains a continuous triangular mesh, the present invention can convert it into a voxel representation. Voxelization is the process of discretizing a continuous geometric surface into a regular grid. In the present invention, the present invention uses a voxelization method to convert a continuous triangular mesh into a regular cubic mesh. During the voxelization process, the present invention determines the density of the voxel and the accuracy of the final volume estimation by specifying the size of the voxel, that is, the voxel side length. Finally, the present invention estimates the volume of the grid by counting the number of non-zero elements in the voxel grid. Each non-zero element represents a small volume unit in the voxel grid, corresponding to a part of the original grid. Therefore, by calculating the number of non-zero elements and multiplying it by the volume of the voxel, the present invention can obtain an estimate of the volume of the original grid. Through the above steps, the present invention can achieve a complete process from point cloud data to grid volume estimation.

[0158] Results and Analysis:

[0159] Feasibility verification of the sea cucumber volume measurement method: In order to verify the feasibility of the method proposed in the present invention, in the actual measurement process, the present invention uses a high-resolution binocular camera to shoot sea cucumbers, obtains two images from different angles, and extracts the three-dimensional point cloud data of the sea cucumber through image processing technology. Subsequently, the program processes the point cloud data and calculates the volume of the sea cucumber. Figure 6 is the relevant result diagram of the volume measurement program example. Figure 6 (a) is the result of the detection. Figure 6 (b) shows the point cloud view and result display after single target object segmentation.

[0160] Validation of the sea cucumber volume measurement method

[0161] Detection model comparison experiment: In order to verify the effectiveness of the YOLOv8-MSA model, the present invention conducted a comparison test of detection accuracy. All experiments were performed on a computer equipped with an NVIDIA RTX 2080ti GPU, the system was Windows 11, the deep learning framework used was PyTorch 1.8.1, and the CUDA version was 11.7. The present invention uses a self-made dataset of underwater in-situ sea cucumber images to compare the detection performance of five models: Faster RCNN, Cascade RCNN, YOLOv7, YOLOv8s, and YOLOv8-MSA.

[0162] The experimental data is shown in Table 1. The results show that Faster RCNN and Cascade RCNN, as two-stage target detection algorithms that perform well at present, have outstanding performance in detection accuracy, with mAP@0.5 reaching 88.9% and 89.4% respectively, which is significantly higher than YOLOv7 by about 10 percentage points. However, the parameters and FLOPs of these two models are relatively large, especially Cascade RCNN, whose parameters reach 69.4M and FLOPs are 256.3G, which greatly reduces the detection speed. This shows that although Faster RCNN and Cascade RCNN are superior in accuracy, they do not have sufficient efficiency. In contrast, YOLOv8s maintains a high accuracy while greatly reducing the number of parameters and the amount of calculation. After the method of the present invention integrates the MSA module on its basis, its mAP@0.5 reaches 92.5%, which is 1.3 percentage points higher than the original YOLOv8, and the number of parameters and FLOPs are only 11.4M and 28.9G. This shows that the YOLOv8-MSA detection model has certain advantages in detecting underwater sea cucumbers.

[0163] Table 1 Comparative analysis of detection model performance

[0164]

[0165] From the data in Table 1, we can see that the R index (recall rate) of YOLOv7 is 0.641, which is significantly lower than other models. This shows that although YOLOv7 performs well in the P index (precision), it is insufficient in the recall rate, and the complex environment in the water causes some targets to be undetected. In comparison, YOLOv8 and YOLOv8-MSA have high R indexes while maintaining high P indexes, which are 0.873 and 0.885 respectively, making their F1 scores significantly higher than other detection models. YOLOv8-MSA introduces the MSA module, which effectively improves the overall detection performance of the model and shows the best performance in comprehensive performance.

[0166] Surface reconstruction effect comparison experiment: In order to verify the effectiveness of the surface reconstruction algorithm of the present invention, the present invention conducted an effect comparison experiment of three surface reconstruction algorithms. The present invention randomly selected an image containing a sea cucumber and obtained its point cloud data, and used three different surface reconstruction algorithms, namely Alpha shapes, Ball pivoting, and Poisson reconstruction, to reconstruct the surface of the sea cucumber. Figure 7 These are the results of sea cucumber surface reconstruction using three different methods. Figure 7 (a) is the Alpha shapes method, Figure 7 (b) is the Ball pivoting method. Figure 7 (c) is the Poisson reconstruction.

[0167] It can be seen that the effect of Alpha shapes surface reconstruction is not satisfactory, the morphological characteristics of the sea cucumber are not well reflected, and there are still large hollow areas; the effect of Ball pivoting reconstruction has improved, but there are still many fragmented hollow areas; Poisson reconstruction has the best effect, which can not only reflect the model representation of the sea cucumber, but also the reconstructed surface is continuous and fits the real morphology of the sea cucumber. This provides more favorable data support for subsequent voxel processing and volume measurement.

[0168] Table 2 Comparison of accuracy of sea cucumber volume measurement by different methods

[0169]

[0170]

[0171] Volume measurement algorithm comparison experiment: In order to verify the effectiveness and accuracy of the measurement method of the present invention, the present invention conducted relevant experiments on volume accuracy measurement. The displacement method is one of the traditional volume measurement methods with high accuracy and reliability. In this experiment, ten sea cucumber models of different sizes were selected, and their actual volumes were measured respectively using the displacement method. Subsequently, the present invention used the volume measurement algorithm proposed in the present invention to measure the same sea cucumber at a depth of 100 cm. The calculation results are shown in Table 2.

[0172] By comparing the different results of manual measurement and measurement by the algorithm of the present invention, it was found that the average error between the volume measured by the program and the actual volume was within six percent. This result shows that the volume measurement program based on binocular vision has high accuracy and reliability, and can effectively replace the traditional drainage method for sea cucumber volume measurement. In addition, the method is easy to operate and has a fast measurement speed, which is suitable for rapid detection needs in large-scale aquaculture scenarios.

[0173] In addition, the present invention uses different measurement algorithms to measure the volume of the point cloud model of the sea cucumber, and uses four different methods, namely the method of the present invention, the minimum bounding box measurement method, the Alpha shapes method and the Ball pivoting method, for comparative analysis to determine the most suitable measurement method. The minimum bounding box method constructs a minimum cuboid to enclose the entire point cloud model to calculate the volume. However, since the bounding box includes part of the space outside the sea cucumber, this leads to a higher upper limit of the volume estimation, and the calculated volume is generally too large, and cannot accurately reflect the true volume of the sea cucumber. Especially for sea cucumbers with irregular shapes, the volume of the bounding box will be significantly larger than the actual volume, resulting in inaccurate measurement results. The Alpha shapes method constructs an overly smooth surface, and a part of the voids appear, which also has a great impact on the reconstruction effect and volume measurement. In actual measurement, the Ball pivoting method has a large volume measurement error and poor stability due to the discreteness and noise of the point cloud data. The Poisson reconstruction method generates a continuous and flawless grid model by densely sampling the point cloud data and inferring the surface continuity using normal information. This method can better fit the true shape of the sea cucumber and provide a more accurate volume measurement result.

[0174] In order to verify the reliability of the algorithm of the present invention at different detection distances, the present invention compares the measurement accuracy at different depths. The present invention uses the above four methods to calculate the accuracy of sea cucumber volume measurement at four different distances of 70cm, 100cm, 120cm, and 150cm. The experimental results are shown in Table 3.

[0175] Table 3 Comparison of sea cucumber volume measurement accuracy at different depths

[0176]

[0177] It can be seen that no matter at which depth, the accuracy of the measurement of the method of the present invention is better than that of the other three methods. Combined with the optimal working distance of the binocular camera, the method of the present invention achieves the highest volume measurement accuracy of 94% when the distance is about 100 cm, which is 22, 32, and 14 percentage points higher than the minimum bounding box method, Alpha shapes, and Ball pivoting methods, respectively.

[0178] The above experimental results show that the deviation of the volume measurement results of the Poisson reconstruction method is within a controllable range and has higher accuracy and stability than other methods.

[0179] The present invention proposes a method for calculating the volume of sea cucumbers based on binocular vision and point cloud processing. The three-dimensional information of the target object is obtained by a binocular camera, and then the YOLOv8-MSA and SAMAD models are used for detection and segmentation. Then, the three-dimensional point cloud data of the sea cucumber is obtained through the mapping relationship between two-dimensional and three-dimensional, and the data is filtered and optimized by the surface reconstruction algorithm, and finally the volume of the sea cucumber is solved. Experiments show that the method has high accuracy and can provide great help in the actual production and breeding process. The method adopted by the present invention does not rely on the selection of feature points and the construction of the target object model, but uses the disparity analysis of the binocular camera combined with the processing of the image pixel level to complete the overall volume measurement. Such a method can not only effectively measure the volume of sea cucumbers, but also perform well in the data measurement tasks of marine treasures such as scallops and sea urchins after certain modifications and improvements. The overall volume measurement algorithm has strong robustness and migration.

[0180] The segmentation model adopted by the present invention is SAM. As a large segmentation model, due to its huge number of parameters and complex calculation process, it has a high demand for computing resources and is difficult to meet tasks with high real-time requirements, resulting in poor overall operation rate of the algorithm. In addition, the Poisson surface reconstruction method also has the problems of poor boundary processing ability and limited accuracy, which needs to be improved. In the next work, the present invention is committed to optimizing the segmentation task, improving the computing efficiency while achieving high accuracy. At the same time, the effect of surface reconstruction is optimized, and the robustness of the volume measurement algorithm in various depths of water is improved, making it more intelligent and universal, and able to adapt to various underwater environments.

[0181] Based on the above embodiments, an embodiment of the present application further provides an electronic device, and when the computer program runs on the electronic device, the electronic device executes the method provided in the above embodiments.

[0182] Based on the above embodiments, the embodiments of the present application further provide a computer storage medium, in which a computer program is stored. When the computer program is executed by a computer, the computer executes the method provided in the above embodiments.

[0183] The storage medium may be any available medium that can be accessed by a computer. For example, but not limited to, a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for measuring the volume of underwater sea cucumbers in situ based on binocular vision, characterized in that: include Detecting a sea cucumber image in an underwater image captured by a binocular camera to obtain a sea cucumber image detection frame; According to the sea cucumber image detection frame, the sea cucumber image detected in the image is segmented, and the mask information of the sea cucumber on the image plane is calculated; Mapping the mask information to a global point cloud constructed by binocular information to obtain point cloud data of the sea cucumber; Reconstructing the point cloud data of the sea cucumber into a grid to obtain a continuous triangular grid; The continuous triangular mesh is converted into a voxel representation, the number of non-zero elements in the voxel mesh and the volume of each element are counted, the volume of the mesh is estimated, and the volume of the sea cucumber is obtained.

2. The underwater in-situ sea cucumber volume measurement method based on binocular vision according to claim 1, characterized in that: in, Detect sea cucumber images in underwater images captured by a binocular camera by integrating a YOLOv8 object detector with a multi-head self-attention mechanism, wherein the multi-head attention mechanism introduces multiple parallel attention mechanisms; The object detector uses a dot product attention mechanism to calculate the similarity between each pair of elements; The object detector calculates the attention weight of each element according to the similarity between each pair of elements; The object detector uses attention weights to compute a contextual representation of each element.

3. The underwater in-situ sea cucumber volume measurement method based on binocular vision according to claim 2, characterized in that: The target detector uses the dot product attention mechanism to calculate the similarity between each pair of elements. i and X j , the similarity between them S(X i ,X j ) is shown in formula (1): Where: d is the dimension of the vector, X i ·X j is the element vector X i and X j The dot product of The target detector uses the Softmax function to take the similarity as an index to obtain the weight of each element. For element X i and X j , the weight between them is shown in formula (2): Where: n is the length of the input sequence, W(X i ,X j ) is the weight of the ith element to the jth element.

4. The underwater in-situ sea cucumber volume measurement method based on binocular vision according to claim 3, characterized in that: in, The multi-head attention mechanism passes the input data through multiple linear transformations, and then divides the output of the linear transformation into multiple heads. Each head independently calculates the attention weight and uses the attention weight to perform weighted summation on the input. The specific expression is shown in formula (3): M h (Q,K,V)=Concat(h1,h2,h3……h h )W0 (3) Where Q, K and V are the input query, key and value, subscript h is the number of heads, h h is the attention output of the h-th head, W0 is a linear transformation, and Concat means connecting multiple heads.

5. The underwater in-situ sea cucumber volume measurement method based on binocular vision according to claim 1, characterized in that: The sea cucumber image detected in the image is segmented by adding a SAM segmentation model with an adapter mechanism, and the mask information of the sea cucumber on the image plane is calculated; wherein the SAM segmentation model with an adapter mechanism includes an image encoder, a hint encoder and a mask decoder; wherein: Image encoder, used to convert the input image into a multi-scale feature vector representation; A hint encoder is used to convert the detection box of the object detector as a box hint into a vector representation of the hint feature; A mask decoder, used for inputting the multi-scale feature vector representation of the image encoder and the vector representation of the prompt feature of the prompt encoder, and outputting the mask information of the sea cucumber on the image plane; The image encoder includes multiple layers of Transformer layers, each of which contains a normalization layer, a multi-head self-attention mechanism, an adapter, and a feedforward neural network. Among them, the adapter includes a low-order fully connected layer Down, a ReLU activation function and a high-order fully connected layer Up.

6. The method for measuring the volume of underwater sea cucumbers in situ based on binocular vision according to claim 5, characterized in that: in, In the Transformer layer of the image encoder, the embedding vector of the image patch is normalized by the first normalization layer and input into the multi-head self-attention mechanism, which calculates the correlation between each image patch and other image patches; Feed the output of the self-attention mechanism into the first adapter; Perform a residual connection between the embedding vector of the image patch and the output of the first adapter, and output an updated embedding vector of the image patch, which contains global information between the image patches; The updated image patch embedding vector is normalized by a second normalization layer and input into a feed-forward neural network, local features are enhanced by the feed-forward neural network, and input into a second adapter; The updated image patch embedding vector, the output of the feedforward neural network, and the second adapter are residually connected, and the output serves as the input of the next Transformer layer.

7. The method for measuring the volume of underwater sea cucumbers in situ based on binocular vision according to claim 5, characterized in that: in, The point cloud data of the sea cucumber is meshed to obtain a continuous triangular mesh, including: Divide the point cloud data of the sea cucumber into spatial regions using an octree to obtain point cloud data under an octree structure; Perform a KD tree search on the local neighborhood of each point cloud under the octree structure and calculate the normal vector representing the local surface direction near each point; By implicitly fitting an indicator function derived from point cloud data with normal vectors; Smoothing the indicator function, and forming a Poisson equation by approximating a gradient field of the smoothed indicator function; Compute approximate solutions to Poisson's equation; The marching cubes algorithm is used to extract the isosurfaces in the approximate solution of Poisson's equation, and the continuous triangular mesh of the sea cucumber is reconstructed from the point cloud data.

8. The method for measuring the volume of underwater sea cucumbers in situ based on binocular vision according to claim 1, characterized in that: The YOLOv8 target detector integrating a multi-head self-attention mechanism comprises a backbone network for extracting features and a neck network and a head network for feature fusion, wherein the backbone network comprises an MSA module, and the MSA module comprises a multi-head attention mechanism; Wherein, the backbone network includes A first convolution module, which takes the underwater image as input, performs preliminary feature extraction on the underwater image, and converts the number of channels of the input image to 64; The second convolution module takes the output of the first convolution module as input and performs further convolution operation; The first C2f module takes the output of the second convolutional module as input and performs feature extraction through a series of convolutions, batch normalization, and activation functions. The third convolution module takes as input the output of the first C2f module and converts the number of feature map channels to 256; The second C2f module takes the output of the third convolutional module as input and performs feature extraction through further convolution, batch normalization and activation function processing; The fourth convolution module takes as input the output of the second C2f module and converts the number of channels of the feature map to 512; The third C2f module takes the output of the fourth convolutional module as input and performs feature extraction through a combination of convolution, batch normalization, and activation function; The fifth convolution module takes as input the output of the third C2f module and converts the number of channels of the feature map to 1024; The input of the MSA module is the output of the fifth convolutional module. Multiple attention heads are used to process the input features in parallel. Each head independently learns the feature interaction information of different dimensions, and the outputs of each head are concatenated into a concatenated vector. The fourth C2f module takes the output of the MSA module as input and generates a feature map through feature extraction; The SPPF module takes as input the output of the fourth C2f module and performs a spatial pyramid pooling operation to capture multi-scale information and enhance spatial resolution.

9. The method for measuring the volume of underwater sea cucumbers in situ based on binocular vision according to claim 4, characterized in that: in, The neck network and the head network include The first upsampling module takes the output of the SPPF module as input, performs upsampling, increases the spatial resolution of the feature map, and inputs its output and the output of the third C2f module to the first connection module in parallel; The first connection module takes as input the output of the first upsampling module and the output of the third C2f module, concatenates them in the channel dimension to form a fused feature map; The fifth C2f module takes the output of the first connection module as input and performs feature extraction; The second upsampling module takes as input the output of the fifth C2f module, performs upsampling, increases the spatial resolution of the feature map, and inputs its output and the output of the second C2f module to the second connection module in parallel; The second connection module, whose input is the output of the second upsampling module and the output of the second C2f module, is concatenated in the channel dimension; a sixth C2f module, whose input is the output of the second connection module; The detection head, whose input is the feature map output by the sixth C2f module, includes multiple convolutional layers and detection layers, and is used to perform target detection and classification from the processed feature map, and generate detection results according to the number of categories; When the target detector is trained: The image size is adaptive 640*640; The initial learning rate is 0.01; Momentum is 0.937; The weight decay is 0.0005; The batch size is 32; The number of training rounds is 150; Each training round is followed by 1 testing round.

10. An electronic device, comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, enable the electronic device to perform any one of the methods described in claims 1-9.

Images