Method and device for measuring size of fish underwater, electronic equipment and storage medium

Abstract

The application provides an underwater fish body size measurement method and device, electronic equipment and storage medium, and belongs to the technical field of image processing. The method comprises the following steps: inputting a target fish group image into a contour extraction model, obtaining key point data and contour data of each fish in the target fish group image output by the contour extraction model; obtaining a body length curve and a body width curve of any fish based on the key point data and the contour data of the fish; determining body shape data of the fish based on the body length curve and the body width curve; and the contour extraction model is obtained after training sample fish group images with key point data labels and contour data labels. The underwater fish body size measurement method, device, electronic equipment and storage medium provided by the application can extract key points of fish in an underwater fish group, fit the spatial posture of the fish body, and measure the size of the underwater free-swimming fish body without contacting the fish body.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to underwater fish size measurement methods, devices, electronic equipment, and storage media. Background Technology

[0002] In aquaculture, fish body length information most directly reflects the growth status of a fish population, and is of great significance for monitoring and assessing the growth status of fish. Fish body dimensions, especially body length, directly reflect the fish's growth status and are important indicators for identifying parameters such as body area, volume, weight, and age. This information can be used as one of the bases for pricing fish fry and is also an important indicator for grading the quality of aquatic products.

[0003] Traditional methods for monitoring fish body length require fish farmers to catch a portion of the fish population and measure the size of these samples to use statistical data to reflect the overall growth status of the fish population.

[0004] The above measurement methods not only cause direct harm to the individual fish caught, but also indirectly harm the entire fish population. Summary of the Invention

[0005] The present invention provides an underwater fish size measurement method, device, electronic device and storage medium to solve the defects of the prior art that cause direct harm to the caught individuals, and to realize the size measurement of free-swimming fish underwater without contact with the fish body.

[0006] This invention provides a method for measuring the body size of underwater fish, comprising:

[0007] The target fish group image is input into the contour extraction model to obtain the key point data and contour data of each fish in the target fish group image output by the contour extraction model.

[0008] Based on the key point data and contour data of any fish, obtain the body length curve and body width curve of any fish.

[0009] Based on the body length curve and the body width curve, the body shape data of any fish is determined; the contour extraction model is obtained by training on sample fish images with key point data labels and contour data labels.

[0010] According to the present invention, an underwater fish body size measurement method is provided, wherein the contour extraction model sequentially comprises: a backbone network, a region proposal network, a nonmaximum suppression module, and a region of interest alignment layer, including:

[0011] The backbone network is used to extract features from the target fish image to obtain a target feature image, which has multiple anchor boxes.

[0012] The region proposal network is used to perform binary classification and bounding box regression on the target feature image to determine the region of interest.

[0013] The nonmaximum suppression module is used to filter the region of interest to obtain the target region of interest;

[0014] The region of interest alignment layer is used to align the feature image with the target region of interest to obtain a region of interest feature map;

[0015] The key point data and contour data of each fish are obtained based on the interest feature map.

[0016] According to a method for measuring the body size of underwater fish provided by the present invention, the backbone network includes ResNet50, a pyramid network, and an attention mechanism module, wherein the attention mechanism module includes a channel attention module and a spatial attention module;

[0017] The ResNet50 is used to perform preliminary feature extraction on the target fish image to obtain an initial feature image;

[0018] The pyramid network is used to perform feature fusion on the initial feature image to obtain a first feature image;

[0019] The channel attention module is used to extract features from the first feature image to obtain the second feature image;

[0020] The spatial attention module is used to extract features from the second feature image to obtain the target feature image.

[0021] According to the present invention, an underwater fish body size measurement method is provided, wherein obtaining the body length curve and body width curve of any fish based on key point data and contour data of any fish includes:

[0022] Based on the intrinsic and extrinsic parameter matrices of the target camera, binocular matching is performed on the contour data to obtain the transformation relationship between the target fish group image and the world coordinate system;

[0023] Based on the transformation relationship, coordinate transformation is performed on each key point data to obtain the world coordinates of each key point; the target fish school image is acquired based on the target camera;

[0024] Based on all world coordinates of any given fish, fit the body length curve and body width curve of any given fish.

[0025] According to a method for measuring the body size of an underwater fish provided by the present invention, the body size data of any fish is determined based on the body length curve and the body width curve, including:

[0026] The body length curve is constructed and solved to obtain the body length data of any fish, and the body width curve is constructed and solved to obtain the body width data of any fish.

[0027] The body shape data is determined based on the body length and body width data.

[0028] According to the present invention, an underwater fish size measurement method further includes, before inputting the target fish group image into the contour extraction model:

[0029] Acquire calibration data sets from multiple azimuths;

[0030] The target camera is calibrated using each calibration data set to obtain the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients;

[0031] Using the extrinsic parameter matrix and the distortion coefficients, the underwater fish school video is corrected to obtain a corrected fish school video, thereby determining the target fish school image. The underwater fish school video is acquired based on the target camera.

[0032] The present invention also provides an underwater fish body measuring device, comprising:

[0033] The first acquisition module is used to input the target fish group image into the contour extraction model and acquire the key point data and contour data of each fish in the target fish group image output by the contour extraction model.

[0034] The second acquisition module is used to acquire the body length curve and body width curve of any fish based on the key point data and contour data of any fish.

[0035] The determination module is used to determine the body shape data of any fish based on the body length curve and the body width curve; the contour extraction model is obtained after training on sample fish images with key point data labels and contour data labels.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the underwater fish size measurement method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the underwater fish size measurement method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the underwater fish body size measurement method as described above.

[0039] The underwater fish size measurement method, device, electronic equipment and storage medium provided by the present invention extract key points of fish in an underwater fish school and then fit the spatial posture of the fish body, so as to realize the size measurement of the fish body swimming freely underwater without contacting the fish body. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating the underwater fish size measurement method provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the outline and key points of the fish provided by the present invention;

[0043] Figure 3 This is the second flowchart illustrating the underwater fish size measurement method provided by the present invention;

[0044] Figure 4 This is the third flowchart illustrating the underwater fish size measurement method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the Mask-RCNN algorithm model provided by the present invention;

[0046] Figure 6 This is a structural diagram of the CBAM attention mechanism module provided by the present invention;

[0047] Figure 7 This is the fourth flowchart illustrating the underwater fish size measurement method provided by the present invention;

[0048] Figure 8 This is a schematic diagram of the underwater fish body measuring device provided by the present invention;

[0049] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Existing fish body measurements rely on relatively heavy human resources, are cumbersome, inefficient, and inaccurate.

[0052] Machine vision technology can be used to measure the size of fish without contact, but fish swimming freely underwater often have distorted postures, and existing technology cannot measure the size of fish swimming freely underwater in a bent state.

[0053] To improve the scientific and efficient nature of fish farming, it is necessary to regularly assess the growth status of fish populations and obtain timely information on fish size in order to determine further farming strategies.

[0054] The following is combined with Figures 1 to 9 The present invention describes the underwater fish size measurement method, apparatus, electronic device, and storage medium provided by embodiments of the present invention.

[0055] Figure 1 This is one of the flowcharts illustrating the underwater fish size measurement method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps:

[0056] First, in step S1, the target fish group image is input into the contour extraction model to obtain the key point data and contour data of each fish in the target fish group image output by the contour extraction model.

[0057] The underwater fish video can be taken by an image acquisition device of the fish school under test. The image acquisition device can be a three-dimensional sensor such as a binocular camera, a structured light camera, or a three-dimensional (3D) depth camera. In the following embodiments, the underwater fish video taken by a binocular camera will be used as an example for illustration, which is not considered as a limitation on the scope of protection of the present invention.

[0058] The underwater fish school video is corrected to obtain a corrected fish school video. The corrected fish school video is then cropped to obtain a target fish school image. The target fish school image includes at least one complete, unobstructed image of the fish to be tested. The target fish school image can be a left-eye image or a right-eye image. In the following embodiments, the processing of the left-eye image is used as an example for illustration, which is not considered as a limitation on the scope of protection of the present invention.

[0059] The target fish school image is input into the contour extraction model. The model performs contour segmentation and keypoint labeling on each complete, unoccluded fish in the target fish school image, and outputs complete keypoint data and contour data for each unoccluded fish. Keypoints are used to mark the body shape of the fish, and all keypoints are located on the contour of the fish.

[0060] Further, in step S2, based on the key point data and contour data of any fish, the body length curve and body width curve of the fish are obtained.

[0061] The pixel coordinates corresponding to the key point data and contour data on the image are transformed to the world coordinate system to obtain the outline and key point data of the fish in the world coordinate system.

[0062] Figure 2 This is a schematic diagram of the outline and key points of the fish provided by the present invention, such as... Figure 2 As shown, the key points of any fish include point A on the head, point B0 at the front of the dorsal fin, point B1 on the pelvic fin, point C0 at the tail end of the dorsal fin, point C1 on the anal fin, and point D at the midpoint of the tail.

[0063] By fitting the coordinates of key points in the world coordinate system to spatial curves, we can obtain the body length curve L1 and the body width curve L2.

[0064] Among them, the body length curve L1 passes through the head A, the anterior end of the dorsal fin and the midpoint B of the pelvic fin, the caudal end of the dorsal fin and the midpoint C of the anal fin, and the midpoint D of the tail; the body width curve L2 passes through the anterior end B0 of the dorsal fin, the anterior end of the dorsal fin and the midpoint B of the pelvic fin, and the pelvic fin B1.

[0065] Further, in step S3, the body shape data of any fish is determined based on the body length curve and the body width curve; the contour extraction model is obtained after training on sample fish images with key point data labels and contour data labels.

[0066] For the body length curve L1 and the body width curve L2, we construct and solve the spatial plane equations respectively, and we can obtain that the length of the body length curve L1 is the body length S1 of the fish, and the length of the body width curve L2 is the body width S2 of the fish.

[0067] Among these methods, multiple sample fish images can be extracted from the corrected fish swarm video, and the contour extraction model can be constructed based on an improved Mask Region-based Convolutional Neural Network (Mask-RCNN).

[0068] Labelme software was used to annotate each sample fish image, marking the complete outline and key points of the unobstructed fish bodies, forming key point data labels and outline data labels. After annotation, the data was exported as JSON format and converted to COCO dataset format to obtain a dataset composed of multiple training samples.

[0069] The training samples in the dataset were divided into training and test sets in an 8:2 ratio. The improved Mask-RCNN model was trained and validated using the training and test sets respectively, resulting in the contour extraction model.

[0070] Figure 3 This is the second flowchart illustrating the underwater fish size measurement method provided by the present invention, as shown below. Figure 3 As shown, it includes:

[0071] First, extract sample fish images from the corrected fish school video;

[0072] Secondly, the fish in the sample fish group images are classified and labeled to create training samples;

[0073] Subsequently, the training samples are input into the improved Mask-RCNN model for training to obtain the contour extraction model.

[0074] The underwater fish size measurement method provided by this invention extracts key points of fish in an underwater fish school and then fits the spatial posture of the fish body, so as to realize the size measurement of the fish body swimming freely underwater without contacting the fish body.

[0075] Optionally, before inputting the target fish image into the contour extraction model, the method further includes:

[0076] Acquire calibration data sets from multiple azimuths;

[0077] The target camera is calibrated using each calibration data set to obtain the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients;

[0078] The underwater fish school video is corrected using the extrinsic parameter matrix and the distortion coefficients to obtain a corrected fish school video, thereby determining the target fish school image. The underwater fish school video is acquired based on the target camera.

[0079] Figure 4 This is the third flowchart illustrating the underwater fish size measurement method provided by the present invention, as shown below. Figure 4 As shown, it includes the following steps:

[0080] First, underwater calibration videos were collected at different distances, angles, and orientations to form a calibration data set. Calibration videos were captured at distances of 0.5m, 1.5m, and 2.0m from the binocular camera. The relative positions of the calibration board and the binocular camera lens when parallel were categorized as directly facing each other, 20cm upward, and 20cm downward. The relative positions of the calibration board and the binocular camera lens when tilted were categorized as tilted forward 20°, tilted backward 20°, tilted left 20°, and tilted right 20°.

[0081] Secondly, the stereo camera is calibrated using calibration video. The captured calibration video is divided into left and right stereo images, and Zhang's calibration method is used to process the left and right stereo images containing the complete calibration plate to detect the corner points of the calibration plate in the images.

[0082] Next, the calibration parameters of the stereo camera are output, including the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients. Based on the actual size of the calibration board and the coordinates of the corner points in the image, the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficients of the stereo camera are calculated. The camera's intrinsic parameter matrix A... in as follows:

[0083]

[0084] Among them, f x f y are the effective focal lengths of the u-axis and v-axis, respectively; u0 and v0 are the optical centers; s is the non-perpendicularity factor of the u-axis and v-axis, and in general, s = 0.

[0085] The extrinsic parameters of a stereo camera include: rotation matrix R and translation vector T, as follows:

[0086]

[0087]

[0088] Where [r1 r4 r7] is a vector rotating about the X-axis; [r2 r5 r8] is a vector rotating about the Y-axis; [r3 r6 r9] is a vector rotating about the Z-axis; t x It is the translation distance along the X-axis; t y It is the translation distance along the Y-axis; t z It is the translation distance along the Z-axis.

[0089] The distortion coefficients of a stereo camera include radial distortion parameters k1 and k2.

[0090] Subsequently, the underwater fish school video was corrected using calibration parameters. The binocular camera, fixed to a camera mount, allows adjustment of its vertical height, horizontal position, and tilt angle. Connected to a computer via a USB extension cable, it records and saves underwater fish school videos in real time. The camera's extrinsic parameter matrix is ​​then used to perform binocular parallelism correction and distortion correction on the captured underwater fish school videos, as detailed below:

[0091] Divide the camera rotation matrix R into a composite matrix r for the left and right cameras. l r r , where r l =R 1 / 2 r r =R -1 / 2 This achieves coplanarity of the image plane.

[0092] Create a rotation matrix R in the direction of the translation vector T. rec R rec =[e1, e2, e3] T ;

[0093] Where, e1=T / ‖T‖ is the pole in the same direction as the translation vector T, and ‖T‖ represents taking the modulus of vector T; T=[T x T y T z ] T T x T y and T z These are the translation vectors in the x, y, and z directions, respectively;

[0094] It is a vector pointing in the direction of the image plane;

[0095] e3 = e1 × e2 is a vector perpendicular to the plane containing e1 and e2.

[0096] Based on the synthesis matrix r of the left camera l The composite matrix r of the right camera r Rotation matrix R in the direction of translation vector T rec The alignment transformation matrix R′ of the left camera can be obtained. l Alignment transformation matrix R′ of the right camera r The details are as follows:

[0097]

[0098] Using the alignment transformation matrix R′ of the left camera l The underwater fish video captured by the left camera is rotated and aligned; similarly, the alignment transformation matrix R′ of the right camera can be used. r Rotate and align the underwater fish video captured by the right camera.

[0099] The transformation is completed by multiplying the left and right cameras by these two matrices respectively, resulting in the corrected pixel coordinates of the image after row rotation alignment. and the image coordinates after row rotation alignment

[0100] The image pixel coordinate system is a rectangular coordinate system with the top left corner of the video frame as the origin, while the image coordinate system is a rectangular coordinate system with the intersection of the diagonals of the video frame as the origin.

[0101] For any frame in the rotated underwater fish video, the image correction pixel coordinates are... Distortion correction is performed using the distortion coefficient, as detailed below:

[0102]

[0103] in, (u, v) represents the pixel coordinates of the image after correction, and (u, v) represents the pixel coordinates of the image without distortion. This indicates the image coordinates after row rotation and alignment.

[0104] The distortion parameters k1 and k2 of the left and right cameras are used to perform distortion correction on each frame of the underwater fish school video after rotation correction, so as to obtain the corrected fish school video. The left eye image is then extracted from the corrected fish school video of the left camera as the target fish school image.

[0105] The underwater fish size measurement method provided by the present invention can obtain more accurate images of the target fish school by performing coordinate correction on the captured underwater fish school video, thus providing a basis for calculating the body size data of the fish.

[0106] Optionally, the contour extraction model sequentially comprises: a backbone network, a region proposal network, a nonmaximum suppression module, and a region of interest alignment layer, including:

[0107] The backbone network is used to extract features from the target fish image to obtain a target feature image, which has multiple anchor boxes.

[0108] The region proposal network is used to perform binary classification and bounding box regression on the target feature image to determine the region of interest.

[0109] The nonmaximum suppression module is used to filter the region of interest to obtain the target region of interest;

[0110] The region of interest alignment layer is used to align the feature image with the target region of interest to obtain a region of interest feature map;

[0111] The key point data and contour data of each fish are obtained based on the interest feature map.

[0112] The Mask-RCNN instance segmentation algorithm was modified. The backbone network of the Mask-RCNN instance segmentation algorithm model uses a ResNet50 with fewer layers to reduce computational cost and ensure computational speed. A Convolutional Block Attention Module (CBAM) was added to the convolutional module to improve detection accuracy. The code in the visualize.py file was modified to add keypoint display functionality to the mask display layer, enabling simultaneous contour segmentation and keypoint detection. The modifications to the visualize.py file are as follows:

[0113] The `display_keypoints` function in the `visualize.py` file is used to display keypoints, but it cannot display the mask for each instance. The `display_instances` function has a mask display module that can display the mask for each instance. By adding a variable "masks" to the `display_keypoints` function and adding the mask display module to the function body, calling the `display_keypoints` function in the `visualize.py` file when outputting the detection results can display both the outline and keypoints simultaneously.

[0114] Optionally, the backbone network includes ResNet50, a pyramid network, and an attention mechanism module, wherein the attention mechanism module includes a channel attention module and a spatial attention module;

[0115] The ResNet50 is used to perform preliminary feature extraction on the target fish image to obtain an initial feature image;

[0116] The pyramid network is used to perform feature fusion on the initial feature image to obtain a first feature image;

[0117] The channel attention module is used to extract features from the first feature image to obtain the second feature image;

[0118] The spatial attention module is used to extract features from the second feature image to obtain the target feature image.

[0119] Figure 5 This is a schematic diagram of the Mask-RCNN algorithm model provided by the present invention, as shown below. Figure 5As shown, the Mask-RCNN network uses ResNet-50 as its backbone network, which is connected to a Feature Pyramid Network (FPN). The input image data first passes through the backbone network composed of ResNet50 and the Feature Pyramid Network to extract the target feature image. Each pixel on the feature image generates several anchor boxes, which are then used for binary classification and bounding box regression in the Region Proposal Network. Finally, the Non-Maximum Suppressing (NMS) module is used to filter the candidate regions.

[0120] The region of interest alignment layer transforms the filtered regions of interest into a uniform size that corresponds to the original image. Finally, object classification, bounding box regression, key point detection, and mask generation are performed on the obtained regions of interest.

[0121] The Region of Interest Alignment (ROIAlign) layer uses a bilinear interpolation algorithm, which introduces no error throughout the process. This means that the pixels in the original image and the pixels in the feature map are perfectly aligned without any deviation, which not only improves the accuracy of detection but also facilitates instance segmentation.

[0122] The region of interest alignment layer connects to the branches in the head network of the Mask-RCNN network structure, which are, from top to bottom, the Category branch, the Coordinates branch, the Mask branch, and the Key points branch.

[0123] The Category branch is used for object classification, and its output is the category of each object in the target fish image; the Coordinates branch is used for bounding box regression, and its output is a rectangle that can select each object in the target fish image; the Mask branch is used for mask prediction, and its output is the specific outline of each object in the target fish image; the Key points branch is used for key point detection, and its output is the key points of each object in the target fish image, where the target object is a complete, unoccluded fish.

[0124] CBAM is an attention mechanism module that focuses on local information in a feature image. It enhances the attention given to useful regions by assigning weights to the feature image in both space and channels.

[0125] Figure 6 This is a structural diagram of the CBAM attention mechanism module provided by the present invention, as shown below. Figure 6 As shown, CBAM includes two modules: channel attention module and spatial attention module.

[0126] The channel attention module increases the weights of effective channels and decreases the weights of ineffective channels. First, the pixel values ​​of the first feature image F are input, and global average pooling and maximum pooling are performed to obtain two channel information values. These are then fed into a two-layer shared neural network for backpropagation training, resulting in two feature images. Finally, the two feature images are superimposed and input into a sigmoid activation function to obtain the weight coefficients M. c Finally, the weighting coefficient M c Multiplying the first feature image F by the second feature image F′ after scaling by the channel attention module yields the second feature image F′.

[0127] The spatial attention module assigns different weights to different regions of the feature image spatially, thereby increasing the weight of useful regions and decreasing the weight of useless regions. The spatial attention module takes the pixel values ​​of the second feature image F′ as input, and performs maximum pooling and average pooling to obtain two feature images. These two feature images are then convolved. The convolved feature image is then input into a sigmoid activation function to obtain the weight coefficients M. s Finally, the weighting coefficient M s Multiply by the second feature image F′ to obtain the scaled target feature image.

[0128] According to the underwater fish size measurement method provided by the present invention, the contour extraction model has been improved by incorporating an attention mechanism module, which speeds up data processing and improves the accuracy of the results.

[0129] Optionally, obtaining the body length curve and body width curve of any fish based on its key point data and contour data includes:

[0130] Based on the intrinsic and extrinsic parameter matrices of the target camera, binocular matching is performed on the contour data to obtain the transformation relationship between the target fish group image and the world coordinate system;

[0131] Based on the transformation relationship, coordinate transformation is performed on each key point data to obtain the world coordinates of each key point; the target fish school image is acquired based on the target camera;

[0132] Based on all world coordinates of any given fish, fit the body length curve and body width curve of any given fish.

[0133] Based on the contour data output by the contour extraction model, a binocular matching algorithm (Semi-Global Block Matching, SGBM) is used to perform binocular matching within the fish body contour. This avoids the complex and computationally intensive nature of global matching and yields a local binocular depth map, which provides the true three-dimensional coordinates of key points.

[0134] The formula for converting from coordinates to world coordinates is as follows:

[0135]

[0136] Where S is the scale factor, used for ease of calculation; (u, v) are the image pixel coordinates in the distortion-free pixel coordinate system; (x w y w , z w () are coordinates in the world coordinate system; pixel coordinates (u, v) are obtained through the intrinsic parameter matrix A. in The operations on the extrinsic matrices R and T yield the coordinates (x, y) in the world coordinate system. w y w , z w ).

[0137] According to the underwater fish size measurement method provided by the present invention, the size measurement is performed by fitting a true spatial curve through the three-dimensional spatial coordinates of key points. The entire measurement process can be automatically realized without human intervention.

[0138] Optionally, based on the body length curve and the body width curve, the body size data of any fish is determined, including:

[0139] The body length curve is constructed and solved to obtain the body length data of any fish, and the body width curve is constructed and solved to obtain the body width data of any fish.

[0140] The body shape data is determined based on the body length and body width data.

[0141] Spatial curve fitting is performed on the three-dimensional spatial coordinates of key points, where the key points include the head A(x) A y A , z A ), midpoint of tail D(x) D y D , z D ), dorsal fin front end B0(x B0 y B0 , z B0 ), pelvic fin B1 (x B1 y B1 , z B1 ), dorsal fin tail end C0, anal fin C1, fit two spatial curves L1 and L2, L1 passes through the head A, the front end of the dorsal fin and the midpoint B of the pelvic fin of the fish body (x B y B , z B ), the caudal tip of the dorsal fin and the midpoint of the anal fin C(x) C y C , z CL1 is the midpoint of the tail (D), L2 passes through the front end of the dorsal fin (B0), the front end of the dorsal fin, the midpoint of the pelvic fin (B), and the pelvic fin (B1). The length of L1 represents the body length of the fish, and the length of L2 represents the body width of the fish.

[0142] For curve L1, construct the spatial plane equation Γ1: A1x + B1y + C1z + D1 = 0 (C ≠ 0), and transform it into... Written as z = a0x + b0y + c0, let:

[0143]

[0144]

[0145]

[0146] According to the solution of the normal equation system x = (H T H) -1 H T b, obtain parameters A1, B1, C1.

[0147] Project points A, B, C, and D onto plane Γ1 to obtain the coordinates of points A′, B′, C′, and D′. Define curve L1: y = a1 + b1x + c1x 2 +d1x 3 The parameters a1, b1, c1, and d1 are obtained by fitting the coordinates of points A′, B′, C′, and D′.

[0148] The length of curve L1 is S1, which serves as an estimate of the fish's body length, i.e., body length data. Specifically:

[0149]

[0150] Similarly, by constructing and solving curve L2, the length S2 of curve L2 is obtained as the estimated value of fish body width, i.e., body width data.

[0151] According to the underwater fish body size measurement method provided by the present invention, by constructing and solving the body length curve and body width curve of the fish, the spatial posture of the fish can be effectively fitted, and relatively accurate body size data of the fish can be obtained.

[0152] Figure 7 This is the fourth flowchart illustrating the underwater fish size measurement method provided by this invention, as shown below. Figure 7 As shown, it includes:

[0153] First, collect underwater videos of fish swimming;

[0154] Secondly, the calibration data of the binocular camera was collected, and the camera intrinsic parameter matrix, extrinsic parameter matrix and distortion coefficient were obtained by Zhang's calibration method. The distortion of the underwater fish school video was corrected to obtain the corrected fish school video, and multiple sample fish school images were extracted from the corrected fish school video.

[0155] Then, data was labeled for each sample fish image to construct a dataset;

[0156] Next, the improved Mask-RCNN was used to perform contour segmentation and keypoint detection on the target fish image;

[0157] Subsequently, binocular matching was performed on the target fish image to obtain the pixel coordinates of key points;

[0158] Next, the pixel coordinates are transformed to obtain world coordinates;

[0159] Finally, spatial curve fitting is performed on the key points, the length of the spatial curve is calculated, and the estimated body size data of the fish is obtained.

[0160] The underwater fish body measuring device provided by the present invention is described below. The underwater fish body measuring device described below can be referred to in correspondence with the underwater fish body size measuring method described above.

[0161] Figure 8 This is a schematic diagram of the underwater fish body measuring device provided by the present invention, as shown below. Figure 8 As shown, it includes:

[0162] The first acquisition module 801 is used to input the target fish group image into the contour extraction model and acquire the key point data and contour data of each fish in the target fish group image output by the contour extraction model.

[0163] The second acquisition module 802 is used to acquire the body length curve and body width curve of any fish based on the key point data and contour data of any fish.

[0164] The determination module 803 is used to determine the body shape data of any fish based on the body length curve and the body width curve; the contour extraction model is obtained after training on sample fish images with key point data labels and contour data labels.

[0165] First, the first acquisition module 801 inputs the target fish group image into the contour extraction model to acquire the key point data and contour data of each fish in the target fish group image output by the contour extraction model.

[0166] The underwater fish video can be taken by an image acquisition device of the fish school under test. The image acquisition device can be a three-dimensional sensor such as a binocular camera, a structured light camera, or a three-dimensional (3D) depth camera. In the following embodiments, the underwater fish video taken by a binocular camera will be used as an example for illustration, which is not considered as a limitation on the scope of protection of the present invention.

[0167] The underwater fish school video is corrected to obtain a corrected fish school video. The corrected fish school video is then cropped to obtain a target fish school image. The target fish school image includes at least one complete, unobstructed image of the fish to be tested. The target fish school image can be a left-eye image or a right-eye image. In the following embodiments, the processing of the left-eye image is used as an example for illustration, which is not considered as a limitation on the scope of protection of the present invention.

[0168] The target fish school image is input into the contour extraction model. The model performs contour segmentation and keypoint labeling on each complete, unoccluded fish in the target fish school image, and outputs complete keypoint data and contour data for each unoccluded fish. Keypoints are used to mark the body shape of the fish, and all keypoints are located on the contour of the fish.

[0169] Furthermore, the second acquisition module 802 acquires the body length curve and body width curve of any fish based on the key point data and contour data of any fish.

[0170] The pixel coordinates corresponding to the key point data and contour data on the image are transformed to the world coordinate system to obtain the outline and key point data of the fish in the world coordinate system.

[0171] like Figure 2 As shown, the key points of any fish include point A on the head, point B0 at the front of the dorsal fin, point B1 on the pelvic fin, point C0 at the tail end of the dorsal fin, point C1 on the anal fin, and point D at the midpoint of the tail.

[0172] By fitting the coordinates of key points in the world coordinate system to spatial curves, we can obtain the body length curve L1 and the body width curve L2.

[0173] Among them, the body length curve L1 passes through the head A, the anterior end of the dorsal fin and the midpoint B of the pelvic fin, the caudal end of the dorsal fin and the midpoint C of the anal fin, and the midpoint D of the tail; the body width curve L2 passes through the anterior end B0 of the dorsal fin, the anterior end of the dorsal fin and the midpoint B of the pelvic fin, and the pelvic fin B1.

[0174] Furthermore, the determination module 803 determines the body shape data of any fish based on the body length curve and the body width curve; the contour extraction model is obtained after training on sample fish group images with key point data labels and contour data labels.

[0175] For the body length curve L1 and the body width curve L2, we construct and solve the spatial plane equations respectively, and we can obtain that the length of the body length curve L1 is the body length S1 of the fish, and the length of the body width curve L2 is the body width S2 of the fish.

[0176] Among these methods, multiple sample fish images can be extracted from the corrected fish swarm video, and the contour extraction model can be constructed based on an improved Mask Region-based Convolutional Neural Network (Mask-RCNN).

[0177] Labelme software was used to annotate each sample fish image, marking the complete outline and key points of the unobstructed fish bodies, forming key point data labels and outline data labels. After annotation, the data was exported as JSON format and converted to COCO dataset format to obtain a dataset composed of multiple training samples.

[0178] The training samples in the dataset were divided into training and test sets in an 8:2 ratio. The improved Mask-RCNN model was trained and validated using the training and test sets respectively, resulting in the contour extraction model.

[0179] like Figure 3 As shown, the training methods include:

[0180] First, extract sample fish images from the corrected fish school video;

[0181] Secondly, the fish in the sample fish group images are classified and labeled to create training samples;

[0182] Subsequently, the training samples are input into the improved Mask-RCNN model for training to obtain the contour extraction model.

[0183] The underwater fish body measuring device provided by the present invention extracts key points of fish in an underwater fish school and then fits the spatial posture of the fish body, so as to realize the size measurement of the fish body swimming freely underwater without contacting the fish body.

[0184] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logic instructions in the memory 930 to execute an underwater fish size measurement method, which includes: inputting a target fish group image into a contour extraction model to obtain key point data and contour data of each fish in the target fish group image output by the contour extraction model; obtaining the body length curve and body width curve of any fish based on the key point data and contour data of any fish; and determining the body shape data of any fish based on the body length curve and body width curve. The contour extraction model is obtained after training on sample fish group images with key point data labels and contour data labels.

[0185] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the underwater fish size measurement method provided by the above methods. The method includes: inputting a target fish group image into a contour extraction model to obtain key point data and contour data of each fish in the target fish group image output by the contour extraction model; obtaining the body length curve and body width curve of any fish based on the key point data and contour data of any fish; and determining the body shape data of any fish based on the body length curve and body width curve. The contour extraction model is obtained after training on sample fish group images with key point data labels and contour data labels.

[0187] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the underwater fish size measurement method provided by the methods described above. This method includes: inputting a target fish group image into a contour extraction model to obtain keypoint data and contour data of each fish in the target fish group image output by the contour extraction model; obtaining the body length curve and body width curve of any fish based on the keypoint data and contour data of any fish; and determining the body shape data of any fish based on the body length curve and body width curve. The contour extraction model is obtained after training on sample fish group images with keypoint data labels and contour data labels.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of underwater fish size measurement, characterized by, include: The target fish group image is input into the contour extraction model to obtain the key point data and contour data of each fish in the target fish group image output by the contour extraction model. Based on the keypoint and contour data of any fish, the body length and body width curves of that fish are obtained, including: performing binocular matching on the contour data based on the intrinsic and extrinsic matrices of the target camera to obtain the transformation relationship from the target fish image to the world coordinate system; performing coordinate transformation on each keypoint data based on the transformation relationship to obtain the world coordinates of each keypoint; the target fish image is acquired based on the target camera; and fitting the body length and body width curves of any fish based on all the world coordinates of that fish. Based on the body length curve and the body width curve, the body shape data of any fish is determined; the contour extraction model is obtained after training on sample fish images with key point data labels and contour data labels; the contour extraction model sequentially includes: a backbone network, a region proposal network, a nonmaximum suppression module, and a region of interest alignment layer, including: The backbone network is used to extract features from the target fish image to obtain a target feature image, which has multiple anchor boxes. The region proposal network is used to perform binary classification and bounding box regression on the target feature image to determine the region of interest. The nonmaximum suppression module is used to filter the region of interest to obtain the target region of interest; The region of interest alignment layer is used to align the feature image with the target region of interest to obtain a region of interest feature map; The key point data and contour data of each fish are obtained based on the interest feature map.

2. The method of underwater fish size measurement according to claim 1, characterized in that, The backbone network includes ResNet50, a pyramid network, and an attention mechanism module, which includes a channel attention module and a spatial attention module. The ResNet50 is used to perform preliminary feature extraction on the target fish image to obtain an initial feature image; The pyramid network is used to perform feature fusion on the initial feature image to obtain a first feature image; The channel attention module is used to extract features from the first feature image to obtain the second feature image; The spatial attention module is used to extract features from the second feature image to obtain the target feature image.

3. The method of claim 1, wherein, Based on the body length curve and the body width curve, the body size data of any fish is determined, including: The body length curve is constructed and solved to obtain the body length data of any fish, and the body width curve is constructed and solved to obtain the body width data of any fish. The body shape data is determined based on the body length and body width data.

4. The method for measuring the body size of underwater fish according to claim 1, characterized in that, Before inputting the target fish image into the contour extraction model, the method further includes: Acquire calibration data sets from multiple azimuths; The target camera is calibrated using each calibration data set to obtain the intrinsic parameter matrix, the extrinsic parameter matrix, and the distortion coefficients; Using the extrinsic parameter matrix and the distortion coefficients, the underwater fish school video is corrected to obtain a corrected fish school video, thereby determining the target fish school image. The underwater fish school video is acquired based on the target camera.

5. An underwater fish body measuring device characterized by comprising: include: The first acquisition module is used to input the target fish group image into the contour extraction model and acquire the key point data and contour data of each fish in the target fish group image output by the contour extraction model. The second acquisition module is used to acquire the body length curve and body width curve of any fish based on key point data and contour data of any fish. This includes: performing binocular matching on the contour data based on the intrinsic and extrinsic parameter matrices of the target camera to acquire the transformation relationship between the target fish image and the world coordinate system; performing coordinate transformation on each key point data based on the transformation relationship to acquire the world coordinates of each key point; the target fish image is acquired based on the target camera; and fitting the body length curve and body width curve of any fish based on all world coordinates of any fish. The determination module is used to determine the body shape data of any fish based on the body length curve and the body width curve; the contour extraction model is obtained after training on sample fish images with key point data labels and contour data labels; the contour extraction model sequentially includes: a backbone network, a region proposal network, a nonmaximum suppression module, and a region of interest alignment layer, including: The backbone network is used to extract features from the target fish image to obtain a target feature image, which has multiple anchor boxes. The region proposal network is used to perform binary classification and bounding box regression on the target feature image to determine the region of interest. The nonmaximum suppression module is used to filter the region of interest to obtain the target region of interest; The region of interest alignment layer is used to align the feature image with the target region of interest to obtain a region of interest feature map; The key point data and contour data of each fish are obtained based on the interest feature map.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the underwater fish size measurement method as described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the underwater fish size measurement method as described in any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the underwater fish size measurement method as described in any one of claims 1-4.

Images