Non-contact fish weight measurement method, device, electronic equipment and medium

Abstract

The application provides a non-contact fish weight measurement method, device, electronic equipment and medium. The method comprises: image registration is performed on a color image and a depth image of a fish to be measured to obtain a target image; based on a Mask R-CNN deep learning algorithm, image segmentation is performed on the target image to obtain a mask image corresponding to the fish to be measured; based on a three-dimensional surface area of the fish to be measured and a weight fitting model, the weight of the fish to be measured is determined, the three-dimensional surface area is determined based on point cloud data of a surface of the fish to be measured, the point cloud data is determined based on the target image and the mask image, and the weight fitting model is obtained by polynomial fitting of three-dimensional surface areas of fish samples and measured weights of the fish samples. The non-contact fish weight measurement method provided by the application can realize non-contact measurement of the weight of underwater fish while ensuring prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a non-contact method, device, electronic device, and medium for measuring fish weight. Background Technology

[0002] Fish weight is not only a crucial indicator of their economic value but also one of the most direct characteristics for breeding purposes, making its accurate measurement essential. To date, manual weighing has been the most commonly used estimation method; however, this method is not only time-consuming and labor-intensive but also invasive, resulting in inconsistent measurement results. It is worth noting that because fish live in aquatic environments, visibility, lighting conditions, and stability are often difficult to control, and fish themselves are highly sensitive and easily stressed. Traditional manual contact measurements can easily cause physical harm to fish, potentially leading to disease and death, thus affecting their normal growth and development. Therefore, exploring and developing non-contact weight measurement methods is particularly urgent and important, ensuring fish welfare in a more scientific and humane way, while also providing more accurate data support for scientific research and industrial development.

[0003] Non-invasive weight estimation methods are widely used in fish farming and can generally be divided into three categories: machine vision methods, acoustic echo detection methods, and environmental DNA methods. When using acoustic echo detection for weight measurement, careful consideration must be given to the size and purpose of the water area, as well as the need to estimate the total population weight. Environmental DNA methods require processing and analyzing large amounts of sampled data, and the results may be affected by other environmental factors. Machine vision methods are applicable to various farming scenarios and can be used to estimate the weight of individual fish; however, the current accuracy of machine vision methods in predicting the weight of individual fish is not high. Summary of the Invention

[0004] This invention provides a non-contact method, device, electronic device, and medium for measuring fish weight, in order to solve the problems of numerous limitations and low accuracy in existing fish weight measurement technologies.

[0005] This invention provides a non-contact method for measuring fish weight, comprising:

[0006] Image registration is performed between the color image and the depth image of the fish to be measured to obtain the target image;

[0007] The target image is input into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured.

[0008] Based on the three-dimensional surface area and weight fitting model of the fish to be measured, the weight of the fish to be measured is determined. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0009] In some embodiments, the Mask R-CNN model includes a backbone network, a Region Proposal Network (RPN), a Region of Interest (RoIAlign) layer, a class prediction branch, a bounding box prediction branch, and a mask prediction branch.

[0010] The Mask R-CNN model is trained in the following way:

[0011] The labeled fish sample image is input into the backbone network to obtain the first feature map;

[0012] The first feature map is input into the Region Proposal Network (RPN) to determine multiple candidate regions;

[0013] The multiple candidate regions are input into the RoIAlign layer of the region of interest to obtain the second feature map;

[0014] The second feature map is input to the category prediction branch, the bounding box prediction branch, and the mask prediction branch respectively to obtain the category prediction result, bounding box prediction result, and mask prediction image corresponding to the fish body sample image;

[0015] The MaskR-CNN model is trained based on the category prediction results, the bounding box prediction results, and the mask prediction image.

[0016] In some embodiments, training the Mask R-CNN model based on the category prediction result, the bounding box prediction result, and the mask prediction image includes:

[0017] Based on the category prediction results, determine the classification loss of the Region Proposal Network (RPN) and the classification loss of the Region of Interest (RoIAlign) layer;

[0018] Based on the bounding box prediction results, the regression loss of the Region Proposal Network (RPN) and the regression loss of the Region of Interest (RoIAlign) layer are determined.

[0019] Based on the mask prediction image, calculate the mask loss of the RoIAlign layer in the region of interest;

[0020] The Mask R-CNN model is trained based on the classification and regression losses of the Region Proposal Network (RPN), as well as the classification, regression, and mask losses of the Region of Interest (RoIAlign) layer.

[0021] In some embodiments, after inputting the target image into a Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured, the method further includes:

[0022] The target image and the mask image are fused to obtain a depth information map of the fish to be measured;

[0023] Based on the depth information map, the three-dimensional coordinates of the fish to be measured are determined;

[0024] The point cloud data is generated based on the three-dimensional coordinates of the fish to be measured.

[0025] In some embodiments, after generating the point cloud data based on the three-dimensional coordinates of the fish to be measured, the method further includes:

[0026] The point cloud data is converted into Alpha Shapes surfaces;

[0027] Based on the Alpha Shapes surface, multiple triangular meshes are generated, which are used to characterize the surface shape of the fish body to be measured corresponding to the Alpha Shapes surface.

[0028] The area of ​​the multiple triangular meshes is calculated to determine the three-dimensional surface area of ​​the fish to be measured.

[0029] In some embodiments, the weight fitting model is represented by the following formula:

[0030] y = -3.49 × 10 -14 x 3 +3.749×10 -9 x 2 -8.955×10 -5 x+0.8956;

[0031] Where y represents the weight of the fish to be measured, and x represents the three-dimensional surface area of ​​the fish to be measured.

[0032] The present invention also provides a non-contact fish weight measuring device, comprising:

[0033] The image registration module is used to register the color image and depth image of the fish to be measured to obtain the target image.

[0034] The image segmentation module is used to input the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured.

[0035] The weight determination module is used to determine the weight of the fish to be measured based on the three-dimensional surface area and weight fitting model of the fish to be measured. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the non-contact fish weight measurement method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the non-contact fish weight measurement method as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the non-contact fish weight measurement methods described above.

[0039] The non-contact fish weight measurement method, device, electronic equipment, and medium provided by this invention obtain a target image by image registration using depth and color images, and successfully achieve precise segmentation of the fish to be measured using deep learning technology. Furthermore, by calculating the three-dimensional surface area of ​​the fish to be measured and establishing a reliable three-dimensional surface area and weight fitting model of the actual measured weight, the weight of the fish to be measured can be accurately predicted simply by taking images above the water surface, while ensuring prediction accuracy. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the non-contact fish weight measurement method provided by the present invention.

[0042] Figure 2This is a schematic diagram of the model parameter update process for the non-contact fish weight measurement method provided by the present invention;

[0043] Figure 3 This is a schematic diagram of a mask image for the non-contact fish weight measurement method provided by the present invention.

[0044] Figure 4 This is a three-dimensional schematic diagram of the non-contact fish weight measurement method provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the Alpha Shapes surface model of the non-contact fish weight measurement method provided by the present invention;

[0046] Figure 6 This is a schematic diagram of the non-contact fish weight measuring device provided by the present invention;

[0047] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] The following is combined with Figures 1-7 This invention describes a non-contact fish weight measurement method, apparatus, electronic device, and medium.

[0050] The non-contact fish weight measurement method provided by this invention can be implemented by an electronic device, a component within an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, handheld computers, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This invention does not impose specific limitations.

[0051] The technical solution of this invention will be described in detail below using the example of a computer executing the non-contact fish weight measurement method provided by this invention.

[0052] Figure 1 This is a flowchart illustrating the non-contact fish weight measurement method provided by the present invention.

[0053] Reference Figure 1 The non-contact fish weight measurement method provided by the present invention includes:

[0054] Step 110: Perform image registration between the color image and the depth image of the fish to be measured to obtain the target image;

[0055] Step 120: Input the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured;

[0056] Step 130: Determine the weight of the fish to be measured based on the fitting model of the three-dimensional surface area and weight of the fish to be measured. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0057] In practice, the fish to be measured is an individual fish whose weight needs to be measured. This invention determines the fish species based on the ability of the camera to capture a complete image of the fish, without making any specific limitations.

[0058] The color image of the fish to be measured can be acquired by an RGB camera, and the depth image of the fish to be measured can be acquired by a depth camera. The purpose of image registration is to fuse the depth image and the color image together, that is, to transform the image coordinate system of the depth image to the image coordinate system of the color image, while maintaining a one-to-one correspondence between the pixels in the color image and the pixels in the depth image.

[0059] Before image registration, the depth camera is calibrated underwater using Intel's official calibration tool, Dynamic Calibrator, to obtain its intrinsic and extrinsic parameters. These parameters are then used to match the depth map and color image, aligning the depth and color information in the same coordinate system. In this embodiment, the camera calibration results are shown below:

[0060] Depth camera intrinsic K depth for:

[0061]

[0062] RGB camera internal parameters K rgb for:

[0063]

[0064] Rotation matrix R from RGB camera coordinate system to left infrared camera coordinate system rgb With translation vector T rgb They are respectively:

[0065]

[0066]

[0067] It should be noted that the depth camera mainly includes a left infrared camera, a right infrared camera, a laser projection module, and a depth computing processor.

[0068] The depth image registration process involves setting the left infrared camera (depth camera) as the origin of the world coordinate system. The registration process between the depth image and the color image mainly consists of four steps:

[0069] (1) Restore the pixels of the depth image to the depth camera coordinate system. Based on any pixel P in the depth image... depth coordinates (u) d ,v d ) and the depth value Z of that point depth (Pixel value), calculate the coordinates P of the corresponding spatial point in the depth camera coordinate system. C_depth for:

[0070] Among them, K depth This is the intrinsic parameter of the depth camera.

[0071] (2) Restore the depth points from the depth camera coordinate system to the world coordinate system. Since the depth camera is the origin of the world coordinate system, then P depth Coordinate P in the world coordinate system W for:

[0072] P W =P depth ;

[0073] (3) Transform the depth points in the world coordinate system to the RGB camera coordinate system. Based on the RGB camera's extrinsic parameters R... rgb Translation vector T rgb By transforming the world coordinate system and the RGB camera coordinate system, P is calculated. depth The coordinates P in the RGB camera coordinate system C_rgb for:

[0074]

[0075] (4) Map the depth points of the RGB camera coordinate system onto the plane of Z=1, which corresponds to the pixels of the color image. Based on the RGB camera intrinsic parameter matrix K... rgb The pixel P can be calculated using equation (1). depth Projection point P in the RGB camera pixel plane coordinate system rgb (u rgb ,v rgb ), which is related to pixel P in the color image. rgb (u r ,v r Corresponding to.

[0076]

[0077] Among them, Z rgb For P C_rgb The Z-axis coordinate, its value is equal to Z depth .

[0078] The calibration and matching results were verified by measuring the underwater calibration plate. The average measurement result had an error of less than 0.1 mm from the true value.

[0079] After image registration is completed, the registered target image can be obtained. The trained Mask-RCNN model is then used to segment the acquired target image, resulting in a mask image containing only the fish to be measured.

[0080] Then, based on the depth camera intrinsic parameters calibrated above, the three-dimensional coordinates of each point in the target image are calculated in combination with the depth information, and point cloud data of the surface of the fish to be measured is generated.

[0081] The total area of ​​the entire point cloud surface, i.e. the three-dimensional surface area of ​​the fish body to be measured, can be determined based on the point cloud data of the fish body to be measured.

[0082] The obtained three-dimensional surface area and the actual measured weight of the fish are fitted using a polynomial method to establish a weight fitting model. Using this pre-established weight fitting model, non-contact measurement of the weight of underwater fish can be achieved.

[0083] The non-contact fish weight measurement method provided by this invention obtains a target image by image registration of depth and color images, and successfully achieves accurate segmentation of the fish to be measured by using deep learning technology. Furthermore, by calculating the three-dimensional surface area of ​​the fish to be measured and establishing a reliable three-dimensional surface area and weight fitting model of the actual measured weight, the weight of the fish to be measured can be accurately predicted simply by taking images above the water surface, while ensuring prediction accuracy.

[0084] In some embodiments, the Mask R-CNN model includes a backbone network, a region proposal network (RPN), a region of interest (RoIAlign) layer, a class prediction branch, a bounding box prediction branch, and a mask prediction branch.

[0085] The Mask R-CNN model is trained in the following way:

[0086] The labeled fish sample images are input into the backbone network to obtain the first feature map;

[0087] The first feature map is input into the Region Proposal Network (RPN) to identify multiple candidate regions.

[0088] Multiple candidate regions are input into the RoIAlign layer of the region of interest to obtain the second feature map;

[0089] The second feature map is input into the category prediction branch, the bounding box prediction branch, and the mask prediction branch respectively to obtain the category prediction result, bounding box prediction result, and mask prediction image corresponding to the fish body sample image;

[0090] The Mask R-CNN model is trained based on the category prediction results, bounding box prediction results, and mask prediction images.

[0091] The following example, using sturgeon as a fish sample image, illustrates the training and application process of the Mask-RCNN model.

[0092] First, input sturgeon sample images. Images of 50 sturgeon in the water can be collected, with 20 images of each sturgeon, for a total of 1000 images. The weight of each sturgeon is measured and recorded during collection. After collection, the images are labeled using the Labelme annotation tool, marking and labeling the fish bodies. Each completed sturgeon sample image yields a labeled fish sample image. The annotation information includes the location of the annotation points and their corresponding labels, and is saved to a JSON file corresponding to the original image. The final result is a dataset of 1000 images, which will be used to train a region-based deep learning object detection algorithm (Mask Region-based Convolutional Neural Network, Mask-RCNN) model.

[0093] Of the fish images used, 80% were used for the training set, 10% for the validation set, and 10% for the test set. The dataset was then fed into the Mask-RCNN model for training.

[0094] During training, model weights are initialized using a pre-trained model (trained on the COCO dataset). This pre-trained model provides a good starting point, accelerating model convergence and improving the final performance in segmenting sturgeon sample images. The model parameter update process is as follows: Figure 2 As shown.

[0095] First, inputting a sturgeon sample image, the model performs forward propagation, sequentially passing through the backbone network, the Region Proposal Network (RPN), the Region of Interest (ROI) Align layer, and the class prediction branch, bounding box prediction branch, and mask prediction branch, to obtain the first feature map, candidate region, second feature map, class prediction result, bounding box prediction result, and mask prediction image, respectively.

[0096] The category prediction branch, or classification (Cls) branch, is used to determine the category information of the target object in the sample image. For example, it determines that the target object in the fish sample image is a sturgeon. The bounding box prediction branch, or regression (Reg) branch, is used to determine the location information of the target object in the image based on the bounding box prediction results. The mask prediction branch, or mask branch, is used to obtain the mask prediction image corresponding to the target object.

[0097] In some embodiments, a MaskR-CNN model is trained based on the category prediction results, bounding box prediction results, and mask prediction images, including:

[0098] Based on the category prediction results, determine the classification loss of the Region Proposal Network (RPN) and the classification loss of the Region of Interest (RoIAlign) layer;

[0099] Based on the bounding box prediction results, the regression loss of the Region Proposal Network (RPN) and the regression loss of the Region of Interest (RoIAlign) layer are determined.

[0100] Based on the mask prediction image, calculate the mask loss of the RoIAlign layer in the region of interest;

[0101] The Mask R-CNN model is trained based on the classification and regression losses of the Region Proposal Network (RPN), as well as the classification, regression, and mask losses of the Region of Interest (RoIAlign) layer.

[0102] In this step, the model calculates three parts of the loss based on these predictions and the true labels: first, the classification and regression losses of the Region Proposal Network (RPN), which reflect the quality of the sturgeon region proposals; second, the classification and regression losses of the Region of Interest (RoIAlign) layer, which reflect the accuracy of sturgeon target detection; and third, the mask loss of the RoIAlign layer, which reflects the accuracy of sturgeon instance segmentation. Next, the model performs backpropagation, calculating the gradients of all parameters based on the losses for each part.

[0103] This process is repeated multiple times across the entire training set, each time bringing the parameters closer to the values ​​that minimize the loss. In this way, the model gradually learns how to detect and segment objects from input images. Training can be stopped when the model's performance on the validation set no longer improves.

[0104] For each pixel in each predicted mask image, a binary cross-entropy loss is calculated between its predicted value (a probability between 0 and 1) and its true value (0 or 1). These losses are then averaged, resulting in a value between 0 and 1. A smaller loss indicates a stronger ability to predict pixel-level masks and a better understanding of object shape. The final model achieves a loss of approximately 0.035 when predicting pixel-level segmentation masks.

[0105] In some embodiments, after inputting the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured, the non-contact fish weight measurement method further includes:

[0106] The target image and the mask image are fused to obtain a depth information map of the fish to be measured;

[0107] Based on the depth information map, the three-dimensional coordinates of the fish to be measured are determined;

[0108] Point cloud data is generated based on the three-dimensional coordinates of the fish to be measured.

[0109] In practice, we will use a sturgeon as an example to illustrate this step.

[0110] The Mask-RCNN model trained as described above can be used to segment images of sturgeon, producing mask images containing only sturgeon, such as... Figure 3 As shown.

[0111] from Figure 3 As can be seen, the segmented image of the sturgeon is a binary image, while calculating the three-dimensional surface area requires the simultaneous use of the mask image and the depth image. Figure 4 This is a depth information map obtained by fusing the depth image and the mask image.

[0112] The calculation of three-dimensional surface area is implemented using the Open3D library, and the process is as follows:

[0113] (1) First, based on the depth camera intrinsic parameters calibrated above, the three-dimensional coordinates of each point in the image are calculated in combination with the depth information, and point cloud data of the sturgeon surface is generated.

[0114] In some embodiments, after generating point cloud data based on the three-dimensional coordinates of the fish to be measured, the non-contact fish weight measurement method further includes:

[0115] Convert point cloud data into Alpha Shapes surfaces;

[0116] Based on the Alpha Shapes surface, multiple triangular meshes are generated. These triangular meshes are used to characterize the surface shape of the fish body to be measured corresponding to the AlphaShapes surface.

[0117] The area of ​​multiple triangular meshes is calculated to determine the three-dimensional surface area of ​​the fish to be measured.

[0118] (2) To convert point cloud data into a surface using the Alpha Shapes algorithm, a triangular mesh model needs to be established based on the point cloud data, and the circumcircle radius R of each triangle needs to be calculated. For a given alpha value, a surface is formed by connecting the regions formed by triangles enclosed by circumcircles with radii smaller than alpha. This surface is the Alpha Shapes, as shown below. Figure 5 As shown.

[0119] The larger the alpha value, the smoother the resulting Alpha Shapes surface, the more triangles it contains, and the closer the surface is to the shape of the original point cloud. An Alpha shape is an infinitely connected region whose shape can adapt to the input point cloud and reproduce its surface shape as accurately as possible.

[0120] (3) Construct a triangular mesh using the generated Alpha-shaped surface.

[0121] (4) Iterate through each triangle in the "mesh" and calculate the area of ​​the triangle corresponding to each triangular mesh. Specifically, first obtain the coordinates of the three vertices of the triangle, and then calculate the vectors of the adjacent sides. and Then calculate the area of ​​the triangle using the formula for the cross product of vectors.

[0122]

[0123] Finally, the areas of all the triangular meshes are added together to obtain the total area of ​​the entire point cloud surface, which is the three-dimensional surface area of ​​the sturgeon.

[0124] In some embodiments, the weight fitting model is represented by the following equation:

[0125] y = -3.49 × 10 -14 x 3 +3.749×10 -9 x 2 -8.955×10 -5 x+0.8956 (2)

[0126] Where y represents the weight of the fish to be measured, and x represents the three-dimensional surface area of ​​the fish to be measured.

[0127] In practice, the three-dimensional surface area obtained by the above method and the actual measured weight of the fish are fitted by a polynomial to complete the non-contact measurement of the weight of underwater fish. The weight fitting model is shown in equation (2).

[0128] It is understandable that the weight fitting model can be determined based on the actual situation, and no specific limitations are made here.

[0129] The non-contact fish weight measurement method used in this invention calculates the fish weight, and the weight measured by an electronic scale is compared with the weight measured by an electronic scale. A quantitative analysis using the correlation coefficient yields a correlation coefficient of 0.973. This predictive accuracy is sufficient to replace manual measurement.

[0130] The non-contact fish weight measurement method provided by this invention constructs a fish dataset through underwater camera calibration and successfully achieves accurate segmentation of individual fish using deep learning technology. Furthermore, by calculating the surface area of ​​the fish and establishing a reliable surface area-weight fitting model, the weight of fish can be accurately predicted simply by taking images above the water surface, while ensuring prediction accuracy.

[0131] The non-contact fish weight measuring device provided by the present invention will be described below. The non-contact fish weight measuring device described below can be referred to in correspondence with the non-contact fish weight measuring method described above.

[0132] Figure 6 This is a schematic diagram of the non-contact fish weight measuring device provided by the present invention. (Refer to...) Figure 6 The non-contact fish weight measuring device provided by the present invention includes:

[0133] The image registration module 610 is used to perform image registration between the color image and the depth image of the fish to be measured, so as to obtain the target image;

[0134] The image segmentation module 620 is used to input the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish body to be measured.

[0135] The weight determination module 630 is used to determine the weight of the fish to be measured based on the three-dimensional surface area and weight fitting model of the fish to be measured. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0136] The non-contact fish weight measurement device provided by this invention obtains a target image by image registration of depth and color images, and successfully achieves accurate segmentation of the fish to be measured by using deep learning technology. Furthermore, by calculating the three-dimensional surface area of ​​the fish to be measured and establishing a reliable three-dimensional surface area and weight fitting model of the actual measured weight, the weight of the fish to be measured can be accurately predicted simply by taking images above the water surface, while ensuring prediction accuracy.

[0137] In some embodiments, the Mask R-CNN model includes a backbone network, a Region Proposal Network (RPN), a Region of Interest (RoIAlign) layer, a class prediction branch, a bounding box prediction branch, and a mask prediction branch.

[0138] The device further includes:

[0139] The model training module is used to input the labeled fish sample images into the backbone network to obtain the first feature map;

[0140] The first feature map is input into the Region Proposal Network (RPN) to determine multiple candidate regions;

[0141] The multiple candidate regions are input into the RoIAlign layer of the region of interest to obtain the second feature map;

[0142] The second feature map is input to the category prediction branch, the bounding box prediction branch, and the mask prediction branch respectively to obtain the category prediction result, bounding box prediction result, and mask prediction image corresponding to the fish body sample image;

[0143] The MaskR-CNN model is trained based on the category prediction results, the bounding box prediction results, and the mask prediction image.

[0144] In some embodiments, the model training module is specifically used for:

[0145] Based on the category prediction results, determine the classification loss of the Region Proposal Network (RPN) and the classification loss of the Region of Interest (RoIAlign) layer;

[0146] Based on the bounding box prediction results, the regression loss of the Region Proposal Network (RPN) and the regression loss of the Region of Interest (RoIAlign) layer are determined.

[0147] Based on the mask prediction image, calculate the mask loss of the RoIAlign layer in the region of interest;

[0148] The Mask R-CNN model is trained based on the classification and regression losses of the Region Proposal Network (RPN), as well as the classification, regression, and mask losses of the Region of Interest (RoIAlign) layer.

[0149] In some embodiments, the apparatus further includes:

[0150] The three-dimensional surface area acquisition module is used to fuse the target image and the mask image after inputting the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish body to be measured, thereby obtaining the depth information map of the fish body to be measured.

[0151] Based on the depth information map, the three-dimensional coordinates of the fish to be measured are determined;

[0152] The point cloud data is generated based on the three-dimensional coordinates of the fish to be measured.

[0153] In some embodiments, the three-dimensional surface area acquisition module is further configured to:

[0154] After generating the point cloud data based on the three-dimensional coordinates of the fish to be measured, the point cloud data is converted into an Alpha Shapes surface.

[0155] Based on the Alpha Shapes surface, multiple triangular meshes are generated, which are used to characterize the surface shape of the fish body to be measured corresponding to the Alpha Shapes surface.

[0156] The area of ​​the multiple triangular meshes is calculated to determine the three-dimensional surface area of ​​the fish to be measured.

[0157] In some embodiments, the weight fitting model is represented by the following formula:

[0158] y = -3.49 × 10 -14 x 3 +3.749×10 -9 x 2-8.955×10 -5 x+0.8956;

[0159] Where y represents the weight of the fish to be measured, and x represents the three-dimensional surface area of ​​the fish to be measured.

[0160] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a non-contact fish weight measurement method, which includes:

[0161] Image registration is performed between the color image and the depth image of the fish to be measured to obtain the target image;

[0162] The target image is input into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured.

[0163] Based on the three-dimensional surface area and weight fitting model of the fish to be measured, the weight of the fish to be measured is determined. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0164] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the non-contact fish weight measurement method provided by the above methods, the method comprising:

[0166] Image registration is performed between the color image and the depth image of the fish to be measured to obtain the target image;

[0167] The target image is input into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured.

[0168] Based on the three-dimensional surface area and weight fitting model of the fish to be measured, the weight of the fish to be measured is determined. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0169] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the non-contact fish weight measurement method provided by the methods described above, the method comprising:

[0170] Image registration is performed between the color image and the depth image of the fish to be measured to obtain the target image;

[0171] The target image is input into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured.

[0172] Based on the three-dimensional surface area and weight fitting model of the fish to be measured, the weight of the fish to be measured is determined. The three-dimensional surface area is determined based on the point cloud data of the surface of the fish to be measured. The point cloud data is determined based on the target image and the mask image. The weight fitting model is obtained by performing a polynomial fitting between the three-dimensional surface area of ​​the fish sample and the measured weight of the fish sample.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact method for measuring fish weight, characterized in that, include: Image registration is performed between the color image and the depth image of the fish to be measured to obtain the target image; The target image is input into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish to be measured; the target image and the mask image are fused to obtain the depth information map of the fish to be measured. Based on the depth information map, the three-dimensional coordinates of the fish to be measured are determined; Point cloud data is generated based on the three-dimensional coordinates of the fish to be measured; The point cloud data is converted into Alpha Shapes surfaces; Based on the Alpha Shapes surface, multiple triangular meshes are generated, which are used to characterize the surface shape of the fish body to be measured corresponding to the Alpha Shapes surface. The area of ​​the multiple triangular meshes is calculated to determine the three-dimensional surface area of ​​the fish to be measured; The weight fitting model is expressed by the following formula: ； Where y represents the weight of the fish to be measured, and x represents the three-dimensional surface area of ​​the fish to be measured; The weight of the fish to be measured is determined based on the fitting model of the three-dimensional surface area and weight of the fish to be measured.

2. The non-contact fish weight measurement method according to claim 1, characterized in that, The Mask R-CNN model includes a backbone network, a Region Proposal Network (RPN), a Region of Interest (RoIAlign) layer, a class prediction branch, a bounding box prediction branch, and a mask prediction branch. The Mask R-CNN model is trained in the following way: The labeled fish sample image is input into the backbone network to obtain the first feature map; The first feature map is input into the Region Proposal Network (RPN) to determine multiple candidate regions; The multiple candidate regions are input into the RoIAlign layer of the region of interest to obtain the second feature map; The second feature map is input to the category prediction branch, the bounding box prediction branch, and the mask prediction branch respectively to obtain the category prediction result, bounding box prediction result, and mask prediction image corresponding to the fish body sample image; The Mask R-CNN model is trained based on the category prediction results, the bounding box prediction results, and the mask prediction image.

3. The non-contact fish weight measurement method according to claim 2, characterized in that, The step of training the Mask R-CNN model based on the category prediction result, the bounding box prediction result, and the mask prediction image includes: Based on the category prediction results, determine the classification loss of the Region Proposal Network (RPN) and the classification loss of the Region of Interest (RoIAlign) layer; Based on the bounding box prediction results, the regression loss of the Region Proposal Network (RPN) and the regression loss of the Region of Interest (RoIAlign) layer are determined. Based on the mask prediction image, calculate the mask loss of the RoIAlign layer in the region of interest; The Mask R-CNN model is trained based on the classification and regression losses of the Region Proposal Network (RPN), as well as the classification, regression, and mask losses of the Region of Interest (RoIAlign) layer.

4. A non-contact fish weight measuring device, characterized in that, include: The image registration module is used to register the color image and depth image of the fish to be measured to obtain the target image. The image segmentation module is used to input the target image into the Mask R-CNN model for image segmentation to obtain the mask image corresponding to the fish body to be measured; and to fuse the target image and the mask image to obtain the depth information map of the fish body to be measured. Based on the depth information map, the three-dimensional coordinates of the fish to be measured are determined; Point cloud data is generated based on the three-dimensional coordinates of the fish to be measured; The point cloud data is converted into Alpha Shapes surfaces; Based on the Alpha Shapes surface, multiple triangular meshes are generated, which are used to characterize the surface shape of the fish body to be measured corresponding to the Alpha Shapes surface. The area of ​​the multiple triangular meshes is calculated to determine the three-dimensional surface area of ​​the fish to be measured; The weight fitting model is expressed by the following formula: ； Where y represents the weight of the fish to be measured, and x represents the three-dimensional surface area of ​​the fish to be measured; The weight determination module is used to determine the weight of the fish to be measured based on a fitting model of the three-dimensional surface area and weight of the fish to be measured.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the non-contact fish weight measurement method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the non-contact fish weight measurement method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the non-contact fish weight measurement method as described in any one of claims 1 to 3.

Images