Fish monitoring method based on binocular vision and deep learning model

Abstract

The invention discloses a fish monitoring method based on binocular vision and a deep learning model. The method comprises the following steps: step 1, collecting image data of fishes in a target area; step 2, constructing a freshwater fish sample library; 3, a fish face and fish tail recognition and segmentation model is built and trained; and 4, monitoring the fishes in the target area. According to the method, machine vision, surveying and mapping science and a deep learning technology are combined at the same time, the fish face and the fish tail of each fish can be recognized and segmented, depth information of the fish face and the fish tail is calculated through a binocular vision distance measurement method, comprehensive analysis is carried out, and the accuracy of the fish face and the fish tail is improved. Therefore, the phenomenon of target loss caused by the back-and-forth wandering behavior of fishes in the visual field range of the underwater camera is reduced, the traditional fish resource acquisition efficiency is improved, the traditional fish resource monitoring cost is reduced, the problem that fish targets are easy to lose in an existing method is solved, and the method is high in practicability and easy to popularize.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a fish monitoring method based on binocular vision and a deep learning model. Background Art

[0002] Intelligent monitoring of fish passage facilities helps to improve the accuracy of fishery resources. Existing methods for monitoring the operation effect of fish passage facilities mainly include physical tagging, sonar, radio tag tracking, ultrasonic tag tracking, transponder tagging, PIT radio frequency tagging, etc. However, these methods are difficult to take into account the three aspects of long-term deployment, species identification, and quantity statistics at the same time.

[0003] With the development of machine vision and deep learning model methods, fish target recognition and tracking algorithms based on the yolo model have emerged. However, when fish perform complex movements, the model will lose the fish target. If the same fish is recognized by this model again, it will be judged as a new fish by the model. The recognition results of existing methods are greatly deviated from reality, so how to achieve accurate monitoring of fish is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0004] The purpose of the present invention is to provide a fish monitoring method based on binocular vision and deep learning model to solve the above technical problems.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention discloses a fish monitoring method based on binocular vision and deep learning model, and the method comprises the following steps:

[0007] Step 1: Collecting fish image data in the target area: Setting a binocular high-definition camera in the underwater target area, calibrating the binocular high-definition camera, and then collecting fish image data in the target area, and calculating and analyzing the collected fish image data to obtain the depth information of the fish in the target area;

[0008] Step 2: Construction of a freshwater fish sample library: Collect different types of freshwater fish in different underwater environments with different posture images as sample data, annotate the sample data through visual interpretation, and use data enhancement methods to expand the sample data, thereby constructing a freshwater fish sample library;

[0009] Step 3: Build and train a fish face and tail recognition segmentation model: Build a fish face and tail recognition segmentation model based on MS-CNN, i.e., a multi-scale convolutional neural network, and train the model using the constructed freshwater fish sample library;

[0010] Step 4, monitor the fish in the target area: use the trained fish face and tail recognition and segmentation model to segment the fish face and tail of the fish in the target area, obtain the segmentation results of the fish in the target area, and identify the species of the fish; comprehensively analyze the segmentation results of the fish in the target area with the depth information, calculate the length of the fish in the target area, monitor the movement direction of the fish, and output the monitoring results.

[0011] Furthermore, the specific process of calibrating the binocular high-definition camera described in step 1 is: adopting the mainstream square calibration method, using an underwater high-definition camera to shoot 20-30 prepared chessboard pictures from different angles and directions, the actual distance of a small square on the chessboard is 2.7 cm, and using the binocular camera calibration tool for calibration. After error analysis of each calibration image, the calibration parameters with an error of no more than 0.3 pixels are obtained, and then the calibration parameters are used to complete the calibration of the underwater high-definition camera, so that the left view and the right view can use a unified camera coordinate system.

[0012] Furthermore, the specific process of calculating and analyzing the collected fish image data in step 1 to obtain the depth information of the fish in the target area is: using the principle of stereoscopic parallax, the distance between the object and the camera is calculated according to the position difference of the object, i.e., the fish, in the two images, that is, by measuring the parallax and using the known distance between the cameras, the triangulation method is used to infer the distance of the target object, i.e., the depth information, and the calculation formula is:

[0013]

[0014] Where d is the distance of the target object; B is the baseline length of the binocular camera; f is the focal length of the binocular camera; D is the parallax of the target object in the left and right views;

[0015] D = x l -x r (2)

[0016] Among them, x l is the horizontal axis coordinate of the target object on the left visual image; r is the horizontal axis coordinate of the target object in the right visual image.

[0017] Furthermore, the different types of freshwater fish described in step 2 include nine types of freshwater fish: grass carp, crucian carp, bighead carp, tilapia, snakehead carp, black carp, seabass, bighead carp, and sturgeon.

[0018] Furthermore, the data enhancement method described in step 2 is used to expand the sample data in two ways. The first is to change the image itself, including rotating, flipping, scaling, and adding random noise to the image to obtain a new sample; the second is to splice the target main part with the environment part in other images to obtain a new sample.

[0019] Furthermore, the fish face and tail recognition and segmentation model described in step 3 includes five modules: backbone, RPN, candidate area screening, FCNs segmentation, and FC layers detection and recognition. The model first extracts multi-scale features in fish images through the backbone module. The backbone module is composed of 5 multi-scale residual modules, which extracts fish features at different scales, and combines the features of the previous scale as auxiliary information with the features of the current scale to solve the problem of information redundancy between different scales. After that, the obtained fish shape, underwater environment, and swimming posture feature map are preliminarily sorted and predicted by the RPN module to obtain three target candidate areas of fish body, fish face, and fish tail. The candidate area screening module compares these candidate areas through IOU calculation, screens out the fish body, fish face, and fish tail areas corresponding to each fish detected by the model, and aligns the corresponding areas of the feature map to a uniform size through the Rol Align algorithm to obtain alignment features. The aligned features are segmented into fish face and fish tail in the target area through the FCNs segmentation module; and the fish body, fish face, fish tail bbox and fish species are detected and recognized through the FC layers detection and recognition module.

[0020] Furthermore, the specific process of comprehensively analyzing the segmentation results of the fish in the target area and the depth information in step 4, calculating the length of the fish in the target area, and monitoring the movement direction of the fish is as follows: assigning depth information to the segmented fish face and tail, and performing two 3x3 filtering operations on the depth information on the fish head and tail respectively; the first filtering uses a 3x3 median filter to remove the depth information with large errors; the second filtering uses a 3x3 mean filter to smooth the depth information;

[0021] After obtaining the depth information of the fish face and tail of the target area after smoothing, a geometric mathematical analysis model is established to simplify the fish face and tail areas to their respective geometric centers, and obtain the fish face point P1 and the fish tail point P2. The calculation formula for the fish body length L is:

[0022]

[0023] Where X1, Y1, and Z1 are the three-dimensional coordinates of the fish face point P1; X2, Y2, and Z2 are the three-dimensional coordinates of the fish tail point P2;

[0024]

[0025] Where i is the pixel in the image; X i , Y i , Z i is the three-dimensional coordinate of pixel i; x l is the horizontal axis coordinate of pixel i on the left visual image; xr is the horizontal axis coordinate of pixel point i on the right visual image; y l is the vertical axis coordinate of pixel point i on the left visual image; x O and y O are the optical center coordinates of the camera;

[0026] Then, monitor the movement direction of the fish. If the fish moves deeper, it means it is moving away from the camera. By comparing the depth information Z1 and Z2 at points P1 and P2, if Z1 > Z2, it indicates that the fish is moving deeper; if Z1 < Z2, the opposite is true. Similarly, by comparing X1 and X2, and Y1 and Y2, determine whether the fish is moving left, right, up, or down.

[0027] The beneficial effects of the present invention are as follows: The method of the present invention combines machine vision, surveying science, and deep learning technologies. It can identify and segment the fish face and fish tail of each fish, calculate the depth information of the fish face and fish tail through binocular vision ranging, and conduct comprehensive analysis, thereby reducing the phenomenon of target loss caused by the fish's wandering behavior in the field of view of the underwater camera, improving the efficiency of traditional fish resource acquisition, reducing the cost of traditional fish resource monitoring, and solving the problem of easy loss of fish targets in existing methods. Moreover, it can estimate the body length of the fish and its movement direction in a high degree of freedom while identifying the fish, with strong practicability and easy to promote.

[0028] The following further elaborates on the present invention in detail in conjunction with the accompanying drawings and specific embodiments. Description of the Drawings

[0029] Figure 1 is the schematic flow chart of the method of the present invention;

[0030] Figure 2 is the schematic diagram before the unification of the left and right views of the camera coordinate system;

[0031] Figure 3 is the schematic diagram after the unification of the left and right views of the camera coordinate system;

[0032] Figure 4 is the schematic diagram before the calibration of the underwater high-definition camera;

[0033] Figure 5 is the schematic diagram after the calibration of the underwater high-definition camera;

[0034] Figure 6 is the schematic diagram of the binocular vision ranging principle;

[0035] Figure 7 is the sample annotation result diagram of grass carp, fish head, and fish tail;

[0036] Figure 8 is the architecture diagram of the fish face and fish tail recognition and segmentation model;

[0037] Fig. 9 Schematic diagram of calculating kernel for median filter;

[0038] Fig.10 Schematic diagram of calculating the kernel for the mean filter. DETAILED DESCRIPTION

[0039] The present invention discloses a fish monitoring method based on binocular vision and deep learning model, such as Figure 1 As shown, the method comprises the following steps:

[0040] Step 1: Collect fish image data in the target area: Set up a binocular high-definition camera in the underwater target area and calibrate the binocular high-definition camera. Then collect fish image data in the target area and perform calculation and analysis on the collected fish image data to obtain the depth information of the fish in the target area.

[0041] When collecting fish image data in the underwater target area, a binocular underwater high-definition camera is used, and the binocular high-definition camera is calibrated to correct the distortion of the camera itself.

[0042] Current underwater high-definition cameras need to use the camera's intrinsic parameter matrix, distortion matrix, rotation matrix, and displacement matrix to solve the offset and distortion problems during underwater camera shooting, and at the same time establish a connection between the image pixel coordinate system and the world's real coordinate system. Before the binocular high-definition camera is calibrated, the camera coordinate systems of the images taken by the left and right cameras cannot be unified. After calibration, the offset and distortion can be corrected, and the left and right views can be expressed in a unified camera coordinate system, such as Figure 2 , 3 As shown. The present invention adopts the mainstream square calibration method, using an underwater high-definition camera to shoot 20-30 prepared chessboard images from different angles and directions. The actual distance of a small square on the chessboard is known to be 2.7 cm. The binocular camera calibration tool is used for calibration. After error analysis of each calibration image, calibration parameters with an error of no more than 0.3 pixels are obtained. Then, the calibration parameters are used to complete the calibration of the underwater high-definition camera, so that the left view and the right view can use a unified camera coordinate system, such as Figure 4 , 5 shown.

[0043] Then, the calibrated binocular high-definition camera is used to collect fish image data in the target area, and the collected fish image data is calculated and analyzed based on binocular vision technology to obtain the depth information of the fish in the target area.

[0044] Binocular vision is a technology that uses two cameras or sensors to simulate the human binocular vision system. By placing the two cameras at a certain distance and simulating the distance between the human eyes, the three-dimensional information acquisition and depth perception of the object can be achieved. The core principle of binocular vision is stereo parallax, that is, when the same object is observed by two cameras, due to the certain distance between the two cameras, the position of the object in the two images will be different. This difference appears as the displacement of the object in the image, which is called parallax. The parallax is related to the distance from the object to the camera and the distance between the cameras. Using the principle of stereo parallax, the distance between the object and the camera can be calculated based on the difference in the position of the object (fish) in the two images. By measuring the parallax and using the known distance between the cameras, such as Figure 6 As shown, the triangulation method can be used to infer the distance or depth information of the target object, and the calculation formula is:

[0045]

[0046] Wherein, d is the distance of the target object; B is the baseline length of the binocular camera; f is the focal length of the binocular camera; and D is the parallax of the target object in the left and right views.

[0047] D = x l -x r (2)

[0048] Among them, x l is the horizontal axis coordinate of the target object on the left visual image; r is the horizontal axis coordinate of the target object in the right visual image.

[0049] Step 2: Construction of a freshwater fish sample library: Collect images of different types of freshwater fish in different underwater environments in different postures as sample data, annotate the sample data through visual interpretation, and use data enhancement methods to expand the sample data, thereby constructing a freshwater fish sample library.

[0050] Since fish passing facilities are usually found at dams to facilitate fish to pass through the dams, and dams are usually built on rivers, and fish are all freshwater fish, a freshwater fish sample library was constructed.

[0051] Specifically, we collected images of grass carp, crucian carp, bighead carp, tilapia, snakehead, black carp, sea bass, bighead carp, and sturgeon in different environments, and annotated the types of fish in the images, as well as the head and tail of each fish, through visual interpretation. Figure 7 shown.

[0052] To establish a high-quality freshwater fish sample library, three conditions must be met:

[0053] 1. Images of fish in different underwater postures are needed.

[0054] 2. Images of fish in different underwater environments are needed.

[0055] 3. Professional visual interpreters are needed to accurately label samples.

[0056] Therefore, establishing a high-quality freshwater fish sample library is a time-consuming and labor-intensive task. In order to ensure that the sample set has a sufficient number to support subsequent model training, the data enhancement method is used to expand the sample library. The data enhancement method used in the present invention includes two methods. One is to rotate, flip, scale, and add random noise to the image, that is, to directly make certain changes to the image itself to obtain a new sample; the other is to splice the target body part with the environment part in other images to obtain a new sample.

[0057] Step 3: Build and train a fish face and tail recognition segmentation model: Build a fish face and tail recognition segmentation model based on MS-CNN (multi-scale convolutional neural network), and use the constructed freshwater fish sample library to train the model.

[0058] The semantic granularity of fish in an image is different at different scales. At the scale of single pixels, it is reflected as fish scales, fish eyes at the scale of tens of pixels, and fish fins, fish faces, and fish tails at the scale of hundreds of pixels. The edge features and spectral features of fish are different at different semantic granularities. The underwater environment is very complex, and the movement patterns of fish are also varied. If the fish in the image are segmented directly, they will be seriously disturbed by noise. Based on the above two considerations, the fish face and tail recognition segmentation model built by the present invention based on MS-CNN is as follows: Figure 8 As shown in the figure, the structure of the multi-scale convolution residual block makes full use of the multi-scale information extraction and semantic information expression capabilities of MS-CNN, so that the model can capture the fish body, fish face, and fish tail areas at different scales, and then achieve instantiation positioning of the fish body, fish face, and fish tail by matching the IOU between the fish body, fish face, and fish tail. Based on the positioning information, the model recognition and segmentation module can accurately grasp the edge features and spectral features at each semantic granularity, and realize the species recognition, fish head, and fish tail segmentation of freshwater fish.

[0059] The fish face and tail recognition and segmentation model includes five modules: backbone, RPN, candidate area screening, FCNs segmentation, and FC layers detection and recognition. Specifically, this model first extracts multi-scale features in fish images through the backbone module. The backbone module consists of 5 multi-scale residual modules. Through rich receptive field convolution layers, fish features are extracted at different scales. At the same time, the features of the previous scale are combined with the features of the current scale as auxiliary information to solve the problem of information redundancy between different scales, so that the features extracted at different scales only focus on the current scale. After that, the feature maps of fish shape, underwater environment, swimming posture, etc. are preliminarily sorted and predicted by the RPN module to obtain three target candidate areas: fish body, fish face, and fish tail. The candidate area screening module compares these candidate areas through IOU calculation, screens out the fish body, fish face, and fish tail areas corresponding to each fish detected by the model, and aligns the corresponding areas of the feature map to a uniform size through the Rol Align algorithm to obtain aligned features, which is convenient for the comprehensive calculation of subsequent features. The aligned features are segmented into fish face and fish tail in the target area by FCNs segmentation module; the fish body, fish face, fish tail bbox and fish species are detected and identified by FC layers detection and recognition module. During the training process, the fish face and fish tail recognition and segmentation model is constrained by the freshwater fish sample library established in step 3.

[0060] Step 4, monitor the fish in the target area: use the trained fish face and tail recognition and segmentation model to segment the fish face and tail of the fish in the target area, obtain the segmentation results of the fish in the target area, and identify the species of the fish; comprehensively analyze the segmentation results of the fish in the target area with the depth information, calculate the length of the fish in the target area, monitor the movement direction of the fish, and output the monitoring results.

[0061] Specifically, the segmentation result and the depth information are comprehensively analyzed, the depth information is assigned to the segmented fish face and fish tail, and the depth information on the fish head and fish tail is subjected to two 3x3 filtering operations respectively.

[0062] The main purpose of the first filtering is to remove the depth information with large errors. The present invention uses a 3x3 median filter, which can replace the depth information in the pixel to be processed with the mode value of the depth information value of the 9 pixel points selected by the filter. Fig. 9 As shown, for the 3x3 sample values ​​to be processed, the pixel value of the central pixel will be replaced by 63, thereby removing the noise.

[0063] The main purpose of the second filtering is to smooth the depth information. The present invention uses a 3x3 mean filter, which can replace the depth information in the pixel to be processed with the average value of the depth information values ​​in the surrounding 9 pixels. Fig.10As shown, they are the values in the filter kernel.

[0064] After obtaining the depth information of the fish face and fish tail in the target area of the fish after smoothing processing, a geometric mathematical analysis model is established, and the fish face and fish tail areas are simplified to their respective geometric centers to obtain the fish face point P1 and the fish tail point P2. Then the calculation formula for the fish body length L is as follows:

[0065]

[0066] In the formula, X1, Y1, and Z1 are the three-dimensional coordinates of the fish face point P1; X2, Y2, and Z2 are the three-dimensional coordinates of the fish tail point P2;

[0067]

[0068]

[0069] In the formula, i is the pixel point in the image; X i , Y i , Z i are the three-dimensional coordinates of the pixel point i; x l is the horizontal axis coordinate of the pixel point i on the left visual image; x r is the horizontal axis coordinate of the pixel point i on the right visual image; y l is the vertical axis coordinate of the pixel point i on the left visual image; x O , y O are the optical center coordinates of the camera.

[0070] Then the movement direction of the fish can be monitored. Here, the fish moving deeper represents moving away from the camera. By comparing the depth information Z1 and Z2 at point P1 and point P2, if Z1 > Z2, it means the fish is moving deeper, and if Z1 < Z2, it is the opposite. Similarly, the fish's movement direction left / right and up / down can be judged by comparing X1 and X2, Y1 and Y2.

[0071] Finally, it should be noted that the above description is only used to illustrate the technical solution of the present invention and not to limit it. Although the present invention has been described in detail with reference to the preferred arrangement, those of ordinary skill in the art should understand that the technical solution of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A fish monitoring method based on binocular vision and deep learning model, characterized in that: The method comprises the following steps: Step 1: Collecting fish image data in the target area: Setting a binocular high-definition camera in the underwater target area, calibrating the binocular high-definition camera, and then collecting fish image data in the target area, and calculating and analyzing the collected fish image data to obtain the depth information of the fish in the target area; Step 2: Construction of a freshwater fish sample library: Collect different types of freshwater fish in different underwater environments with different posture images as sample data, annotate the sample data through visual interpretation, and use data enhancement methods to expand the sample data, thereby constructing a freshwater fish sample library; Step 3: Build and train a fish face and tail recognition segmentation model: Build a fish face and tail recognition segmentation model based on MS-CNN, i.e., a multi-scale convolutional neural network, and train the model using the constructed freshwater fish sample library; Step 4, monitor the fish in the target area: use the trained fish face and tail recognition and segmentation model to segment the fish face and tail of the fish in the target area, obtain the segmentation results of the fish in the target area, and identify the species of the fish; comprehensively analyze the segmentation results of the fish in the target area with the depth information, calculate the length of the fish in the target area, monitor the movement direction of the fish, and output the monitoring results.

2. A fish monitoring method based on binocular vision and deep learning model according to claim 1, characterized in that: The specific process of calibrating the binocular high-definition camera described in step 1 is: adopt the mainstream square calibration method, use the underwater high-definition camera to shoot 20-30 prepared chessboard pictures from different angles and directions, the actual distance of a small square on the chessboard is 2.7cm, use the binocular camera calibration tool for calibration, after error analysis of each calibration image, obtain the calibration parameters with an error of no more than 0.3 pixels, and then use the calibration parameters to complete the calibration of the underwater high-definition camera, so that the left view and the right view can use a unified camera coordinate system.

3. The fish monitoring method based on binocular vision and deep learning model according to claim 1 is characterized in that: The specific process of calculating and analyzing the collected fish image data in step 1 to obtain the depth information of the fish in the target area is as follows: using the principle of stereoscopic parallax, the distance between the object and the camera is calculated according to the position difference of the object, i.e., the fish, in the two images. That is, by measuring the parallax and using the known distance between the cameras, the triangulation method is used to infer the distance of the target object, i.e., the depth information. The calculation formula is: Where d is the distance of the target object; B is the baseline length of the binocular camera; f is the focal length of the binocular camera; D is the parallax of the target object in the left and right views; D=x l -x r (2) Among them, x l is the horizontal axis coordinate of the target object on the left visual image; r is the horizontal axis coordinate of the target object in the right visual image.

4. The fish monitoring method based on binocular vision and deep learning model according to claim 1 is characterized in that: The different types of freshwater fish described in step 2 include nine types of freshwater fish: grass carp, crucian carp, bighead carp, tilapia, snakehead carp, black carp, seabass, bighead carp, and sturgeon.

5. The fish monitoring method based on binocular vision and deep learning model according to claim 1 is characterized in that: There are two ways to expand the sample data using the data enhancement method described in step 2. The first is to change the image itself, including rotating, flipping, scaling, and adding random noise to obtain new samples; the second is to splice the target body part with the environment part in other images to obtain a new sample.

6. The fish monitoring method based on binocular vision and deep learning model according to claim 1 is characterized in that: The fish face and fish tail recognition and segmentation model described in step 3 includes five modules: backbone, RPN, candidate region screening, FCNs segmentation, and FC layers detection and recognition. First, the model extracts multi-scale features from fish images through the backbone module. The backbone module consists of 5 multi-scale residual modules, which extract fish features at different scales and combine the features of the previous scale as auxiliary information with the features of the current scale to solve the problem of information redundancy between different scales. Then, the obtained feature maps of fish shape, underwater environment, and swimming posture are preliminarily sorted and predicted by the RPN module to obtain three target candidate regions: fish body, fish face, and fish tail. The candidate region screening module calculates and compares these candidate regions through IOU, screens out the fish body, fish face, and fish tail regions corresponding to each fish detected by the model, and aligns the corresponding regions of the feature map to a unified size through the Rol Align algorithm to obtain aligned features. The aligned features are segmented into fish face and fish tail in the target region through the FCNs segmentation module. The fish body, fish face, fish tail bbox, and fish species are detected and recognized through the FC layers detection and recognition module.

7. The fish monitoring method based on binocular vision and deep learning model according to claim 1 is characterized in that: The specific process of comprehensively analyzing the segmentation results of fish in the target region and depth information and calculating the body length of fish in the target region and monitoring the movement direction of fish in step 4 is as follows: Depth information is assigned to the segmented fish face and fish tail, and the depth information on the fish head and fish tail is respectively subjected to two 3x3 filtering operations. The first filtering uses a 3x3 median filter to remove depth information with large errors. The second filtering uses a 3x3 mean filter to smooth the depth information. After obtaining the depth information of the fish face and fish tail of the fish in the target region after smoothing, a geometric mathematical analysis model is established, and the fish face and fish tail regions are simplified to their respective geometric centers to obtain the fish face point P1 and the fish tail point P2. Then the formula for calculating the fish body length L is: In the formula, X1, Y1, Z1 are the three-dimensional coordinates of the fish face point P1; X2, Y2, Z2 are the three-dimensional coordinates of the fish tail point P2. Where i is the pixel in the image; X i , Y i , Z i is the three-dimensional coordinate of pixel i; x l is the horizontal axis coordinate of pixel i on the left visual image; x r is the horizontal axis coordinate of pixel i on the right visual image; l is the vertical coordinate of pixel i on the left visual image; x O ,y O is the optical center coordinate of the camera; Then, the movement direction of the fish is monitored. The fish moving deeper represents moving away from the camera. By comparing the depth information Z1 and Z2 at point P1 and point P2, if Z1>Z2, it means the fish is moving deeper, and vice versa for Z1<Z2. Similarly, by comparing X1 and X2, Y1 and Y2, it is judged whether the fish is moving left or right, up or down.

Images