Intelligent identification method and device for morphological characteristics of underwater fish

Abstract

According to the intelligent identification method and device for the morphological characteristics of the underwater fish, the morphological characteristics of the underwater fish are intelligently identified based on target detection, key point detection and a three-dimensional reconstruction method, and a target detection network is used for processing a side view angle data set to position a fish body to obtain a target frame of target detection; carrying out key point detection of a key point identification network on the obtained image in the target frame to obtain key point information of the target fish; a binocular disparity map is obtained through a binocular matching algorithm, and then two-dimensional key point position information of the fish is calculated into three-dimensional key point position information; fitting a three-dimensional coordinate through a space curve, and analyzing the angle and length information between the key points to obtain the local deformation, pose and other characteristics of the fish; the data set of the overlook angle is analyzed and processed, and non-rigid deformation information of the fish body at the two-dimensional overlook angle is obtained; therefore, the spatial attitude of the fish body can be effectively fitted, and the morphological characteristics of the underwater free swimming fish body can be identified.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and in particular to a method and a device for intelligently identifying the morphological features of underwater fish. Background Art

[0002] In aquaculture, the morphological characteristics of fish can most intuitively observe the behavioral characteristics and movement of fish, and are of great significance to the evaluation of fish behavior and movement. The morphological characteristics of fish, especially the angle and posture of the fish tail bending, are important references for reflecting the non-rigid deformation, movement ability, movement mode, etc. of fish movement. They can be used as one of the important movement information of fish, and can also be used as an important indicator of fish health characteristics.

[0003] As important basic parameters, morphological characteristic parameters provide direct data reference for quantitative analysis of research processes such as breeding, selection, and trait classification. In order to improve the scientificity and efficiency of fish breeding, the morphological characteristic parameters of fish during movement are obtained in time to assess the status of the fish in real time.

[0004] At present, the traditional method of obtaining the morphological characteristics of fish relies on manpower and naked eye observation, which lacks accuracy and efficiency. In addition, since the morphological characteristics of fish may change rapidly during movement, the naked eye cannot timely process the morphological characteristics of the observed object. When observing manually, it is also necessary to catch the fish as samples and observe the experimental objects separately, and use the observation results to reflect their growth and health status. This method cannot avoid potential infection and harm to the experimental objects, and may also indirectly harm the entire fish school.

[0005] Machine vision and deep learning technologies can be used to intelligently measure the morphological characteristics of fish. However, the position of a fish swimming freely underwater moves in three-dimensional space, and the shape of the fish is mostly non-rigid when moving. Existing technologies cannot accurately measure the morphological characteristics of fish swimming freely underwater. Summary of the invention

[0006] The present invention provides a method and device for intelligently identifying the morphological characteristics of underwater fish, so as to solve the problem that the prior art cannot well realize the accurate measurement of the morphological characteristic information of the fish body swimming freely underwater.

[0007] The present invention provides an intelligent method for identifying underwater fish morphological features, comprising:

[0008] Obtain monocular video from above water and binocular video from below water.

[0009] The above-water top-view monocular video and the underwater side-view binocular video are respectively captured frame by frame to obtain an above-water top-view angle picture set and an underwater side-view angle picture set;

[0010] After data annotation, the above-water top-view angle picture set and the underwater side-view angle picture set are input into the target detection network and the key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle and the two-dimensional key point position information of the underwater side-view angle of the fish body;

[0011] Based on the position information of the two-dimensional key points of the fish body from the top-down angle above water, the non-rigid deformation information of the fish body is obtained;

[0012] Based on the two-dimensional position information of the key points of the side view angle of the fish body underwater and the visual difference of the binocular image, the three-dimensional position information of the key points of the side view angle is obtained; based on the three-dimensional position information, the fish body posture information is obtained;

[0013] Based on the non-rigid deformation information of the fish body and the fish body posture information, the fish body morphological feature information is obtained.

[0014] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, the method of obtaining a monocular video from an overhead view angle above water and a binocular video from a side view angle underwater of a fish body swimming comprises:

[0015] Use an underwater monocular camera to obtain a top-down view of the fish swimming in the water;

[0016] The underwater binocular camera is used to obtain underwater side-view binocular videos of fish swimming.

[0017] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, before obtaining an underwater side-view binocular video of a fish swimming through an underwater binocular camera, the method further includes:

[0018] Calibrate the underwater binocular camera;

[0019] Get the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera;

[0020] After obtaining the underwater side-view binocular video of the fish body swimming by the underwater binocular camera, the method further includes:

[0021] The video taken by the underwater side-viewing binocular camera is corrected by obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera.

[0022] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, after the above-water top-view monocular video and the underwater side-view binocular video are respectively intercepted frame by frame, the method further includes:

[0023] Selecting clear and representative pictures from the side-view angle pictures captured frame by frame, splitting the pictures in half, and using the pictures after the half-split as a training data set;

[0024] Clear and representative pictures are selected from the side-view pictures captured frame by frame as training data sets.

[0025] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, after data annotation is performed on the above-water top-view angle picture set and the underwater side-view angle picture set, the above-water top-view angle picture set and the underwater side-view angle picture set are input into a target detection network and a key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle of the fish body and the two-dimensional key point position information of the underwater side-view angle, including:

[0026] The above-water top-view image set and the underwater side-view image set are input into the target detection network to obtain the positioning frame for fish target detection; the key point recognition network is used to detect the key points of the image in the positioning frame to obtain the two-dimensional key point position information of the fish body.

[0027] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, the non-rigid deformation information of the fish body is obtained based on the two-dimensional key point position information of the fish body from the top-down angle above water, including:

[0028] Get the connecting line of the two-dimensional key points of the water top-down angle at the previous moment and the next moment;

[0029] The angle between the two connecting lines is obtained, which is the non-rigid deformation angle of the fish body.

[0030] According to the method for intelligent recognition of underwater fish morphological features provided by the present invention, the visual disparity of the binocular video includes depth information of the fish body based on the binocular camera, and the depth information is obtained based on the binocular image at a side view angle combined with the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera, specifically including:

[0031] S71, obtaining the angle of the fish's tail swing and the time from the tail swing to the return to the original position, and obtaining the fish's tail swing speed;

[0032] S72, calculating the number of maximum values ​​of the swing angle within a certain time period to obtain the number of swings of the tail fin of the fish body;

[0033] S73, obtaining a tail swing frequency based on the total number of acquired frames and the acquired frame rate;

[0034] S74, based on the minimum distance from the binocular camera center to the fish body surface, the minimum distance from the binocular camera to the fish body surface and the fish body height, obtain the distance from the binocular camera center to the fish body center of mass;

[0035] S75. Obtain depth information based on the fish's tail-swinging speed, number of swings, tail-swinging frequency, and the distance from the center of the binocular camera to the fish's center of mass.

[0036] According to the method for intelligently identifying underwater fish morphological features provided by the present invention, based on the two-dimensional key point position information of the fish body's underwater side view angle and the visual difference of the binocular image, the three-dimensional position information of the key point of the side view angle is obtained; based on the three-dimensional position information, the fish body posture information is obtained, including:

[0037] Based on the depth information and the two-dimensional position information of the key points of the side view angle of the fish body underwater, the three-dimensional position information of the key points of the side view angle is obtained;

[0038] The three-dimensional position information of the key points is subjected to spatial curve fitting to obtain the position and posture information of the fish body.

[0039] The present invention also provides an intelligent device for identifying underwater fish morphological characteristics, the device comprising:

[0040] An acquisition module is used to acquire monocular video from an overhead angle above water and binocular video from a side angle underwater of the fish body swimming;

[0041] The acquisition module is further used to capture the above-water top-view monocular video and the underwater side-view binocular video frame by frame to obtain an above-water top-view angle picture set and an underwater side-view angle picture set;

[0042] A training module is used to annotate the above-water top-view angle picture set and the underwater side-view angle picture set, and then input the data into a target detection network and a key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle and the two-dimensional key point position information of the underwater side-view angle of the fish body;

[0043] A determination module is used to obtain non-rigid deformation information of the fish body based on the position information of two-dimensional key points of the fish body from an overhead view angle above water;

[0044] The determination module is also used to obtain the three-dimensional position information of the key points of the side view angle based on the two-dimensional key point position information of the fish body underwater side view angle and the visual difference of the binocular image; and obtain the fish body posture information based on the three-dimensional position information;

[0045] The determination module is also used to obtain fish body morphological feature information based on the fish body non-rigid deformation information and fish body posture information.

[0046] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for intelligently identifying the morphological features of underwater fish as described above is implemented.

[0047] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for intelligently identifying the morphological characteristics of underwater fish as described in any one of the above methods is implemented.

[0048] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for intelligently identifying underwater fish morphological features.

[0049] The method and device for intelligently identifying the morphological features of underwater fish provided by the present invention intelligently identify the morphological features of underwater fish based on target detection, key point detection and three-dimensional reconstruction methods. The underwater binocular camera is first calibrated to obtain the internal and external parameters of the binocular camera, and the underwater video image taken by the binocular camera is corrected by the internal and external parameters; the video taken by the underwater binocular camera from a side view angle is intercepted frame by frame, and the intercepted clear and representative pictures are split in half into two pictures as a training data set; a clear picture is intercepted from the video taken synchronously by an above-water monocular camera, and the intercepted picture from an overhead view angle above water is used as a supplement to the underwater side view angle data set; and the target detection is used to detect the target point of interest and the target point of interest is used to detect the target point of interest. The detection network processes the side-view angle data set to locate the fish body and obtain the target frame for target detection; the key point detection of the key point recognition network is performed on the image in the obtained target frame to obtain the key point information of the target fish; the binocular disparity map is obtained by the binocular matching algorithm, and then the two-dimensional key point position information of the fish is calculated into the three-dimensional key point position information; the three-dimensional coordinates are fitted by the spatial curve, and the angle and length information between the key points are analyzed to obtain the local deformation, posture and other characteristics of the fish; the data set of the top-down angle is analyzed and processed to obtain the non-rigid deformation information of the fish body at a two-dimensional top-down angle; thereby, the spatial posture of the fish body can be effectively fitted, and the morphological characteristics of the fish body swimming freely underwater can be recognized. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flow chart of an intelligent identification method of underwater fish morphological characteristics provided by the present invention;

[0052] Figure 2 is a flow chart of a method for intelligently identifying underwater fish morphological features provided by an embodiment of the present invention;

[0053] Figure 3is a schematic diagram of an intelligent method for identifying underwater fish morphological features provided by an embodiment of the present invention;

[0054] Figure 4 is a mapping diagram from the world coordinate system to the camera coordinate system of the binocular camera provided in an embodiment of the present invention;

[0055] Figure 5 is a mapping diagram from the camera coordinate system to the image coordinate system of the binocular camera provided in an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of the depth information acquisition principle provided by an embodiment of the present invention;

[0057] Figure 7 is a schematic diagram of the depth information correction principle provided by an embodiment of the present invention;

[0058] Figure 8 is a schematic diagram of the distortion of the original image of the binocular camera provided by an embodiment of the present invention;

[0059] Fig. 9 is a flow chart of a synthetic data set provided by an embodiment of the present invention;

[0060] Fig.10 It is a label map trained by the PP-PicoDet network and the HRNET network provided by an embodiment of the present invention;

[0061] Fig.11 A schematic diagram of depth error of binocular estimation provided by an embodiment of the present invention;

[0062] Fig.12 It is an analysis diagram of the swing frequency and angle of a fish body provided by an embodiment of the present invention;

[0063] Fig.13 It is a principle diagram of fish body swing frequency and angle error based only on a two-dimensional top-view angle image provided by an embodiment of the present invention;

[0064] Fig.14 The embodiment of the present invention provides a determination diagram for determining the specific position and posture of a fish by using a binocular image at a top-down angle based on spatial coordinates;

[0065] Fig.15 A schematic diagram of the structure of an intelligent device for identifying underwater fish morphological characteristics provided by an embodiment of the present invention;

[0066] Fig.16 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Combine the following Figure 1-Figure 16 The method and device for intelligently identifying underwater fish morphological features of the present invention are described. The execution subject of the embodiment of the present invention can be any device capable of intelligently identifying underwater fish morphological features, for example, electronic devices such as terminals or servers.

[0069] In order to more clearly describe the key point information and position information of the fish body in the figure, the present invention adopts the form of color drawings.

[0070] Figure 1 A schematic diagram of a flow chart of an underwater fish morphological feature intelligent recognition method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0071] Step 101: Obtaining a monocular video of a fish swimming from an overhead angle above water and a binocular video of a fish swimming from a side angle underwater;

[0072] Step 102: intercepting the above-water top-view monocular video and the underwater side-view binocular video frame by frame respectively to obtain an above-water top-view angle picture set and an underwater side-view angle picture set;

[0073] Step 103: After data annotation, the above-water top-view angle picture set and the underwater side-view angle picture set are input into a target detection network and a key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle and the two-dimensional key point position information of the underwater side-view angle of the fish body;

[0074] Step 104: obtaining non-rigid deformation information of the fish body based on the two-dimensional key point position information of the fish body from an overhead view angle above water;

[0075] Step 105: based on the two-dimensional key point position information of the fish body's underwater side view angle and the visual difference of the binocular image, obtain the three-dimensional position information of the key point of the side view angle; based on the three-dimensional position information, obtain the fish body posture information;

[0076] Step 106: Based on the non-rigid deformation information of the fish body and the fish body posture information, obtain the fish body morphological feature information.

[0077] The above steps are described in detail below in conjunction with specific embodiments.

[0078] Figure 2is a flow chart of a method for intelligently identifying underwater fish morphological features provided by an embodiment of the present invention; Figure 3 FIG. 1 is a schematic diagram of an intelligent method for identifying underwater fish morphological features provided by an embodiment of the present invention. Figure 3 As shown, in a specific embodiment, the method for intelligently identifying underwater fish morphological features can be specifically divided into the following steps:

[0079] S1: Calibrate with an underwater binocular camera and photograph the calibration plate from different angles and distances;

[0080] It is understandable that before obtaining the monocular video of the fish swimming from a top-down angle above water and the binocular video of the fish swimming from a side-view angle underwater, an underwater binocular camera can be used to calibrate a plate to shoot images at different angles and distances, or videos can be taken, and the videos can be captured as pictures later.

[0081] In this step, for example, the position relationship between the calibration plate and the binocular camera lens in step S1 is: the center of the calibration plate is facing the center of the binocular camera at a distance of 40 cm, the calibration plate is offset by 20 degrees up, down, left and right, and horizontally translated by 20 cm in each direction, the calibration plate is facing the center of the binocular camera at a distance of 60 cm and 80 cm, and the above steps are performed again, a total of 48 situations (3*4*4 situations).

[0082] S2: Capture the video shot in the calibration step frame by frame into images, and obtain the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera through MATLAB analysis;

[0083] In this step, the calibration data collected in S1 is put into the binocular camera calibration model, and the binocular camera is calibrated using Zhang's calibration method to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera.

[0084] Specifically, the steps of obtaining the internal and external parameters in step S2 are as follows:

[0085] Step 1: Convert the image coordinate system to the pixel coordinate system:

[0086] In a specific embodiment, a direct coordinate system uv is established with the upper left corner of the image as the origin and the pixel as the unit. The horizontal coordinate u and the vertical coordinate v of the pixel are the column number and the row number of the pixel in the image array respectively.

[0087] It can be understood that in this embodiment, since (u, v) only represents the number of columns and rows of pixels, and the position of the pixel in the image is not expressed in physical units, it is necessary to establish physical units, and set dx and dy to represent the physical size of each pixel on the horizontal axis x and the vertical axis y respectively. Then, the coordinates of each pixel in the image in the uv coordinate system and the coordinates in the xy coordinate system have the following relationship:

[0088]

[0089]

[0090] It can be expressed as a homogeneous matrix:

[0091]

[0092] Among them, u0 and v0 are called intrinsic parameters.

[0093] Step 2: Obtain the transformation from the world coordinate system to the camera coordinate system:

[0094] In a specific embodiment, the world coordinate system and the camera coordinate system are not the same coordinate system, and the two coordinate systems need to be transformed together. Figure 4 is a mapping diagram from the world coordinate system of the binocular camera provided in an embodiment of the present invention to the camera coordinate system, such as Figure 4 As shown, for example, the camera coordinate system is set to be the O1-X1Y1Z1 coordinate system, and the world coordinate system is set to be the O-XYZ coordinate system. There is a rigid body transformation relationship between the two:

[0095]

[0096] The corresponding homogeneous expression is:

[0097]

[0098] Here R(3*3) is the selection matrix, T(3*1) is the translation matrix, these two matrices are independent of the camera, so these two parameters are called the external parameters of the camera; O is a 3*1 matrix with all zeros.

[0099] Step 3: Convert the camera coordinate system to the image coordinate system:

[0100] Since the camera coordinate system and the image coordinate system are not the same coordinate system, if the two are linked, a perspective projection is required, and the shape is projected onto the projection surface using the central projection method, thereby obtaining a single-sided projection image that is closer to the visual effect. Figure 5 is a mapping diagram from the camera coordinate system to the image coordinate system of the binocular camera provided in an embodiment of the present invention, such as Figure 5 As shown, the camera coordinate system is set to O1-X1Y1Z1, and the homogeneous coordinates of the spatial point X1 are: , which is projected into the image coordinate system: , through similar triangles we can get: , let the length O1O be f, f is the intrinsic parameter of the camera, and we get:

[0101] Then the transformation relationship from the camera coordinate system to the image coordinate system can be expressed as:

[0102]

[0103] S3: Synchronously shoot monocular videos from the top-down angle of the water and binocular videos from the side-view angle of the underwater fish swimming, calibrate the captured images according to the calibration results of the binocular videos, and integrate the above-water videos taken by the above-water monocular camera and the underwater videos taken by the underwater binocular camera into a dataset containing above-water monocular and underwater binocular angles.

[0104] In this step, it is necessary to obtain monocular video and binocular video from the top view angle and side view angle respectively, and then capture the video into images by frame, select the clear images, and correct the binocular image through the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera obtained in S2.

[0105] Figure 6 is a schematic diagram of the depth information acquisition principle provided by an embodiment of the present invention, such as Figure 6 As shown, the depth information formula is:

[0106]

[0107] Where P1 and P2 are the optical centers of the left and right cameras respectively, D1 and D2 are the imaging planes of the cameras, h is the distance from the optical center plane of the camera to the imaging plane, also known as the focal length, L is the optical center distance of the left and right cameras, L1 and L2 are the horizontal distances from the optical center of the left and right cameras to the imaging point, and H is the distance from the target object to the camera baseline, which is the depth information to be calculated.

[0108] Figure 7 is a schematic diagram of the depth information acquisition and correction principle provided by an embodiment of the present invention, such as Figure 7 As shown in the figure, the video shot by the underwater side-viewing angle binocular camera is corrected by obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera. The correction principle is: to estimate the depth information of the target point through the binocular image, it is necessary to accurately focus on the same point in the two images. In this way, the depth principle of the object can be inferred based on the position relationship of the object point in the two images.

[0109] Step S3: Collect underwater fish swimming data and calibrate the collected images according to the calibration results;

[0110] In this step, the binocular camera is fixed on the camera bracket and connected to the computer host via a USB extension cable to record and save the underwater images in real time. When the camera takes pictures, the real object is projected onto the image plane through the lens. However, the lens will introduce distortion due to manufacturing accuracy and deviation in assembly process, resulting in distortion of the original image. Figure 8is a schematic diagram of the distortion of the original image of the binocular camera provided by an embodiment of the present invention, such as Figure 8 Write a Python program to use the camera calibration parameters obtained in step S2 to perform distortion correction on the underwater images.

[0111] S4: Split the binocular video from the side view angle into two monocular videos in half and capture them frame by frame with the monocular video from the overhead view angle to form a monocular image, and perform target recognition and key point recognition annotation on the fish body; Fig. 9 is a flow chart of a synthetic data set provided by an embodiment of the present invention, such as Fig. 9 As shown, in this step, the corrected binocular image is split in half into two monocular images and merged with the monocular image at a bird's-eye view angle to form training data. Labelme software can be used to annotate the image data, annotate the target detection frame and key point position information of the unobstructed fish body, and after the annotation is completed, export the data to json format and convert it to coco dataset format to synthesize the dataset.

[0112] Fig.10 is a label graph trained by the PP-PicoDet network and the HRNET network provided by an embodiment of the present invention, such as Fig.10 As shown in the figure, the ratio of training set to validation set is set to 9:1.

[0113] S5: The side view monocular image after binocular camera image segmentation and the top view monocular image of the water monocular camera are used for target detection using the PP-PicoDet network to obtain the target frame of the fish body, and the key point recognition technology based on the HRNET network is used to detect the key points of the fish body (there are 9 key points for the side view angle and 7 key points for the top view angle), and the two-dimensional key point information of the side view angle and the top view angle is obtained;

[0114] In this step, the labeled data in the dataset is first trained for target detection using the PP-PicoDet network, and the target detection results are transmitted to the second-layer network. The second-layer network is trained to detect the key points of the fish body using the key point recognition technology based on the HRNET network.

[0115] Specifically, in a specific embodiment, in step S5, the data constructed in step S4 is divided into a training set and a test set in a ratio of 9:1, and the data set is trained using the PP-PicoDet network target recognition algorithm. PP-PicoDet is a mobile-friendly and high-precision Anchor-Free target detection algorithm proposed by Baidu, and its measured performance is very superior. The main advantage of this network comes from its Neck part - CSP-PAN. The CSP-PAN neck uses a PAN structure to obtain multi-level feature maps, and uses a CSP network to connect and fuse features between adjacent feature maps. Picodet also adds a feature map scale on top of the CSP-PAN to detect more targets. For key point detection networks, existing fusion schemes are mostly low-level and high-level fusion methods. HRNet can connect high-to-low resolution subnetworks in parallel instead of connecting them in series. This can maintain high resolution instead of covering it through low-to-high processing, and the predicted heat map may be more accurate in space.

[0116] S6: The side view binocular image is combined with the internal parameter matrix and the external parameter matrix of the binocular camera to obtain a binocular disparity map, and combined with the two-dimensional key point information of the side view angle to generate three-dimensional key point information (three-dimensional key point coordinates).

[0117] In this step, the SGBM binocular matching algorithm is used to perform binocular matching within the fish body contour segmented in S5, avoiding the huge amount of computation caused by global matching, obtaining a local binocular depth map, and obtaining the real space three-dimensional coordinates of the key points in step S5. When obtaining the depth information of the fish here, it is necessary to improve the SGBM binocular matching algorithm calculation method and integrate the information of the bird's-eye view angle into the calculation. Fig.11 A schematic diagram of the depth error of binocular estimation provided by an embodiment of the present invention is provided in an embodiment of the present invention, such as Fig.11As shown in the figure, taking the spotted rock bream among the flattened fish as an example, since the body width of the flattened fish is small but the body height is high, the adult spotted rock bream is about 16cm tall, about 25cm long and only 4cm wide. Therefore, when estimating the depth information, if the general method refers to the distance from the center of the camera to the surface of the fish body, rather than the distance to the center of the fish body, then the error of this difference is only 4cm of the fish body width when the fish is viewed from the side. However, when the fish swims sideways, the belly of the fish faces the camera viewed from the side, then the error is 16cm of the fish body height, which is obviously a 4-fold error. When the fish head faces the camera viewed from the side, the error is 25cm of the body length, which is nearly 6 times the error. These data are obviously very serious errors, so at this time, it is necessary to analyze the posture of the fish in combination with a monocular camera at a top-down angle, and then proportionally infer the real depth information based on the fish body width or height and the binocular inferred depth. It is worth noting that the depth information here is no longer the distance from the center of the camera to the surface of the fish body, but the distance from the center of the camera to the center of mass of the fish.

[0118] Specifically, in step S6, the specific operation of calculating the real depth information in proportion according to the fish body width or fish body height and binocular depth calculation has the following three situations:

[0119] like Fig.11 In the case of Figure A, the depth error estimated by binocular vision is generally within 2 cm, which is half the width of the fish body, so it can be ignored.

[0120] like Fig.11 In the case of Figure B, since the depth error estimated by the binocular camera is generally about half of the fish height, 8 cm, it will cause errors if ignored. Here, the distance H from the center of the binocular camera to the fish surface is measured by binocular, and then the distance from the fish surface to the binocular camera is obtained by combining the top view of the monocular camera above water, which is G. The height of the fish is g, so the distance h from the center of the binocular camera to the center of mass of the fish can be calculated as:

[0121]

[0122] like Fig.11 In the case of Figure C, since the depth error estimated by the binocular camera is generally about half the fish body length, 13cm, it will cause a large error if ignored. Here, the distance H from the center of the binocular camera to the fish surface is measured by binocular, and then the distance from the fish surface to the binocular camera is obtained by combining the top view of the monocular camera above water, which is G. The length of the fish is g, so the distance h from the center of the binocular camera to the fish mass center can be calculated as:

[0123]

[0124] Through the above method, combined with the depth information formula in step 3:

[0125]

[0126] Get the final depth information formula:

[0127]

[0128] The depth information of the fish can be accurately obtained, thereby accurately obtaining the three-dimensional position information of the fish.

[0129] S7: The position and posture information of the fish is calculated by fitting the spatial curve through the three-dimensional coordinates of the key points.

[0130] In this step, spatial curve fitting is performed through the three-dimensional coordinates of key points to fit a smooth curve to represent the fish body, especially the tail of the fish. The non-rigid deformation of the fish is analyzed by the changes in this curve, and the non-rigid motion of the fish is analyzed by the frequency and angle of the fish body swinging within a certain period of time. Fig.12 is an analysis diagram of the swing frequency and angle of the fish body provided by an embodiment of the present invention, such as Fig.12 As shown. The calculation method includes: tail swing angle x, time t from the start of tail swing to return to the original position, tail swing speed: , by calculating the number of maximum values ​​of the swing angle α within a certain time T, the number of times the left pectoral fin swings n is counted. n represents the number of tail fin swings, N represents the total number of frames collected, and 30 represents the acquisition frame rate, then the tail swing frequency is: The value corresponding to the tail-wagging frequency is the speed of the fish's tail swinging. Combined with the swinging angle of the upper swing, α can make a detailed analysis of the fish's swimming morphological characteristics. This technology can serve as the basis for many technologies. If there is a study that needs to study the relationship between the speed of fish swimming and the frequency and speed of the tail-wagging, then this technology can very conveniently analyze the required data, and then through quantitative training, the computer can infer the fish's movement method and even predict the fish's position in the next few seconds based on the way the fish's tail swings.

[0131] Fig.13 This is a principle diagram of the swing frequency and angle error of the fish body based only on a two-dimensional top-view angle image provided by an embodiment of the present invention. Fig.13 As shown, in this embodiment, for the non-rigid deformation of the fish tail in the top view, many studies determine the swinging frequency and angle of the fish tail or fin by analyzing the top view from top to bottom, but this has a very large limitation, because the movement of the fish is three-dimensional. If the fish moves in a horizontal plane, the two-dimensional top view is sufficient to predict the non-rigid deformation information of the fish, but when the fish moves in three dimensions, only using two-dimensional information will cause large errors.

[0132] Specifically, in step S7, the fish's position information is determined by setting spatial coordinates with the help of binocular images at a bird's-eye view. Fig.14The embodiment of the present invention provides a judgment diagram for determining the specific posture of a fish based on a binocular image at a top-down angle under spatial coordinate conditions; the specific judgment information is as follows: Fig.14 As shown in the figure, through two types of information: posture and non-rigid deformation, this technology can completely restore the morphological characteristics of fish in motion.

[0133] Furthermore, the non-rigid deformation motion information of fish is inferred through the two-dimensional information of key points at the top-down angle. When identifying the morphological characteristics of fish, the side view angle and the top view angle are videos shot at the same time, and then the non-rigid deformation motion information and posture information of the fish are integrated to obtain the morphological characteristics of underwater laterally flattened fish.

[0134] The intelligent recognition device for underwater fish morphological characteristics provided by the present invention is described below. The intelligent recognition device for underwater fish morphological characteristics described below and the intelligent recognition method for underwater fish morphological characteristics described above can be referenced to each other.

[0135] Fig.15 A schematic diagram of the structure of an intelligent device for identifying underwater fish morphological characteristics provided by an embodiment of the present invention is shown in FIG. Fig.15 As shown, the device comprises:

[0136] An acquisition module 1501 is used to acquire a monocular video of a fish swimming from an overhead angle above water and a binocular video of a fish swimming from a side angle under water;

[0137] The acquisition module 1501 is further used to capture the above-water top-view monocular video and the underwater side-view binocular video frame by frame to obtain an above-water top-view angle picture set and an underwater side-view angle picture set;

[0138] The training module 1502 is used to annotate the above-water top-view angle picture set and the underwater side-view angle picture set, and input them into the target detection network and the key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle and the two-dimensional key point position information of the underwater side-view angle of the fish body;

[0139] A determination module 1503 is used to obtain non-rigid deformation information of the fish body based on the two-dimensional key point position information of the fish body from an overhead view angle above water;

[0140] The determination module 1503 is further used to obtain the three-dimensional position information of the key points of the side view angle based on the two-dimensional key point position information of the underwater side view angle of the fish body and the visual difference of the binocular image; and obtain the fish body posture information based on the three-dimensional position information;

[0141] The determination module 1503 is further used to obtain fish body morphological feature information based on the fish body non-rigid deformation information and fish body posture information.

[0142] Fig.16A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.16 As shown, the electronic device may include: a processor (processor) 1610, a communication interface (Communications Interface) 1620, a memory (memory) 1630 and a communication bus 1640, wherein the processor 1610, the communication interface 1620, and the memory 1630 communicate with each other through the communication bus 1640. The processor 1610 can call the logic instructions in the memory 1630 to execute the intelligent recognition method of underwater fish morphological features, which includes: obtaining an overhead monocular video and an underwater side-view binocular video of a fish swimming; intercepting the overhead monocular video and the underwater side-view binocular video frame by frame to obtain an overhead picture set and an underwater side-view picture set; annotating the overhead picture set and the underwater side-view picture set, inputting them into the target detection network and the key point detection network to learn and obtain the two-dimensional key point position information of the fish body at the overhead angle and the two-dimensional key point position information of the underwater side-view angle; obtaining the non-rigid deformation information of the fish body based on the two-dimensional key point position information of the fish body at the overhead angle; obtaining the three-dimensional position information of the key points at the side-view angle based on the two-dimensional key point position information of the fish body at the underwater side-view angle and the visual difference of the binocular image; obtaining the fish body posture information based on the three-dimensional position information; obtaining the fish body morphological feature information based on the non-rigid deformation information and the fish body posture information.

[0143] In addition, the logic instructions in the above-mentioned memory 1630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0144] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the underwater fish morphological feature intelligent recognition method provided by the above-mentioned methods, the method comprising: obtaining an above-water top-down angle monocular video and an underwater side-view angle binocular video of a fish swimming; capturing the above-water top-down angle monocular video and the underwater side-view angle binocular video frame by frame respectively to obtain an above-water top-down angle picture set and an underwater side-view angle picture set; capturing the above-water top-down angle monocular video and the underwater side-view angle binocular video frame by frame ... After data annotation of the picture set and the underwater side view angle picture set, they are input into the target detection network and the key point detection network to learn the two-dimensional key point position information of the fish body at the overhead view angle above water and the two-dimensional key point position information of the underwater side view angle; based on the two-dimensional key point position information of the fish body at the overhead view angle above water, the non-rigid deformation information of the fish body is obtained; based on the two-dimensional key point position information of the fish body at the underwater side view angle and the visual difference of the binocular image, the three-dimensional position information of the key points of the side view angle is obtained; based on the three-dimensional position information, the fish body posture information is obtained; based on the non-rigid deformation information and the fish body posture information of the fish body, the morphological feature information of the fish body is obtained.

[0145] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the underwater fish morphological feature intelligent recognition method provided by the above-mentioned methods, the method comprising: obtaining an above-water top-down angle monocular video and an underwater side-view angle binocular video of a fish swimming; intercepting the above-water top-down angle monocular video and the underwater side-view angle binocular video frame by frame respectively to obtain an above-water top-down angle picture set and an underwater side-view angle picture set; and digitizing the above-water top-down angle picture set and the underwater side-view angle picture set. After being marked, the information is input into the target detection network and the key point detection network to learn the two-dimensional key point position information of the fish body at the overhead view angle above water and the two-dimensional key point position information of the underwater side view angle; based on the two-dimensional key point position information of the fish body at the overhead view angle above water, the non-rigid deformation information of the fish body is obtained; based on the two-dimensional key point position information of the fish body at the side view angle underwater and the visual difference of the binocular image, the three-dimensional position information of the key points at the side view angle is obtained; based on the three-dimensional position information, the fish body posture information is obtained; based on the non-rigid deformation information and the fish body posture information, the fish body morphological feature information is obtained.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent method for identifying underwater fish morphological characteristics, characterized in that: include: Obtain monocular video from above water and binocular video from below water. The above-water top-view monocular video and the underwater side-view binocular video are respectively captured frame by frame to obtain an above-water top-view angle picture set and an underwater side-view angle picture set; After data annotation, the above-water top-view angle picture set and the underwater side-view angle picture set are input into the target detection network and the key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle and the two-dimensional key point position information of the underwater side-view angle of the fish body; Based on the position information of the two-dimensional key points of the fish body from the top-down angle above water, the non-rigid deformation information of the fish body is obtained; Based on the two-dimensional position information of the key points of the side view angle of the fish body underwater and the visual difference of the binocular image, the three-dimensional position information of the key points of the side view angle is obtained; Based on the three-dimensional position information, obtaining fish body posture information; Based on the non-rigid deformation information of the fish body and the fish body posture information, the fish body morphological feature information is obtained.

2. The method for intelligently identifying underwater fish morphological characteristics according to claim 1, characterized in that: The method of obtaining the monocular video from the top-down angle of the water and the binocular video from the side-down angle of the water of the fish body swimming comprises: Use an underwater monocular camera to obtain a top-down view of the fish swimming in the water; The underwater binocular camera is used to obtain underwater side-view binocular videos of fish swimming.

3. The method for intelligently identifying underwater fish morphological characteristics according to claim 2, characterized in that: Before obtaining the underwater side-view binocular video of the fish body swimming by the underwater binocular camera, the method further includes: Calibrate the underwater binocular camera; Get the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera; After obtaining the underwater side-view binocular video of the fish body swimming by the underwater binocular camera, the method further includes: The video taken by the underwater side-viewing binocular camera is corrected by obtaining the intrinsic parameter matrix and extrinsic parameter matrix of the binocular camera.

4. The method for intelligently identifying underwater fish morphological characteristics according to claim 1, characterized in that: After the above-water top-view monocular video and the underwater side-view binocular video are respectively captured frame by frame, the method further includes: Selecting clear and representative pictures from the side-view angle pictures captured frame by frame, splitting the pictures in half, and using the pictures after the half-split as a training data set; Clear and representative pictures are selected from the side-view pictures captured frame by frame as training data sets.

5. The method for intelligently identifying underwater fish morphological characteristics according to claim 1, characterized in that: After data annotation, the above-water top-view angle picture set and the underwater side-view angle picture set are input into the target detection network and the key point detection network to learn and obtain the two-dimensional key point position information of the above-water top-view angle of the fish body and the two-dimensional key point position information of the underwater side-view angle, including: The above-water top-view image set and the underwater side-view image set are input into the target detection network to obtain the positioning frame for fish target detection; the key point recognition network is used to detect the key points of the image in the positioning frame to obtain the two-dimensional key point position information of the fish body.

6. The method for intelligently identifying underwater fish morphological characteristics according to claim 1, characterized in that: The non-rigid deformation information of the fish body is obtained based on the two-dimensional key point position information of the fish body from the top view angle above water, including: Get the connecting line of the two-dimensional key points of the water top-down angle at the previous moment and the next moment; The angle between the two connecting lines is obtained, which is the non-rigid deformation angle of the fish body.

7. The method for intelligently identifying underwater fish morphological characteristics according to claim 1, characterized in that: The visual disparity of the binocular video includes the depth information of the fish body based on the binocular camera, and the depth information is obtained based on the side-view binocular image combined with the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera, specifically including: S71, obtaining the angle of the fish's tail swing and the time from the tail swing to the return to the original position, and obtaining the fish's tail swing speed; S72, calculating the number of maximum values ​​of the swing angle within a certain time period to obtain the number of swings of the tail fin of the fish body; S73, obtaining a tail swing frequency based on the total number of acquired frames and the acquired frame rate; S74, based on the minimum distance from the binocular camera center to the fish body surface, the minimum distance from the binocular camera to the fish body surface and the fish body height, obtain the distance from the binocular camera center to the fish body center of mass; S75. Obtain depth information based on the fish's tail-swinging speed, number of swings, tail-swinging frequency, and the distance from the center of the binocular camera to the fish's center of mass.

8. The method for intelligently identifying underwater fish morphological characteristics according to claim 7, characterized in that: Based on the two-dimensional position information of the key points of the side view angle of the fish body underwater and the visual difference of the binocular image, the three-dimensional position information of the key points of the side view angle is obtained; Based on the three-dimensional position information, the fish body posture information is obtained, including: Based on the depth information and the two-dimensional position information of the key points of the side view angle of the fish body underwater, the three-dimensional position information of the key points of the side view angle is obtained; The three-dimensional position information of the key points is subjected to spatial curve fitting to obtain the position and posture information of the fish body.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for intelligently identifying the morphological characteristics of underwater fish as described in any one of claims 1 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for intelligently identifying the morphological characteristics of underwater fish as described in any one of claims 1 to 8 is implemented.

Images