A method and device for automatically estimating the length of underwater fish based on a monocular camera

Abstract

The present invention belongs to the technical field of automatic estimation of fish body length, and discloses an automatic estimation method and device for underwater fish body length based on a monocular camera. The method comprises obtaining an underwater RGB color image containing fish based on a monocular camera; performing depth estimation on the underwater RGB color image to obtain absolute depth information of each area in the underwater RGB color image to form a depth map; detecting the specific position of the fish in the depth map, and calculating and obtaining the mass point and target part of the fish; taking the mass point as the center, taking the depth value of each block from the depth map, and taking the average value of the depth value of each block as the average distance D of the fish body from the camera, and judging whether the inclination angle of the fish body is within the set angle threshold range, and if so, estimating the body length of the fish. The present invention greatly reduces the cost and maintenance cost, has good applicability, can estimate the length of swimming fish, and gets rid of the limitation of fixed equipment shooting.

Description

Technical Field

[0001] The invention relates to the technical field of automatic estimation of fish body length, and in particular to a method and device for automatic estimation of underwater fish body length based on a monocular camera. Background Art

[0002] Among a series of methods for estimating fish body length, the simpler method relies on traditional manual measurement and artificial underwater observation. The manual measurement method is time-consuming and labor-intensive, and repeatability and accuracy are difficult to guarantee. The total production can only be estimated through partial measurement, which will lead to large deviations in the results. Artificial underwater observation is highly subjective and has large errors. Some fish production estimation devices use a monocular camera to take pictures of a fixed conveyor belt for transporting fish from a bird's-eye view on the water surface, and estimate its length based on the number of pixels occupied by the fish in the picture. This method has an ideal error only when the fish is at a fixed distance, and there will be a large error when the fish's swimming position changes, so it cannot be actually applied to underwater observation platforms.

[0003] In recent years, with the rapid development of computer technology and artificial intelligence, some studies have used convolutional neural networks to detect and segment fish to improve the accuracy of measurement. There is a method that deploys a monocular camera looking down in the atmosphere, and uses a deep learning model to regress the physical depth and imaging length of the fish to obtain the fish body length data. The disadvantage is that it needs to overcome the influence of optical factors between different media. There is also a method based on binocular vision, using binocular ranging to calculate the distance using the parallax of the pictures taken by the two cameras, and then combine it with the body length of the picture to calculate the actual body length of the fish. However, the binocular method has high requirements for the accuracy of the calibration parameters, and the maintenance cost is very high in actual operation. The size of the equipment often conflicts with the vehicle platform. The "depth" described here is a professional term for monocular estimation, which means the distance of the target from the camera, not the depth underwater.

[0004] Due to the high cost of deploying underwater equipment and the difficulty of maintenance, the current underwater equipment is mainly deployed with a single camera. Therefore, the existing stereoscopic body length estimation algorithm based on binocular cameras is not suitable for this environment, and a more difficult monocular fish body length estimation algorithm is needed. The monocular vision system mainly relies on a camera to capture images and interprets and analyzes these images through computer vision algorithms. The principle of using a monocular camera to estimate the body length of fish is to first estimate the distance from the fish target to the camera, then extract the fish in the video through the target detection algorithm, and then estimate the actual body length of the fish through the three-dimensional change algorithm. In this process, the existing algorithms are deployed on land to measure the body length of fish, not directly deployed underwater. The distance from the target to the camera is fixed, so the difficulty is relatively low. Secondly, in the underwater environment, there are color casts and image blur. Accurately estimating the distance from the fish target to the camera is the most critical factor affecting the final result. Our algorithm is already in a leading position in the field of underwater monocular estimation and has practical use value. In general, the current image processing algorithms have the following problems: 1. They are mainly used on water platforms, with fixed conveyor belts and camera distances, which reduces the algorithm difficulty and enables fish body estimation, but are difficult to apply to underwater equipment because underwater equipment will be in a moving state during the observation process, and fish will swim freely, making the algorithm difficult. 2. Binocular cameras are mainly used because the triangulation method of binocular cameras has high accuracy, but since most underwater equipment platforms are difficult to maintain and the product price is high, it is necessary to develop a high-precision fish body length estimation algorithm based on a monocular camera. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for automatically estimating the length of underwater fish based on a monocular camera in view of the deficiencies in the prior art.

[0006] To achieve the above object, in a first aspect, the present invention provides a method for automatically estimating the length of underwater fish based on a monocular camera, comprising:

[0007] Step 1: Obtain an underwater RGB color image containing fish based on a monocular camera;

[0008] Step 2: performing depth estimation on the underwater RGB color image to obtain absolute depth information of each area in the underwater RGB color image to form a depth map;

[0009] Step 3, detecting the specific position of the fish in the depth map, and calculating and obtaining the mass point and target part of the fish;

[0010] Step 4: Taking the mass point as the center, extract the The depth value of the block, and The average depth value of each block is taken as the average distance D between the fish body and the camera, and it is judged whether the tilt angle of the fish body is within the set angle threshold range. If the tilt angle of the fish body is within the set angle threshold range, the estimated fish body length is:

[0011]

[0012] in, is the calculated length of the fish body, is the width of the underwater RGB color image, and f is the focal length of the lens.

[0013] Furthermore, the step 2 specifically includes:

[0014] Extracting deep features from underwater RGB color images using an improved EfficientNet B5 encoder ;

[0015] The underwater RGB color image is segmented using a segmentation prior generator to obtain segmentation prior features. ;

[0016] Adaptive feature adjustment module is used to segment the prior features With deep features Perform fusion to obtain fused features;

[0017] An improved decoder is used to generate a feature map F based on the fused features;

[0018] The visual transformation module is used to generate the range attention map R and the depth box center vector c(b) according to the feature map F. The range attention map R is adjusted to n channels through the convolution layer, and the final depth map is calculated through Softmax activation and the depth box center vector c(b).

[0019] Furthermore, the improved EfficientNet B5 encoder uses discrete wavelet transform for downsampling instead of the original convolution method in the traditional EfficientNet B5; the improved decoder uses inverse discrete wavelet transform for upsampling instead of the bilinear interpolation method of the traditional decoder.

[0020] Furthermore, the adaptive feature adjustment module is formed by stacking eight layers of identical residual blocks, and the residual blocks include a network structure formed by sequentially connecting two groups of stacked spatial adaptive denormalization layers, activation layers, and convolutional layers, and the input end of the topmost spatial adaptive denormalization layer and the output end of the bottommost convolutional layer are both connected to the addition operation module;

[0021] The spatial adaptive denormalization layer is used to spatially transform the segmentation prior features With deep features Fusion, as follows:

[0022] The segmentation prior features The size is adjusted to the depth feature Consistency;

[0023] Use convolutional layers to extract prior features from segmentation Learning the transformation parameters , ;

[0024] For deep features Perform instance normalization to obtain , Indicates instance normalization;

[0025] Using Transform Parameters , right Perform affine transformation to obtain fused features for:

[0026] .

[0027] Furthermore, the depth box center vector c(b) is calculated as follows:

[0028]

[0029] in, is the i-th value in the depth box center vector c(b), are the minimum and maximum pixel depth values ​​in the entire dataset, respectively. , are the i-th and j-th values ​​in the depth box width vector b, respectively. , Respectively vector The i-th and j-th values ​​in .

[0030] Furthermore, the target parts include eyes, mouth, pelvic fins, dorsal fins, anal fins and caudal fins of fish.

[0031] Furthermore, the value of m is 3.

[0032] Furthermore, the set angle threshold range is 75°-105°.

[0033] In a second aspect, the present invention provides an automatic estimation device for underwater fish body length based on a monocular camera, comprising a storage medium and a processor, wherein the storage medium stores a computer program, and the computer program is used to implement the above method when executed by the processor.

[0034] Beneficial effects: 1. The use of monocular cameras greatly reduces costs and maintenance expenses, and has good applicability for various usage scenarios;

[0035] 2. In the depth estimation module, the discrete wavelet transform layer is used to replace the traditional convolution layer, which retains more feature information and optimizes the shortcomings of large depth estimation errors at the edges of underwater objects;

[0036] 3. Using segmentation priors to guide depth estimation tasks enhances the network's ability to distinguish different objects, improves the consistency of depth within the same area and the differences between different areas;

[0037] 4. It can estimate the length of fish swimming freely underwater, getting rid of the limitations of fixed equipment shooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a residual block according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of a spatially adaptive denormalization layer according to an embodiment of the present invention;

[0040] Figure 3 It is a schematic diagram of the principle of estimating the body length of fish based on the average distance;

[0041] Figure 4 It is a schematic diagram for estimating the tilt angle of the fish body. DETAILED DESCRIPTION

[0042] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0043] like Figures 1 to 4 As shown, an embodiment of the present invention provides a method for automatically estimating the length of underwater fish based on a monocular camera, comprising:

[0044] Step 1: Obtain an underwater RGB color image containing fish based on a monocular camera. The size of the underwater RGB color image is H×W×3.

[0045] Step 2: perform depth estimation on the underwater RGB color image to obtain absolute depth information of each area in the underwater RGB color image to form a depth map. The above step 2 specifically includes:

[0046] Extracting deep features from underwater RGB color images using an improved EfficientNet B5 encoder , the above is a real number, H is the height of the underwater RGB color image, and W is the width of the underwater RGB color image. The above-mentioned improved EfficientNet B5 encoder uses discrete wavelet transform for downsampling instead of the original convolution method in the traditional EfficientNet B5. First, the DWT module in the discrete wavelet transform module decomposes the input into multiple sub-bands through a set of two-dimensional filters to obtain information of different frequencies and directions. Row processing applies low-pass and high-pass filters to each row of the input and downsamples the results; column processing applies the same low-pass and high-pass filters to each column of the row processing results and downsamples the results. The order of row processing and column processing does not affect the results and can be swapped. After row and column processing, the sub-bands will pass through the following four filters: , , , , their combination relationship reflects the frequency information decomposition in different directions, which is defined as follows:

[0047]

[0048] Input X is decomposed into rows and columns and After four filters, four sub-bands are calculated: LL, LH, HL, HH. LL represents the low-frequency part of the image, which contains the overall structure and general shape information. LH, HL, and HH represent the high-frequency detail information in the vertical, horizontal, and diagonal directions respectively. The expressions for the values ​​of any point (p, q) on the four sub-bands are:

[0049]

[0050] Finally, the four subbands are concatenated into a tensor along the channel dimension and passed through the BN layer and activation function. This method can effectively extract image features and improve the performance of the network on the edge. Although X is downsampled in the encoder, due to the biorthogonal properties of the wavelet transform, X can be restored through its inverse transform IWT:

[0051]

[0052] For example, for the value on X(2p-1,2q-1), it can be restored by the following formula:

[0053]

[0054] Similarly, the values ​​of other points can be obtained by similar methods.

[0055] The Segmentation Prior Generator is used to segment the underwater RGB color image to obtain the segmentation prior features. The segmentation prior generator generates segmentation prior feature maps by using a pre-trained qualified segmentation network encoder. , helping the network to distinguish different underwater objects. In the selection of the pre-trained segmentation network encoder, the present invention selected the encoder of Efficientsam. The segmentation prior generator learns the segmentation prior features to dynamically adjust the distribution of depth features, effectively utilizing the guiding role of the segmentation prior features on depth estimation; improving the consistency of the depth inside the object, while highlighting the depth differences between different regions. In severely degraded underwater scenes, the network's ability to recognize object structures is enhanced.

[0056] Specifically, in the segmentation prior generator, the underwater RGB color image is upsampled to 1024×1024×3 size after being input into the segmentation prior generator to retain more local structures. The upsampled result enters the segmentation network encoder to generate the segmentation prior feature map , .

[0057] Adaptive feature adjustment module (AFR Module) is used to segment the prior features With deep features The adaptive feature adjustment module is formed by stacking eight layers of identical residual blocks. For details, see Figure 1 , Figure 1 The diagram shows the structural principle of one of the residual blocks. The residual block consists of a network structure formed by two sets of stacked spatially-adaptive denormalization layers, activation layers (RELU) and convolutional layers (Conv). The input of the top spatially-adaptive denormalization layer and the output of the bottom convolutional layer are both connected to the addition operation module. Figure 2 , Figure 2 The schematic diagram shows the structural principle of the spatial adaptive anti-normalization layer, which is used to spatially transform the segmentation prior features. With deep features Fusion. Specifically, segmentation prior features After entering the spatial adaptive anti-normalization layer, it is adjusted to the depth feature Consistent , and use convolutional layers to learn transformation parameters from segmentation priors , , in the depth feature After instance normalization, the depth features Perform instance normalization to obtain , Represents instance normalization; using transformation parameters , right Perform affine transformation to obtain fused features for:

[0058] .

[0059] The improved decoder is used to generate feature map F based on the fused features. The above improved decoder uses inverse discrete wavelet transform for upsampling instead of the bilinear interpolation method of the traditional decoder. In the upsampling process, the high-frequency components are lost as little as possible, and the local detail features of the image are better preserved. The network obtains better structural texture clues, and the generated depth map is more refined and more accurate at the edges.

[0060] The vision transformer is used to transform the feature map Generate a range attention map R and a depth box center vector c(b), where the depth box center vector c(b) defines how to divide the depth interval of the input image, and the range attention map R contains useful information that can be used for pixel-level depth calculation. It cuts off the output part of the MLP head and the original feature map through the visual transformation module. Fusion generation is performed. The range attention map R is adjusted to n channels through the convolution layer, and the final depth map is obtained through Softmax activation and depth box center vector calculation. The above n can be set as needed, preferably 256.

[0061] Specifically, the visual transformation module first passes the decoded features through a convolution block with a convolution kernel size of p×p, a step size of p, and an output channel number of E. The convolution output is a size of The tensor of is flattened to obtain a vector , called patch embeddings. According to the normal operation, the position encoding is added to the patch embeddings and fed into the transformer to output The MLP head is used on the first output, using ReLU activation and outputting an N-dimensional vector Finally, the vector Normalize and obtain the following depth box width vector b:

[0062]

[0063] in, is the i-th value in the depth box width vector b, , Respectively vector The i-th and j-th values ​​in , used to ensure that the width of each box is strictly non-negative.

[0064] The subsequent part of the output is used as a 1×1 convolution kernel and is combined with the decoded feature map after the 3×3 convolution layer. Convolution is performed to obtain the range attention map R. R then passes through a 1×1 convolution layer to change the number of channels to N, and then enters the Softmax activation layer to obtain N Softmax scores , respectively represent the probability of the N depth box centers at each pixel, k = 1, ..., N. The calculation method of the above depth box center vector c(b) is as follows:

[0065]

[0066] in, is the i-th value in the depth box center vector c(b), are the minimum and maximum pixel depth values ​​in the entire dataset, respectively. is the jth value in the depth box width vector b.

[0067] Finally, at each pixel, the N Softmax scores of the pixel The final depth value is calculated by linear combination with the depth box center vector c(b) for:

[0068]

[0069] in, is the kth value in the center vector c(b) of the depth box. The generated depth map is initially half the original size and is upsampled to the original resolution H×W×1 by bilinear interpolation to output the final depth map.

[0070] Step 3: Detect the specific position of the fish in the depth map, and calculate the mass point and target part of the fish. The specific implementation method is as follows:

[0071] Using the YOLOpose network (feature extraction network) and the HRnet network (feature fusion network), the position information box (x, y, w, h, confidence) and the target part coordinate keypot of each fish in the picture are directly predicted, where x, y are the coordinates of the center point of the fish position box, w, h are the width and height of the position box, and condfidence is the confidence level of the current position box. Specifically, the above target parts include the eyes, mouth, pelvic fins, dorsal fins, anal fins and caudal fins of fish.

[0072] Step 4: Take the mass point as the center and extract it from the depth map The depth value of the block, and The average depth value of each block is taken as the average distance D between the fish body and the camera, and it is judged whether the tilt angle of the fish body is within the set angle threshold range. If the tilt angle of the fish body is within the set angle threshold range, the estimated fish body length is:

[0073]

[0074] in, is the calculated length of the fish body, is the width of the underwater RGB color image, and f is the focal length of the lens. The value of m is preferably 3. The calculation principle is as follows Figure 3 As shown, Figure 3 The figure shows the situation where the center of mass of the fish body coincides with the center of the image (CCD optical center).

[0075] In addition, the calculation method of the tilt angle of the fish body is as follows Figure 4 As shown, Figure 4 The θ in the equation is the angle of inclination of the fish body. The specific estimation method is as follows: First, the length calculated by the formula for estimating the length of the fish body is used as Figure 4 The length of side C in Figure 4 In the figure, the coordinates of the fish's eyes and tail have been calculated before. The depth values ​​of 3×3 blocks are taken from the depth map with the coordinate points of the fish's eyes and tail as the center, and the average of the depth values ​​of the 3×3 blocks is taken as the length of the A side and the B side. Then, the cosine theorem can be used to estimate the inclination angle θ of the fish body. When the inclination angle θ of the fish body is about 90°, the estimated length of the fish body is close to the actual length. The angle threshold range set above can be set according to the accuracy requirements. The narrower the angle threshold range, the higher the accuracy of the evaluation, but the probability of successful evaluation will decrease; the wider the angle threshold range, the lower the accuracy of the evaluation, but the probability of successful evaluation will increase. The set angle threshold range is preferably 75°-105°.

[0076] Based on the above embodiments, those skilled in the art can easily understand that the present invention also provides an automatic estimation device for underwater fish body length based on a monocular camera, including a storage medium and a processor, wherein the storage medium stores a computer program, and the computer program is used to implement the above method when executed by the processor.

[0077] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for automatically estimating the length of underwater fish based on a monocular camera, characterized in that: include: Step 1: Obtain an underwater RGB color image containing fish based on a monocular camera; Step 2: performing depth estimation on the underwater RGB color image to obtain absolute depth information of each area in the underwater RGB color image to form a depth map; Step 3, detecting the specific position of the fish in the depth map, and calculating and obtaining the mass point and target part of the fish; Step 4: With the particle as the center, take out the depth values ​​of m*m blocks from the depth map, and take the average of the depth values ​​of the m*m blocks as the average distance D of the fish body from the camera, and judge whether the tilt angle of the fish body is within the set angle threshold range. If the tilt angle of the fish body is within the set angle threshold range, the estimated fish body length is: ； in, is the calculated length of the fish body, is the width of the underwater RGB color image, and f is the focal length of the lens; The step 2 specifically includes: Extracting deep features from underwater RGB color images using an improved EfficientNet B5 encoder ; The underwater RGB color image is segmented using a segmentation prior generator to obtain segmentation prior features. ; Adaptive feature adjustment module is used to segment the prior features With deep features Perform fusion to obtain fused features; An improved decoder is used to generate a feature map F based on the fused features; The visual transformation module is used to generate the range attention map R and the depth box center vector c(b) according to the feature map F. The range attention map R is adjusted to n channels through the convolution layer, and the final depth map is obtained through Softmax activation and the depth box center vector c(b). The improved EfficientNet B5 encoder uses discrete wavelet transform for downsampling instead of the original convolution method in the traditional EfficientNet B5; the improved decoder uses inverse discrete wavelet transform for upsampling instead of the bilinear interpolation method of the traditional decoder; The adaptive feature adjustment module is formed by stacking eight layers of identical residual blocks, and the residual blocks include a network structure formed by sequentially connecting two groups of stacked SPADE blocks, activation modules and convolution modules, and the input end of the top SPADE block and the output end of the bottom convolution module are both connected to the addition operation module; The SPADE block is used to spatially segment the prior features With deep features Fusion, as follows: The segmentation prior features The size is adjusted to the depth feature Consistency; Use convolutional layers to extract prior features from segmentation Learning the transformation parameters , ; For deep features Perform instance normalization to obtain , Indicates instance normalization; Using Transform Parameters , right Perform affine transformation to obtain fused features for: 。 2. The method for automatically estimating the length of underwater fish based on a monocular camera according to claim 1, characterized in that: The depth box center vector c(b) is calculated as follows: ； in, is the i-th value in the depth box center vector c(b), are the minimum and maximum pixel depth values ​​in the entire dataset, respectively. , are the i-th and j-th values ​​in the depth box width vector b, respectively. , Respectively vector The i-th and j-th values ​​in .

3. The method for automatically estimating the length of underwater fish based on a monocular camera according to claim 1, characterized in that: The target parts include the eyes, mouth, pelvic fins, dorsal fins, anal fins and caudal fins of the fish.

4. The method for automatically estimating the length of underwater fish based on a monocular camera according to claim 1, characterized in that: The value of m is 3.

5. The method for automatically estimating the length of underwater fish based on a monocular camera according to claim 1, characterized in that: The set angle threshold range is 75°-105°.

6. An underwater fish body length automatic estimation device based on a monocular camera, comprising a storage medium and a processor, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 5.

Images