A Double-Layer High-Throughput Fish Counting Device and Method Based on Machine Vision

Abstract

The present invention discloses a double-layer high-throughput fish counting device and method based on machine vision. The device includes a front channel, a double-layer counting channel, and a double-layer fish outlet. One end of the double-layer counting channel is connected to the front channel, and the other end is connected to the double-layer fish outlet. The double-layer counting channel and the double-layer fish outlet form an upper and lower double-layer fish passage. The double-layer counting channel includes a counting channel housing and a double-layer channel composed of a white U-shaped channel and a transparent U-shaped channel, each provided with an image acquisition module. The device of the present invention makes full use of the space in the vertical direction, counts fish through the upper and lower double-layer channels, can at least double the counting throughput. The provided counting method improves the operation efficiency through a dual-process mode and does not require precise positioning of each fish individual. Newly emerged fish individuals are judged and counted through the front and back frames, without continuous tracking of the target, which can greatly reduce the calculation amount and algorithm complexity and improve the counting speed.

Description

Technical Field

[0001] The present invention relates to the technology of fishery aquaculture equipment, and particularly to a double-layer high-throughput fish counting device and method based on machine vision. Background Art

[0002] Accurately estimating the number of fish is an important part of fish aquaculture production, and it is of great significance in aspects such as aquaculture density control and growth status assessment. With the development of modern fishery, the intensification and industrialization levels of China's fish aquaculture industry have been continuously improved. While the aquaculture scale and density are increasing, the aquaculture area is continuously decreasing. Therefore, how to efficiently count fish in the narrow aisle space of fish aquaculture factories and improve the utilization rate of aquaculture space and the transfer efficiency of equipment has attracted more and more attention.

[0003] Chinese Invention Application Publication No. CN113408687A discloses a high-throughput fry online counting device and method. Based on the machine vision method, the partition in the counting channel in the traditional infrared counting scheme is removed, the effective area of the counting channel is increased, and through the tracking and prediction strategy of multiple consecutive frames, high-throughput counting can be achieved. Its main motivation is to increase the effective area in the horizontal direction, but the utilization of the space in the vertical direction (such as the space in the middle of the bracket) is not good. In addition, the complexity of the strategy algorithm using continuous multi-frame tracking is relatively high, which is not conducive to improving the calculation throughput.

[0004] Chinese Invention Application Publication No. CN114938790A discloses a portable fry automatic counting device. The fish inlet adopts a structure similar to a drawer, and the bracket adopts a foldable form. Without changing the throughput, the volume of the counting device can be greatly reduced, which is convenient for the transfer and transportation of the counting device. However, it only maintains the throughput and reduces the volume, and cannot further improve the throughput.

[0005] The improvement of fish counting throughput mainly lies in the improvement of space utilization rate and counting algorithm efficiency. Facing the current situation of increasing large-scale aquaculture counting requirements and decreasing aquaculture area, there is an urgent need for a more efficient counting device and method. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to improve the fish counting throughput and efficiency within the effective floor area, overcome the deficiencies in the prior art, and provide a double-layer high-throughput fish counting device and method based on machine vision.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] A double-layer high-throughput fish counting device based on machine vision, comprising a front channel, a double-layer counting channel and a double-layer fish outlet; the double-layer counting channel is the main part of the counting device, one end is connected to the front channel, and the other end is connected to the double-layer fish outlet. The double-layer counting channel and the double-layer fish outlet form an upper and lower fish passage, and a display and control module is also arranged on the double-layer counting channel; the front channel is the entrance of the fish, one end is closed, and the other end is placed near the upper part of the double-layer counting channel with a flip cover, which can be used to block external light sources; the double-layer counting channel includes a counting channel housing, a white U-shaped channel, a transparent U-shaped channel and a white light supplementary lamp; the counting channel housing includes a hollow passage, and the upper and lower parts of the passage bulge outwards, and image acquisition modules are arranged in the protruding housings; the white U-shaped channel and the transparent U-shaped channel are both inserted and fixed in the hollow passage of the double-layer counting channel housing, and the transparent U-shaped channel is arranged below the white U-shaped channel, and the two divide the hollow passage of the double-layer counting channel housing into upper and lower channels; the white light supplementary lamp is arranged on the inner walls on both sides of the hollow passage for providing sufficient light inside.

[0009] In the above technical solution, further, the front channel includes a front channel housing, a fish stratifier and a fish receiver; the fish receiver is a container with a strip-shaped opening at the bottom for receiving fish supplied from above; an interface for connecting a fish suction pump is arranged at the bottom near the closed end of the front channel housing, and a row of strip-shaped water leakage holes is arranged in the middle part. Below the water leakage holes is a U-shaped groove, and a water pipe connection port is arranged at the bottom of the groove for connecting a water pump to supply water or serving as a drainage port for drainage; the fish receiver is detachably installed on the front channel housing, and the fish stratifier is installed at the end of the front channel housing. The fish stratifier is provided with several channels for diverting fish so that the fish are dispersed into the upper and lower channels of the double-layer counting channel.

[0010] Furthermore, the fish stratifier is divided into upper and lower layers, and the upper layer is spaced by cylindrical tubes, which can preliminarily screen the size of the fish. The fish with a thickness less than that of the cylindrical tube enter the lower channel of the double-layer counting channel, and the rest of the fish enter the upper channel of the double-layer counting channel.

[0011] Furthermore, the front channel includes a double-layer front channel housing and a double-hole fish receiver; the double-layer front channel housing is divided into upper and lower layers, and an opening is arranged near the initial end of the upper layer for arranging the double-hole fish receiver. The double-hole fish receiver is a receiving container with two openings at the bottom. One opening diverts a part of the fish to the upper layer of the front channel housing, and the other opening diverts the fish to the lower layer of the front channel housing.

[0012] A double-layer high-throughput fish counting method based on machine vision, implemented based on the above-mentioned device. The method includes two processes. One process processes the video stream collected by the image acquisition module at the top of the double-layer counting channel, and the other process processes the video stream collected by the image acquisition module at the bottom of the double-layer counting channel. The processing flows of the two processes are the same, and both include image preprocessing, fish detection, association of fish in consecutive frames, fish counting, and result display. Finally, the counting results of the two processes are superimposed and displayed.

[0013] In the above technical solution, further, the image preprocessing specifically includes mean filtering and background modeling.

[0014] Further, the fish detection specifically includes segmenting the foreground region according to the difference between the image and the background model, feature extraction, determination and localization of the number of fish; the feature extraction includes but is not limited to the area A, contour perimeter l l of the foreground region, the skeleton length l g of the foreground region, the length l r and width w r of the minimum bounding rectangle of the foreground region, the area A t and perimeter l t of the convex hull of the foreground region, the area ratio complexity The determination and localization of the number of fish include using the extracted feature vector as input, using a machine learning classification model to judge the number n of fish in the foreground region, and using the center of the minimum bounding rectangle of the foreground region as the position coordinates of n fish in the foreground region;

[0015] Further, the association of fish in consecutive frames includes constructing a cost function cv ij , a cost matrix CM and an objective function Z, and using the Hungarian algorithm to calculate the optimal solution of the objective function; the cost function cv ij , the cost matrix CM and the objective function Z are:

[0016]

[0017]

[0018]

[0019] Set boundary conditions

[0020] In the formula, (x i , y i )(i = 1, 2,.., n) are the coordinates of fish individuals in the previous frame, (x j , y j(j = 1, 2,.., m) are the coordinates of fish individuals in the current frame, D is the average moving distance of the fish in the x - direction between two consecutive frames. By introducing the parameter D, the moving distance of fry in the x - axis direction between two consecutive frames is estimated, and there is no need to require the image acquisition module to sample the channel at a high frame rate, that is, this method can be used for fry association under low - frame - rate sampling conditions; n is the number of fish individuals in the previous frame; m is the number of fish individuals in the current frame; T is the set threshold.

[0021] Further, the fish counting refers to judging the newly - emerged fish individuals in the first half of the current frame image in two consecutive frames, and adding the number of newly - emerged fish individuals to the total number; the newly - emerged fish individuals are those that have not been associated with the previous frame image or the loss function value between them and the associated individuals is T.

[0022] Compared with the existing counting schemes, the present invention makes full use of the space in the vertical direction under the condition of the same floor area, and conducts fish counting through upper and lower double - layer channels, which can at least double the counting throughput, providing a fish high - throughput counting device; the double - layer high - throughput fish counting method based on machine vision of the present invention processes the video streams of two image acquisition modules in a dual - process manner, improving the operation efficiency of the controller; by introducing a machine - learning classification algorithm, there is no need to accurately locate each fish individual, judge and count the newly - emerged fish individuals through the front and rear frames, and there is no need to continuously track the target, which can greatly reduce the calculation amount and algorithm complexity and improve the counting speed. Brief Description of the Drawings

[0023] Figure 1 is an isometric view of the double - layer high - throughput fish counting device based on machine vision of the present invention

[0024] Figure 2 is an isometric view of the double - layer high - throughput fish counting device from another perspective

[0025] Figure 3 is a sectional view of the double - layer high - throughput fish counting device

[0026] Figure 4 is an exploded view of the double - layer counting channel of the double - layer high - throughput fish counting device

[0027] Figure 5 is a schematic diagram of a possible improved structure of the fish layer - separator of the double - layer high - throughput fish counting device, (a) is the lower - layer structure, (b) is the upper - layer structure, (c) is the combined upper - and - lower - layer structure

[0028] Figure 6 is a schematic diagram of a possible improved structure of the front - end channel of the double - layer high - throughput fish counter device (exploded view)

[0029] Figure 7Flowchart of the double-layer high-throughput fish counting method based on machine vision of the present invention

[0030] Description of the reference numerals:

[0031] 1 - Front channel housing; 2 - Fish stratifier; 3 - Flap; 4 - Display screen; 5 - Controller protection housing; 6 - Image acquisition module; 7 - Counting channel housing; 8 - Double-layer fish outlet; 9 - Fish receiver; 10 - White U-shaped channel; 11 - Transparent U-shaped channel; 12 - White light fill light; 13 - Improved fish stratifier; 14 - Double-hole fish receiver; 15 - Double-layer front channel housing Specific implementation mode

[0032] The following details the implementation mode of the present invention. The examples of the implementation mode are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end.

[0033] Example 1

[0034] As Figure 1 shown, the double-layer high-throughput fish counting device based on machine vision includes a front channel, a double-layer counting channel, a double-layer fish outlet 8, a flap 3, and a display and control module. The double-layer counting channel is the main part of the counting device. One end is connected to the front channel by bolts, and the other end is connected to the double-layer fish outlet 8. The connection method can be that one side is connected by a hinge, and the other side can be fixed by bolts or buckles. The double-layer fish outlet 8 can rotate around the hinge; the double-layer counting channel and the double-layer fish outlet 8 can form upper and lower fish passage channels. The double-layer fish outlet 8 can divert the fish in the upper and lower layers of the double-layer counting channel to different breeding ponds; a display and control module is also arranged on the side of the double-layer counting channel close to the front channel. Among them, the controller is placed inside the controller protection housing 5, and the display 4 is embedded in the upper opening of the controller protection housing 5. The display 4 can simultaneously display the counting pictures and the number of fish in the upper and lower layers of the double-layer counting channel and perform basic control of the counting device; the front channel is the entrance of the fish. One end is closed, and a flap 3 is provided at the other end close to the upper part of the double-layer counting channel. It is installed on the two hinge mounting holes on the side of the double-layer counting channel housing 7 through a hinge and can be used to block external light sources;

[0035] As Figure 3 、 4As shown in the figure, the double-layer counting channel includes a counting channel housing 7, a white U-shaped channel 10, a transparent U-shaped channel 11, and a white light fill light 12; the counting channel housing 7 includes a hollow aisle, and the upper and lower housings of the aisle are trapezoidally convex. The image acquisition module 6 is fixed in the upper and lower trapezoidally convex housings by bolts; the transparent U-shaped channel 11 is arranged at the bottom of the hollow aisle of the double-layer counting channel housing 7, and the white U-shaped channel 10 is arranged above the transparent U-shaped channel 11. The two divide the hollow aisle of the double-layer counting channel housing into upper and lower channels; the white light fill light 12 is arranged on the inner walls of both sides of the aisle of the counting channel housing 7 to provide sufficient light inside the aisle; bolt mounting holes are left on both sides of the aisle of the transparent U-shaped channel 11, the white U-shaped channel 10, the white light fill light 12, and the counting channel housing 7, and the four are fixed by bolts; the image acquisition module 6 at the top of the counting channel housing 7 can capture the fish passing through the white U-shaped channel 10 and is not affected by the shadows of the fish individuals in the lower layer. Similarly, the image acquisition module 6 below can capture the fish passing through the transparent U-shaped channel 11, with the bottom of the upper white U-shaped channel 10 as the background, but is not affected by the shadows of the fish passing through in the upper layer. In addition, in this embodiment, an opaque white film is also pasted on the bottom of the white U-shaped channel 10.

[0036] The front channel includes a front channel housing 1, a fish stratifier 2, and a fish receiver 9 ( Figure 1 not shown in the figure); as Figure 2 shown, the fish receiver 9 is a container for receiving fish supplied from above. The supply method can be manual fish pouring, waterwheel fish pouring, etc. There is an opening at the bottom, and the opening spacing is set according to the size of the fish. Its main function is to prevent a large number of fish from entering the double-layer counting channel at the same time; the outer edge of the fish receiver 9 has a certain extension and is provided with bolt mounting holes, and it is fixed on the front channel housing 1 by bolts; the front channel housing 1 is provided with an interface (i.e., Figure 1 the cylindrical interface in the shown example) for connecting a fish suction pump near the bottom of the closed end. A row of strip-shaped water leakage holes are arranged in the middle part. The gap of the water leakage holes should be smaller than the thickness of the fish, and a screen is arranged above the strip-shaped water leakage holes to prevent the fish from being scratched and damaged when passing through the water leakage holes; as an improvement, the strip-shaped water leakage holes can be changed to dense small holes; below the water leakage holes is a U-shaped groove, and a water pipe connection port is arranged at the bottom of the U-shaped groove; when using a waterwheel or a fish suction pump to supply fish, the water pipe connection port serves as a drainage hole, and when using manual fish pouring, the water pipe connection port is connected to a water pump for water supply; when using a fish suction pump to supply fish, there is no need to install the fish receiver 9.

[0037] As Figure 1As shown, the fish stratifier 2 is provided with a number of evenly distributed channels. Some of these channels divert fish to the upper layer of the double-layer counting channel, and the remaining channels divert fish to the lower layer of the double-layer counting channel. Triangular vertical plates are provided at the edges of each upper channel to divert the fish; all sharp places need to be polished to prevent damage to the fish.

[0038] As Figure 5 shown, as an improvement, the improved fish stratifier 13 is divided into upper and lower layers, and the upper and lower layers can be directly welded and fixed. The upper layer is distributed at intervals through cylindrical tubes, which can preliminarily screen the size of the fish. Fish with a thickness less than that of the cylindrical tube directly fall into the lower layer and then enter the lower channel of the double-layer counting channel, and the remaining fish enter the upper channel of the double-layer counting channel;

[0039] As Figure 6 shown, the front channel includes a double-layer front channel housing 15 and a double-hole fish receiver 14. The double-layer front channel housing 15 is divided into upper and lower layers. An opening is provided near the initial end of the upper layer. The double-hole fish receiver 14 is a receiving container provided at this opening. Two openings are provided at its lower part. One opening diverts a part of the fish to the upper layer of the front channel housing 15, and the other opening diverts the fish to the lower layer of the front channel housing 15. The two openings are of the same size;

[0040] As Figure 7 shown, it is a schematic flow diagram of a double-layer high-throughput fish counting method based on machine vision. This method includes two processes. One process processes the video stream collected by the image acquisition module at the top of the double-layer counting channel, and the other process processes the video stream collected by the image acquisition module at the bottom of the double-layer counting channel. The processing flows of the two processes are the same, and both include image preprocessing, fish detection, association of fish in the front and back frames, fish counting, and result display. The counting images and the number of fish of the two processes are displayed independently, and finally the counting results of the two processes are superimposed and the total number is displayed; in the image preprocessing part, it specifically includes mean filtering and background modeling. The background modeling method can be constructed using pictures without counting, or the Gaussian mixture model separation algorithm (MOG).

[0041] When performing formal counting, it is necessary to obtain in advance a data sample library composed of the characteristics and adhesion quantities (labels) of fish of different specifications and varieties, and select different machine learning classification algorithms (support vector machine SVM, random forest RF, fully connected neural network BPNN, decision tree DT, AdaBoost) for training and comparison to select the best classification model.

[0042] Fish detection specifically includes segmenting the foreground area according to the difference between the image and the background model, feature extraction, fish quantity classification and positioning; feature extraction includes but is not limited to the area A of the foreground area, the contour perimeter l l , the skeleton length lg , the length l of the minimum bounding rectangle r and the width w r , the area A of the convex hull t and the perimeter l t , the area ratio complexity are constructed into the feature vector of the foreground region; subsequently, the extracted feature vector is used as the input, and the best machine learning classification model is adopted to judge the number n of fish in the foreground region, and the center of the minimum bounding rectangle of the foreground region is used as the position coordinates of n fish in the foreground region.

[0043] The association of fish between consecutive frames includes constructing a loss function cv ij , a loss matrix CM and an objective function Z, and using the Hungarian algorithm to calculate the optimal solution of the objective function; the loss function cv ij , the loss matrix CM and the objective function Z are respectively:[[]]

[0044]

[0045]

[0046]

[0047] Set boundary conditions

[0048] where (x i , y i )(i = 1, 2,.., n) are the coordinates of fish individuals in the previous frame, (x j , y j )(j = 1, 2,.., m) are the coordinates of fish individuals in the current frame, D is the average moving distance of fish in the x direction between two consecutive frames, and by introducing the parameter D, the moving distance of fry in the x-axis direction between two consecutive frames is estimated. There is no need to require the image acquisition module to sample the channel at a high frame rate, that is, this method can be used for fry association under low frame rate sampling conditions; n is the number of fish individuals in the previous frame; m is the number of fish individuals in the current frame; T is the set threshold, which can be infinite. To reduce the computational complexity, in the embodiment, T is set to 1000. In the loss function of the present invention, only distance features are used. Under low frame rate sampling conditions, the moving distance of fry between two consecutive frames is relatively large. If D (average inter-frame moving distance) is not introduced, when the fry density is relatively large, there is likely to be a large error in the association of fry between consecutive frames. By using D, it means that the fry in the previous frame are pre-moved by a distance of D and then associated with the fry in the current frame. Such an operation can effectively improve the association accuracy under low frame rate conditions. Thus, the requirements for equipment can be greatly reduced.

[0049] In this embodiment, in order to obtain an accurate D value, before counting, 20 fish can be manually placed into the counting device one by one to obtain the moving distance of fish individuals in the x direction in two consecutive frames, and the average value is automatically calculated as the calibrated D value, which is then written into the counting code.

[0050] During the fish counting process, only the previous frame and the current frame are concerned. Newly emerged fish individuals in the counting area in the current frame are judged, and the number of newly emerged fish individuals is added to the total number. Newly emerged fish individuals refer to fish individuals that have not been associated or have a loss function value of T with the associated individuals.

[0051] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A double-layer high-throughput fish counting device based on machine vision, characterized in that, it includes a front channel, a double-layer counting channel and a double-layer fish outlet; the double-layer counting channel is the main part of the counting device, one end is connected to the front channel, and the other end is connected to the double-layer fish outlet. The double-layer counting channel and the double-layer fish outlet form an upper and lower fish passage. A display and control module is also arranged on the double-layer counting channel; the front channel is the entrance of the fish, one end is closed, and a flip cover is placed near the upper part of the double-layer counting channel at the other end, which can be used to block external light sources; the double-layer counting channel includes a counting channel housing, a white U-shaped channel, a transparent U-shaped channel and white light supplementary lamps; the counting channel housing includes a hollow aisle, and the housing bulges outwards above and below the aisle, and image acquisition modules are arranged in the bulging housings; the white U-shaped channel and the transparent U-shaped channel are both inserted and fixed in the hollow aisle of the double-layer counting channel housing, and the transparent U-shaped channel is arranged below the white U-shaped channel, and the two divide the hollow aisle of the double-layer counting channel housing into upper and lower channels; the white light supplementary lamps are arranged on the inner walls on both sides of the hollow aisle to provide sufficient light inside; the front channel includes a front channel housing, a fish stratifier and a fish receiver; the fish receiver is a container with an opening at the bottom for receiving fish supplied from above; an interface for connecting a fish suction pump is arranged at the bottom near the closed end of the front channel housing, and a row of strip-shaped water leakage holes are arranged in the middle part. Below the water leakage holes is a U-shaped groove, and a water pipe connection port is arranged at the bottom of the groove for connecting a water pump for water supply or serving as a drainage port for drainage; the fish receiver is detachably installed on the front channel housing, and the fish stratifier is installed at the end of the front channel housing. The fish stratifier is provided with several channels for diverting fish so that the fish are dispersed into the upper and lower channels of the double-layer counting channel; the fish stratifier is divided into upper and lower layers, and the upper layer is spaced by cylindrical tubes, which can preliminarily screen the size of the fish. The fish with a thickness less than the cylindrical tube enter the lower channel of the double-layer counting channel, and the rest of the fish enter the upper channel of the double-layer counting channel; the front channel includes a double-layer front channel housing and a double-hole fish receiver; the double-layer front channel housing is divided into upper and lower layers. An opening is provided near the initial end of the upper layer for arranging the double-hole fish receiver. The double-hole fish receiver is a receiving container with two openings at the bottom. One opening diverts a part of the fish to the upper layer of the front channel housing, and the other opening diverts the fish to the lower layer of the front channel housing.

2. A double-layer high-throughput fish counting method based on machine vision, characterized in that, it is realized based on the device described in claim 1. The method includes two processes. One process processes the video stream collected by the image acquisition module at the top of the double-layer counting channel, and the other process processes the video stream collected by the image acquisition module at the bottom of the double-layer counting channel. The processing flows of the two processes are the same, and both include image preprocessing, fish detection, fish association between the front and back frames, fish counting, result display, and finally the counting results of the two processes are superimposed and displayed.

3. The method according to claim 2, wherein, the image preprocessing specifically includes mean filtering and background modeling.

4. The method according to claim 2, wherein, The fish detection specifically includes segmenting the foreground area based on the difference between the image and the background model, feature extraction, determining the number of fish, and positioning; the feature extraction includes but is not limited to the area of the foreground area , contour perimeter , skeleton length , length of the minimum bounding rectangle and width , convex hull area and perimeter , area ratio , complexity ; the fish number classification and positioning includes using the extracted feature vector as input, adopting a machine learning classification model to judge the number of fish in the foreground area , and using the center of the minimum bounding rectangle of the foreground area as the position coordinates of the fish in the foreground area.

5. The method according to claim 2, wherein, The front and rear frame fish association includes constructing a loss function , a loss matrix and an objective function , and using the Hungarian algorithm to calculate the optimal solution of the objective function; the loss function the loss matrix and the objective function are as follows: , , , Set boundary conditions , Wherein ( )( ) are the fish individual coordinates in the previous frame, ( )( ) are the fish individual coordinates in the current frame, D is the average moving distance of the fish in the direction between two consecutive frames; is the number of fish individuals in the previous frame; is the number of fish individuals in the current frame; is the set threshold.

6. The method according to claim 5, wherein, The fish counting refers to judging the newly emerged fish individuals in the first half of the current frame in two consecutive frames, and adding the number of the newly emerged fish individuals to the total number; the newly emerged fish individuals are those that have not been associated with the previous frame or the loss function value between the associated individuals is of the fish individuals.

Images