A fish school detection method, system and device based on image recognition

Abstract

The present invention discloses a fish school detection method, system and device based on image recognition, the method comprising: collecting fish school images and performing target segmentation on the fish school images; performing target extraction and classification on the background difference map; performing contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence; constructing cross fish samples according to the head and tail positions of individual fish; constructing a training set and training a Yolo v4 network; obtaining an image to be tested and inputting it into a trained detection network. The system comprises: a segmentation module, a classification module, an extraction module, a construction module, a training module and a detection module. The device comprises a memory and a processor for executing the above-mentioned fish school detection method based on image recognition. By using the present invention, the problem of local occlusion, missed detection and wrong detection of the target caused by the dense distribution of a huge number of highly similar individuals is solved. The present invention can be widely used in the field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a fish school detection method, system and device based on image recognition. Background Art

[0002] In recent years, the improvement of computer hardware and the development of computer vision technology have provided important technical support for the detection and tracking of group animals. As a classic model organism, zebrafish is widely used as a subject for studying group behavior due to its simple structure and easy feeding. The problems of traditional fish school detection methods include: low efficiency, easy to make mistakes, easy to make false detections, or limited methods, requiring manual setting of multiple parameters, ineffective target segmentation, and severe occlusion. These methods have a great impact on the accuracy and integrity of detection and tracking. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a fish school detection method, system and device based on image recognition, which solves the problems of local occlusion, missed detection and wrong detection of targets due to the dense distribution of huge numbers of individuals with high similarity.

[0004] The first technical solution adopted by the present invention is: a fish school detection method based on image recognition, comprising the following steps:

[0005] Collect fish school images and perform target segmentation on the fish school images to obtain background difference images;

[0006] Extract and classify the target from the background difference image to obtain individual fish samples;

[0007] Perform contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence to obtain the head and tail positions of individual fish;

[0008] Based on the translation target method, a cross-fish sample was constructed according to the head and tail positions of individual fish;

[0009] A training set is constructed based on individual fish samples and cross-fish samples, and the Yolov4 network is trained to obtain a trained detection network;

[0010] Get the image to be tested and input it into the trained detection network, and output the detection result.

[0011] Furthermore, before the step of collecting the fish school image and performing target segmentation on the fish school image to obtain the background difference image, the following steps are also included:

[0012] Build a light-blocking shooting system;

[0013] The light-shielding shooting system comprises a light-shielding curtain, an LED light bar, an outer frame bracket, a vessel and a shooting device.

[0014] Furthermore, the step of collecting the fish school image and performing target segmentation on the fish school image to obtain the background difference map specifically includes:

[0015] Determine the segmentation threshold based on Otsu's adaptive segmentation method;

[0016] Extracting the background image of the school of fish to obtain the background image;

[0017] The fish school image is differentiated from the background image, and the image is segmented based on the segmentation threshold to obtain a background difference image.

[0018] Furthermore, the step of extracting and classifying the background difference image to obtain individual fish samples specifically includes:

[0019] Perform target extraction on the background difference image to obtain target data;

[0020] The average pixel area of ​​individual fish was calculated and the individual blocks and cross blocks in the target data were distinguished to obtain individual fish samples.

[0021] Furthermore, the calculation formula for the average pixel area of ​​the individual fish is as follows:

[0022]

[0023] In the above formula, a mean is the average pixel area, n is the number of fish, j is the jth fish among n fish, a mj Represents the average area of ​​the jth fish among n fish.

[0024] Further, the step of sequentially performing contour extraction, skeleton extraction and head and tail extraction on the individual fish samples to obtain the head and tail positions of the individual fish specifically includes:

[0025] The target contours of individual fish samples were extracted and the head-tail node positions were calculated based on the contour curvature method;

[0026] The skeleton points of the target contour of the individual fish sample are extracted based on the distance transformation, and the skeleton points are fitted with a curve to obtain the skeleton information;

[0027] The head and tail images are obtained according to the head and tail node positions and skeleton information, and the head and tail positions of individual fish are calculated.

[0028] Furthermore, the step of constructing a cross fish sample according to the head and tail positions of individual fish based on the translation target method specifically includes:

[0029] Select two individual fish in the individual fish sample to obtain a first individual fish and a second individual fish;

[0030] Calculate the pixel centroids of the first individual fish and the second individual fish respectively to obtain the pixel centroids of the first individual fish and the second individual fish;

[0031] Move the second fish pixel centroid to coincide with the first fish pixel centroid, and calculate the movement amount;

[0032] Binarize and invert the second fish, use it as a mask and add the shift amount to perform a bitwise AND operation with the first fish;

[0033] All pixels of the second individual fish are superimposed onto the first individual fish according to the movement amount and the head and tail positions after the movement are calculated;

[0034] Circularly move the overlapping steps to get cross-fish samples.

[0035] The second technical solution adopted by the present invention is: a fish school detection system based on image recognition, comprising:

[0036] A segmentation module is used to collect fish school images and perform target segmentation on the fish school images to obtain a background difference map;

[0037] The classification module is used to extract and classify the target from the background difference image to obtain individual fish samples;

[0038] The extraction module performs contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence to obtain the head and tail positions of individual fish;

[0039] The construction module, based on the translation target method, constructs cross-fish samples according to the head and tail positions of individual fish;

[0040] A training module is used to construct a training set based on individual fish samples and cross-fish samples and train the Yolo v4 network to obtain a trained detection network;

[0041] The detection module obtains the image to be tested and inputs it into the trained detection network, and outputs the detection result.

[0042] The third technical solution adopted by the present invention is: a fish school detection device based on image recognition, comprising:

[0043] at least one processor;

[0044] at least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned fish school detection method based on image recognition.

[0046] The beneficial effects of the method, system and device of the present invention are as follows: the present invention realizes the detection of the whole target by means of local detection and skeleton constraint, solves the problems of local occlusion, missed detection and wrong detection of the target due to the dense distribution of a huge number of highly similar individuals, and simulates the zebrafish crossing scene to meet the diversity requirements of the training set by automatically obtaining the head and tail positions of individual fish and combining the translation target algorithm. The Yolo v4 network is trained based on the training set, and the Yolo v4 network is used to detect complex crossing data that is difficult to be effectively solved by the automatic labeling algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the steps of a fish school detection method based on image recognition of the present invention;

[0048] Figure 2 is a fish school image according to a specific embodiment of the present invention;

[0049] Figure 3 is a background image extracted by a specific embodiment of the present invention;

[0050] Figure 4 is a background difference map of a specific embodiment of the present invention;

[0051] Figure 5 This is a characteristic diagram of the appearance of a fish according to a specific embodiment of the present invention;

[0052] Figure 6 It is a schematic diagram of the positions of the head and tail nodes of a specific embodiment of the present invention;

[0053] Figure 7 It is a schematic diagram of the skeleton of a specific embodiment of the present invention;

[0054] Figure 8 It is a schematic diagram of the head and tail positions of a specific embodiment of the present invention;

[0055] Fig. 9 This is a Yolo v4 training result diagram of a specific embodiment of the present invention;

[0056] Fig.10 It is a structural block diagram of a fish school detection system based on image recognition of the present invention; DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] like Figure 1As shown, the present invention provides a fish school detection method based on image recognition, the method comprising the following steps:

[0059] S0. Construct a light-shielding shooting system, wherein the light-shielding shooting system includes a light-shielding curtain, an LED light bar, an outer frame bracket, a vessel and a shooting device.

[0060] Specifically, the blackout curtain is used to prevent the influence of external light, the LED light strip is used to provide stable lighting, the outer frame bracket is used to fix the blackout curtain, the shooting device is used to fix the camera and take images, the container size is 65cm*45cm, which is used to place the fish school. The experiment uses a high-speed camera Basler acA1440-220uc, and the camera is fixed above the container, perpendicular to the water surface.

[0061] S1, collecting fish school images and performing target segmentation on the fish school images to obtain a background difference image;

[0062] S1.1, determine the segmentation threshold based on Otsu's adaptive segmentation method;

[0063] S1.2, extracting the background image of the fish school image to obtain a background image;

[0064] S1.3, subtracting the fish school image from the background image, and performing image segmentation based on a segmentation threshold to obtain a background difference image.

[0065] Specifically, the fish school image reference Figure 2 , background image reference Figure 3 , background difference map reference Figure 4 .

[0066] S2, extracting and classifying the target from the background difference image to obtain individual fish samples;

[0067] S2.1, extracting the target from the background difference image to obtain the target data;

[0068] S2.2. Calculate the average pixel area of ​​individual fish and distinguish individual blocks and cross blocks in the target data to obtain individual fish samples.

[0069] Specifically, the target data includes individual fish and cross fish, where individual fish is a single zebrafish pixel set, and cross fish is two or more zebrafish pixel sets. Since the appearance characteristics (pixel area, outline, etc.) of individual fish are stable, it is easier to extract the head and tail of fish than cross fish, so the present invention uses an individual fish extraction head and tail algorithm. The classification of individual fish and cross fish can be judged by pixel area.

[0070] First, the average pixel area of ​​individual fish is calculated. Assume that the total number of targets obtained after target segmentation in the mth frame is equal to the number of fish schools. At this time, it can be guaranteed that each image block is an individual block. The average pixel area a of the individual block is calculated.mean , the average pixel area of ​​individual fish is calculated as follows:

[0071]

[0072] In the above formula, a mean is the average pixel area, n is the number of fish, j is the jth fish among n fish, a mj Represents the average area of ​​the jth fish among n fish.

[0073] The individual blocks and cross blocks are distinguished by the average pixel area of ​​individual fish. As follows:

[0074]

[0075] In the above formula, a mean is the average pixel area, a ij is the pixel area of ​​j image blocks obtained by background subtraction in the i-th frame image, T m is the area difference threshold, J ij When it is 0, it is an individual block, otherwise it is a cross block.

[0076] S3, performing contour extraction, skeleton extraction and head and tail extraction on the individual fish samples in sequence to obtain the head and tail positions of the individual fish;

[0077] S3.1, extract the target contour of individual fish samples and calculate the head and tail node positions according to the contour curvature method;

[0078] Specifically, refer to Figure 5 The fish head is usually wide and round (U-shaped), while the tail shows a sharp tip (V-shaped). Since the curvature represents the degree of bending of the curve, the greater the curvature, the more curved it is. Therefore, the curvature of the tail node is the largest, and the curvature of the head node is the second largest.

[0079] For an individual fish, let the contour be a set of points P, which can form a curve. The inverse of the curvature is the radius of curvature:

[0080]

[0081] In the above formula, k represents the curvature, Δθ represents the change in the tangent direction angle, l represents the arc length, and r represents the radius of curvature.

[0082] Let a point on the contour be p, then the expression of the curvature k of point p is:

[0083]

[0084] In the above formula, x i With y i are the x and y coordinates of point p, x i+1 With yi+1 , x i+2 With y i+2 are the x and y coordinates of the points adjacent to point p, is the distance between two points, is the difference in tangent direction angles.

[0085] Let K be the curvature set of the target contour, let k1 be the maximum curvature on the contour, the corresponding point p1 is the target tail point, let k2 be the second largest curvature on the contour, the corresponding point p2 is the target head point, the schematic diagram of the head and tail node positions is referenced Figure 6 .

[0086] S3.2, extracting skeleton points of the target contour of the individual fish sample based on the distance transformation, and using curve fitting to obtain skeleton information;

[0087] Specifically, the trunk of a zebrafish is composed of three parts: the head, the body, and the tail. The topological structure of the fish can be simplified into "two points and one line", namely the head and the tail, and the skeleton line. The skeleton line is selected as the simplified target. The skeleton line simplifies the model while maintaining the length, curvature, posture and other information of the target, and the representation is simpler and clearer. The method selected by the present invention is a skeleton extraction method based on distance transformation. The above algorithm is used to extract the skeleton of an individual fish, and a curve is used to fit the skeleton points. The schematic diagram is shown in FIG. Figure 7 .

[0088] S3.3. Obtain the head and tail images according to the head and tail node positions and skeleton information, and calculate the head and tail positions of individual fish.

[0089] Specifically, after extracting the skeleton, the skeleton information is combined with the individual and the head and tail point position information to obtain the head and tail images. The fish is divided into three parts, with the rear edge of the fish's gills as the head part boundary, the fish fin near the tail of the fish's abdomen as the tail part boundary, and the middle part as the abdomen. Using the proportion information, combined with the length information of the fish skeleton line, the head and tail images can be obtained. The proportion length of the fish head is 19.59%, the proportion length of the fish abdomen is 45.28%, and the proportion length of the fish tail is 35.62%. Figure 8 This is a schematic diagram after extracting the head and tail positions of individual fish.

[0090] S4, based on the translation target method, construct cross-fish samples according to the head and tail positions of individual fish;

[0091] Specifically, in order to ensure the diversity of training data and the generalization ability of the network in crossing fish detection, the present invention extracts the target to obtain individual fish for algorithm synthesis to form a simulated crossing fish image, and uses the synthesized crossing fish image as a training set to improve the diversity of the training set and facilitate subsequent training.

[0092] S4.1. Select two individual fish from the individual fish sample to obtain a first individual fish and a second individual fish;

[0093] S4.2, respectively calculating the pixel centroids of the first individual fish and the second individual fish to obtain the pixel centroids of the first individual fish and the second individual fish;

[0094] Specifically, calculate two individual blocks of fish i,j With fish i,j+1 The pixel centroid is denoted as (x i,j ,y i,j ) and (x i,j+1 ,y i,j+1 ), among which fish i,j Represents the j-th individual fish image block obtained using target detection in the i-th frame image.

[0095] S4.3, moving the second fish pixel centroid to coincide with the first fish pixel centroid, and calculating the movement amount;

[0096] Specifically, (x i,j+1 ,y i,j+1 ) moves to the position corresponding to (x i,j ,y i,j ) overlap, calculate the movement x mov and mov , as shown below:

[0097] x mov =x ij -x i,j+1

[0098] y mov =y ij -y i,j+1

[0099] S4.4, binarize and invert the second fish, use it as a mask and add the shift amount to perform a bitwise AND operation with the first fish;

[0100] Specifically, fish i,j+1 The individual blocks are binarized, and the inversion (0 to 1, 1 to 0) is used as a mask, and the mask is added with the displacement (x mov ,y mov ), and fish i,j Perform a bitwise AND operation to convert fish i,j+1 Move to fish i,jThe pixel values ​​on the left and right sides of the first fish are set to 0 in advance to facilitate the subsequent pixel overlay operation. This step is mainly to set the overlapping position of the first fish and the second fish to 0, so that the pixel values ​​of the second fish can be directly overlaid on it. If this step is not performed and the overlay is performed directly, the visual effect of the second fish above the first fish cannot be simulated.

[0101] S4.5, superimposing all pixels of the second individual fish onto the first individual fish according to the movement amount and calculating the head and tail positions after the movement;

[0102] Specifically, according to the amount of movement, the fish i,j+1 All pixels of the individual block are added with the displacement (x mov ,y mov ), fish i,j+1 Superimpose on fish i,j On; set the fish before moving i,j The coordinates of the head bounding box are (x hi,j ,y hi,j ,w hi,j ,h hi,j ), the tail is (x ti,j ,y ti,j ,w ti,j ,h ti,j ), fish i,j+1 The coordinates of the head bounding box are (x hi,j+1 ,y hi,j+1 ,w hi,j+1 ,h hi,j+1 ), the tail is (x ti,j+1 ,y ti,j+1 ,w ti,j+1 ,h ti,j+1 ), fish i,j+1 The movement amount x mov and mov , the fish i,j+1 Move to fish i,j On. Move after fish i,j The head and tail positions remain unchanged, fish i,j+1 The coordinates of the tail bounding box are (x ti,j ,y ti,j ,w ti,j ,h ti,j ) and (x ti,j+1 +x mov ,y ti,j+1 +y mov ,w ti,j+1 ,h ti,j+1 ).

[0103] S4.6. Circularly move the overlapping steps to obtain cross-fish samples.

[0104] S5. Construct a training set based on individual fish samples and cross fish samples and train the Yolo v4 network to obtain a trained detection network.

[0105] Specifically, a training set was constructed based on individual fish samples and cross fish samples combined with head and tail annotations, and the Yolo v4 network was used to detect complex cross data that are difficult for automatic annotation algorithms to effectively solve.

[0106] This paper uses mAP (mean average precision) as an evaluation indicator to measure the quality of the Yolo v4 model. Fig. 9 This is the Yolo v4 training result graph, the horizontal axis is the training iteration, the vertical axis of the area line at the top of the graph is the mAP (%) value, the larger the value, the better the network detection result, the vertical axis of the curve at the bottom of the graph is the training loss value, the smaller the value, the better the network detection result. It can be seen that after about 1800 iterations, the training loss value and mAP are in a stable state, and mAP99.5% means that the network training effect is good.

[0107] S6. Obtain the image to be tested and input it into the trained detection network, and output the detection result.

[0108] like Fig.10 As shown, a fish school detection system based on image recognition includes:

[0109] A segmentation module is used to collect fish school images and perform target segmentation on the fish school images to obtain a background difference map;

[0110] The classification module is used to extract and classify the target from the background difference image to obtain individual fish samples;

[0111] The extraction module performs contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence to obtain the head and tail positions of individual fish;

[0112] The construction module, based on the translation target method, constructs cross-fish samples according to the head and tail positions of individual fish;

[0113] A training module is used to construct a training set based on individual fish samples and cross-fish samples and train the Yolo v4 network to obtain a trained detection network;

[0114] The detection module obtains the image to be tested and inputs it into the trained detection network, and outputs the detection result.

[0115] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] A fish school detection device based on image recognition:

[0117] at least one processor;

[0118] at least one memory for storing at least one program;

[0119] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned fish school detection method based on image recognition.

[0120] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0121] A storage medium storing processor-executable instructions, characterized in that the processor-executable instructions are used to implement a fish school detection method based on image recognition as described above when executed by the processor.

[0122] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0123] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A fish school detection method based on image recognition, characterized in that: The following steps are involved: Collect fish school images and perform target segmentation on the fish school images to obtain background difference images; Extract and classify the target from the background difference image to obtain individual fish samples; Perform contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence to obtain the head and tail positions of individual fish; Based on the translation target method, a cross-fish sample was constructed according to the head and tail positions of individual fish; A training set is constructed based on individual fish samples and cross-fish samples, and the Yolo v4 network is trained to obtain a trained detection network; Obtain the image to be tested and input it into the trained detection network, and output the detection result; The step of constructing a cross fish sample according to the head and tail positions of individual fish based on the translation target method specifically includes: Select two individual fish in the individual fish sample to obtain a first individual fish and a second individual fish; Calculate the pixel centroids of the first individual fish and the second individual fish respectively to obtain the pixel centroids of the first individual fish and the second individual fish; Move the second fish pixel centroid to coincide with the first fish pixel centroid, and calculate the movement amount; Binarize and invert the second fish, use it as a mask and add the shift amount to perform a bitwise AND operation with the first fish; All pixels of the second individual fish are superimposed onto the first individual fish according to the movement amount and the head and tail positions after the movement are calculated; Circularly move the overlapping steps to get cross-fish samples.

2. The fish school detection method based on image recognition according to claim 1, characterized in that: Before the step of collecting the fish school image and performing target segmentation on the fish school image to obtain the background difference image, the following steps are also included: Build a light-blocking shooting system; The light-shielding shooting system comprises a light-shielding curtain, an LED light bar, an outer frame bracket, a vessel and a shooting device.

3. The fish school detection method based on image recognition according to claim 2, characterized in that: The step of collecting the fish school image and performing target segmentation on the fish school image to obtain the background difference map specifically includes: Determine the segmentation threshold based on Otsu's adaptive segmentation method; Extracting the background image of the school of fish to obtain the background image; The fish school image is differentiated from the background image, and the image is segmented based on the segmentation threshold to obtain a background difference image.

4. The fish school detection method based on image recognition according to claim 3 is characterized in that: The step of extracting and classifying the target from the background difference image to obtain individual fish samples specifically includes: Perform target extraction on the background difference image to obtain target data; The average pixel area of ​​individual fish was calculated and the individual blocks and cross blocks in the target data were distinguished to obtain individual fish samples.

5. The fish school detection method based on image recognition according to claim 4, characterized in that: The calculation formula for the average pixel area of ​​the individual fish is as follows: In the above formula, a mean is the average pixel area, n is the number of fish, j is the jth fish among n fish, a mj Represents the average area of ​​the jth fish among n fish.

6. The fish school detection method based on image recognition according to claim 5, characterized in that: The step of sequentially performing contour extraction, skeleton extraction and head and tail extraction on the individual fish samples to obtain the head and tail positions of the individual fish specifically includes: The target contours of individual fish samples were extracted and the head-tail node positions were calculated based on the contour curvature method; The skeleton points of the target contour of the individual fish sample are extracted based on the distance transformation, and the skeleton points are fitted with a curve to obtain the skeleton information; The head and tail images are obtained according to the head and tail node positions and skeleton information, and the head and tail positions of individual fish are calculated.

7. A fish school detection system based on image recognition, characterized in that: Used to perform a fish school detection method based on image recognition as claimed in claim 1, comprising: A segmentation module is used to collect fish school images and perform target segmentation on the fish school images to obtain a background difference map; The classification module is used to extract and classify the target from the background difference image to obtain individual fish samples; The extraction module performs contour extraction, skeleton extraction and head and tail extraction on individual fish samples in sequence to obtain the head and tail positions of individual fish; The construction module, based on the translation target method, constructs cross-fish samples according to the head and tail positions of individual fish; A training module is used to construct a training set based on individual fish samples and cross-fish samples and train the Yolo v4 network to obtain a trained detection network; The detection module obtains the image to be tested and inputs it into the trained detection network, and outputs the detection result.

8. A fish school detection device based on image recognition, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the fish school detection method based on image recognition as described in any one of claims 1-6.

Images