Fish image intelligent identification method for improving Faster R-CNN target detection

Abstract

The invention relates to a fish image intelligent identification method for improving Faster R-CNN target detection, and the method comprises the steps: extracting the bottom-layer feature information of fish in image data through a deep learning principle, carrying out the continuous permutation and combination of the bottom-layer fish image feature information at a higher level through a multi-layer neural network, obtaining the combined advanced fish feature information, and carrying out the recognition of the fish image through the improved Faster R-CNN target detection. A boundary regression algorithm is constructed by using a linear regression principle, and fish feature information is extracted from image data, identified and classified by combining with a target positioning area after frame regression optimization and adding a classification algorithm of next connection matching. According to the method, intelligent identification and classification of fish image data are realized, extraction and classification of various fish features in large-scale image data with densely distributed underwater fishes are realized, and the problems of low identification precision, low identification rate and poor positioning precision in the field of intelligent fish identification are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image analysis, and in particular relates to a fish image intelligent recognition method for improving Faster R-CNN target detection. Background Art

[0002] The purpose of intelligent fish identification is to achieve automatic, accurate and efficient identification and classification of fish. By using computer vision, machine learning, deep learning and other technologies, the appearance, shape, color and other information of fish are analyzed and compared, so as to accurately determine the species and quantity of fish.

[0003] At present, intelligent fish recognition technology mainly relies on the development of computer vision and machine learning technology. By analyzing the appearance, shape, color and other information of fish, and combining with the monitoring of fish behavior, automatic recognition and classification of fish can be achieved. The traditional machine vision method evaluates a threshold based on the contour features of fish in the image to detect image data. This method has high accuracy, but it is very dependent on experts to manually summarize effective fish features. It is only applicable to target groups with a single species and highly consistent effective features. It cannot cope with the problem of fish target recognition when the number and type of fish increase, and the movement of fish causes fish overlap and shape changes.

[0004] Compared with machine learning, deep learning can automatically extract features from images and has strong adaptability by continuously learning the difference between predicted values ​​and true values. The existing deep learning models for intelligent fish recognition mainly include the Yolov3 model algorithm based on network structure optimization and the Yolov4 algorithm based on the Resblockbodyl module to improve positioning accuracy. There are still the following technical problems: (1) The recognition accuracy is not high. The main reason is that there are many types of fish and they coexist. It is difficult to associate fish characteristics through fish behavior. (2) The recognition speed is slow. Due to the large amount of data of fish images, it has a strong dependence on the calculation amount of the model. Excessive data will reduce the accuracy and recognition rate. (3) The positioning accuracy is not enough. The underwater environment is complex. The light and water quality will interfere with the clarity and color information of the fish image, resulting in inaccurate recognition and positioning accuracy. (4) Most underwater fish are gregarious creatures. In the collected fish image data, most of the fish are in group activities. The existing Faster R-CNN model cannot handle the recognition and classification of dense targets. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for intelligent fish image recognition based on improved Faster R-CNN target detection, so as to achieve rapid and high-precision recognition of fish image data in complex environments.

[0006] The present invention solves the technical problem by the following technical solutions:

[0007] A method for intelligent fish image recognition based on improved Faster R-CNN target detection, the method comprising the following steps:

[0008] S1. Fish image data acquisition and cleaning: Collect and expand fish image data sets, preprocess fish image data, and expand sample images of target fish;

[0009] S2. Target fish sample labeling: Select image data with obvious target fish features from the original data to create label information, ensuring that the target category to be detected contains at least 4 sample data in the sample data for model training and feature learning;

[0010] S3. Construction of fish intelligent identification model: Add boundary regression module and sub-joint matching classification module to the existing Faster R-CNN model to build an untrained fish intelligent identification basic model;

[0011] S4. Training the convolutional neural network: using the back propagation algorithm and the gradient descent method, according to the change of the loss value of the forward propagation of the neural network, back propagation updates the weight of each layer of the neural network. When the loss value of the fish intelligent recognition model tends to converge, stop training the model and complete the optimal weight parameter setting of the fish intelligent recognition model that meets the target sample;

[0012] S5, fish image feature extraction: input each image in the data set into the fish intelligent recognition deep learning model trained in S4, and extract deep learning features at the terminal convolution layer of the deep learning model;

[0013] S6, image recognition: Based on the fish image features extracted by the S5 model, the model's boundary regression module and sub-joint matching classification module are used to complete image classification, output fish image classification results, and realize intelligent recognition of fish images in the data set.

[0014] Furthermore, the S1 preprocessing removes image data with insufficient resolution, no objects, and no complete objects in the data set.

[0015] Furthermore, the S2 selects an image containing obvious target fish, segments the target area, and uses a rectangular area to segment the target area and the background area; and converts the image with the segmented area mark into a vol file format.

[0016] Moreover, the fish intelligent identification model of S3 is composed of the existing Faster-R-CNN basic model, boundary regression module, and sub-joint matching classification module.

[0017] The principle of the boundary regression module is:

[0018] The boundary regression module fine-tunes the inaccurately positioned target detection frame to make the positive target and the correctly labeled data closer. The specific calculation process is as follows. The target window is represented by a four-dimensional vector (x, y, w, h), which corresponds to the center coordinates and width and height of the window respectively. Assume that the inaccurately positioned detection frame A represents the original positive target, and the fine-tuned detection frame B represents the detection frame GT of the target. Therefore, it is necessary to define a relationship so that the original anchor A outputs a regression window G' that is closer to the real window G through function mapping, then:

[0019] 1) Given anchor anchorA = (Ax, Ay, Aw, Ah) and target detection box GT = (Gx, Gy, Gw, Gh)

[0020] 2) Define the variation function F, F(Ax, Ay, Aw, Ah) = (G'x, G'y, G'w, G'h), and (G'x, G'y, G'w, G'h) ≈ (Gx, Gy, Gw, Gh)

[0021] In order to solve the problem of inaccurate target positioning and incomplete positioning area, linear regression processing is performed. The calculation process is as follows: First, the original frame is translated:

[0022] G'x=Aw*dx(A)+Ax (1)

[0023] G'y=Ah*dy(A)+Ay (2)

[0024] Then do the zoom:

[0025] G'w=Aw*exp(dw(A)) (3)

[0026] G'h=Ah*exp(dh(A)) (4)

[0027] According to linear regression Y = WX, let X = cnn feature map be defined as Define the intermediate transformation quantities (tx, ty, tw, th), and output four transformation quantities dx(A)dy(A)dw(A)dh(A). Therefore, define the objective function as:

[0028]

[0029] in: is the feature vector composed of the feature map corresponding to the anchor, W* is the parameter learned by the model, d*(A) is the predicted value, * represents x, y, w, h, that is, each transformation corresponds to the above objective function. In order to minimize the gap between the predicted value d*(A) and the true value t*, the loss function L1 is defined:

[0030]

[0031] The function optimization goal is:

[0032]

[0033] The operation process of the secondary matching classification module is as follows:

[0034] (1) Create the corresponding target information from the first detected result, initialize the motion variable of the Kalman filter, and predict the corresponding anchor frame through the Kalman filter. At this time, the target information must be in an unconfirmed state;

[0035] (2) Perform IOU matching on the anchor box of the current target detection and the target information detected in the first time, and then calculate the cost matrix based on the result of IOU matching;

[0036] (3) All the cost matrices obtained in (2) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target information are successfully matched, indicating that the previous and next recognition matches are successful, and the corresponding target detection anchor frame is updated with its corresponding target information through Kalman filtering;

[0037] (4) Repeat steps (2) to (3) until the target information in the confirmation state appears or the data set ends;

[0038] (5) Predict the confirmed target information and the corresponding frame of the unconfirmed Tracks through Kalman filtering, and cascade match the confirmed target information and the target detection anchor frame until the information is successfully matched;

[0039] (6) All the cost matrices obtained in (5) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target frame are successfully matched, indicating that the previous and next matches are successful, and the corresponding target detection frame is updated with its corresponding target information variable through Kalman filtering;

[0040] (7) Repeat steps (5)-(6) until the data set classification is completed.

[0041] The positive effects that the present invention can produce are:

[0042] The present invention can realize rapid intelligent identification of multiple mixed fish species, and provides an automated and intelligent monitoring and identification method for fish farming or fishway operation management. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the present invention;

[0044] Figure 2 The candidate network structure diagram of the region of the present invention;

[0045] Figure 3 Schematic diagram of the boundary regression window calculation principle of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.

[0047] A method for intelligent fish image recognition based on improved Faster R-CNN target detection, the method comprising the following steps:

[0048] S1. Fish image data acquisition and cleaning: Collect and expand fish image data sets, preprocess fish image data, remove image data with insufficient resolution, no target, and no complete target in the data set, and expand sample images of target fish;

[0049] S2. Target fish sample labeling: Select image data with obvious target fish features from the original data to create label information, ensure that the target category to be detected contains at least 4 sample data in the sample data for model training and feature learning, select images containing obvious target fish, segment the target area, and use rectangular areas to segment the target area and background area; convert the image with segmentation area labels into vol file format;

[0050] S3. Construction of fish intelligent identification model: Based on the existing Faster R-CNN model, a boundary regression module and a sub-joint matching classification module are added to build a basic model for fish intelligent identification. Figure 2 Schematic diagram of the regional candidate network structure used in the model;

[0051] The principle of the boundary regression module is:

[0052] The boundary regression module fine-tunes the inaccurately positioned target detection frame to make the positive target and the correctly labeled data closer. The logical diagram of boundary regression is as follows: Figure 3 The specific calculation process is as follows: the target window is represented by a four-dimensional vector (x, y, w, h), which corresponds to the center coordinates and width and height of the window respectively. Assume that the inaccurately positioned detection frame A represents the original positive target, and the fine-tuned detection frame B represents the detection frame GT of the target. Therefore, it is necessary to define a relationship so that the original anchor A outputs a regression window G' that is closer to the real window G through function mapping, then:

[0053] 1) Given anchor anchorA = (Ax, Ay, Aw, Ah) and target detection box GT = (Gx, Gy, Gw, Gh)

[0054] 2) Define the variation function F, F(Ax, Ay, Aw, Ah) = (G'x, G'y, G'w, G'h), and (G'x, G'y, G'w, G'h) ≈ (Gx, Gy, Gw, Gh)

[0055] In order to solve the problem of inaccurate target positioning and incomplete positioning area, linear regression processing is performed. The calculation process is as follows: First, the original frame is translated:

[0056] G'x=Aw*dx(A)+Ax (1)

[0057] G'y=Ah*dy(A)+Ay (2)

[0058] Then do the zoom:

[0059] G'w=Aw*exp(dw(A)) (3)

[0060] G'h=Ah*exp(dh(A)) (4)

[0061] According to linear regression Y = WX, let X = cnn feature map be defined as Define the intermediate transformation quantities (tx, ty, tw, th), and output four transformation quantities dx(A)dy(A)dw(A)dh(A). Therefore, define the objective function as:

[0062]

[0063] in: is the feature vector composed of the feature map corresponding to the anchor, W* is the parameter learned by the model, d*(A) is the predicted value, * represents x, y, w, h, that is, each transformation corresponds to the above objective function. In order to minimize the gap between the predicted value d*(A) and the true value t*, the loss function L1 is defined:

[0064]

[0065] The function optimization goal is:

[0066]

[0067] The operation process of the secondary matching classification module is as follows:

[0068] (1) Create the corresponding target information from the first detected result, initialize the motion variable of the Kalman filter, and predict the corresponding anchor frame through the Kalman filter. At this time, the target information must be in an unconfirmed state;

[0069] (2) Perform IOU matching on the anchor box of the current target detection and the target information detected in the first time, and then calculate the cost matrix based on the result of IOU matching;

[0070] (3) All the cost matrices obtained in (2) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target information are successfully matched, indicating that the previous and next recognition matches are successful, and the corresponding target detection anchor frame is updated with its corresponding target information through Kalman filtering;

[0071] (4) Repeat steps (2) to (3) until the target information in the confirmation state appears or the data set ends;

[0072] (5) Predict the confirmed target information and the corresponding frame of the unconfirmed Tracks through Kalman filtering, and cascade match the confirmed target information and the target detection anchor frame until the information is successfully matched;

[0073] (6) All the cost matrices obtained in (5) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target frame are successfully matched, indicating that the previous and next matches are successful, and the corresponding target detection frame is updated with its corresponding target information variable through Kalman filtering;

[0074] (7) Repeat steps (5) to (6) until the data set classification is completed;

[0075] S4. Training the convolutional neural network: using the back propagation algorithm and the gradient descent method, according to the change of the loss value of the forward propagation of the neural network, back propagation updates the weight of each layer of the neural network. When the loss value of the fish intelligent recognition model tends to converge, the model training is stopped, and the optimal weight parameter setting of the fish intelligent recognition model that meets the target sample is completed;

[0076] S5, fish image feature extraction: input each image in the data set into the fish intelligent recognition deep learning model trained in S4, and extract deep learning features at the terminal convolution layer of the deep learning model;

[0077] S6, image recognition: Based on the fish image features extracted by the S5 model, the model's boundary regression module and sub-joint matching classification module are used to complete image classification, output fish image classification results, and realize intelligent recognition of fish images in the data set.

[0078] The present invention completes fish image recognition through boundary regression and secondary matching allocation modules. Experiments have shown that this method can effectively improve the recognition accuracy and recognition speed of fish images.

[0079] Although the embodiments and drawings of the present invention are disclosed for illustrative purposes, those skilled in the art will appreciate that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. An intelligent fish image recognition method based on improved Faster R-CNN target detection, characterized by: The steps of the method are: S1. Fish image data acquisition and cleaning: Collect and expand fish image data sets, preprocess fish image data, and expand sample images of target fish; S2. Target fish sample labeling: Select image data with obvious target fish features from the original data to create label information, ensuring that the target category to be detected contains at least 4 sample data in the sample data for model training and feature learning; S3. Construction of fish intelligent identification model: Add boundary regression module and sub-joint matching classification module to the existing Faster R-CNN model to build an untrained fish intelligent identification basic model; S4. Training the convolutional neural network: using the back propagation algorithm and the gradient descent method, according to the change of the loss value of the forward propagation of the neural network, back propagation updates the weight of each layer of the neural network. When the loss value of the fish intelligent recognition model tends to converge, the model training is stopped, and the optimal weight parameter setting of the fish intelligent recognition model that meets the target sample is completed; S5, fish image feature extraction: input each image in the data set into the fish intelligent recognition deep learning model trained in S4, and extract deep learning features at the terminal convolution layer of the deep learning model; S6, image recognition: Based on the fish image features extracted by the S5 model, the model's boundary regression module and sub-joint matching classification module are used to complete image classification, output fish image classification results, and realize intelligent recognition of fish images in the data set.

2. The method for intelligent fish image recognition based on improved Faster R-CNN target detection according to claim 1, characterized in that: The S1 preprocessing removes image data with insufficient resolution, without objects, or without complete objects in the data set.

3. The method for intelligent fish image recognition based on improved Faster R-CNN target detection according to claim 1, characterized in that: The S2 selects an image containing obvious target fish, segments the target area, and uses a rectangular area to segment the target area and the background area; and converts the image with the segmented area mark into a vol file format.

4. The method for intelligent fish image recognition based on improved Faster R-CNN target detection according to claim 1, characterized in that: The fish intelligent recognition model of S3 is composed of the existing Faster-R-CNN basic model, boundary regression module, and sub-joint matching classification module. The principle of the boundary regression module is: The boundary regression module fine-tunes the inaccurately positioned target detection frame to make the positive target and the correctly labeled data closer. The specific calculation process is as follows. The target window is represented by a four-dimensional vector (x, y, w, h), which corresponds to the center coordinates and width and height of the window respectively. Assume that the inaccurately positioned detection frame A represents the original positive target, and the fine-tuned detection frame B represents the detection frame GT of the target. Therefore, it is necessary to define a relationship so that the original anchor A outputs a regression window G' that is closer to the real window G through function mapping, then: 1) Given anchor anchorA = (Ax, Ay, Aw, Ah) and target detection box GT = (Gx, Gy, Gw, Gh) 2) Define the variation function F, F(Ax, Ay, Aw, Ah) = (G'x, G'y, G'w, G'h), and (G'x, G'y, G'w, G'h) ≈ (Gx, Gy, Gw, Gh) In order to solve the problem of inaccurate target positioning and incomplete positioning area, linear regression processing is performed. The calculation process is as follows: First, the original frame is translated: G'x=Aw*dx(A)+Ax(1) G'y=Ah*dy(A)+Ay(2) Then do the zoom: G'w=Aw*exp(dw(A))(3) G'h=Ah*exp(dh(A))(4) According to linear regression Y = WX, let X = cnnfeaturemap be defined as Define the intermediate transformation quantities (tx, ty, tw, th), and output four transformation quantities dx(A)dy(A)dw(A)dh(A). Therefore, define the objective function as: in: is the feature vector composed of the featuremap corresponding to the anchor, W* is the parameter learned by the model, d*(A) is the predicted value, * represents x, y, w, h, that is, each transformation corresponds to the above objective function. In order to minimize the gap between the predicted value d*(A) and the true value t*, the loss function L1 is defined: The function optimization goal is: The operation process of the secondary matching classification module is as follows: (1) Create the corresponding target information from the first detected result, initialize the motion variable of the Kalman filter, and predict the corresponding anchor frame through the Kalman filter. At this time, the target information must be in an unconfirmed state; (2) Perform IOU matching on the anchor box of the current target detection and the target information detected in the first time, and then calculate the cost matrix based on the result of IOU matching; (3) All the cost matrices obtained in (2) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target information are successfully matched, indicating that the previous and next recognition matches are successful, and the corresponding target detection anchor frame is updated with its corresponding target information through Kalman filtering; (4) Repeat steps (2) to (3) until the target information in the confirmation state appears or the data set ends; (5) Predict the confirmed target information and the corresponding frame of the unconfirmed Tracks through Kalman filtering, and cascade match the confirmed target information and the target detection anchor frame until the information is successfully matched; (6) All the cost matrices obtained in (5) are used as the input of the Hungarian algorithm to obtain the linear matching result until the detection frame and the predicted target frame are successfully matched, indicating that the previous and next matches are successful, and the corresponding target detection frame is updated with its corresponding target information variable through Kalman filtering; (7) Repeat steps (5)-(6) until the data set classification is completed.

Images