A method and device for dynamic recognition of underwater fish posture

Abstract

The present invention provides a method and device for dynamic recognition of underwater fish posture, the method comprising: determining a fish image to be recognized; performing target detection on a spliced ​​image based on a fish posture recognition model, and after obtaining a target detection frame image, downsampling the target detection frame image, generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on the features in the feature pyramid. The present invention can determine the posture of a fish in a moving state with high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and a device for dynamic recognition of underwater fish posture. Background Art

[0002] Aquaculture is the use of available waters for breeding, and the use of aquaculture techniques and facilities to engage in aquatic economic breeding according to the ecological habits of the breeding objects and the requirements of the water environment conditions. The behavior of fish when swimming is closely related to the environment in which they live. Detecting and identifying the behavior of fish can help determine their health.

[0003] At present, traditional image processing techniques (such as binarization, shape fitting, and grayscale value detection) and convolutional neural networks are mostly used for fish posture recognition. However, the above methods can only recognize the posture of fish in a static state, and cannot accurately recognize the posture of fish in a moving state. Summary of the invention

[0004] The present invention provides a method and device for dynamic recognition of underwater fish posture, which are used to solve the defect of low accuracy in posture recognition of fish in a moving state in the prior art.

[0005] The present invention provides a method for dynamic recognition of underwater fish posture, comprising:

[0006] Determine a fish body image to be identified;

[0007] Based on the fish posture recognition model, target detection is performed on the spliced ​​image to obtain a target detection frame image, the target detection frame image is downsampled, a feature pyramid is generated based on the downsampled target detection frame image, and the fish posture is determined based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be identified after data enhancement;

[0008] The fish posture recognition model is trained based on sample fish images and sample fish posture labels.

[0009] According to a method for dynamic recognition of underwater fish posture provided by the present invention, the method performs target detection on a spliced ​​image based on a fish posture recognition model, obtains a target detection frame image, downsamples the target detection frame image, generates a feature pyramid based on the downsampled target detection frame image, and determines the fish posture based on features in the feature pyramid, including:

[0010] Based on the input layer of the fish posture recognition model, data enhancement is performed on the fish image to be identified, and multiple images obtained after the data enhancement are spliced ​​to obtain the spliced ​​image, and target detection is performed on the spliced ​​image to obtain the target detection frame image;

[0011] Based on the sampling layer of the fish posture recognition model, the target detection frame image is sliced ​​and convolved in sequence to obtain an initial feature map, and the initial feature map is downsampled multiple times to obtain a downsampled feature map;

[0012] Based on the pyramid layer of the fish posture recognition model, upsampling the feature map of the previous layer and fusing it with the feature map of the next layer to obtain a first feature pyramid, and downsampling the feature map of the next layer and fusing it with the feature map of the previous layer to obtain a second feature pyramid, wherein the feature map of the first layer is the downsampled feature map;

[0013] Based on the fact that the fish body posture is a prediction layer of a recognition model, the fish body posture is determined according to features in the first feature pyramid and the second feature pyramid.

[0014] According to a method for dynamic recognition of underwater fish posture provided by the present invention, before performing target detection on the spliced ​​image, the method further includes: scaling the spliced ​​image to a preset standard size.

[0015] According to a method for dynamic recognition of underwater fish posture provided by the present invention, the initial feature map is downsampled multiple times to obtain a downsampled feature map, including:

[0016] The current feature map is downsampled to obtain a current initial downsampled feature map, and the previous downsampled feature map and the current initial downsampled feature map are fused based on the residual component of the sampling layer to obtain the current downsampled feature map; the first current feature map is the initial feature map.

[0017] According to a method for dynamic recognition of underwater fish posture provided by the present invention, the down-sampling feature graph is obtained, and then the method further includes:

[0018] The downsampled feature map is sequentially subjected to convolution processing, normalization processing and maximum pooling processing.

[0019] According to a method for dynamic recognition of underwater fish posture provided by the present invention, the method for determining the fish posture based on the features in the feature pyramid comprises:

[0020] Based on the features in the feature pyramid, determining multiple candidate detection boxes;

[0021] Based on the non-maximum suppression algorithm, a fish body detection frame is screened from multiple candidate detection frames, and posture recognition is performed based on the fish body detection frame to determine the fish body posture.

[0022] According to a method for dynamic recognition of underwater fish posture provided by the present invention, the loss value of the fish posture recognition model is determined based on the following formula:

[0023] Loss=-1 / n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

[0024] Among them, Loss represents the loss value of the fish posture recognition model, o[i] represents the sample predicted fish posture obtained by the fish posture recognition model predicting the sample fish image, and t[i] represents the sample fish posture label.

[0025] The present invention also provides an underwater fish posture dynamic recognition device, comprising:

[0026] A determination unit, used for determining a fish body image to be identified;

[0027] The recognition unit is used to perform target detection on the spliced ​​image based on the fish posture recognition model, and after obtaining the target detection frame image, downsample the target detection frame image, generate a feature pyramid based on the downsampled target detection frame image, and determine the fish posture based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be recognized after data enhancement;

[0028] The fish posture recognition model is trained based on sample fish images and sample fish posture labels.

[0029] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for dynamic recognition of underwater fish posture as described in any one of the above is implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for dynamic recognition of underwater fish posture as described in any one of the above is implemented.

[0031] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for dynamic recognition of underwater fish postures.

[0032] The underwater fish posture dynamic recognition method and device provided by the present invention performs target detection on the spliced ​​image based on the fish posture recognition model, and after obtaining the target detection frame image, downsamples the target detection frame image, so that the downsampled target detection frame image not only retains the feature information of the image, but also can reduce the calculation parameters of the model and improve the recognition efficiency of the model. In addition, a feature pyramid is generated based on the downsampled target detection frame image, so that the fish posture can be effectively determined based on the scale features in the feature pyramid. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 This is one of the flow charts of the underwater fish posture dynamic recognition method provided by the present invention;

[0035] Figure 2 It is a structural schematic diagram of the sampling layer of the fish posture recognition model provided by the present invention;

[0036] Figure 3 It is a schematic diagram of the structure of Focus in the sampling layer provided by the present invention;

[0037] Figure 4 It is a schematic diagram of the structure of CSP1_X in the sampling layer provided by the present invention;

[0038] Figure 5 It is a structural schematic diagram of the convolutional layer CBL provided by the present invention;

[0039] Figure 6 It is a schematic diagram of the structure of CSP2_X in the pyramid layer provided by the present invention;

[0040] Figure 7 It is a structural schematic diagram of the residual component provided by the present invention;

[0041] Figure 8 It is a schematic diagram of the structure of the SPP provided by the present invention;

[0042] Fig. 9 Schematic diagram of the model performance index curve provided by the present invention;

[0043] Fig.10 This is the second flow chart of the underwater fish posture dynamic recognition method provided by the present invention;

[0044] Fig.11 It is a structural schematic diagram of the underwater fish posture dynamic recognition device provided by the present invention;

[0045] Fig.12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] At present, traditional image processing techniques (such as binarization, shape fitting, and grayscale value detection) and convolutional neural networks are mostly used for fish posture recognition. However, the above methods can only recognize the posture of fish in a static state, and cannot accurately recognize the posture of fish in a moving state.

[0048] In addition, fish posture recognition is also performed using support vector machine technology, but this method relies on complex algorithms such as extracting feature vectors and sliding windows, which is inefficient. In addition, this method requires additional corresponding hardware facilities and has high maintenance costs.

[0049] To this end, the present invention provides a method for dynamic recognition of underwater fish posture. Figure 1 FIG. 1 is one of the flow charts of the underwater fish posture dynamic recognition method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0050] Step 110: Determine the fish image to be identified.

[0051] Here, the fish image to be identified is an image for which fish posture recognition is required. The fish image to be identified can be acquired by an image acquisition device, where the image acquisition device can be a camera or a webcam, or can be a smart device such as a smart phone, tablet, or computer equipped with a camera.

[0052] Step 120: Based on the fish posture recognition model, target detection is performed on the spliced ​​image. After obtaining the target detection frame image, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and the fish posture is determined based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be identified after data enhancement;

[0053] The fish posture recognition model is trained based on sample fish images and their sample fish posture labels.

[0054] Specifically, the stitched image is obtained by stitching the fish images to be identified after data enhancement. For example, after determining the fish image to be identified, data enhancement can be performed on it to obtain multiple enhanced images. Among them, these enhanced images can be obtained by randomly scaling, randomly cropping, and randomly arranging the fish image to be identified based on the Mosaic data enhancement algorithm. After data enhancement of the fish image to be identified, the obtained stitched image contains posture information of different angles, sizes, etc. of the fish to be identified, so that more detailed posture information of the fish to be identified when moving can be obtained based on the stitched image.

[0055] After obtaining the spliced ​​image, target detection is performed on the spliced ​​image, so that the target object that needs to be recognized can be accurately extracted from the spliced ​​image to obtain a target detection frame image. After obtaining the target detection frame image, downsampling it can not only not lose the feature information of the image, but also reduce the calculation parameters of the model and improve the recognition efficiency of the model.

[0056] After downsampling the target detection frame image, a feature pyramid is generated based on the downsampled target detection frame image, and the fish posture is determined based on the features in the feature pyramid. The feature pyramid can be a top-down FPN (feature pyramid networks) feature pyramid and a bottom-up PAN (Pyramid Attention Network) feature pyramid. The combination of the two adds bottom-up enhancement on the basis of the FPN feature pyramid, so that the top-level features obtain the position information of the bottom-level features, thereby improving the posture recognition accuracy.

[0057] It is understood that the embodiment of the present invention can pre-train a fish posture recognition model, which can be implemented by performing the following steps: first, a large number of sample fish images are collected, and their corresponding sample fish posture labels are determined by manual annotation. Then, the initial model is trained based on the sample fish images and their sample fish posture labels to obtain a fish posture recognition model.

[0058] As an optional embodiment, the sample fish image comes from a fish pond with a diameter of 5 meters and a depth of 80 centimeters. The water in the fish pond comes from filtered seawater. The water temperature is heated to 15 degrees. The camera is placed at the bottom of the pond. The sample fish image video is obtained by recording method. The video pixel is 1920*1080, 25 frames per second, and the format is mp4. The video is recorded for 1 hour, and Python is used to intercept and process the video obtained by the video. A series of initial sample fish images in jpg format are obtained by intercepting one every two seconds. A total of 1,800 initial sample fish images are obtained and saved to the hard disk, and then some low-quality initial sample fish images are deleted, such as missing fish, a large number of fish overlapping, no fish, and blurred fish images. Finally, 800 sample fish images are obtained.

[0059] In addition, after obtaining the sample fish image, the image enhancement algorithm ACE (Automatic Color Equalization) can be used to process the sample fish image. The ACE algorithm is derived from the retinex algorithm, which can adjust and stretch the contrast of the image and achieve the standardization of color and saturation. The pixel value is corrected by differentially calculating the relative brightness relationship between the target point and the surrounding pixels, which has a good enhancement effect, can correct the color cast, achieve the defogging effect, the noise reduction effect, and can also achieve the brightness enhancement of the dark scene. The processed sample fish image has no blue-green color cast of the original image, the brightness has been improved, and the underwater blur has also been improved. In addition, the sample fish image can be annotated using make sense, and different labels are marked for fish in different postures. The overlapping and missing fish are not marked to obtain the sample fish posture label. Then use python to divide the sample fish image, such as randomly dividing it into a training set and a test set in a ratio of 8:2.

[0060] The underwater fish posture dynamic recognition method provided by the embodiment of the present invention performs target detection on the spliced ​​image based on the fish posture recognition model, and after obtaining the target detection frame image, downsamples the target detection frame image, so that the downsampled target detection frame image not only retains the feature information of the image, but also can reduce the calculation parameters of the model and improve the recognition efficiency of the model. In addition, a feature pyramid is generated based on the downsampled target detection frame image, so that the fish posture can be effectively determined based on the scale features in the feature pyramid.

[0061] Based on the above embodiment, based on the fish posture recognition model, target detection is performed on the spliced ​​image, and after obtaining the target detection frame image, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and the fish posture is determined based on the features in the feature pyramid, including:

[0062] Based on the input layer of the fish posture recognition model, data enhancement is performed on the fish image to be identified, and multiple images obtained after the data enhancement are spliced ​​to obtain a spliced ​​image, and target detection is performed on the spliced ​​image to obtain a target detection frame image;

[0063] Based on the sampling layer of the fish posture recognition model, the target detection frame image is sliced ​​and convolved in sequence to obtain an initial feature map, and the initial feature map is downsampled multiple times to obtain a downsampled feature map;

[0064] Based on the pyramid layer of the fish posture recognition model, the feature map of the previous layer is upsampled and fused with the feature map of the next layer to obtain a first feature pyramid, and the feature map of the next layer is downsampled and fused with the feature map of the previous layer to obtain a second feature pyramid, and the feature map of the first layer is a downsampled feature map;

[0065] Based on the prediction layer of the fish body posture recognition model, the fish body posture is determined according to the features in the first feature pyramid and the second feature pyramid.

[0066] Specifically, the fish posture recognition model can adopt the YOLOv5 model structure, which can include a four-layer structure of an input layer, a sampling layer (Backbone layer), a pyramid layer (Neck layer) and a prediction layer (Prediction layer).

[0067] The input layer can include the Mosaic data enhancement algorithm, the adaptive image scaling algorithm, and the adaptive anchor box calculation algorithm. The sampling layer can include the focus structure, the CSP1_X structure, and the SPP structure. The pyramid layer can include the CSP2_X, FPN, and PAN structures. The prediction layer can include the GIOU_Loss and NMS non-maximum suppression algorithms.

[0068] Based on the input layer of the fish posture recognition model, the fish image to be identified is data enhanced, and the multiple images obtained after data enhancement are spliced ​​to obtain a spliced ​​image, and the spliced ​​image is subjected to target detection to obtain a target detection frame image. Among them, when the fish image to be identified is data enhanced, the Mosaic data enhancement algorithm can be used, such as randomly scaling, randomly cropping, and randomly arranging any four images and then splicing them, which improves the detection accuracy of small targets and occluded targets and the robustness of the model, reduces the memory requirements of the model, and enriches the data set. Mosaic9 is based on Mosaic, and the number of randomly scaled, randomly cropped, and randomly arranged images is increased to 9, which further improves the robustness and reduces the memory requirements of the model. In the YOLO algorithm, there will be an initial length and width anchor frame for different data sets. In the process of training the fish posture recognition model, the predicted frame is output based on the initial anchor frame, and then compared with the real frame, the difference between the two is calculated, and then reversely updated, and the network parameters are iterated. During each training, the best anchor frame value in different training sets is adaptively calculated.

[0069] like Figure 2 As shown in the figure, the sampling layer includes the focus structure, CSP1_X structure and SPP structure. The Focus structure inputs the original 608×608×3 image into the Focus structure, and after slicing, it becomes a 304×304×12 feature map, and then undergoes a convolution operation with 32 convolution kernels, and finally becomes a 304×304×32 feature map. The function of this structure is to avoid the loss of image feature information during image downsampling, so as to extract more sufficient features when using convolution operations. This module performs slicing operations before backbone, such as Figure 3 As shown in the figure, every other pixel value is selected for downsampling, and four pictures with complementary features are obtained without losing information. The W and H information are concentrated in the channel space, and the output space is expanded four times. The original three channels become twelve channels, so that after the convolution operation, a double downsampled feature map with no information loss is obtained. The Focus structure reduces the cost of convolution and improves the speed. In addition, Figure 4 and Figure 5 As shown in the figure, a convolution layer is set before the CSP1_X structure. The size of the convolution kernel is 3×3 and the step size is 2, so that the size of the feature map is reduced by half, which plays a role of downsampling. There are 4 CSP1_X structures in Backbone, and the final output feature map size is 19×19.

[0070] Based on the pyramid layer of the fish posture recognition model, the feature map of the previous layer is upsampled and fused with the feature map of the next layer to obtain the first feature pyramid (i.e., FPN feature pyramid), and the feature map of the next layer is downsampled and fused with the feature map of the previous layer to obtain the second feature pyramid (i.e., PAN feature pyramid).

[0071] For example, the pyramid layer can be an FPN+PAN structure, which combines downsampling and upsampling to generate a feature pyramid, making the network scale invariant. The feature pyramid generated by the FPN structure is top-down, and the feature maps extracted by the sampling layer are downsampled to obtain features of sizes 76×76×255, 38×38×255, and 19×19×255, respectively, where 255 represents a 255-dimensional output. FPN fuses the upsampling results of the previous layer of features with the next layer of features to obtain a feature pyramid. The embodiment of the present invention adds a PAN structure on the basis of FPN. In contrast to FPN, the PAN feature pyramid is bottom-up, and is obtained by fusing the downsampling results of the next layer of features of FPN with the previous layer of features. FPN mainly improves the effect of target detection by fusing high-level and low-level features, especially the detection effect of small-sized targets; PAN structure segments the shallow features of the network. The shallow feature information of the network is very important for target detection, because target detection is pixel-level classification, and shallow information is mostly edge features. The combination of the two adds bottom-up enhancement on the basis of FPN, so that the top-level features can obtain the position information of the bottom-level features, which improves the detection effect of large objects. Among them, the pyramid layer also includes Figure 6 The CSP2_X structure shown.

[0072] After obtaining the first pyramid and the second pyramid, the fish posture is identified by the prediction layer of the recognition model according to the features in the first feature pyramid and the second feature pyramid, and the fish posture is identified and determined.

[0073] Based on any of the above embodiments, target detection is performed on the stitched image, and before that, the method further includes: scaling the stitched image to a preset standard size.

[0074] Specifically, the input layer may also include an adaptive image scaling algorithm, which is used to scale the stitched image to a preset standard size before performing target detection on the stitched image to add as few black edges as possible, thereby greatly improving the calculation speed of the model.

[0075] Based on any of the above embodiments, the initial feature map is downsampled multiple times to obtain a downsampled feature map, including:

[0076] The current feature map is downsampled to obtain the current initial downsampled feature map, and the previous downsampled feature map and the current initial downsampled feature map are fused based on the residual component of the sampling layer to obtain the current downsampled feature map; the first current feature map is the initial feature map.

[0077] like Figure 7 As shown in the figure, the sampling augmentation also includes multiple residual components and a tensor splicing module, and uses SiLU as the activation function. The residual component uses the identity mapping to fuse the previous downsampled feature map and the current initial downsampled feature map to obtain the current downsampled feature map to prevent gradient explosion and network degradation during the convolution process, thereby enhancing the learning ability of the model, reducing computational bottlenecks, and reducing memory consumption.

[0078] Based on any of the above embodiments, a downsampled feature map is obtained, and then the following steps are further included:

[0079] The downsampled feature map is sequentially subjected to convolution, normalization and maximum pooling.

[0080] Specifically, the sampling layer may also include an SPP structure, such as Figure 8 As shown in the figure, the SPP structure is first processed by 1×1 convolution and then normalized. The main steps of normalization are to find the mean and variance of all data, then divide the pixel value by the variance after finding the difference between the mean and the pixel value for normalization, and add the offset factor and scale change factor to control the normalized value. The value of the factor is learned by the model during training. Then, 5 / 9 / 13 can be used for maximum pooling, and then concat tensor splicing can be performed. Finally, SiLU is used as the activation function, which effectively avoids problems such as image distortion caused by cropping and scaling operations of the image area, improves the receptive field, solves the problem of repeated extraction of image features by the model, greatly improves the operation speed, and saves computing costs.

[0081] Based on any of the above embodiments, determining the posture of the fish body based on the features in the feature pyramid includes:

[0082] Based on the features in the feature pyramid, multiple candidate detection boxes are determined;

[0083] Based on the non-maximum suppression algorithm, the fish detection frame is screened from multiple candidate detection frames, and posture recognition is performed based on the fish detection frame to determine the fish posture.

[0084] Specifically, based on the features in the feature pyramid, multiple candidate detection frames can be obtained. Usually, these candidate detection frames contain the same content, so it is necessary to screen out the best detection frame as the fish body detection frame for posture recognition, thereby simplifying the detection result. Therefore, the embodiment of the present invention is based on the non-maximum suppression algorithm (NMS) to screen out the fish body detection frame from multiple candidate detection frames, so that some occluded and overlapping targets can be improved to a certain extent, and the originally undetected targets can be detected and recognized after the NMS operation.

[0085] Among them, when performing non-maximum suppression operations, the corresponding loss value can be determined as: GIoUloss = 1-(IoU-|Ac-U|) / |Ac|, where IoU is the intersection-union ratio, Ac is the area of ​​the minimum closure of the predicted box and the true box, and U is the union area of ​​the predicted box and the true box. The formula means: first calculate the minimum closure area of ​​the two boxes, then calculate the proportion of the area that does not belong to the two boxes in the IoU and closure area to the closure area, and finally subtract this proportion from IoU to get GIoU. The final Boundingbox loss is 1-GIoU.

[0086] Based on any of the above embodiments, the loss value of the fish posture recognition model is determined based on the following formula:

[0087] Loss=-1 / n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

[0088] Among them, Loss represents the loss value of the fish posture recognition model, o[i] represents the sample predicted fish posture obtained by the fish posture recognition model predicting the sample fish image, and t[i] represents the sample fish posture label.

[0089] In addition, the SiLU activation function used in the embodiment of the present invention has the characteristics of no upper bound but lower bound, smoothness, and non-monotonicity, and is a special case of the Swish activation function: f(x)=x*σ(x)f′(x)=f(x)+σ(x)(1-f(x)).

[0090] As an optional embodiment, the fish posture recognition model operating environment can use Google's colab platform, the data and YOLOv5 model files are stored on Google Cloud Disk, and the model is imported into the colab platform for training. YOLOv5 uses version 6.1, the python version is 3.7, and the required library files are installed using the requirements file. The cloud disk capacity is 15GB, the memory capacity is 8GB, the GPU is Tesla K80, the video memory size is 11441MB, and CUDA is turned on during training.

[0091] The parameters of the YOLOv5 model are set as follows: use the yolov5n model, use the hyperparameters optimized for scratch-low, set the epoch to 100, the batch-size to 16, the image size to 640, enable cache, enable agnostic-nms, enable augmentation, enable FP16 half, set the optimizer to AdamW, enable label-smoothing and set it to 0.1, set conf-thres to 0.5, and set iou-thres to 0.5.

[0092] After Cache is turned on, the data set will be imported into the video memory before training, and it does not need to be read from the hard disk every time it is used, which speeds up the training speed. agnostic-nms is nms non-maximum suppression. After turning it on, it can reduce repeated and redundant target boxes. iou-thres is the intersection-over-union ratio used when non-maximum suppression is turned on. conf-thres is the confidence threshold setting. Only target boxes with confidence greater than this value are displayed in the training results. augmentation is a data enhancement method based on the HSV color space. After turning it on, it can expand the data volume to a certain extent and improve the robustness of the model. AdamW is a gradient descent algorithm obtained by adding L2 regularization to Adam, which reduces the overfitting phenomenon and has a significantly better convergence speed than Adam and SGD. FP16 half uses half floating point precision, which can greatly reduce memory usage and speed up the calculation speed without basically affecting the accuracy of the model. Label-smoothing is a regularization strategy that mainly adds noise through soft one-hot, reducing the weight of the category of the real sample label when calculating the loss function, and ultimately suppressing overfitting and improving the generalization ability of the model.

[0093] In addition, in order to improve the convergence speed of the model, improve the training efficiency, and reduce the number of iterations, the hyperparameters of the YOLOv5 model are obtained using the evolve hyperparameter evolutionary algorithm. For example, a total of 100 epochs were trained for different and mutually orthogonal hyperparameters, and each epoch was trained until the 10th generation.

[0094] The initial value of the hyperparameter is the value in the hyp.scratch-low file in YOLOv5. The hyperparameter evaluation index fitness is set to 0.1×mAP(0.5)+0.9×mAP(0.5:0.95). This method obtains the optimal hyperparameter by maximizing the fitness index. This method uses a genetic algorithm. For each generation in the evolution process, the previous generation with the highest fitness score is selected for mutation. All hyperparameters will have a 20% chance of mutating to other values.

[0095] The optimized values ​​are: lr0 learning rate 0.008, lr1 learning rate 0.07, learning momentum 0.85, weight decay 0.0006, warm-up algebra 3.4, warm-up learning momentum 0.95, warm-up learning initial bias learning rate 0.09, anchor box size 0.065, iou threshold 0.2, hue transformation 0.001, saturation enhancement 0.63, brightness enhancement 0.43, image rotation probability 0.0009, image translation probability 0.12, image scaling probability 0.9, image flipping probability 0.001, image mirroring probability 0.5, mosaic image enhancement probability 0.85, mixup image mixing probability 0.05, and the number of anchor boxes in each layer 4.6.

[0096] Next, the images were trained using a computer equipped with the deployed YOLOv5 6.1 version. The model has its own image enhancement algorithm, which can expand the data set by mirroring, flipping, cropping, splicing, rotating, translating, copying, mixing, adjusting brightness, hue, saturation, etc. of the original image. The YOLOv5 algorithm iteratively updates the weight matrix and bias in the forward propagation through gradient descent to reduce the loss between the predicted box and the true box, and obtains the value of the error cost function based on the updated weights, and uses the optimizer to adjust the learning rate, and finally obtains the fish posture recognition model.

[0097] Among them, the following parameters can be used to evaluate the performance of the model:

[0098] FN: It is judged as a negative sample, but it is actually a positive sample.

[0099] FP: It is judged as a positive sample, but in fact it is a negative sample.

[0100] TN: It is judged as a negative sample and is actually a negative sample.

[0101] TP: It is judged as a positive sample and is actually a proof sample.

[0102] Intersection over Union (IoU) = A∩B / A∪B.

[0103] Precision precision = TP / (TP+FP).

[0104] Recall rate recall = TP / (TP+FN).

[0105] F1-score=2(Precision×Recall) / (Precision+Recall).

[0106] mAP = 1 / m∑AP(i), i∈[0,m), i∈N+, m is the number of categories, and AP(i) is the average precision of the i-th category.

[0107] mAP(0.5) represents the mAP when IoU is 0.5, TP represents the number of detection boxes with IoU>0.5, and FP represents the number of detection boxes with IoU<=0.5.

[0108] mAP(0.5:0.95) represents the average mAP at different IoU thresholds (from 0.5 to 0.95, with a step size of 0.05).

[0109] YOLOv5 is divided into five structures according to depth and width, namely v5n, v5s, v5m, v5l, and v5x. The embodiment of the present invention can adopt the simplest v5n structure, which runs the fastest. Compared with the other four models, the prediction accuracy is not significantly reduced.

[0110] like Fig. 9 As shown, the fish posture recognition model achieved 95.1% accuracy, 94.9% recall, 97.8% mAP (0.5) score and 83.0% mAP (0.5: 0.95) score, and all evaluation indicators were basically converged at 100 epochs. The evaluation of the model mAP score further proves that the method provided by the embodiment of the present invention has good robustness and accuracy, and the recognition speed can be as high as 120FPS, which can realize quasi-real-time fully automatic recognition and positioning of moving targets of fish in underwater environments.

[0111] Based on any of the above embodiments, the present invention also provides a method for dynamic recognition of underwater fish posture, such as Fig.10 As shown, the method includes:

[0112] First, video image data is obtained, and image preprocessing and image annotation are performed on the video image data in sequence. Then, the annotated video image data is randomly divided into a training set and a test set, and the training set is used to build a model and train a fish posture detection model (the model can be a YOLOv5-F model).

[0113] After the model training is completed, the test set is used to evaluate the performance and accuracy of the model to determine whether it meets the standards. If so, posture recognition can be performed on real-time video data; if not, it is necessary to continue collecting data to train the model.

[0114] The underwater fish posture dynamic recognition device provided by the present invention is described below. The underwater fish posture dynamic recognition device described below and the underwater fish posture dynamic recognition method described above can correspond to each other.

[0115] Based on any of the above embodiments, the present invention also provides a device for dynamic recognition of underwater fish posture, such as Fig.11 As shown, the device comprises:

[0116] A determination unit 1110 is used to determine a fish body image to be identified;

[0117] The recognition unit 1120 is used to perform target detection on the spliced ​​image based on the fish posture recognition model, and after obtaining the target detection frame image, downsample the target detection frame image, generate a feature pyramid based on the downsampled target detection frame image, and determine the fish posture based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be recognized after data enhancement;

[0118] The fish posture recognition model is trained based on sample fish images and sample fish posture labels.

[0119] Based on any of the above embodiments, the identification unit 1120 includes:

[0120] An input unit, configured to perform data enhancement on the to-be-identified fish image based on the input layer of the fish posture recognition model, splice multiple images obtained after the data enhancement to obtain the spliced ​​image, and perform target detection on the spliced ​​image to obtain the target detection frame image;

[0121] A sampling unit, configured to sequentially slice and convolve the target detection frame image based on the sampling layer of the fish posture recognition model to obtain an initial feature map, and perform multiple downsampling on the initial feature map to obtain a downsampled feature map;

[0122] A pyramid unit is used for upsampling a feature map of an upper layer based on the pyramid layer of the fish posture recognition model and fusing it with a feature map of a lower layer to obtain a first feature pyramid, and downsampling a feature map of the lower layer and fusing it with a feature map of an upper layer to obtain a second feature pyramid, wherein the feature map of the first layer is the downsampled feature map;

[0123] A prediction unit is used to determine the fish posture based on the fact that the fish posture is a prediction layer of a recognition model and according to features in the first feature pyramid and the second feature pyramid.

[0124] Based on any of the above embodiments, the device further includes a scaling unit, configured to scale the stitched image to a preset standard size before performing target detection on the stitched image.

[0125] Based on any of the above embodiments, the sampling unit is used to:

[0126] The current feature map is downsampled to obtain a current initial downsampled feature map, and the previous downsampled feature map and the current initial downsampled feature map are fused based on the residual component of the sampling layer to obtain the current downsampled feature map; the first current feature map is the initial feature map.

[0127] Based on any of the above embodiments, the device further includes:

[0128] The processing unit is used to perform convolution processing, normalization processing and maximum pooling processing on the downsampled feature map in sequence after obtaining the downsampled feature map.

[0129] Based on any of the above embodiments, the identification unit 1120 includes:

[0130] A candidate unit, used to determine a plurality of candidate detection boxes based on the features in the feature pyramid;

[0131] The screening unit is used to screen a fish body detection frame from multiple candidate detection frames based on a non-maximum suppression algorithm, and perform posture recognition based on the fish body detection frame to determine the fish body posture.

[0132] Based on any of the above embodiments, the loss value of the fish posture recognition model is determined based on the following formula:

[0133] Loss=-1 / n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

[0134] Among them, Loss represents the loss value of the fish posture recognition model, o[i] represents the sample predicted fish posture obtained by the fish posture recognition model predicting the sample fish image, and t[i] represents the sample fish posture label.

[0135] Fig.12 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig.12 As shown, the electronic device may include: a processor 1210, a memory 1220, a communications interface 1230 and a communications bus 1240, wherein the processor 1210, the memory 1220 and the communications interface 1230 communicate with each other through the communications bus 1240. The processor 1210 may call the logic instructions in the memory 1220 to execute the underwater fish posture dynamic recognition method, the method comprising: determining a fish image to be recognized; performing target detection on the spliced ​​image based on a fish posture recognition model, after obtaining a target detection frame image, downsampling the target detection frame image, and generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish image to be recognized after data enhancement; the fish posture recognition model is trained based on the sample fish image and its sample fish posture label.

[0136] In addition, the logic instructions in the above-mentioned memory 1220 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the underwater fish posture dynamic recognition method provided by the above-mentioned methods, and the method includes: determining a fish image to be identified; based on a fish posture recognition model, performing target detection on the spliced ​​image, and after obtaining a target detection frame image, downsampling the target detection frame image, and generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be identified after data enhancement; the fish posture recognition model is trained based on sample fish images and their sample fish posture labels.

[0138] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned underwater fish posture dynamic recognition method, the method comprising: determining a fish image to be identified; performing target detection on a spliced ​​image based on a fish posture recognition model, and after obtaining a target detection frame image, downsampling the target detection frame image, and generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on the features in the feature pyramid; the spliced ​​image is obtained by splicing the fish images to be identified after data enhancement; the fish posture recognition model is trained based on sample fish images and their sample fish posture labels.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamic recognition of underwater fish posture, characterized in that: include: Determine a fish body image to be identified; Based on the fish posture recognition model, target detection is performed on the spliced ​​image to obtain a target detection frame image, then the target detection frame image is downsampled, a feature pyramid is generated based on the downsampled target detection frame image, and the fish posture is determined based on features in the feature pyramid; The spliced ​​image is obtained by splicing the fish body images to be identified after data enhancement; The fish posture recognition model is trained based on sample fish images and sample fish posture labels; The method of performing target detection on the spliced ​​image based on the fish posture recognition model to obtain a target detection frame image, downsampling the target detection frame image, generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on features in the feature pyramid includes: Based on the input layer of the fish posture recognition model, data enhancement is performed on the fish image to be identified, and multiple images obtained after the data enhancement are spliced ​​to obtain the spliced ​​image, and target detection is performed on the spliced ​​image to obtain the target detection frame image; Based on the sampling layer of the fish posture recognition model, the target detection frame image is sliced ​​and convolved in sequence to obtain an initial feature map, and the initial feature map is downsampled multiple times to obtain a downsampled feature map; Based on the pyramid layer of the fish posture recognition model, upsampling the feature map of the previous layer and fusing it with the feature map of the next layer to obtain a first feature pyramid, and downsampling the feature map of the next layer and fusing it with the feature map of the previous layer to obtain a second feature pyramid, wherein the feature map of the first layer is the downsampled feature map; Based on the fact that the fish body posture is a prediction layer of a recognition model, the fish body posture is determined according to features in the first feature pyramid and the second feature pyramid.

2. The method for dynamic recognition of underwater fish posture according to claim 1, characterized in that: The performing target detection on the stitched image also includes: scaling the stitched image to a preset standard size.

3. The method for dynamic recognition of underwater fish posture according to claim 1, characterized in that: The step of downsampling the initial feature map multiple times to obtain a downsampled feature map includes: The current feature map is downsampled to obtain a current initial downsampled feature map, and the previous downsampled feature map and the current initial downsampled feature map are fused based on the residual component of the sampling layer to obtain the current downsampled feature map; the first current feature map is the initial feature map.

4. The method for dynamic recognition of underwater fish posture according to claim 1, characterized in that: The down-sampled feature map is obtained, and then the following steps are further included: The downsampled feature map is sequentially subjected to convolution processing, normalization processing and maximum pooling processing.

5. The method for dynamic recognition of underwater fish posture according to any one of claims 1 to 4, characterized in that: The determining of the fish posture based on the features in the feature pyramid comprises: Based on the features in the feature pyramid, determining multiple candidate detection boxes; Based on the non-maximum suppression algorithm, a fish body detection frame is screened from multiple candidate detection frames, and posture recognition is performed based on the fish body detection frame to determine the fish body posture.

6. The method for dynamic recognition of underwater fish posture according to any one of claims 1 to 4, characterized in that: The loss value of the fish posture recognition model is determined based on the following formula: Loss=-1 / n∑(t[i]log(o[i])+(1-t[i])log(1-o[i])); Among them, Loss represents the loss value of the fish posture recognition model, o[i] represents the sample predicted fish posture obtained by the fish posture recognition model predicting the sample fish image, t[i] represents the sample fish posture label, and n represents the number of the sample fish images.

7. An underwater fish posture dynamic recognition device, characterized in that: include: A determination unit, used for determining a fish body image to be identified; A recognition unit, configured to perform target detection on the spliced ​​image based on a fish posture recognition model, downsample the target detection frame image after obtaining the target detection frame image, generate a feature pyramid based on the downsampled target detection frame image, and determine the fish posture based on features in the feature pyramid; The spliced ​​image is obtained by splicing the fish body images to be identified after data enhancement; The fish posture recognition model is trained based on sample fish images and sample fish posture labels; The method of performing target detection on the spliced ​​image based on the fish posture recognition model to obtain a target detection frame image, downsampling the target detection frame image, generating a feature pyramid based on the downsampled target detection frame image, and determining the fish posture based on features in the feature pyramid includes: Based on the input layer of the fish posture recognition model, data enhancement is performed on the fish image to be identified, and multiple images obtained after the data enhancement are spliced ​​to obtain the spliced ​​image, and target detection is performed on the spliced ​​image to obtain the target detection frame image; Based on the sampling layer of the fish posture recognition model, the target detection frame image is sliced ​​and convolved in sequence to obtain an initial feature map, and the initial feature map is downsampled multiple times to obtain a downsampled feature map; Based on the pyramid layer of the fish posture recognition model, upsampling the feature map of the previous layer and fusing it with the feature map of the next layer to obtain a first feature pyramid, and downsampling the feature map of the next layer and fusing it with the feature map of the previous layer to obtain a second feature pyramid, wherein the feature map of the first layer is the downsampled feature map; Based on the fact that the fish body posture is a prediction layer of a recognition model, the fish body posture is determined according to features in the first feature pyramid and the second feature pyramid.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the underwater fish posture dynamic recognition method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for dynamic recognition of underwater fish postures as described in any one of claims 1 to 6 is implemented.

Images