A Method for Detecting and Recognizing Koi Fish Using Multi-Scale Feature Extraction

Abstract

The present invention provides a method for detecting and recognizing koi fish using multi-scale feature extraction, belonging to the technical field of deep learning image detection, and particularly relates to a method for detecting and recognizing the quality of koi fish. It is characterized by steps such as a data collection process, a picture processing and feature extraction process, multi-scale feature extraction, a training process, etc. The purpose is to improve the accuracy of koi fish recognition; the method used in the present invention has characteristics such as a high detection rate and a high accuracy rate when applied to underwater koi fish detection, making up for a series of problems generated in the traditional manual screening of koi fish, such as a low manual detection rate over time; it can ensure that the data set is strictly unified in format overall; it can solve the problem of insufficient koi fish data sets nowadays; it can effectively fuse the feature map information of different scales, further increase the detection rate of underwater koi fish, and can detect underwater koi fish targets with multiple occlusions.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning image detection, and particularly relates to a method for detecting and identifying the quality of koi carp. Background Art

[0002] In recent years, deep learning has become extremely popular. It is a research direction in the field of machine learning. It can learn the internal laws and representation levels of sample data, and the information obtained during these learning processes is very helpful for the interpretation of data such as text, images, and sounds. Its ultimate goal is to enable machines to have the ability of analysis and learning like humans, and be able to recognize data such as text, images, and sounds. Deep learning is a complex machine learning algorithm, and the effects achieved in speech and image recognition far exceed those of previous related technologies. In the field of object detection based on deep learning, it is defined as a computer's judgment on whether a given image or video contains a target object. If a target object is contained, the position and category information of the target in the image or video need to be given. The application fields of object detection are diverse, such as security prevention and control, enterprise business, autonomous driving, etc.

[0003] China will eventually become the largest koi carp market in the world. The koi carp market has huge development potential. There are many varieties of koi carp, with beautiful appearances and high ornamental values, and the sales volume in the market is relatively stable. Koi carp have high reproductive abilities, strong adaptability to the environment, are easy to raise, and have high yields. From the current market conditions and economic benefits of koi carp, the breeding profit is quite good. Calculated according to the current market price, the profit from breeding one mu of koi carp is about 5000 - 10000 yuan, which is specifically related to the breeding variety. The better the variety, the higher the price and the profit. The market demand for koi carp is continuously increasing, but currently, there are relatively few places for large-scale breeding of koi carp. Coupled with the fact that koi carp breeding belongs to a sunrise industry, the prospect is relatively good.

[0004] At present, the target recognition technology for marine fish in the world has been very mature, so the publicly available marine fish data sets are very rich. However, for the object detection and recognition of koi carp, and the corresponding data sets are extremely scarce. Moreover, there is no experience in the recognition methods for different sizes and species, including the recognition of occluded and small targets, and there is no good detection effect. There are also problems with the accuracy of batch detection of the species of koi carp. And manual detection will also produce a large number of errors during the recognition process, which will have a significant impact on improving the overall efficiency of related industries. Therefore, in order to be able to better distinguish the quality of koi carp, an efficient koi carp detection algorithm is very important. Summary of the Invention

[0005] Object of the Invention: The present invention aims to provide a method for detecting and identifying koi carp using multi-scale feature extraction, aiming to improve the accuracy of koi carp recognition.

[0006] Technical solution:

[0007] A method for detecting and identifying koi fish using multi-scale feature extraction is characterized in that the method is performed according to the following steps:

[0008] Data collection process: Obtain and annotate several pictures of koi fish, and divide them into a training set and a validation set with a certain ratio;

[0009] Image processing and feature extraction: The labeled images are pre-processed with mosaic data augmentation and adaptive scaling before being fed into the main model's backbone network, where image features are extracted. A multi-scale feature extraction module (Nonlocal-pro) then extracts multi-scale features of koi fish, enabling the method to detect koi of different sizes.

[0010] The processing of the main model includes: the feature map processed by the backbone network is processed by a multi-scale feature extraction module (Nonlocal-pro) and a codec module (Transformer). The multi-scale feature extraction module extracts multi-scale features of koi fish, enabling the method to detect koi fish of different sizes; the codec module improves the performance of small target detection; the feature map processed by the backbone network is passed to the neck module, where the feature map is connected through a bidirectional weighted feature pyramid to enhance feature extraction. At the same time, a CA attention module is added to the detection head to enable it to embed the position information of the koi fish in the image into the channel. At this point, the main model detection is completed;

[0011] Next, we enter the training process: the output end gives the bounding box and confidence through the 4 valid feature layers output from the last CA attention module:

[0012] The non-maximum suppression method is used to filter out duplicate bounding boxes and obtain the predicted box. The predicted box is compared with the box generated by the annotation tool, and the GIoU loss function is used to calculate the loss. The loss function is then used for backpropagation to adjust the weight. The GIoU loss formula is as follows:

[0013]

[0014] A: Marking box rectangle;

[0015] B: prediction box rectangle;

[0016] C: The minimum bounding rectangle of the image formed by the two boxes, that is, the area of the minimum box that contains both the predicted box and the real box;

[0017] Repeat the above process to make it gradually converge, and continuously adjust the parameters through the test of the validation set to make it have generalization ability and accuracy improvement;

[0018] Preferably, in the small-scale feature extraction process step of the multi-scale feature extraction module (Nonlocal-pro), when the original feature map is input into the small-scale feature extraction process, the number of channels of the picture will be compressed separately first, and the feature extraction of the small-scale convolution kernel will be carried out twice. Then, the three results will be merged in the dimensions other than the number of channels, which refers to the width and height dimensions. At this time, the dimension of the picture becomes THW*C; T represents the number of pictures input in batches, H represents the height, W represents the width, and C represents the number of channels. The dimensions of the three results just mentioned all become THW*C;

[0019] Next, perform a matrix dot product operation on the result of transposing the matrix after compressing the number of channels and the result of the first small-scale convolution kernel feature extraction to obtain the relationship between each pixel in each frame and all pixels in all other frames. The dimension of the result obtained by the dot product becomes THW*THW, that is, the dot product of THW*C and C*THW. Then, perform a softmax operation on this self-correlation feature to obtain a result with a value range of [0,1];

[0020] Then, perform a matrix dot product operation on this result and the result of the second small-scale convolution kernel feature extraction. At this time, the dimension is the dot product operation of THW*THW and THW*C, and the obtained result dimension is THW*C; So far, the small-scale feature extraction process in the multi-scale feature extraction module (Nonlocal-pro) has been completed. The processing processes of the medium-scale and large-scale are exactly the same as the small-scale feature extraction process. The difference is that the size of the convolution kernel used in the medium-scale and large-scale extraction processes is different from that of the small-scale feature extraction convolution kernel. Therefore, this module can extract features of different scales. After extracting the features of three scales, three feature maps of different scales will be generated. At this time, stack these three feature maps in the channel dimension to get a new feature map, then adjust the number of channels of this feature map to be the same as that of the original input feature map, and finally add this feature map and the original feature map through matrix addition to output. So far, the entire multi-scale feature extraction module (Nonlocal-pro) has been explained;

[0021] Preferably, the input picture of this method first undergoes Mosaic data augmentation to expand the dataset for the subsequent model training process; Next, the picture that has undergone Mosaic data augmentation will go through a series of feature extraction processes and be sent to the above-mentioned multi-scale feature extraction module (Nonlocal-pro), where the main model extracts multi-scale features.

[0022] Preferably, the codec module (Transformer) of this method is a residual structure, which is divided into a backbone part and a residual edge part. The residual edge part is directly connected end to end without any processing or after a small amount of processing.

[0023] Preferably, the neck module of this method adopts the connection method of a bidirectional weighted feature pyramid to re-fuse the features of the four effective feature layers of the backbone network to generate four new effective feature layers, and respectively send them to the detection head through the CA attention module; the connection method of the bidirectional weighted feature pyramid has two paths, one of which is a bottom-up path that transmits the position information of low-level features; the other path is a top-down path that transmits the semantic information of high-level features; in addition to the two paths, this structure also adds an additional edge between the input and output nodes of the same layer and deletes those nodes with only one input edge; the specific input-output relationship is shown in the following formula:

[0024]

[0025]

[0026] As Figure 5 shown, the above two formulas take Feature Layer 3 as an example, represents the output of this layer, represents the output of the intermediate layer, represents the input of this layer, represents the output of Feature Layer 2, represents the input of Feature Layer 4, Resize represents upsampling or downsampling, ω represents the learned parameter used to distinguish the importance of different features in the feature fusion process, and Conv represents feature convolution.

[0027] Preferably, the CA attention module of this method adds position information to the channel attention. This attention mechanism decomposes the channel attention into two parallel one-dimensional feature encoding processes, aggregating features in two directions respectively; one direction obtains the long-range dependence relationship, and the other direction obtains the accurate position information; encodes the generated feature map to form a pair of direction-aware and position-sensitive features; the CA attention module mainly includes two steps: coordinate information embedding and coordinate attention generation; for coordinate information embedding, the global pooling is divided into two one-dimensional feature encoding operations; for the input feature map X with the dimension of C×H×W, first use the pooling kernels of size (H,1) and (1,W) to encode each channel along the horizontal coordinate and the vertical coordinate respectively, that is, the output of the c-th channel with height h and the c-th channel with width w, and the output formulas are shown in the following two formulas:

[0028]

[0029]

[0030] The above formula aggregates features along two directions and returns a pair of direction-aware attention features Z h and Z w , which helps the network locate the target to be detected more accurately; for coordinate attention generation, the module first generates two feature layers before cascading, and then uses a shared 1×1 convolution to transform F1. The formula is as follows:

[0031] f=δ(F1([Z h ,Z w ]))

[0032] Where f∈R C / r×(H+W) It is the intermediate feature map of spatial information in the horizontal and vertical directions. r represents the downsampling ratio and is set to 16 to control the size of the module, which is the most balanced between overall performance and computational complexity. δ represents the nonlinear activation function, [·,·] represents the connection operation along the spatial dimension, and then f is split into two separate tensors f along the spatial dimension. w ∈R C / r×W and f h ∈R C / r×W Then use two 1×1 convolution F h and F w The feature map f h and f w Transformed to the same number of channels as input X, the result is as follows:

[0033] g h =σ(F h (f h ))

[0034] g w =σ(F w (f w ))

[0035] Finally, for g h and g w Expanding on this, as the attention weight, the final output of the CA attention module can be expressed as follows:

[0036]

[0037] At this point, the entire CA attention module is executed.

[0038] Beneficial effects: Mosaic image enhancement expands the dataset through methods such as splicing, rotation, and symmetry, facilitating the subsequent training process. The addition of the multi-scale feature extraction module (Nonlocal-pro) can effectively detect koi fish of different scales underwater, while the addition of the encoder-decoder module (Transformer) can detect smaller koi fish targets. Using the connection method of bidirectional weighted feature pyramids in the neck module can effectively fuse feature information of different scales and greatly improve the performance of multi-occluded koi fish target detection underwater. This invention uses deep learning methods for koi fish detection, making up for a series of problems in the traditional manual screening of koi fish, such as the low detection rate by humans over time.

[0039] The final achieved effect is that when the camera captures real-time koi fish images underwater, the monitor will detect the quality of each koi fish in real-time. Specifically, each detected koi fish will be framed by a rectangular box, and the quality of the koi fish will be marked on each rectangular box, and the confidence level will also be displayed after the quality of the koi fish. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is the dataset collection flow chart of the present invention;

[0041] Figure 2 is the main method flow chart of the present invention;

[0042] Figure 3 is the schematic diagram of the structure of the main model components of the present invention;

[0043] Figure 4 is the schematic diagram of the connection relationship of the multi-scale feature extraction module (Nonlocal-pro) in the main model of the present invention;

[0044] Figure 5 is the schematic diagram of the connection relationship of the neck (BIFPN + CA attention) in the main model of the present invention;

[0045] Figure 6 is the schematic diagram of the CA attention module of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0046] The present invention will be described in detail below in conjunction with the accompanying drawings through specific embodiments.

[0047] A method for detecting and recognizing koi fish using multi-scale feature extraction, characterized in that: the detection and recognition method is carried out according to the following steps:

[0048] (1) Data collection process: Obtain a number of pictures of koi fish and label them, and divide them into a training set and a validation set in a certain proportion;

[0049] (2) Image processing and feature extraction process: The transmitted marked images are pre-processed by mosaic data augmentation and adaptive scaling, and then fed into the backbone network. In the backbone network, feature extraction of the images is performed. After that, a multi-scale feature extraction module (Nonlocal-pro) is used to perform multi-scale feature extraction of the koi fish, so that the method can detect koi fish of different scales;

[0050] The main model processing process includes: The feature map processed by the backbone network is processed by a multi-scale feature extraction module (Nonlocal-pro) and an encoder-decoder module (Transformer). The multi-scale feature extraction module performs multi-scale feature extraction of the koi fish, so that the method can detect koi fish of different scales; the encoder-decoder module improves the performance of small target detection; the feature map processed by the backbone network is fed into the neck module, where the feature map is enhanced by bidirectional weighted feature pyramid connection method, and a CA attention module is added at the detection head, so that it can embed the position information of the koi fish in the image into the channel. So far, the main model detection is completed;

[0051] (3) Enter the training process: The output end gives the bounding box and confidence through the 4 effective feature layers finally output from the CA attention module:

[0052] The non-maximum suppression method is used to filter out duplicate bounding boxes, and the predicted boxes are obtained. The predicted boxes are compared with the boxes generated by the annotation tool, and the GIoU loss function is used to calculate the loss, and then the loss function is used for backpropagation to adjust the weights. The GIoU loss formula is as follows:

[0053]

[0054] A: The annotated box rectangle;

[0055] B: The predicted box rectangle;

[0056] C: The smallest circumscribed rectangle of the image formed by the two boxes, that is, the area of the smallest box that contains both the predicted box and the true box;

[0057] Repeat the above process to make it gradually converge, and continuously adjust the parameters through the test of the validation set to make it have generalization ability and accuracy improvement, and finally obtain the recognition of the koi fish.

[0058] Some preliminary preparations are required before using this method, that is, the preparation process of the dataset is as Figure 1 . As shown, since there is little research on koi fish in the world at present, it is impossible to find a publicly available training dataset. Therefore, the acquisition and preprocessing of images and the production of the dataset in the early stage are particularly important.

[0059] After collecting several koi pictures, the first thing to do is to preprocess the pictures. Because the collected pictures contain multiple koi, they need to be cropped to increase the training effect and model accuracy.

[0060] After the preprocessing of the dataset is completed, you need to use the annotation tool to frame the koi fish with a rectangular frame for each processed image and label the koi species name. Due to reasons such as saving format and compatibility, the dataset will be temporarily saved in VOC format.

[0061] Since the dataset format required by this invention is TXT format, it is necessary to convert the pre-processed dataset into TXT format through a specific script, and at the same time divide the dataset into a training set and a validation set with a specific ratio.

[0062] Once the data set is ready, you can input the image:

[0063] like Figure 2 As shown, the processed data set is sent to the input of this method. Because the sizes of all pictures are different, and this method requires all pictures to be the same size to generate the feature layer, the picture needs to be adaptively scaled at this time, that is, reduced according to the input size required by this method. Then, the black bars added to the shorter sides become squares to meet the input specifications of 640 pixels x 640 pixels. If the data is insufficient, a data enhancement method is required. The mosaic method is used to splice the four pictures into one picture by randomly scaling, randomly cropping, and randomly arranging them. In this way, when this method receives a picture, it means that it accepts the targets of the original four pictures at the same time, enriching the data set. At this point, the preliminary picture and processing work of this method has been completed.

[0064] The processed pictures will then be Figure 2 The main model part shown is also the core part of the present invention. Figure 3 The network shown is divided into the backbone network (Backbone) and the neck (Neck).

[0065] The backbone network of the main model mainly includes a multi-scale feature extraction module (Nonloacl-pro) and a codec module (Transformer). The Transformer is mainly used to improve the performance of small target detection of this method. Since the codec module is already very common, it will not be elaborated here. Here we focus on the multi-scale feature extraction module in this invention. Figure 4 shown.

[0066] like Figure 4As shown, the original feature map is input into the small-scale feature extraction process. First, the number of channels of the compressed image is separated and the features are extracted by two small-scale convolutional kernels. Then, the dimensions other than the number of channels of these three results are merged, which refers to the dimensions of width and height. At this time, the dimension of the image becomes THW*C (T represents the number of batch input images, H represents height, W represents width, and C represents the number of channels). Therefore, the dimensions of the three results just now all become THW*C.

[0067] Since pixels are essentially matrices, one of the results will be transposed next (here, the result after compressing the number of channels is selected for transposition). At this time, the matrix dimension of this result becomes C*THW, and the dimensions of the other two results remain unchanged, that is, still THW*C.

[0068] Next, the result of the matrix transpose after compressing the number of channels is dot-multiplied with the result of the first small-scale convolutional kernel feature extraction. The purpose is to calculate the self-correlation in the features, that is, to obtain the relationship between each pixel in each frame and all pixels in all other frames. The dimension of the result obtained by dot multiplication becomes THW*THW (dot multiplication of THW*C and C*THW). Then, a softmax operation is performed on this self-correlation feature, and a result with a value range of [0,1] can be obtained.

[0069] Then, this result is dot-multiplied with the result of the second small-scale convolutional kernel feature extraction. At this time, the dimension is the dot multiplication operation of THW*THW and THW*C, and the dimension of the obtained result is THW*C.

[0070] So far, the small-scale feature extraction process in the multi-scale feature extraction module (Nonlocal-pro) has been completed.

[0071] The processes of the medium-scale and large-scale feature extraction modules are exactly the same as the small-scale feature extraction process. The only difference is that the convolutional kernels used for scale feature extraction are different.

[0072] The dimensions of the three results obtained through the small, medium, and large scale feature extraction processes are all THW*C. At this time, the three results will be merged on the channel to fuse information of different scales.

[0073] After fusing the scale information, the number of channels needs to be adjusted. At this time, the number of channels is expanded, and the fully processed feature map is added to the far feature map by matrix addition. The obtained result is the output of the entire multi-scale feature extraction module (Nonlocal-pro).

[0074] So far, the principle of the multi-scale extraction module (Nonlocal-pro) in the backbone network of the multi-scale extraction process in the present invention has been completed.

[0075] After the picture passes through the backbone network of the main model, four effective feature layers will be generated, and these four effective feature layers will be passed into the neck for further enhanced feature extraction.

[0076] The neck module adopts the connection method of a bidirectional weighted feature pyramid to re-fuse the features of the four effective feature layers of the backbone network to generate four new effective feature layers, and respectively send them into the detection head through the CA attention module; the connection method of the bidirectional weighted feature pyramid has two paths, one of which is the bottom-up path, which transmits the position information of the low-level features; the other path is the top-down path, which transmits the semantic information of the high-level features; in addition to the two paths, this structure also adds an extra edge between the input and output nodes of the same layer, and deletes those nodes with only one input edge; the specific input-output relationship is shown in the following formula:

[0077]

[0078]

[0079] The above two formulas take feature layer 3 as an example. represents the output of this layer, represents the output of the intermediate layer, represents the input of this layer, represents the output of feature layer 2, represents the input of feature layer 4, Resize represents upsampling or downsampling, ω represents the learned parameter used to distinguish the importance of different features in the feature fusion process, and Conv represents feature convolution.

[0080] In the neck part, the connection method of the bidirectional weighted feature pyramid as shown in Figure 5 is adopted. The purpose is to enable this method to complete the recognition and detection task of koi in various complex or occluded environments, and a channel attention CA is added at the detection head, as shown in Figure 6 . CA can encode the channel relationship and long-term dependence through accurate position information. Specifically, it is divided into two steps as shown in Figure 6 - coordinate information embedding and coordinate attention generation: respectively through two global pooling operations, it is transformed into a one-dimensional feature encoding operation. Specifically, given the input X, pooling kernels with sizes of (H,1) or (1,W) are used to encode each channel along the horizontal and vertical coordinates respectively, and the obtained results are stacked on the channel, that is Figure 6Perform the concat operation as shown, then use a 1×1 convolution to transform it, and next use normalization and non-linear activation functions. The resulting outputs are respectively transformed into tensors with the same number of channels using two 1×1 convolution kernels, and finally, a sigmoid activation function is used for activation.

[0081] The CA attention module of this method incorporates position information into channel attention. This attention mechanism decomposes channel attention into two parallel one-dimensional feature encoding processes, aggregating features in two directions respectively; one direction obtains long-range dependencies, and the other direction obtains accurate position information; the generated feature maps are encoded to form a pair of direction-aware and position-sensitive features; the CA attention module mainly includes two steps: coordinate information embedding and coordinate attention generation; for coordinate information embedding, global pooling is divided into a pair of one-dimensional feature encoding operations; for the input feature map X with dimensions C×H×W, first use pooling kernels of sizes (H,1) and (1,W) to encode each channel along the horizontal and vertical coordinates respectively, that is, the outputs of the c-th channel with height h and the c-th channel with width w, and the output formulas are shown as the following two equations:

[0082]

[0083]

[0084] The above formulas perform feature aggregation along two directions, returning a pair of direction-aware attention features Z h and Z w , which helps the network to more accurately locate the target to be detected; for coordinate attention generation, the previous module first generates two feature layers, and then uses a shared 1×1 convolution for transformation F1, and the formula is as follows:

[0085] f = δ(F1([Z h ,Z w ))

[0086] where f ∈ R C / r×(H+W) is the intermediate feature map of the spatial information in the horizontal and vertical directions, r represents the downsampling ratio, which is set to 16 to control the size of the module, achieving the best balance between overall performance and computational cost. δ represents the non-linear activation function, [·,·] represents the concatenation operation along the spatial dimension. Then, f is sliced into two separate tensors f w ∈ R C / r×W and f h ∈ R C / r×W Then, two 1×1 convolutions F h and F w are used to transform the feature map fh and f w Transform to the same number of channels as the input X, and the result is as follows:

[0087] g h = σ(F h (f h ))

[0088] g w = σ(F w (f w ))

[0089] Finally, expand g h and g w as attention weights, and the final output of the CA attention module can be expressed as follows:

[0090]

[0091] The entire CA attention module has been executed.

[0092] So far, the multi-scale extraction process of the present invention has been completely completed.

[0093] Next, the picture will enter the final output stage:

[0094] During the feature extraction process, four new effective feature layers will be generated. The output end gives the bounding boxes (the initial prediction of the model, there are multiple bounding boxes for one category) and confidence scores (indicating the confidence level that there is indeed an object in the bounding box and the confidence level that the bounding box includes all the features of the entire object) according to the four newly generated feature layers. Then, as Figure 2 Using the non-maximum suppression method, the duplicate bounding boxes are filtered out. The steps of non-maximum suppression are as follows: First, sort according to the confidence score, select the bounding box with the highest confidence score and add it to the final output list, delete it from the bounding box list, calculate the area of all bounding boxes, calculate the intersection over union IoU (which refers to the ratio of the intersection area of two boxes to the union area of two boxes, indicating the intersection degree of two boxes) between the bounding box with the highest confidence score and other candidate boxes. Delete the bounding boxes with IoU greater than a certain value, and repeat the above process until the bounding box list is empty. The remaining bounding boxes are the prediction boxes, and then compare them with the boxes manually marked before, and calculate the loss using the GIoU loss function as shown in Figure 2 (The loss function reflects the gap between the predicted box and the true box, and by using the loss function, the weights can be continuously adjusted to narrow the gap, thereby improving the accuracy). Then use the loss function for backpropagation to adjust the weights of this method.

[0095] Finally, it will judge whether the model has converged according to the law of weight adjustment. If it has converged, the final model will be output. If it has not converged, all the above processes will be repeated.

[0096] So far, the overall method of this invention has been completed.

[0097] This invention can directly give the category probability and position of koi, and has a faster recognition speed. It can process a large number of koi pictures, promoting the development of related industries and contributing to the screening of koi. It greatly reduces the huge amount of manpower and material resources consumed by traditional manual screening. When the dataset is large enough and after training, this method can be improved to a very high accuracy, and then it can be deployed on relevant hardware with cameras for real-time detection.

Claims

1. A method for detecting and identifying koi fish using multi-scale feature extraction, characterized in that: The detection and recognition method is carried out according to the following steps: (1) Data collection process: Obtain several pictures of koi fish and label them, and divide them into a training set and a validation set in a certain proportion; (2) Picture processing and feature extraction process: Transmit the labeled pictures to the preprocessing of mosaic data augmentation and adaptive scaling, and then input them into the backbone network of the main model. Feature extraction of the pictures is carried out in the backbone network. After that, a multi-scale feature extraction module is used to perform multi-scale feature extraction of koi fish, so that the method can detect koi fish of different scales; The main model processing process includes: The feature map processed by the backbone network is processed by a multi-scale feature extraction module and a codec module. The multi-scale feature extraction module performs multi-scale feature extraction of koi fish, so that the method can detect koi fish of different scales; the codec module improves the performance of small target detection; the feature map processed by the backbone network is input into the neck module. In the neck module, the feature map is enhanced by the connection method of the bidirectional weighted feature pyramid. At the same time, a CA attention module is added at the detection head to embed the position information of the koi fish in the picture into the channel. So far, the main model detection is completed; (3) Enter the training process: The output end gives the bounding box and confidence through the 4 effective feature layers finally output from the CA attention module: The non-maximum suppression method is used to filter out the duplicate bounding boxes to obtain the predicted boxes. The predicted boxes are compared with the boxes generated by the annotation tool, and the GIoU loss function is used to calculate the loss. Then the loss function is used for backpropagation to adjust the weights. The GIoU loss formula is as follows: ； A: The annotated box rectangle; B: The predicted box rectangle; C: The smallest enclosing rectangle of the image formed by the two boxes, that is, the area of the smallest box that contains both the predicted box and the true box; Repeat the above process to make it gradually converge, and continuously adjust the parameters through the test of the validation set to make it have generalization ability and accuracy improvement, and finally obtain the recognition of koi fish; The CA attention module of this method adds position information to the channel attention. This attention mechanism decomposes the channel attention into two parallel one-dimensional feature encoding processes, aggregating features in two directions respectively; one direction obtains the long-range dependence relationship, and the other direction obtains the accurate position information; the generated feature map is encoded to form a pair of direction-aware and position-sensitive features; the CA attention module mainly includes two steps: coordinate information embedding and coordinate attention generation; for coordinate information embedding, global pooling is divided into a pair of one-dimensional feature encoding operations; for the input feature map X with dimensions C×H×W, first use pooling kernels of size (H,1) and (1,W) to encode each channel along the horizontal direction coordinates and vertical direction coordinates respectively, that is, the output of the c-th channel with height h and the c-th channel with width w. The output formulas are shown as the following two formulas: ； The above formula aggregates features in two directions and returns a pair of direction-aware attention features and , which helps the network more accurately locate the target to be detected; for coordinate attention generation, the previous module first generates two feature layers and then uses a shared 1×1 convolution for transformation , and the formula is as follows: ； Among them, is the intermediate feature map of the spatial information in the horizontal and vertical directions, represents the downsampling ratio, which is set to 16 to control the size of the module, and the overall performance and computational complexity are the most balanced; represents the non-linear activation function, represents the connection operation along the spatial dimension. Then, f is split into two separate tensors along the spatial dimension and Then, two 1×1 convolutions are used and to transform the feature maps and to the same number of channels as the input X, and the result is as follows: ； Finally, expand and as the attention weights, and the final output of the CA attention module is expressed as follows: ； So far, the entire CA attention module has been executed.

2. The koi detection and recognition method using multi-scale feature extraction according to claim 1, wherein: In the small-scale feature extraction process in the multi-scale feature extraction module, the original feature map is input into the small-scale feature extraction process. First, the number of channels of the compressed image and the feature extraction of the small-scale convolution kernel are performed separately. Then, the three results are combined into the dimensions other than the number of channels, which refers to the width and height dimensions. At this time, the dimensions of the image become THW*C; T represents the number of batch input images, H represents height, W represents width, and C represents the number of channels. The dimensions of the three results just now all become THW*C; Then, the matrix transpose result after compressing the number of channels is multiplied by the result of the first small-scale convolution kernel feature extraction to obtain the relationship between each pixel in each frame and all pixels in all other frames. The dimension of the result of the dot product becomes THW*THW, that is, the dot product of THW*C and C*THW. Then, a softmax operation is performed on the autocorrelation feature to obtain a result with a value range of [0,1]. Then perform a matrix dot product operation on the result and the result of the second small-scale convolution kernel feature extraction. The dimension at this time is the dot product operation of THW*THW and THW*C, and the result dimension is THW*C. At this point, the small-scale feature extraction process in the multi-scale feature extraction module has been completed, and the medium-scale and large-scale processing processes are exactly the same as the small-scale feature extraction process. The difference is that the convolution kernel size used in the medium-scale and large-scale extraction processes is different from the small-scale feature extraction convolution kernel, so this module can extract features of different scales. After extracting the three-scale features, three feature maps of different scales will be generated. At this time, these three feature maps are stacked in the channel dimension to obtain a new feature map, and then the feature map is adjusted to the same number of channels as the original input feature map. Finally, this feature map and the original feature map are output by matrix addition. At this point, the entire multi-scale feature extraction module has been explained.

3. The koi detection and recognition method using multi-scale feature extraction according to claim 1, characterized in that: The input image of this method first undergoes mosaic data augmentation, image adaptive scaling, rotation, and splicing to expand the data set to facilitate the subsequent model training process; then the image after mosaic data augmentation is transmitted to the above-mentioned multi-scale feature extraction module through a series of feature extraction processes, where the main model extracts multi-scale features.

4. A koi fish detection and recognition method using multi-scale feature extraction according to claim 1, characterized in that: The codec module of the method is a residual structure, which is divided into a main part and a residual side part. The residual side part is directly connected end to end without any processing or with a small amount of processing.

5. A method for detecting and identifying koi fish using multi-scale feature extraction according to claim 1, characterized in that: The neck module of this method adopts the connection method of a bidirectional weighted feature pyramid, re-fuses the features of the four effective feature layers of the backbone network to generate four new effective feature layers, and sends them to the detection head through the CA attention module respectively; the connection method of the bidirectional weighted feature pyramid has two paths, one of which is the bottom-up path that transmits the position information of low-level features; the other path is the top-down path that transmits the semantic information of high-level features; in addition to the two paths, this structure also adds an extra edge between the input and output nodes of the same layer and deletes those nodes with only one input edge; the specific input-output relationship is shown in the following formula: ； Taking the feature layer 3 as an example for the above two formulas, represents the output of this layer, represents the output of the intermediate layer, represents the input of this layer, represents the output of the feature layer 2, represents the input of the feature layer 4, Resize represents upsampling or downsampling, represents the learned parameter, which is used to distinguish the importance of different features in the feature fusion process, Conv represents feature convolution.

Images