Intelligent zooplankter identification method and system based on machine learning

Abstract

The invention discloses a machine learning-based zooplankton intelligent identification method and system. The method comprises the steps of constructing a target detection model for aquatic organisms based on deep learning; wherein the target detection model comprises an image fusion layer, a DenseNet network and a region candidate device RPN; performing multi-focus image fusion processing on the multi-dimensional image sample of the zooplankton by using the image fusion layer to obtain a fusion feature map; performing feature extraction on the fused feature map through the DenseNet network to obtain feature maps of different levels, and positioning target information of the feature maps; generating a plurality of candidate frames based on the target information by using a region candidate RPN, and performing screening through a non-maximum suppression method based on Giou to obtain a target candidate frame with the highest score; and the target detection model classifies and locates the target candidate box to realize zooplankter identification.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and system for intelligent zooplankton recognition based on machine learning. Background Art

[0002] As an important component of aquatic ecosystems, the identification and study of zooplankton can reveal information such as the species composition, population structure, and ecological role of the ecosystem, providing important data for ecological research.

[0003] However, the current research methods based on morphology and DNA sequences for artificial recognition research require a high level of professional knowledge from the institute, and the experimental preparation time in the early stage of the research is long, and the research efficiency is low. In addition, considering that in current practical applications, when the image is sampled centrally through machine automatic focusing acquisition, the same field of view will produce multi-dimensional sample images due to the change in focal length, and different layers will have a certain impact on the subsequent test results;

[0004] Therefore, there is an urgent need for an intelligent identification method to accurately identify aquatic zooplankton in order to improve the efficiency of experimental research. Summary of the invention

[0005] The purpose of this application is to provide a method and system for intelligent zooplankton identification based on machine learning to achieve accurate identification of aquatic zooplankton.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for intelligent identification of zooplankton based on machine learning, comprising:

[0008] Constructing a target detection model for aquatic organisms based on deep learning; wherein the target detection model includes an image fusion layer, a DenseNet network, and a region candidate RPN;

[0009] Using the image fusion layer to perform multi-focus image fusion processing on the multi-dimensional image samples of zooplankton to obtain a fusion feature map;

[0010] Extract features from the fused feature map through the DenseNet network, obtain feature maps at different levels, and locate target information of the feature map;

[0011] Based on the target information, the region candidate RPN is used to generate multiple candidate boxes, and the target candidate box with the highest score is obtained by screening through the non-maximum suppression method based on Giou;

[0012] The target detection model classifies and locates the target candidate frame to achieve zooplankton recognition.

[0013] Furthermore, the image fusion layer is used to perform multi-focus image fusion processing on the multi-dimensional image samples in the zooplankton sample library to obtain a fusion feature map, including:

[0014] Inputting the multi-dimensional image samples in the zooplankton sample library into the target detection model;

[0015] The image fusion layer uses a two-layer convolutional neural network to perform secondary convolution on the multidimensional image samples to extract shallow feature information of the multidimensional image samples and obtain multiple feature dimension maps;

[0016] Among them, both layers of convolutional neural networks use the initial 7*7*64 initial convolution kernel weights in ResNet-101; and before fusion, the multi-dimensional graphic samples that are not suitable for fusion need to be adjusted;

[0017] The image fusion layer performs dimension superposition on the feature dimension map along the channel information, reduces the dimension of the feature dimension map, and fuses multiple feature dimension maps using the weighted average method in the element fusion strategy to obtain a fused feature map;

[0018] After the multi-dimensional image samples are fused, the fused feature maps are reshaped and resized using 5*5*64 and 3*3*64 convolution kernels and maximum pooling convolution.

[0019] Furthermore, the weighted average method in the element fusion strategy is used to fuse multiple feature dimension graphs, including:

[0020] Based on the weighted average method, an initial weight is assigned to the feature matrix of each feature dimension graph according to the importance of each feature dimension graph to the fusion result; wherein the initial weight conforms to a normal distribution with a predetermined mean value;

[0021] When performing feature matrix fusion on the feature dimension graph, each feature matrix is ​​multiplied by the corresponding initial weight and added together to achieve processing of different focus layers of multi-dimensional image samples;

[0022] Wherein, the initial weight is:

[0023]

[0024] In the formula, represents the weight of the initial weight after multiple iterations, w i represents the weight of the i-th focusing layer, α represents the learning rate, and e p Represents the loss value of the target detection model;

[0025] The weight fusion formula of the feature matrix is:

[0026]

[0027] In the formula, represents the fusion matrix, where x and y represent the feature information involved in the fusion; N represents the number of multi-focus layers, that is, the number of feature dimension graphs obtained corresponding to the input multi-dimensional image samples; represents the convolution output of the i-th focusing layer.

[0028] Furthermore, feature extraction is performed on the fused feature map through the DenseNet network to obtain feature maps of different levels, and target information of the feature map is located, including:

[0029] The DenseNet network is used to extract features from the fused feature map to obtain feature maps of different levels; wherein the DenseNet network includes DenseBlock dense blocks, DenseLayer dense layers and Transition transition blocks;

[0030] Introduce asymmetric convolution in the DenseLayer dense layer, replace the original 3*3 convolution kernel with a parallel 3*3+1*3+3*1 parallel convolution kernel, and superimpose the convolution information of the two asymmetric convolution kernels to the center of the original 3*3 convolution kernel to enhance the feature information of the cross skeleton of the 3*3 convolution kernel;

[0031] Introduce channel attention mechanism and spatial attention mechanism in DenseLayer dense layer to build channel attention module and spatial attention module;

[0032] The spatial information of the feature map is processed by using the maximum pooling and mean pooling based on the channel attention module, and the target information of the feature map is located in the process of reducing and increasing the dimension of the feature map by the multi-layer perceptron module;

[0033] Among them, the spatial attention module is used to supplement the channel attention: the maximum pooling and mean pooling in the spatial attention module are used to splice the feature map, and the channel dimension of the feature map is reduced by a 7*7 convolution kernel.

[0034] Furthermore, the convolution information superposition formula is:

[0035] F*K (3*3) +F*K (1*3) +F*K (3*1) =F*(K (3*3) ⊕K (1*3) ⊕K (3*1) ),

[0036] In the formula, F represents the feature matrix of the feature map, K represents the convolution kernel, and ⊕ represents element-by-element accumulation;

[0037] The output of the channel attention module is calculated as follows:

[0038] A c (F) = ε(MLP(F max )+MLP(F avg )),

[0039] In the formula, A c (F) represents the output of the feature matrix F of the feature map in the channel attention module, ε represents the sigmoid activation function, MLP represents the multi-layer perceptron module, and F max Represents the feature matrix of the maximum pooling, F avg Represents the feature matrix of average pooling;

[0040] The output of the spatial attention module is calculated as follows:

[0041] A s (F) = ε(conv(add(F max ,F avg ))),

[0042] In the formula, A s (F) represents the output of the feature matrix F in the spatial attention module, and add represents the addition of the channel dimension.

[0043] Furthermore, after the DenseNet network locates the target information of the feature map, the feature map is fused and further located using the feature pyramid FPN, including:

[0044] The target detection model also includes a feature pyramid FPN;

[0045] The feature pyramid FPN fuses feature maps of different levels to construct a multi-scale feature pyramid;

[0046] The feature pyramid FPN acquires layer feature information at different levels of the multi-scale feature pyramid, thereby realizing target information positioning at different scales of the multi-scale feature pyramid;

[0047] The feature pyramid FPN transmits the multi-scale feature pyramid to the region candidate RPN.

[0048] Furthermore, based on the target information, the region candidate RPN is used to generate multiple candidate boxes, and the target candidate boxes with the highest scores are screened by the Giou-based non-maximum suppression method, including:

[0049] Based on target information at different scales, a plurality of candidate boxes are generated on the multi-scale feature pyramid using a region candidate RPN and an anchor box;

[0050] In the RoIPooling stage, the Giou-based non-maximum suppression method is used to screen the candidate frames by calculating the overlap between the candidate frames to obtain the target candidate frame with the highest score, while suppressing other candidate frames with high overlap with the target candidate frame;

[0051] Among them, simultaneously suppressing other candidate frames with a high degree of overlap with the target candidate frame includes:

[0052] Set an overlap threshold, and keep the original processing unchanged for other candidate frames whose values ​​are smaller than the overlap threshold; for other candidate frames whose values ​​are larger than the overlap threshold, multiply the scores of other candidate frames by half of the overlap between other candidate frames and the target candidate frame, so as to update the scores of other candidate frames;

[0053] If the updated scores of other candidate boxes are less than the specified threshold, the scores of other candidate boxes are directly assigned to 0;

[0054] The candidate box score formula is:

[0055]

[0056] In the formula, S i represents the candidate box score, Giou(M,b i ) represents the target candidate box M and the i-th other candidate box b i The overlap degree, N t represents the overlap threshold, and S represents the specified threshold.

[0057] In the second aspect, based on the same inventive concept, the embodiment of the present application also provides a zooplankton intelligent identification system based on machine learning, including:

[0058] A model building unit, used for building a target detection model for aquatic organisms based on deep learning; wherein the target detection model includes an image fusion layer, a DenseNet network and a region candidate RPN;

[0059] A fusion processing unit, used for performing multi-focus image fusion processing on the multi-dimensional image samples of zooplankton using the image fusion layer to obtain a fusion feature map;

[0060] A feature extraction unit, used to extract features from the fused feature map through the DenseNet network to obtain feature maps of different levels, and locate target information of the feature map;

[0061] A screening unit is used to generate multiple candidate boxes using the region candidate RPN based on the target information, and screen the target candidate box with the highest score using the Giou-based non-maximum suppression method;

[0062] The recognition unit is used for the target detection model to classify and locate the target candidate frame to realize the recognition of zooplankton.

[0063] On the third aspect, based on the same inventive concept, an embodiment of the present application also provides an electronic device, including a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program or instructions to implement the steps of the aforementioned method for intelligent identification of zooplankton based on machine learning.

[0064] In a fourth aspect, based on the same inventive concept, an embodiment of the present application also provides a computer storage medium, in which a computer program or instructions are stored, and when the computer program or instructions are executed by a processor, the steps of the aforementioned method for intelligent identification of zooplankton based on machine learning are implemented.

[0065] The technical effects and advantages of the present application are as follows: the present application adopts a feature-based image fusion layer to perform fusion preprocessing on the input multi-dimensional image, so as to realize the aggregation of feature information focused on different focus layers; and uses a convolutional neural network based on DENSENET to obtain feature maps of different levels, so as to provide richer and more accurate feature maps for FPN module fusion; the present application enables the RPN network structure and the Giou-based non-maximum suppression method to perform target prediction and recognition on feature maps of different levels, so as to solve the problem of missed detection caused by dense sample targets in the sample data set; the present application has certain improvements in evaluation indicators such as model coverage, Map index, and detection speed compared with existing target detection network models.

[0066] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 This is a flow chart of the method for intelligent zooplankton identification based on machine learning according to an embodiment of the present application;

[0069] Figure 2 This is a schematic diagram of the structure of the target detection model in the embodiment of the present application;

[0070] Figure 3 This is a schematic diagram of the multi-focus image fusion processing process in an embodiment of the present application;

[0071] Figure 4 This is a schematic diagram of the DenseNet network structure in the embodiment of this application;

[0072] Figure 5 This is a schematic diagram of asymmetric convolution in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a calculation method based on Giou in an embodiment of the present application;

[0074] Figure 7 This is a schematic diagram of the structure of the zooplankton intelligent identification system based on machine learning in the embodiment of the present application;

[0075] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] To address the deficiencies of the prior art, this application discloses an intelligent zooplankton identification method based on machine learning, such as Figure 1 As shown, the following steps are included:

[0078] Step S1: Construct a target detection model for aquatic organisms based on deep learning, where Figure 2 As shown, the target detection model includes an image fusion layer, a DenseNet network and a region candidate RPN;

[0079] Step S2: using the image fusion layer to perform multi-focus image fusion processing on the multi-dimensional image samples of zooplankton to obtain a fusion feature map;

[0080] Step S3: extracting features from the fused feature map through the DenseNet network, obtaining feature maps at different levels, and locating target information of the feature map;

[0081] Step S4: Generate multiple candidate boxes using the region candidate network (RPN) based on the target information, and select the target candidate box with the highest score by using the Giou-based non-maximum suppression method;

[0082] Step S5: the target detection model classifies and locates the target candidate frame to achieve zooplankton recognition.

[0083] In some specific embodiments, before performing step S1, it is necessary to construct a relatively complete and annotated zooplankton sample library based on the collected zooplankton image samples;

[0084] The present application collects zooplankton image samples by artificial microscopic collection method and machine automatic focusing collection method, and the machine automatic focusing collection method is the main method;

[0085] Among them, the manual microscopic collection method collects the clearest picture of an area each time;

[0086] The machine automatic focusing acquisition method is to acquire a multi-dimensional image of each area from blurry to clear and then to blurry; therefore, the zooplankton image sample includes a multi-dimensional image sample.

[0087] In actual applications, when faced with multi-dimensional images produced by focal length changes, if a single-dimensional image is used, the stability of image clarity cannot be guaranteed in the experiment, and there are large differences in the experimental results. Therefore, it is necessary to process the multi-dimensional image samples.

[0088] Generally, there are two ways to process multidimensional images: pixel-level image fusion (PIF) and feature-level image fusion;

[0089] Pixel-level image fusion decomposes multidimensional images into different feature regions according to different decomposition rules, but this method is heavily dependent on the accuracy of image decomposition;

[0090] Feature-level image fusion focuses on the feature weights of different images, which to a certain extent reduces the impact of individual prominent images on multi-dimensional image recognition effects.

[0091] Therefore, in some specific embodiments, step S2: using the image fusion layer to perform multi-focus image fusion processing on the multi-dimensional image samples in the zooplankton to obtain a fusion feature map; specifically includes:

[0092] Step S21: input the multidimensional image samples in the zooplankton sample library into the target detection model, and the image fusion layer uses a two-layer convolutional neural network to perform secondary convolution on the multidimensional image samples to extract shallow feature information of the multidimensional image samples and obtain multiple feature dimension graphs;

[0093] Among them, both layers of convolutional neural networks use the initial 7*7*64 initial convolution kernel weights in ResNet-101; and multi-dimensional graphic samples that are not suitable for fusion need to be adjusted before fusion.

[0094] Step S22: The image fusion layer performs dimension superposition on the feature dimension map along the channel information, and then uses a 1*1 convolution kernel to reduce the dimension of the feature dimension map to a certain extent, and uses the weighted average method in the element fusion strategy to fuse multiple feature dimension maps to obtain a fused feature map, so as to realize the fusion of feature information of different layers into the same image.

[0095] In some specific embodiments, a weighted average method in the element fusion strategy is used to fuse multiple feature dimension graphs to obtain a fused feature graph, such as Figure 3 As shown, specifically including:

[0096] Using the weighted average method, an initial weight is assigned to the feature matrix of each feature dimension map according to the importance of each feature dimension map to the fusion result; wherein the initial weight conforms to a normal distribution with a mean value of a predetermined value (i.e., 0.3);

[0097] When performing feature matrix fusion on the feature dimension graph, each feature matrix is ​​multiplied by the corresponding initial weight and added together to achieve processing of different focus layers of multi-dimensional image samples;

[0098] Wherein, the initial weight is:

[0099]

[0100] In the formula, represents the weight of the initial weight after multiple iterations, w i represents the weight of the i-th focusing layer, α represents the learning rate, and e p Represents the loss value of the target detection model;

[0101] The weight fusion formula of the feature matrix is:

[0102]

[0103] In the formula, represents the fusion matrix, where x and y refer to the information involved in the fusion, and N represents the number of multi-focus layers, that is, the number of feature dimension maps obtained corresponding to the input multi-dimensional image samples; represents the convolution output of the i-th focusing layer.

[0104] Step S23: After the multi-dimensional image samples are fused in the image fusion layer, the fused feature map is reshaped using 5*5*64 and 3*3*64 convolution kernels and maximum pooling convolution, and the size is adjusted to 320*320; and in order to connect to the subsequent DenseNet network, it is necessary to ensure that the number of channels is maintained at 64 layers.

[0105] Step S3: extracting features from the fused feature map through the DenseNet network, obtaining feature maps at different levels, and locating the target information of the feature map; specifically including:

[0106] Step S31: extract features from the fused feature map using a convolutional neural network based on AC / AM_DenseNet to obtain feature maps at different levels (i.e., feature maps);

[0107] Among them, Figure 4 As shown, the DenseNet network includes DenseBlock dense blocks, DenseLayer dense layers and Transition transition blocks;

[0108] Due to the resizing of the fused feature map in step S2, the original DenseBlock of AC / AM_DenseNet is optimized in the target detection model of this application, and the Transition block, as a transition between dense blocks, has the function of reducing the feature map dimension channel and the feature map dimension size, and is composed of BN+Relu+1*1Conv+2*2AvgPool;

[0109] DenseLayer, as the basic network structure of the DenseNet network, performs a 32-channel feature extraction and can also be used as a link between post-order prediction and regression feature maps.

[0110] Step S32: In order to improve the accuracy of the overall prediction of the model and consider zooplankton targets at different angles, Figure 5 As shown, an asymmetric convolution (AC) is introduced in the DenseLayer dense layer:

[0111] The original 3*3 convolution kernel is replaced with a parallel 3*3+1*3+3*1 parallel convolution kernel, and the convolution information of the two asymmetric convolution kernels is superimposed on the center of the original 3*3 convolution kernel to enhance the feature information of the cross skeleton of the 3*3 convolution kernel, and at the same time, it has better robustness to the target information at different angles in the feature map;

[0112] The convolution information superposition formula is:

[0113] F*K (3*3) +F*K (1*3) +F*K (3*1) =F*(K (3*3) ⊕K (1*3) ⊕K (3*1) ),

[0114] In the formula, F represents the feature matrix of the feature map, K represents the convolution kernel, and ⊕ represents element-by-element accumulation.

[0115] Step S33: Due to the dense connection of the DenseNet network, it often takes a long time to train and predict. Therefore, the channel attention mechanism and the spatial attention mechanism are introduced in the DenseLayer dense layer to construct the channel attention module and the spatial attention module to form AC / AM_DenseLayer.

[0116] The spatial information of the feature map is processed by using the maximum pooling and mean pooling based on the channel attention module, and the target information in the feature map is located in the process of reducing and increasing the dimension of the feature map by the multi-layer perceptron module;

[0117] The output of the channel attention module is calculated as follows:

[0118] A c (F) = ε(MLP(F max )+MLP(F avg )),

[0119] In the formula, A c (F) represents the output of the feature matrix F in the channel attention module, ε represents the sigmoid activation function, MLP represents the multi-layer perceptron module, and F max Represents the feature matrix of the maximum pooling, F avg Represents the feature matrix of average pooling.

[0120] The spatial attention module is a supplement to the channel attention, reflecting the importance of the feature information of the feature map in the spatial dimension. Therefore, the maximum pooling and mean pooling of the spatial attention module are used in turn to splice the feature map, and the channel dimension of the feature map is reduced to 1 through a 7*7 convolution kernel.

[0121] The output of the spatial attention module is calculated as follows:

[0122] A s (F) = ε(conv(add(F max ,F avg ))),

[0123] In the formula, A s (F) represents the output of the feature matrix F in the spatial attention module, and add represents the addition of the channel dimension.

[0124] This application transforms the original DenseNet network structure into the AC / AM_DenseNet network structure by introducing the channel attention mechanism and the spatial attention mechanism.

[0125] In some specific embodiments, after locating the target information of the feature map by the DenseNet network in step S3, the feature map is fused and further located by using the feature pyramid FPN, including:

[0126] The target detection model also includes a feature pyramid FPN; the feature pyramid FPN fuses feature maps of different levels to construct a multi-scale feature pyramid, so that the feature pyramid FPN can simultaneously obtain layer feature information of different levels (i.e., low-level and high-level layers), thereby realizing the positioning of target information at different scales of the multi-scale feature pyramid, that is, being able to accurately detect targets at different scales;

[0127] The feature pyramid FPN transmits the multi-scale feature pyramid to the region candidate RPN. For the region candidate RPN, the feature pyramid FPN provides a rich feature representation, so that the RPN can better locate and generate candidate targets; specifically, the feature pyramid FPN fuses the high-resolution feature map from the bottom layer with the semantic information from the top layer to obtain more expressive and semantically rich features. These multi-scale feature pyramids are sent to the region candidate RPN to generate candidate target boxes.

[0128] This application uses the multi-scale information provided by the feature pyramid FPN to enable the region candidate RPN to effectively propose objects at different scales, including small and large objects;

[0129] In addition, the feature pyramid FPN also enables the communication and fusion of feature information at different levels through top-down and bottom-up feedback connections, further enhancing the semantic expression ability of the features. Because objects of different scales require feature information at different levels for accurate positioning and classification, this multi-level feature fusion is particularly important for the target detection task of the region candidate RPN, enabling the region candidate RPN to accurately generate candidate target boxes at different scales, thereby providing stronger performance and accuracy for the target detection task.

[0130] Step S4: Generate multiple candidate boxes using the region candidate network (RPN) based on the target information, and select the target candidate box with the highest score by using the Giou-based non-maximum suppression method; specifically, the following steps are included:

[0131] Step S41: Based on target information at different scales, a plurality of candidate boxes are generated on the multi-scale feature pyramid using a region candidate network (RPN) and an anchor box;

[0132] Among them, the essence of the region candidate RPN is a classless obejct detector based on sliding windows. Each point on the feature map (i.e., feature information matrix) can correspond to a specific point on the input multi-dimensional image sample.

[0133] For ease of processing, anchor boxes of a certain size ratio are pre-set, and each point on the feature map (i.e., feature information matrix) will generate an anchor box; therefore, an m×n feature map (i.e., feature information matrix) will generate m×n×k anchor boxes, where k is the pre-set number of boxes, i.e., the number of initial anchor box shapes;

[0134] RPN determines which anchor boxes may contain target objects based on the target information of different scales in the multi-scale feature pyramid, and regresses the coordinates of these boxes to generate proposals so that the target can be further classified and located in subsequent steps.

[0135] Step S42: Figure 6 As shown, the target detection model also includes ROIPooling. In the RoI Pooling stage, the Giou-based non-maximum suppression method is used to screen the candidate frames by calculating the overlap between the candidate frames to obtain the target candidate frame with the highest score, while suppressing other candidate frames with a high overlap with the target candidate frame;

[0136] Among them, simultaneously suppressing other candidate frames with a high degree of overlap with the target candidate frame includes:

[0137] This application can reduce the problem of false detection or repeated detection by suppressing redundant candidate boxes, and at the same time reduce the number of target candidate boxes that need to be regressed, making the output results more concise;

[0138] However, in the prior art, when the overlap between other candidate frames and the target candidate frame with the highest score is greater than a set threshold, other candidate frames will be directly deleted. This may easily lead to the possibility of mistakenly deleting other target candidate frames (not the target candidate frame with the highest score) when processing complex scenes with crowded and truncated targets to be detected.

[0139] Therefore, the present application sets an overlap threshold, and keeps the original processing unchanged for other candidate frames that are smaller than the overlap threshold; for other candidate frames that are larger than the overlap threshold, the corresponding scores are not directly reset to zero, but the scores of other candidate frames are multiplied by half of the overlap between the other candidate frames and the target candidate frame, so that the scores of other candidate frames are updated to smaller scores;

[0140] If the updated scores of other candidate boxes are less than the specified threshold, the scores of other candidate boxes are directly assigned to 0;

[0141] The candidate box score formula is:

[0142]

[0143] In the formula, S i represents the candidate box score, Giou(M,b i ) represents the target candidate box M and the i-th other candidate box b i The overlap degree, N t represents the overlap threshold, and S represents the specified threshold.

[0144] pass Figure 6 It can be seen that the calculation formula of Giou is:

[0145] Giou=C / (A+BC)-(UA-B+C) / U,

[0146] In the formula, A represents the area of ​​the light green region, that is, candidate box A; B represents the area of ​​the dark green region, that is, candidate box B; C represents the intersection area of ​​A and B, that is, the overlapping part of candidate box A and candidate box B; U represents the area enclosed by the red dotted frame, that is, the area enclosed by candidate box A and candidate box B (including the area of ​​A and B).

[0147] Step S5: the target detection model classifies and locates the target candidate frame to realize the recognition of plankton, and the target detection model can be iteratively updated according to the classification and positioning losses.

[0148] The Flask library can also be used to provide the sample prediction, sample training, threshold adjustment and other functions of the target detection model to web applications with separated front-end and back-end in the form of interfaces for researchers to use.

[0149] Based on the same inventive concept, the present application also provides an intelligent zooplankton identification system based on machine learning, such as Figure 7 As shown, including:

[0150] A model building unit, used for building a target detection model for aquatic organisms based on deep learning; wherein the target detection model includes an image fusion layer, a DenseNet network and a region candidate RPN;

[0151] A fusion processing unit, used for performing multi-focus image fusion processing on the multi-dimensional image samples of zooplankton using the image fusion layer to obtain a fusion feature map;

[0152] A feature extraction unit, used to extract features from the fused feature map through the DenseNet network to obtain feature maps of different levels, and locate target information of the feature map;

[0153] A screening unit is used to generate multiple candidate boxes using the region candidate RPN based on the target information, and screen the target candidate box with the highest score using the Giou-based non-maximum suppression method;

[0154] The recognition unit is used for the target detection model to classify and locate the target candidate frame to realize the recognition of zooplankton.

[0155] In some specific embodiments, the system further includes a collection unit for constructing a relatively complete and annotated zooplankton sample library based on the collected zooplankton image samples.

[0156] In some specific embodiments, the system further includes a feature fusion unit for fusing and further locating the feature map using a feature pyramid FPN.

[0157] Regarding the system in the above embodiment, the specific manner in which each unit module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0158] Based on the same inventive concept, the present application also provides an electronic device, whose structure is as follows: Figure 8 As shown, it includes a memory, a processor and a computer program stored in the memory, and the processor executes the computer program or instructions to implement the aforementioned intelligent identification method of zooplankton based on machine learning.

[0159] Based on the same inventive concept, an embodiment of the present application also provides a computer storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the aforementioned method for intelligent identification of zooplankton based on machine learning are implemented.

[0160] The method provided in this application is used to verify the feasibility of detecting and identifying plankton, and a comparison table of experimental results is established to evaluate and compare the target detection model of this application; the following Table 1 shows the influence of different optimization strategies provided in this application on the average precision (Map) of the prior art model, whether they play a positive role, and whether the detection effect of the final target detection model can reach the highest; as shown in Table 1:

[0161] Table 1 Comparison of experimental results

[0162]

[0163] It can be seen from the third column of Table 1 that through all the optimization strategies provided in this application, the average accuracy of the target detection model can reach a maximum value of 75.8, while without using the optimization strategies provided in this application, the average accuracy of the target detection model is 72.6, which is less than the maximum value of 75.8; therefore, by using the detection method of this application, the detection accuracy of plankton can be improved.

[0164] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. The method for intelligent identification of zooplankton based on machine learning is characterized by: include: Constructing a target detection model for aquatic organisms based on deep learning; wherein the target detection model includes an image fusion layer, a DenseNet network, and a region candidate RPN; Using the image fusion layer to perform multi-focus image fusion processing on the multi-dimensional image samples of zooplankton to obtain a fusion feature map; Extract features from the fused feature map through the DenseNet network, obtain feature maps at different levels, and locate target information of the feature map; Based on the target information, the region candidate RPN is used to generate multiple candidate boxes, and the target candidate box with the highest score is obtained by screening through the non-maximum suppression method based on Giou; The target detection model classifies and locates the target candidate frame to achieve zooplankton recognition.

2. The method for intelligent zooplankton identification based on machine learning according to claim 1, characterized in that: The image fusion layer is used to perform multi-focus image fusion processing on the multi-dimensional image samples in the zooplankton sample library to obtain a fusion feature map, including: Inputting the multi-dimensional image samples in the zooplankton sample library into the target detection model; The image fusion layer uses a two-layer convolutional neural network to perform secondary convolution on the multidimensional image samples to extract shallow feature information of the multidimensional image samples and obtain multiple feature dimension maps; Among them, both layers of convolutional neural networks use the initial 7*7*64 initial convolution kernel weights in ResNet-101; and before fusion, the multi-dimensional graphic samples that are not suitable for fusion need to be adjusted; The image fusion layer performs dimension superposition on the feature dimension map along the channel information, reduces the dimension of the feature dimension map, and fuses multiple feature dimension maps using the weighted average method in the element fusion strategy to obtain a fused feature map; After the multi-dimensional image samples are fused, the fused feature maps are reshaped and resized using 5*5*64 and 3*3*64 convolution kernels and maximum pooling convolution.

3. The method for intelligent zooplankton identification based on machine learning according to claim 2, characterized in that: The weighted average method in the element fusion strategy is used to fuse multiple feature dimension maps to obtain a fused feature map, including: Based on the weighted average method, an initial weight is assigned to the feature matrix of each feature dimension graph according to the importance of each feature dimension graph to the fusion result; wherein the initial weight conforms to a normal distribution with a predetermined mean value; When performing feature matrix fusion on the feature dimension graph, each feature matrix is ​​multiplied by the corresponding initial weight and added together to achieve processing of different focus layers of multi-dimensional image samples; Wherein, the initial weight is: In the formula, represents the weight of the initial weight after multiple iterations, w i represents the weight of the i-th focusing layer, α represents the learning rate, and e p Represents the loss value of the target detection model; The weight fusion formula of the feature matrix is: In the formula, represents the fusion matrix, where x and y represent the feature information involved in the fusion; N represents the number of multi-focus layers, that is, the number of feature dimension graphs obtained corresponding to the input multi-dimensional image samples; represents the convolution output of the i-th focusing layer.

4. The method for intelligent zooplankton identification based on machine learning according to claim 1 or 3, characterized in that: The fused feature map is subjected to feature extraction through the DenseNet network to obtain feature maps of different levels, and the target information of the feature map is located, including: The DenseNet network is used to extract features from the fused feature map to obtain feature maps of different levels; wherein the DenseNet network includes DenseBlock dense blocks, DenseLayer dense layers and Transition transition blocks; Introduce asymmetric convolution in the DenseLayer dense layer, replace the original 3*3 convolution kernel with a parallel 3*3+1*3+3*1 parallel convolution kernel, and superimpose the convolution information of the two asymmetric convolution kernels to the center of the original 3*3 convolution kernel to enhance the feature information of the cross skeleton of the 3*3 convolution kernel; Introduce channel attention mechanism and spatial attention mechanism in DenseLayer dense layer to build channel attention module and spatial attention module; The spatial information of the feature map is processed by using the maximum pooling and mean pooling based on the channel attention module, and the target information of the feature map is located in the process of reducing and increasing the dimension of the feature map by the multi-layer perceptron module; Among them, the spatial attention module is used to supplement the channel attention: the maximum pooling and mean pooling in the spatial attention module are used to splice the feature map, and the channel dimension of the feature map is reduced by a 7*7 convolution kernel.

5. The method for intelligent zooplankton identification based on machine learning according to claim 4, characterized in that: The convolution information superposition formula is: F*K (3*3) +F*K (1*3) +F*K (3*1) =F*(K (3*3) ⊕K (1*3) ⊕K (3*1) ), In the formula, F represents the feature matrix of the feature map, K represents the convolution kernel, and ⊕ represents element-by-element accumulation; The output of the channel attention module is calculated as follows: A c (F)=ε(MLP(F max )+MLP(F avg )), In the formula, A c (F) represents the output of the feature matrix F of the feature map in the channel attention module, ε represents the sigmoid activation function, MLP represents the multi-layer perceptron module, and F max Represents the feature matrix of the maximum pooling, F avg Represents the feature matrix of average pooling; The output of the spatial attention module is calculated as follows: A s (F)=ε(conv(add(F max ,F avg ))), In the formula, A s (F) represents the output of the feature matrix F in the spatial attention module, and add represents the addition of the channel dimension.

6. The method for intelligent zooplankton identification based on machine learning according to claim 1 or 5, characterized in that: After the DenseNet network locates the target information of the feature map, the feature map is fused and further located using the feature pyramid FPN, including: The target detection model also includes a feature pyramid FPN; The feature pyramid FPN fuses feature maps of different levels to construct a multi-scale feature pyramid; The feature pyramid FPN acquires layer feature information at different levels of the multi-scale feature pyramid, thereby realizing target information positioning at different scales of the multi-scale feature pyramid; The feature pyramid FPN transmits the multi-scale feature pyramid to the region candidate RPN.

7. The method for intelligent zooplankton identification based on machine learning according to claim 6, characterized in that: Based on the target information, the region candidate RPN is used to generate multiple candidate boxes, and the non-maximum suppression method based on Giou is used to filter and obtain the target candidate boxes with the highest scores, including: Based on target information at different scales, a plurality of candidate boxes are generated on the multi-scale feature pyramid using a region candidate RPN and an anchor box; In the RoIPooling stage, the Giou-based non-maximum suppression method is used to screen the candidate frames by calculating the overlap between the candidate frames to obtain the target candidate frame with the highest score, while suppressing other candidate frames with high overlap with the target candidate frame; Among them, simultaneously suppressing other candidate frames with a high degree of overlap with the target candidate frame includes: Set an overlap threshold, and keep the original processing unchanged for other candidate frames whose values ​​are smaller than the overlap threshold; for other candidate frames whose values ​​are larger than the overlap threshold, multiply the scores of other candidate frames by half of the overlap between other candidate frames and the target candidate frame, so as to update the scores of other candidate frames; If the updated scores of other candidate boxes are less than the specified threshold, the scores of other candidate boxes are directly assigned to 0; The candidate box score formula is: In the formula, S i represents the candidate box score, Giou(M,b i ) represents the target candidate box M and the i-th other candidate box b i The overlap degree, N t represents the overlap threshold, and S represents the specified threshold.

8. The intelligent zooplankton identification system based on machine learning is characterized by: include: A model building unit, used for building a target detection model for aquatic organisms based on deep learning; wherein the target detection model includes an image fusion layer, a DenseNet network and a region candidate RPN; A fusion processing unit, used for performing multi-focus image fusion processing on the multi-dimensional image samples of zooplankton using the image fusion layer to obtain a fusion feature map; A feature extraction unit, used to extract features from the fused feature map through the DenseNet network to obtain feature maps of different levels, and locate target information of the feature map; A screening unit is used to generate multiple candidate boxes using the region candidate RPN based on the target information, and screen the target candidate box with the highest score using the Giou-based non-maximum suppression method; The recognition unit is used for the target detection model to classify and locate the target candidate frame to realize the recognition of zooplankton.

9. An electronic device, characterized in that: It comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program or instructions to implement the steps of the method for intelligent identification of zooplankton based on machine learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by the processor, the steps of the method for intelligent identification of zooplankton based on machine learning described in any one of claims 1-7 are implemented.

Images