Method and apparatus for classifying multi-size plankton images

Abstract

The application relates to a method and device for classifying multi-size plankton pictures, which comprises the following steps: obtaining a first plankton picture; converting the first plankton picture into a second plankton picture according to the size category to which the size of the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; inputting the second plankton picture into a processing model to process the second plankton picture through the processing model; wherein the processing model comprises multiple processing branches, and the processing branches are respectively used for processing second plankton pictures of different sizes; determining the classification result of the second plankton picture according to the processing result of the processing model, so as to determine the classification result of the first plankton picture corresponding to the second plankton picture. Through the technical scheme provided by the application, the classification effect of multi-size plankton pictures can be improved.

Description

Technical Field

[0001] This application relates to the field of image classification technology, and in particular to a method and apparatus for classifying multi-sized plankton images. Background Technology

[0002] The classification of plankton is of great significance in scientific research and ecological monitoring. With the continuous development of machine vision technology, it is now possible to automatically identify plankton in plankton images through neural network-based classification models, thus achieving automatic classification of plankton.

[0003] Typically, when automatically classifying plankton, the plankton images input into the classification model contain only one plankton. However, due to the vast size variation among plankton, the sizes of the plankton images input into the classification model also vary significantly.

[0004] For input data, classification models typically have certain size specifications. When processing plankton images of different sizes, the model needs to significantly scale down images that deviate from the size specifications to conform to them. However, this approach leads to the loss of some features in the plankton images, resulting in poor classification performance for multi-sized plankton images. Summary of the Invention

[0005] This application provides a method and apparatus for classifying multi-sized plankton images to solve the technical problem that classification models do not perform well in classifying multi-sized plankton images.

[0006] In a first aspect, this application provides a method for classifying multi-size plankton images, the method comprising:

[0007] Obtain the first image of planktonic organisms;

[0008] The first plankton image is converted into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein the size of the second plankton image is the standard size corresponding to the size category of the first plankton image;

[0009] The second plankton image is input into the processing model, and the second plankton image is processed by the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are respectively used to process the second plankton images of different sizes;

[0010] The classification result of the second plankton image is determined based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0011] In one feasible embodiment of this application, converting the first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs includes:

[0012] Determine the standard size corresponding to the size category to which the size of the first plankton image belongs; wherein, the standard size corresponding to the size category to which the size of the first plankton image belongs is the average size or the maximum size of each of the first plankton images in the size category to which the size of the first plankton image belongs;

[0013] The first plankton image is compressed or stretched to convert it into the second plankton image; or,

[0014] Add pixels with a value of 0 to the first plankton image to convert the first plankton image into the second plankton image.

[0015] In one feasible embodiment of this application, the second plankton image is processed using the processing model, including:

[0016] The processing branch corresponding to the size of the second plankton image is determined, and the second plankton image is transmitted to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein, the processing branch is located in the hidden layer of the processing model;

[0017] The feature map of the second plankton image is extracted by the processing branch in the hidden layer of the processing model that corresponds to the size of the second plankton image, and the feature map is scaled and fully connected before being transmitted to the output layer of the processing model; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size;

[0018] The feature map is processed by the output layer of the processing model to output the processing result of the second plankton image.

[0019] In one feasible embodiment of this application, a feature sharing module is set in the hidden layer of the processing model, and the feature map of the second plankton image is extracted through the processing branch in the hidden layer of the processing model corresponding to the size of the second plankton image, including:

[0020] The feature sharing module enables feature sharing between the first processing branch and the second processing branch; wherein, the first processing branch is the processing branch corresponding to the size of the second plankton image, and the second processing branch is the processing branch not corresponding to the size of the second plankton image; the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer.

[0021] When the second planktonic image passes through the first convolutional layer, the first feature tensor of the second planktonic image is extracted through the first convolutional kernel, and the second feature tensor of the second planktonic image is extracted through the second convolutional kernel; wherein, the first convolutional kernel is a convolutional kernel from the first convolutional layer, and the second convolutional kernel is a convolutional kernel from both the first convolutional layer and the second convolutional layer.

[0022] The first feature tensor and the second feature tensor are fused to obtain the third feature tensor;

[0023] The feature map of the second plankton image is determined based on the third feature tensor.

[0024] In one feasible embodiment of this application, feature sharing is configured between the first processing branch and the second processing branch through the feature sharing module, including:

[0025] The first convolutional layer is determined in the first processing branch, and the second convolutional layer is determined in the second processing branch; wherein the size of the convolutional kernel contained in the first convolutional layer is the same as the size of the convolutional kernel contained in the second convolutional layer;

[0026] Shared convolutional kernels are marked in the first convolutional layer and the second convolutional layer; wherein the number of shared convolutional kernels is not greater than the total number of convolutional kernels contained in the first convolutional layer and is not greater than the total number of convolutional kernels contained in the second convolutional layer.

[0027] In the first convolutional layer, all convolutional kernels other than the shared convolutional kernels are marked as non-shared convolutional kernels;

[0028] The non-shared convolutional kernel is determined to be the first convolutional kernel, and the shared convolutional kernel is determined to be the second convolutional kernel.

[0029] In one feasible embodiment of this application, before acquiring the first plankton image, the method further includes:

[0030] Multiple images of third planktonic organisms are acquired; wherein the classification results corresponding to the third planktonic organism images are labeled on the third planktonic organism images;

[0031] The multiple third planktonic images are divided into multiple training datasets based on their dimensions; wherein, each training dataset includes multiple third planktonic images of different dimensions.

[0032] Construct the initial processing model;

[0033] The initial processing model is trained multiple times using the training dataset to obtain the processing model.

[0034] In one feasible embodiment of this application, the initial processing model includes N initial processing branches, and each training dataset is divided into N sub-training datasets. Each sub-training dataset includes multiple images of the third plankton of uniform size, where N is an integer greater than or equal to 2. The initial processing model is trained multiple times using the training datasets, including:

[0035] During a single training session, N sub-training datasets from one training dataset are simultaneously input into N initial processing branches to train the initial processing branches into the processing branches.

[0036] The initial processing model is trained multiple times using multiple training datasets to obtain the processing model.

[0037] Secondly, this application provides a classification device for multi-size plankton images, the device comprising:

[0038] The first acquisition module is used to acquire the first plankton image;

[0039] A first conversion module is used to convert the first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein the size of the second plankton image is a standard size corresponding to the size category of the first plankton image;

[0040] The processing module is used to input the second plankton image into the processing model and process the second plankton image through the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are respectively used to process the second plankton images of different sizes;

[0041] The determination module is used to determine the classification result of the second plankton image based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0042] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to perform the classification method for multi-size plankton images described in the first aspect of this application.

[0043] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the classification method for multi-size plankton images described in the first aspect of this application.

[0044] The technical solutions provided in this application have the following advantages compared with the prior art:

[0045] The technical solution provided in this application converts a first plankton image into a second plankton image based on the size category to which the size of the first plankton image belongs. The size of the second plankton image is a standard size corresponding to the size category to which the size of the first plankton image belongs. The second plankton image is processed by a processing model, which includes multiple processing branches for processing second plankton images of different sizes. The classification result of the second plankton image is determined based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0046] First, the size category of the first plankton image is identified, and image processing is performed based on this category, avoiding large-scale scaling of the first plankton image. Second, the processing model includes multiple processing branches, allowing for adaptive processing of the second plankton image through the corresponding branches. The technical solution provided in this application, based on reasonable transformation of the plankton image and adaptive settings of the processing model, enhances the processing model's ability to handle multi-sized plankton images, thereby improving the classification effect of multi-sized plankton images. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0050] Figure 1 A flowchart illustrating a method for classifying multi-size plankton images provided in this application embodiment;

[0051] Figure 2 A schematic diagram of the processing model used in a classification method for multi-size plankton images provided in this application embodiment;

[0052] Figure 3 This diagram illustrates the setting of shared and non-shared convolutional kernels in the feature sharing module of a classification method for multi-size plankton images provided in this application embodiment.

[0053] Figure 4 A schematic diagram of a multi-size plankton image classification device provided in this application embodiment;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0057] To address the technical problem that existing classification models do not perform well in classifying multi-sized plankton images, this application provides a method and apparatus for classifying multi-sized plankton images, thereby improving the classification performance of multi-sized plankton images.

[0058] Figure 1A flowchart illustrating a method for classifying multi-size plankton images provided in this application embodiment is shown below. Figure 1 The classification method for multi-size plankton images provided in this application specifically includes the following steps:

[0059] S1: Obtain the first image of planktonic organisms;

[0060] Specifically, the size of the first plankton image can be any size. For any first plankton image, there is only one plankton individual. The size of the first plankton image is also specifically related to the size of the plankton individual contained in the first plankton image.

[0061] In some specific examples, the first plankton image can be obtained from the FlowCAM imaging system. The FlowCAM imaging system is a flow cytometry particle analysis system based on digital imaging technology, commonly used to monitor plankton community composition. By using the FlowCAM imaging system to perform high-speed imaging and data processing on the plankton in the sample, the FlowCAM imaging system outputs multiple first plankton images, thus obtaining the first plankton image from the FlowCAM imaging system.

[0062] S2: Convert the first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein, the size of the second plankton image is the standard size corresponding to the size category of the first plankton image;

[0063] Specifically, the first plankton image is processed to convert it into a second plankton image, the size of which is a standard size corresponding to the size category of the first plankton image.

[0064] In one feasible embodiment of this application, converting a first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs includes:

[0065] Determine the standard size corresponding to the size category to which the size of the first plankton image belongs; wherein, the standard size corresponding to the size category to which the size of the first plankton image belongs is the average size or maximum size of all first plankton images in the size category to which the size of the first plankton image belongs;

[0066] Compress or stretch the first plankton image to convert it into a second plankton image; or,

[0067] Add pixels with a value of 0 to the first plankton image to convert the first plankton image into a second plankton image.

[0068] Specifically, the size of the first plankton image belongs to any one of several size categories. The size category can be manually set by technicians in advance, or it can be automatically generated based on the sizes of multiple first plankton images.

[0069] In one feasible embodiment of this application, when there are multiple first plankton images to be processed, the multiple first plankton images can be clustered according to their size to determine multiple size categories.

[0070] As an example, in the above embodiment, multiple first plankton images can be processed by a clustering algorithm to obtain multiple clusters. Each cluster corresponds to a size category. The size of multiple first plankton images associated with a cluster belongs to the size category corresponding to that cluster. The size category can be identified by the centroid of the cluster corresponding to the size category.

[0071] As an example, the clustering algorithm can be any of the K-Means algorithm, DBSCAN algorithm, or hierarchical clustering algorithm. Since the specific details of the clustering algorithm are existing technologies, they will not be elaborated here.

[0072] A size category corresponds to a standard size. The standard size corresponding to a size category can be the average size of each first plankton image in that size category, the maximum size of each first plankton image in that size category, or the size of the centroid of the cluster corresponding to that size category.

[0073] When the average size of each first plankton image in the size category is used as the standard size, the length of the standard size is the average length of each first plankton image in the size category, and the width of the standard size is the average width of each first plankton image in the size category. When the maximum size of each first plankton image in the size category is used as the standard size, the length of the standard size is the maximum length of each first plankton image in the size category, and the width of the standard size is the maximum width of each first plankton image in the size category. When the size of the centroid of the cluster is used as the standard size, the length of the standard size is the length of the first plankton image that serves as the centroid in the corresponding cluster of the size category, and the width of the standard size is the width of the first plankton image that serves as the centroid in the corresponding cluster of the size category.

[0074] In one feasible embodiment of this application, a first plankton image is converted into a second plankton image through image transformation. Specifically, the image transformation includes compressing or stretching the first plankton image to a standard size, and adding pixels with a pixel value of 0 around the first plankton image to the standard size. The image transformation also includes normalizing and standardizing the image, limiting the maximum, minimum, mean, variance, and other statistical quantities of the image pixels to a certain range. As an example, normalization transforms the pixel values ​​from the [0, 255] interval to the [-1, 1] interval, and standardization transforms the data into a distribution with a mean of 0 and a standard deviation of 1.

[0075] In step S2, for any first plankton image, the size of the first plankton image is determined, the size category to which the size of the first plankton image belongs is determined based on the size of the first plankton image, the standard size corresponding to the size category is determined based on the size category to which the size of the first plankton image belongs, and after determining the standard size, the first plankton image is converted into a second plankton image.

[0076] As can be seen, first-size plankton images belonging to different size categories are converted into second-size plankton images of different standard dimensions. Therefore, although the first-size plankton images have undergone image processing, they are not significantly scaled, and the proportions and sizes of the second-size plankton images relative to the first-size images actually change only slightly. This technical solution standardizes the multi-size first-size plankton images for easier subsequent processing while preserving the proportional characteristics of the individual plankton within the first-size images.

[0077] S3: Input the second plankton image into the processing model, and process the second plankton image through the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are used to process second plankton images of different sizes respectively;

[0078] Specifically, the second plankton image is input into the processing model, which processes the second plankton image to obtain the processing result.

[0079] Figure 2 A schematic diagram of the processing model used in a classification method for multi-size plankton images provided in this application embodiment is shown below. Figure 2 The processing model includes an input layer, a hidden layer, and an output layer. The hidden layer of the processing model contains multiple parallel processing branches.

[0080] Specifically, the input layer of the processing model is used to input the second plankton image. After determining the processing branch corresponding to the second plankton image based on its size, the input layer of the processing model transmits the second plankton image to the corresponding processing branch.

[0081] The hidden layer of the processing model includes multiple processing branches (e.g.) Figure 2 The system consists of A1-An, multiple scaling networks, and a fully connected network. Each processing branch is connected to a scaling network, and each scaling network is connected to a fully connected network. Different processing branches process second plankton images of different sizes to extract feature maps. Since the sizes of the second plankton images transmitted to different processing branches differ, the sizes of the feature maps extracted by different processing branches also differ. The scaling networks process the feature maps of different sizes, converting the size of each feature map to a uniform feature map size. The fully connected network fully connects the various feature maps that have been unified to the feature map size and transmits them to the output layer.

[0082] The output layer of the processing model is used to output the processing results. The output layer includes multiple neurons, and the number of neurons in the output layer is equal to the number of categories in the second plankton image.

[0083] In one feasible embodiment of this application, the network structures of each processing branch are different. The larger the standard size of the second plankton image processed by a processing branch, the more complex the network structure of that branch. This allows different processing branches to have different processing capabilities, with more complex branches exhibiting stronger processing power. Based on this, processing branches with higher processing capabilities can process second plankton images with larger standard sizes, while processing branches with lower processing capabilities can process second plankton images with smaller standard sizes, thus improving the overall processing capability of the processing model while avoiding waste of computational resources.

[0084] As an example, the network structure of any processing branch includes, but is not limited to, VGG16, ResNet34, ResNet50, MobileNetv2, EfficientNetB0, etc.; any processing branch can also add some additional modules to improve the processing capability of the processing branch. These additional modules include, but are not limited to, Feature Pyramid Network, Context Module, ECA Net (Efficient Channel Attention), RPN Net (Candidate Box Extraction), etc.

[0085] It is understandable that the specific structure of any processing branch can be freely combined from the various network structures and additional modules provided in the above examples, without any specific restrictions. Of course, based on this freely combinable implementation method, the number of convolutional layers, the number of convolutional kernels, the convolutional kernel size, and the type of activation function can all be different for different processing branches.

[0086] As an example, scaling networks in the hidden layers of a model can be implemented using Spatial Pyramid Pooling.

[0087] Based on the structure of the processing model described in the above embodiments, in a feasible embodiment of this application, the second plankton image is processed by the processing model, including:

[0088] A processing branch corresponding to the size of the second plankton image is determined, and the second plankton image is transmitted to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein, the processing branch is located in the hidden layer of the processing model;

[0089] The feature map of the second plankton image is extracted by processing the processing branch in the hidden layer of the processing model that corresponds to the size of the second plankton image. The feature map is then scaled and fully connected before being transmitted to the output layer of the processing model. The size of the feature map transmitted to the output layer of the processing model is a preset feature map size.

[0090] The feature map is processed by the output layer of the processing model, and the processing result of the second plankton image is output.

[0091] In one feasible embodiment of this application, the feature map of the second plankton image is extracted by processing a branch in the hidden layer of the processing model that corresponds to the size of the second plankton image, including:

[0092] Feature sharing is configured between the first processing branch and the second processing branch through a feature sharing module; wherein, the first processing branch is a processing branch corresponding to the size of the second plankton image, and the second processing branch is a processing branch not corresponding to the size of the second plankton image; the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer.

[0093] When the second planktonic image passes through the first convolutional layer, the first feature tensor of the second planktonic image is extracted through the first convolutional kernel, and the second feature tensor of the second planktonic image is extracted through the second convolutional kernel; wherein, the first convolutional kernel is a convolutional kernel from the first convolutional layer, and the second convolutional kernel is a convolutional kernel from both the first and second convolutional layers.

[0094] The first feature tensor and the second feature tensor are fused to obtain the third feature tensor;

[0095] The feature map of the second plankton image is determined based on the third feature tensor.

[0096] Specifically, continue to refer to Figure 2 The hidden layer of the processing model also includes a feature sharing module. This module is used to set up feature sharing between the first processing branch and at least one second processing branch. The first processing branch is the processing branch corresponding to the standard size of the currently input second plankton image, and the second processing branch is the processing branch not corresponding to the standard size of the currently input second plankton image. It can be seen that there can be multiple second processing branches.

[0097] Meanwhile, since each processing branch of the processing model can be set as the first processing branch to establish a feature sharing module, it is evident that there can be multiple feature sharing modules.

[0098] Specifically, the feature sharing module sets up a first processing branch and at least one second processing branch to share features. The first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer.

[0099] For example, if the currently input second plankton image is processed in processing branch A1, and the feature sharing module sets processing branches A2, A3, and A4 to share features with A1, then A1 is the first processing branch, and A2, A3, and A4 are all the second processing branches.

[0100] After setting up feature sharing between the first and second processing branches in the feature sharing module, when the second plankton image passes through the first convolutional layer, the first feature tensor of the second plankton image is extracted through the first convolutional kernel, which is a convolutional kernel from the first convolutional layer; at the same time, when the second plankton image passes through the first convolutional layer, the second feature tensor of the second plankton image is extracted through the second convolutional kernel, which is a convolutional kernel from both the first and second convolutional layers.

[0101] It should be noted that when feature sharing is set between the first processing branch and the second processing branch in the feature sharing module, the first processing branch may include multiple first convolutional layers, and feature sharing is performed when passing through each of the first convolutional layers.

[0102] After extracting the first and second feature tensors from the second plankton image, the first processing branch fuses the first and second feature tensors to obtain the third feature tensor of the second plankton image. The third feature tensor is then transmitted to the next level convolutional layer of the first convolutional layer for further processing until it reaches the end of the first processing branch. The feature map of the second plankton image is obtained from the third feature tensor.

[0103] If the first convolutional layer is located at the beginning of the first processing branch, the input to the first convolutional layer is the second plankton image, and the output feature tensor is sent to the next level convolutional layer of the first convolutional layer for further processing in the first processing branch. If the first convolutional layer is located at the end of the first processing branch, the input to the first convolutional layer is the feature tensor output by the previous level convolutional layer of the first convolutional layer, and the output feature map is sent to the scaling network. If the first convolutional layer is located in the middle of the first processing branch, the input to the first convolutional layer is the feature tensor output by the previous level convolutional layer of the first convolutional layer, and the output feature tensor is sent to the next level convolutional layer of the first convolutional layer for further processing in the first processing branch.

[0104] For example, refer to Figure 3 The Tensor (the original feature tensor) is processed in the first processing branch. When the Tensor passes through the first convolutional layer, it is processed by both the first and second convolutional kernels. The first convolutional kernel comes only from the first convolutional layer, while the second convolutional kernel comes from both the first and second convolutional layers. After processing by the first convolutional kernel, the Tensor outputs Tensor1 (the first feature tensor). After processing by the second convolutional kernel, it outputs Tensor2 (the second feature tensor). Tensor1 and Tensor2 are fused to obtain Tensor3 (the third feature tensor). The processing of the Tensor in the first convolutional layer ends, and it continues to be passed to other convolutional layers in the first processing branch until the feature map of the second plankton image corresponding to the Tensor is output at the end of the first processing branch.

[0105] It is understandable that, since the second plankton image is processed simultaneously through the first and second convolutional kernels, the second plankton image can refer to features from the processing branch corresponding to its own standard size, or it can refer to features from the processing branches corresponding to other standard sizes. Setting feature sharing between processing branches can ensure that common features between similar images of different size categories can be shared between different processing branches, which is beneficial to improving the classification effect of the processing model on plankton images.

[0106] In one feasible embodiment of this application, feature sharing is configured between the first processing branch and the second processing branch through a feature sharing module, including:

[0107] A first convolutional layer is determined in a first processing branch, and a second convolutional layer is determined in a second processing branch; wherein the size of the convolutional kernel contained in the first convolutional layer is the same as the size of the convolutional kernel contained in the second convolutional layer;

[0108] Shared convolutional kernels are marked in the first and second convolutional layers; wherein the number of shared convolutional kernels is no greater than the total number of convolutional kernels contained in the first convolutional layer and no greater than the total number of convolutional kernels contained in the second convolutional layer.

[0109] In the first convolutional layer, convolutional kernels other than shared kernels are labeled as non-shared kernels;

[0110] The non-shared convolutional kernel is determined as the first convolutional kernel, and the shared convolutional kernel is determined as the second convolutional kernel.

[0111] Specifically, the feature sharing module first determines the first convolutional layer in the first processing branch and the second convolutional layer in the second processing branch. The size of the convolutional kernels in the first convolutional layer and the size of the convolutional kernels in the second convolutional layer need to be consistent.

[0112] After determining the first and second convolutional layers, the feature sharing module marks shared convolutional kernels in both layers. These shared kernels can be any number of kernels in the first and second convolutional layers. The feature sharing module also marks the kernels other than the shared kernels in the first convolutional layer as non-shared kernels.

[0113] As can be seen, the non-shared convolutional kernel in the first convolutional layer is the first convolutional kernel, and the other convolutional kernels in the first convolutional layer besides the non-shared convolutional kernel, as well as the other convolutional kernels in the second convolutional layer besides the non-shared convolutional kernel, are the second convolutional kernels.

[0114] S4: Determine the classification result of the second plankton image based on the processing result of the processing model, and then determine the classification result of the first plankton image corresponding to the second plankton image;

[0115] Specifically, the processing result output by the output layer of the processing model represents the classification result of the second plankton image. After determining the classification result of the second plankton image, the classification result of the first plankton image corresponding to the second plankton image can be determined, thereby completing the classification of the first plankton image.

[0116] Based on the technical solutions provided in the above embodiments, firstly, the size category of the first plankton image is identified, and image processing is performed based on the size category, avoiding large-scale scaling of the first plankton image; secondly, the processing model includes multiple processing branches, thereby adaptively processing the second plankton image through the corresponding processing branches. In summary, the technical solutions provided in this application, based on reasonable transformation of the plankton image and adaptive settings of the processing model, enhance the processing model's ability to process multi-sized plankton images, thereby improving the classification effect of multi-sized plankton images.

[0117] In one feasible embodiment of this application, the processing model is trained before S1. The training process of the processing model specifically includes:

[0118] Multiple images of third-order plankton were obtained; the third-order plankton images were labeled with their corresponding classification results.

[0119] Multiple third-order planktonic images are divided into multiple training datasets based on their dimensions; each training dataset contains multiple third-order planktonic images of different sizes.

[0120] Construct the initial processing model;

[0121] The initial processing model is trained multiple times using the training dataset to obtain the final processing model.

[0122] Specifically, multiple images of third-order plankton are obtained. The third-order plankton images already have corresponding classification results labeled on them, and the classification of the third-order plankton images can be done manually by technicians.

[0123] Multiple training datasets were identified, each containing multiple images of third-order plankton of varying sizes.

[0124] In one feasible embodiment of this application, the third planktonic images can first be clustered according to their size using a clustering algorithm to determine multiple clusters. Each cluster contains multiple third planktonic images, and the third planktonic images in the same cluster belong to the same size category. Then, the size of the third planktonic images in a cluster is uniformly converted to the standard size corresponding to the size category of the current cluster. Finally, multiple third planktonic images of different standard sizes are selected to form a training set.

[0125] In one feasible embodiment of this application, the clustering results of the third plankton image can be used as the basis for classifying the size category of the first plankton image. That is, by clustering the third plankton image to determine multiple clusters, and using these clusters as size categories, when it is necessary to determine the size category of the first plankton image, it is only necessary to calculate the distance from the first plankton image to the centroid of these clusters. When the distance from the first plankton image to the centroid of a certain cluster is the smallest, the first plankton image belongs to the size category corresponding to that cluster.

[0126] To ensure data balance, data balancing is performed on each training dataset. As an example, data balancing could involve assigning specific weight values ​​to the third plankton images in all training datasets to be balanced, ensuring a consistent total weight across all balanced training datasets. Alternatively, data balancing could involve copying the third plankton images from training datasets with fewer images, ensuring a consistent total number of images across all balanced training datasets.

[0127] To ensure training efficiency, the third plankton images were normalized and standardized, limiting the maximum, minimum, mean, and variance of the image pixels to a certain range. As an example, normalization transformed the pixel values ​​from the [0, 255] interval to the [-1, 1] interval, and standardization converted the data into a distribution with a mean of 0 and a standard deviation of 1.

[0128] Set the parameters of the initial processing model and construct the initial processing model. As an example, setting the parameters of the initial processing model includes, but is not limited to, setting the stride, padding, learning rate, momentum, loss function, optimization algorithm, mini-batch size, number of training epochs, and whether to use regularization techniques.

[0129] After constructing the initial processing model, the initial processing model is trained multiple times using multiple training datasets to obtain the final processing model.

[0130] The processing model, derived from the initial processing model, requires not only training but also validation and testing. Therefore, it is necessary to construct test and validation datasets, and then divide them into standard-sized third-generation plankton images for the test and validation datasets.

[0131] In one feasible embodiment of this application, after dividing the multiple third planktonic images into multiple training datasets according to their sizes, the method further includes:

[0132] Based on a preset partitioning ratio, several training datasets are designated as validation datasets, and several training datasets are designated as test datasets.

[0133] In some specific examples, the preset split ratio is set by the technical staff. For example, if the current split ratio is set to 20% and 10%, then 20% of the training dataset will be used as the validation dataset to validate the initial processing model after training, and 10% of the training dataset will be used as the test dataset to test the initial processing model after training.

[0134] In one feasible embodiment of this application, the initial processing model includes N initial processing branches, the training dataset is divided into N sub-training datasets, and each sub-training dataset includes multiple third planktonic images of the same size, where N is an integer greater than or equal to 2. The initial processing model is trained multiple times using the training datasets, including:

[0135] During a training session, N sub-training datasets from a training dataset are simultaneously input into N initial processing branches to train the initial processing branches into processing branches.

[0136] The initial processing model is trained multiple times using multiple training datasets to obtain the final processing model.

[0137] Specifically, the initial processing model is trained multiple times to obtain the processing model. The initial processing model includes N initial processing branches, and the training dataset is divided into N sub-training datasets. When training the initial processing model once, the N sub-training datasets from one training dataset are simultaneously input into the N initial processing branches to train the initial processing branches into processing branches.

[0138] It can be seen that when training the initial processing model once, the total number of third planktonic images M that need to be input into the initial processing model is equal to the sub-batch size × the number of initial processing branches N.

[0139] When training the initial processing model, different initial processing branches are trained using different sub-training datasets. One initial processing branch is trained using a third plankton image of the same standard size. In this way, the initial processing branch can extract the features of the standard-sized third plankton image of the current input, and the processing branch can extract the feature map of the second plankton image of the same standard size.

[0140] The structure of the initial processing model is consistent with that of the processing model, and it also has a feature sharing module. Therefore, when the initial processing model is trained once, since the feature sharing module sets up feature sharing between the initial processing branches, each initial processing branch learns some features used to process other standard-sized initial processing branches during training, so that the processed model after training can realize feature sharing between processing branches.

[0141] Figure 4 This application provides a schematic diagram of a multi-size plankton image classification device. Corresponding to the above method embodiment, this application also provides a multi-size plankton image classification device. (Refer to...) Figure 4 The device specifically includes:

[0142] The first acquisition module 401 is used to acquire the first plankton image;

[0143] The first conversion module 402 is used to convert the first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein the size of the second plankton image is a standard size corresponding to the size category of the first plankton image;

[0144] The processing module 403 is used to input the second plankton image into the processing model and process the second plankton image through the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are used to process second plankton images of different sizes respectively;

[0145] The determination module 404 is used to determine the classification result of the second plankton image based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0146] In one feasible embodiment of this application, the first conversion module 402 includes:

[0147] The determining unit is used to determine the standard size corresponding to the size category to which the size of the first plankton image belongs; wherein, the standard size corresponding to the size category to which the size of the first plankton image belongs is the average size or the maximum size of each first plankton image in the size category to which the size of the first plankton image belongs;

[0148] The first conversion unit is used to compress or stretch the first plankton image to convert it into a second plankton image; or,

[0149] The second conversion unit is used to add pixels with a pixel value of 0 to the first plankton image to convert the first plankton image into a second plankton image.

[0150] In one feasible embodiment of this application, the processing module 403 includes:

[0151] The first processing unit is used to determine the processing branch corresponding to the size of the second plankton image, and to transmit the second plankton image to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein, the processing branch is located in the hidden layer of the processing model;

[0152] The second processing unit is used to extract the feature map of the second plankton image through the processing branch in the hidden layer of the processing model that corresponds to the size of the second plankton image, and to transmit the feature map to the output layer of the processing model after scaling and full connection; wherein, the size of the feature map transmitted to the output layer of the processing model is a preset feature map size;

[0153] The third processing unit is used to process the feature map through the output layer of the processing model and output the processing result of the second plankton image.

[0154] In one feasible embodiment of this application, a feature sharing module is set in the hidden layer of the processing model, and the second processing unit includes:

[0155] A subunit is set up to enable feature sharing between the first processing branch and the second processing branch through the feature sharing module; wherein, the first processing branch is a processing branch corresponding to the size of the second plankton image, and the second processing branch is a processing branch that does not correspond to the size of the second plankton image; the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer.

[0156] An extraction subunit is used to extract the first feature tensor of the second plankton image through the first convolutional kernel and the second feature tensor of the second plankton image through the second convolutional kernel when the second plankton image passes through the first convolutional layer; wherein, the first convolutional kernel is a convolutional kernel from the first convolutional layer, and the second convolutional kernel is a convolutional kernel from both the first and second convolutional layers.

[0157] A fusion subunit is used to fuse the first feature tensor and the second feature tensor to obtain the third feature tensor;

[0158] A sub-unit is determined to identify the feature map of the second plankton image based on the third feature tensor.

[0159] In one feasible embodiment of this application, a subunit is provided, including:

[0160] The first determining subunit is used to determine the first convolutional layer in the first processing branch and the second convolutional layer in the second processing branch; wherein the size of the convolutional kernel contained in the first convolutional layer is the same as the size of the convolutional kernel contained in the second convolutional layer;

[0161] The first labeling subunit is used to label shared convolutional kernels in the first convolutional layer and the second convolutional layer; wherein the number of shared convolutional kernels is no greater than the total number of convolutional kernels contained in the first convolutional layer and no greater than the total number of convolutional kernels contained in the second convolutional layer.

[0162] The second labeling subunit is used to label convolutional kernels other than shared convolutional kernels as non-shared convolutional kernels in the first convolutional layer;

[0163] The second determining unit is used to determine the non-shared convolution kernel as the first convolution kernel and the shared convolution kernel as the second convolution kernel.

[0164] In one feasible embodiment of this application, the device further includes:

[0165] The second acquisition module is used to acquire multiple third planktonic images; wherein the third planktonic images are labeled with the corresponding classification results;

[0166] The partitioning module is used to divide multiple third-party planktonic images into multiple training datasets based on their dimensions; wherein, a training dataset includes multiple third-party planktonic images of different sizes;

[0167] Build modules are used to construct the initial processing model;

[0168] The training module is used to train the initial processing model multiple times using the training dataset to obtain the final processing model.

[0169] In one feasible embodiment of this application, the initial processing model includes N initial processing branches, the training dataset is divided into N sub-training datasets, each sub-training dataset includes multiple third planktonic images of the same size, where N is an integer greater than or equal to 2, and the training module includes:

[0170] The input unit is used to simultaneously input N sub-training datasets from a training dataset into N initial processing branches during a single training session, so as to train the initial processing branches into processing branches.

[0171] The training unit is used to train the initial processing model multiple times using multiple training datasets to obtain the processing model.

[0172] like Figure 5As shown in the figure, this application provides an electronic device, including a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0173] Memory 503 is used to store computer programs;

[0174] In one embodiment of this application, when the processor 501 executes the program stored in the memory 503, it implements a method for classifying multi-size plankton images provided in any of the foregoing method embodiments, for example including:

[0175] Obtain the first image of planktonic organisms;

[0176] The first planktonic image is converted into a second planktonic image based on its size category; wherein the size of the second planktonic image is the standard size corresponding to the size category of the first planktonic image.

[0177] The second planktonic image is input into the processing model, which processes the second planktonic image. The processing model includes multiple processing branches, each of which is used to process second planktonic images of different sizes.

[0178] The classification result of the second plankton image is determined based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0179] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of a method for classifying multi-size plankton images as provided in any of the foregoing method embodiments.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0182] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0183] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for classifying multi-size plankton images, characterized in that, The method includes: Obtain the first image of planktonic organisms; The first plankton image is converted into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein the size of the second plankton image is the standard size corresponding to the size category of the first plankton image; The second plankton image is input into the processing model, and the second plankton image is processed by the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are respectively used to process the second plankton images of different sizes; The classification result of the second plankton image is determined based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image; The processing of the second plankton image using the processing model includes: determining the processing branch corresponding to the size of the second plankton image; transmitting the second plankton image to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein the processing branch is located in the hidden layer of the processing model; extracting a feature map of the second plankton image through the processing branch corresponding to the size of the second plankton image in the hidden layer of the processing model, and transmitting the feature map to the output layer of the processing model after scaling and full connection; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; processing the feature map through the output layer of the processing model, and outputting the processing result of the second plankton image.

2. The method according to claim 1, characterized in that, Converting the first plankton image into a second plankton image based on its size category includes: Determine the standard size corresponding to the size category to which the size of the first plankton image belongs; wherein, the standard size corresponding to the size category to which the size of the first plankton image belongs is the average size or the maximum size of each of the first plankton images in the size category to which the size of the first plankton image belongs; The first plankton image is compressed or stretched to convert it into the second plankton image; or, Add pixels with a value of 0 to the first plankton image to convert the first plankton image into the second plankton image.

3. The method according to claim 1, characterized in that, The processing model includes a feature sharing module in its hidden layer. This module extracts a feature map of the second plankton image through the processing branch in the hidden layer that corresponds to the size of the second plankton image. This includes: The feature sharing module enables feature sharing between the first processing branch and the second processing branch; wherein, the first processing branch is the processing branch corresponding to the size of the second plankton image, and the second processing branch is the processing branch not corresponding to the size of the second plankton image; the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer. When the second planktonic image passes through the first convolutional layer, the first feature tensor of the second planktonic image is extracted through the first convolutional kernel, and the second feature tensor of the second planktonic image is extracted through the second convolutional kernel; wherein, the first convolutional kernel is a convolutional kernel from the first convolutional layer, and the second convolutional kernel is a convolutional kernel from both the first convolutional layer and the second convolutional layer. The first feature tensor and the second feature tensor are fused to obtain the third feature tensor; The feature map of the second plankton image is determined based on the third feature tensor.

4. The method according to claim 3, characterized in that, The feature sharing module enables feature sharing between the first processing branch and the second processing branch, including: The first convolutional layer is determined in the first processing branch, and the second convolutional layer is determined in the second processing branch; wherein the size of the convolutional kernel contained in the first convolutional layer is the same as the size of the convolutional kernel contained in the second convolutional layer; Shared convolutional kernels are marked in the first convolutional layer and the second convolutional layer; wherein the number of shared convolutional kernels is not greater than the total number of convolutional kernels contained in the first convolutional layer and is not greater than the total number of convolutional kernels contained in the second convolutional layer. In the first convolutional layer, all convolutional kernels other than the shared convolutional kernels are marked as non-shared convolutional kernels; The non-shared convolutional kernel is determined to be the first convolutional kernel, and the shared convolutional kernel is determined to be the second convolutional kernel.

5. The method according to claim 1, characterized in that, Before acquiring the first plankton image, the method further includes: Multiple images of third planktonic organisms are acquired; wherein the classification results corresponding to the third planktonic organism images are labeled on the third planktonic organism images; The multiple third planktonic images are divided into multiple training datasets based on their dimensions; wherein, each training dataset includes multiple third planktonic images of different dimensions. Construct the initial processing model; The initial processing model is trained multiple times using the training dataset to obtain the processing model.

6. The method according to claim 5, characterized in that, The initial processing model includes N initial processing branches. Each training dataset is divided into N sub-training datasets, each sub-training dataset containing multiple images of the third plankton of uniform size. N is an integer greater than or equal to 2. The initial processing model is trained multiple times using the training datasets, including: During a single training session, N sub-training datasets from one training dataset are simultaneously input into N initial processing branches to train the initial processing branches into the processing branches. The initial processing model is trained multiple times using multiple training datasets to obtain the processing model.

7. A classification device for multi-size plankton images, characterized in that, The device includes: The first acquisition module is used to acquire the first plankton image; A first conversion module is used to convert the first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs; wherein the size of the second plankton image is a standard size corresponding to the size category of the first plankton image; The processing module is used to input the second plankton image into the processing model and process the second plankton image through the processing model; wherein, the processing model includes multiple processing branches, and the processing branches are respectively used to process the second plankton images of different sizes; The determination module is used to determine the classification result of the second plankton image based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image; The processing module includes: a first processing unit, configured to determine a processing branch corresponding to the size of the second plankton image, and transmit the second plankton image to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein the processing branch is located in the hidden layer of the processing model; a second processing unit, configured to extract a feature map of the second plankton image through the processing branch corresponding to the size of the second plankton image in the hidden layer of the processing model, and transmit the feature map to the output layer of the processing model after scaling and full connection; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; and a third processing unit, configured to process the feature map through the output layer of the processing model and output the processing result of the second plankton image.

8. An electronic device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to implement the classification method for multi-size plankton images as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the classification method for multi-size plankton images as described in any one of claims 1-6.

Images