Classification method and device for plankton pictures of multiple sizes

Abstract

The invention relates to a multi-size plankton picture classification method and device. The method comprises the following steps: acquiring a first plankton picture; converting the first plankton picture into a second plankton picture according to the size category of the size of the first plankton picture; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; inputting the second plankton picture into a processing model, and processing the second plankton picture through the processing model; wherein the processing model comprises a plurality of processing branches, and the processing branches are respectively used for processing second plankton pictures with different sizes; and determining a classification result of the second plankton picture according to a processing result of the processing model, thereby determining a classification result of the first plankton picture corresponding to the second plankton picture. Through the technical scheme provided by the invention, the classification effect of plankton pictures of multiple sizes can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to a method and device for classifying multi-sized plankton images. Background Art

[0002] The classification of plankton is of great significance in scientific research and ecological monitoring. With the continuous development of machine vision technology, it is now possible to automatically identify plankton in plankton images through a neural network-based classification model, thereby achieving automatic classification of plankton.

[0003] Typically, when automatically classifying plankton, the plankton image fed into the classification model contains only one plankton. However, because plankton vary greatly in size, the size of the plankton images fed into the classification model also varies greatly.

[0004] Classification models typically set a certain size specification for input data. When processing plankton images of varying sizes, the model must resize those that significantly deviate from the specified size specification to ensure they meet the specified size specification. However, this approach can lead to the loss of some of the plankton image's features, resulting in poor classification results for plankton images of varying sizes. Summary of the Invention

[0005] The present application provides a method and apparatus for classifying multi-sized plankton images to solve the technical problem that classification models have poor classification effects on multi-sized plankton images.

[0006] In a first aspect, the present application provides a method for classifying multi-scale plankton images, the method comprising: Get the first plankton picture; converting the first plankton picture into a second plankton picture according to a size category to which the size of the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; inputting the second plankton image into a processing model, and processing the second plankton image by the processing model; wherein the processing model includes a plurality of processing branches, each of which is used to process the second plankton image of different sizes; The classification result of the second plankton picture is determined according to the processing result of the processing model, thereby determining the classification result of the first plankton picture corresponding to the second plankton picture.

[0007] In a feasible embodiment of the present application, converting the first plankton picture into a second plankton picture according to a size category to which the first plankton picture belongs includes: determining the standard size corresponding to the size category to which the size of the first plankton picture belongs; wherein the standard size corresponding to the size category to which the size of the first plankton picture belongs is the average size or maximum size of each of the first plankton pictures in the size category to which the size of the first plankton picture belongs; compressing or stretching the first plankton picture to convert the first plankton picture into the second plankton picture; or, Pixels with a pixel value of 0 are added to the first plankton picture to convert the first plankton picture into the second plankton picture.

[0008] In a feasible embodiment of the present application, processing the second plankton image by the processing model includes: determining the processing branch corresponding to the size of the second plankton image, and transmitting the second plankton image to the processing branch corresponding to the size of the second plankton image through the input layer of the processing model; wherein the processing branch is located in the hidden layer of the processing model; extracting a feature map of the second plankton image through the processing branch corresponding to the size of the second plankton image in the hidden layer of the processing model, scaling and fully connecting the feature map, and then transmitting it to the output layer of the processing model; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; The feature map is processed by an output layer of the processing model, and the processing result of the second plankton image is output.

[0009] In a feasible embodiment of the present application, a feature sharing module is provided in the hidden layer of the processing model, and a feature map of the second plankton image is extracted through the processing branch corresponding to the size of the second plankton image in the hidden layer of the processing model, including: Setting feature sharing between a first processing branch and a second processing branch through the feature sharing module; wherein the first processing branch is the processing branch corresponding to the size of the second plankton image, the second processing branch is the processing branch not corresponding to the size of the second plankton image, the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer; When the second plankton image passes through the first convolutional layer, extracting a first feature tensor of the second plankton image using a first convolution kernel, and extracting a second feature tensor of the second plankton image using a second convolution kernel; wherein the first convolution kernel is a convolution kernel from the first convolutional layer, and the second convolution kernel is a convolution kernel from both the first convolutional layer and the second convolutional layer; fusing the first feature tensor and the second feature tensor to obtain a third feature tensor; The feature map of the second plankton image is determined according to the third feature tensor.

[0010] In a feasible embodiment of the present application, setting feature sharing between the first processing branch and the second processing branch through the feature sharing module includes: Determining the first convolutional layer in the first processing branch, and determining the second convolutional layer in the second processing branch; wherein the size of the convolution kernel included in the first convolutional layer is consistent with the size of the convolution kernel included in the second convolutional layer; Marking shared convolution kernels in the first convolution layer and the second convolution layer; wherein the number of the shared convolution kernels is not greater than the total number of convolution kernels included in the first convolution layer, and is not greater than the total number of convolution kernels included in the second convolution layer; Marking convolution kernels other than the shared convolution kernel in the first convolution layer as non-shared convolution kernels; The non-shared convolution kernel is determined to be the first convolution kernel, and the shared convolution kernel is determined to be the second convolution kernel.

[0011] In a feasible embodiment of the present application, before obtaining the first plankton image, the method further includes: Acquire a plurality of third plankton pictures; wherein the third plankton pictures are marked with the classification results corresponding to the third plankton pictures; Dividing the plurality of third plankton images into a plurality of training data sets according to the sizes of the plurality of third plankton images; wherein one of the training data sets includes a plurality of third plankton images of inconsistent sizes; Build an initial processing model; The initial processing model is trained multiple times using the training data set to obtain the processing model.

[0012] In a feasible embodiment of the present application, the initial processing model includes N initial processing branches, each of the training data sets is divided into N sub-training data sets, each of the sub-training data sets includes multiple third plankton images of the same size, N is an integer greater than or equal to 2, and the initial processing model is trained multiple times using the training data sets, including: During one training, N sub-training data sets in one training data set are simultaneously input into N initial processing branches to train the initial processing branches into the processing branches; The initial processing model is trained multiple times using multiple training data sets to obtain the processing model.

[0013] In a second aspect, the present application provides a device for classifying multi-sized plankton images, the device comprising: A first acquisition module, configured to acquire a first plankton image; a first conversion module, configured to convert the first plankton picture into a second plankton picture according to a size category to which the size of the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; a processing module, configured to input the second plankton image into a processing model, and process the second plankton image using the processing model; wherein the processing model includes a plurality of processing branches, each of which is configured to process the second plankton image of different sizes; A determination module is configured to determine a classification result of the second plankton picture according to a processing result of the processing model, thereby determining a classification result of the first plankton picture corresponding to the second plankton picture.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the method for classifying multi-sized plankton images described in the first aspect of the present application.

[0015] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the classification method of multi-sized plankton images described in the first aspect of the present application.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art: The technical solution provided in the embodiment of the present application converts a first plankton image into a second plankton image according to the size category to which the size of the first plankton image belongs, wherein the size of the second plankton image is a standard size corresponding to the size category to which the size of the first plankton image belongs. The second plankton image is processed by a processing model, and the processing model includes multiple processing branches for processing second plankton images of different sizes. The classification result of the second plankton image is determined based on the processing result of the processing model, thereby determining the classification result of the first plankton image corresponding to the second plankton image.

[0017] First, the size category of the first plankton image is identified and image processing is performed based on the size category, avoiding large-scale scaling of the first plankton image. Second, a processing model is configured to include multiple processing branches, so that the second plankton image can be adaptively processed through the corresponding processing branches. The technical solutions provided in the embodiments of this application enhance the processing model's ability to process plankton images of multiple sizes based on the rational transformation of the plankton images and the adaptive configuration of the processing model, thereby improving the classification effect of these multi-sized plankton images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 A schematic flow chart of a method for classifying multi-sized plankton images provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a processing model used in a method for classifying multi-sized plankton images provided in an embodiment of the present application; Figure 3 A schematic diagram of setting shared convolution kernels and non-shared convolution kernels in a feature sharing module in a method for classifying multi-size plankton images provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of a device for classifying multi-sized plankton images provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The disclosure below provides many different embodiments or examples for implementing different structures of the present application. In order to simplify the disclosure of the present application, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present application. In addition, the present application may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0024] In order to solve the technical problem that the classification model in the existing technology has poor classification effect on multi-sized plankton images, the present application provides a classification method and device for multi-sized plankton images to improve the classification effect of multi-sized plankton images.

[0025] Figure 1 A schematic diagram of a method for classifying multi-size plankton images provided in an embodiment of the present application is provided. Figure 1 The present invention provides a method for classifying multi-scale plankton images, which specifically includes the following steps: S1: Get the first plankton picture; Specifically, the size of the first plankton picture can be any size. For any first plankton picture, the first plankton picture only contains one plankton individual. The size of the first plankton picture is also specifically related to the size of the plankton individual contained in the first plankton picture.

[0026] In some specific examples, a first plankton image can be obtained from a FlowCAM imaging system. The FlowCAM imaging system is a flow imaging particle analysis system based on digital imaging technology, and is commonly used to monitor plankton community composition. The FlowCAM imaging system performs high-speed imaging and data processing on plankton in a test sample, and the FlowCAM imaging system outputs multiple first plankton images, and the first plankton image is obtained from the FlowCAM imaging system.

[0027] S2: converting the first plankton picture into a second plankton picture according to the size category to which the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; Specifically, the first plankton picture is processed and converted into a second plankton picture, and the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture.

[0028] In a feasible embodiment of the present application, converting the first plankton picture into the second plankton picture according to the size category to which the first plankton picture belongs includes: Determining a standard size corresponding to a size category to which the size of the first plankton image belongs; wherein the standard size corresponding to the size category to which the size of the first plankton image belongs is an average size or a maximum size of each first plankton image in the size category to which the size of the first plankton image belongs; compressing or stretching the first plankton picture to convert the first plankton picture into a second plankton picture; or, Pixels with a pixel value of 0 are added to the first plankton picture to convert the first plankton picture into a second plankton picture.

[0029] Specifically, the size of the first plankton picture belongs to any one of a plurality of size categories. The size category may be manually set in advance by a technician, or may be automatically generated based on the sizes of a plurality of first plankton pictures.

[0030] In a feasible embodiment of the present application, when there are multiple first plankton pictures to be processed, the multiple first plankton pictures may be clustered according to their sizes to determine multiple size categories.

[0031] As an example, in the above embodiment, multiple first plankton pictures can be processed by a clustering algorithm to obtain multiple clusters, where one cluster corresponds to a size category. The sizes of multiple first plankton pictures associated with a cluster belong to the size category corresponding to the cluster, and the size category can be identified by the centroid of the cluster corresponding to the size category.

[0032] As an example, the clustering algorithm may be any one of the K-Means algorithm, the DBSCAN algorithm, or the hierarchical clustering algorithm. Since the specific content of the clustering algorithm is an existing technology, it will not be described in detail here.

[0033] A size category corresponds to a standard size. The standard size corresponding to the size category can be the average size of each first plankton image in the size category, the maximum size of each first plankton image in the size category, or the size of the centroid of the cluster corresponding to the size category.

[0034] When the average size of each first plankton image in a size category is used as the standard size, the length of the standard size is the average length of each first plankton image in the size category, and the width of the standard size is the average width of each first plankton image in the size category; when the maximum size of each first plankton image in a size category is used as the standard size, the length of the standard size is the maximum length of each first plankton image in the size category, and the width of the standard size is the maximum width of each first plankton image in the size category; when the size of the centroid of the cluster is used as the standard size, the length of the standard size is the length of the first plankton image as the centroid in the cluster corresponding to the size category, and the width of the standard size is the width of the first plankton image as the centroid in the cluster corresponding to the size category.

[0035] In a feasible embodiment of the present application, the first plankton picture is converted into the second plankton picture by performing a picture transformation on the first plankton picture. Specifically, the picture transformation includes compressing or stretching the first plankton picture according to a standard size, and includes adding pixels with a pixel value of 0 around the first plankton picture according to a standard size. The picture transformation also includes normalizing and standardizing the picture, limiting the statistical quantities such as the maximum value, minimum value, mean, and variance of the picture pixels to a certain range. As an example, the pixel value is changed from the interval [0, 255] to the interval [-1, 1] by normalization, and the data is converted into a distribution with a mean of 0 and a standard deviation of 1 by standardization.

[0036] In step S2, for any first plankton picture, the size of the first plankton picture is determined, the size category to which the size of the first plankton picture belongs is determined based on the size of the first plankton picture, and the standard size corresponding to the size category is determined based on the size category to which the size of the first plankton picture belongs. After determining the standard size, the first plankton picture is converted into a second plankton picture.

[0037] As can be seen, first plankton images of different sizes belonging to different size categories are converted into second plankton images of different standard sizes. Therefore, although the first plankton images undergo image processing, they are not significantly scaled, and the proportions and sizes of the second plankton images are actually limited compared to the first. This technical solution not only standardizes the first plankton images of multiple sizes for subsequent processing, but also preserves the relevant proportional characteristics of the individual plankton included in the first plankton images.

[0038] S3: Inputting the second plankton image into the processing model, and processing the second plankton image through the processing model; wherein the processing model includes a plurality of processing branches, and the processing branches are respectively used to process the second plankton images of different sizes; Specifically, the second plankton picture is input into the processing model, the second plankton picture is processed by the processing model, and a processing result of the processing model is obtained.

[0039] Figure 2 A schematic diagram of the structure of the processing model used in the classification method of multi-size plankton images provided in the embodiment of the present application, referring to Figure 2 The processing model includes an input layer, a hidden layer and an output layer, and the hidden layer of the processing model includes multiple parallel processing branches.

[0040] Specifically, the input layer of the processing model is used to input the second plankton picture. After determining the processing branch corresponding to the second plankton picture based on the size of the second plankton picture, the input layer of the processing model transmits the second plankton picture to the corresponding processing branch.

[0041] The hidden layer of the processing model includes multiple processing branches (e.g. Figure 2A1-An in the image), multiple scale-up networks, and a fully connected network. One processing branch is connected to a scale-up network, and each scale-up network is connected to the fully connected network. Different processing branches are used to process second plankton images of different sizes and extract feature maps of the second plankton images. Because the sizes of the second plankton images transmitted to different processing branches vary, the sizes of the feature maps extracted by different processing branches also vary. The scale-up network is used to process feature maps of different sizes, converting the sizes of each feature map to a unified feature map size. The fully connected network is used to fully connect the feature maps, which have been unified to the feature map size, and transmit them to the output layer.

[0042] The output layer of the processing model is used to output the processing results. The output layer includes multiple neurons, and the number of neurons in the output layer is the number of categories of the second plankton image.

[0043] In one feasible embodiment of the present application, the network structures of each processing branch are different. The larger the standard size of the second plankton image that the processing branch is used to process, the more complex the network structure of the processing branch. In this way, different processing branches can have different processing capabilities, and the more complex the network structure, the stronger the processing capability of the processing branch. Based on this, the second plankton image with a large standard size can be processed by a processing branch with strong processing capabilities, and the second plankton image with a small standard size can be processed by a processing branch with weak processing capabilities, thereby improving the overall processing capability of the processing model while avoiding waste of computing resources.

[0044] As an example, the network structure of any processing branch includes but is not limited to VGG16, ResNet34, ResNet50, MobileNetv2, EfficientNetB0, etc.; any processing branch can also add some additional modules to enhance the processing capability of the processing branch. These additional modules include but are not limited to Feature Pyramid Network, Context Module, ECA Net (Efficient Channel Attention), RPN Net (Extraction of Candidate Boxes), etc.

[0045] It is understood that the specific structure of any processing branch can be obtained by freely combining the various network structures and various additional modules provided in the above examples, and there is no specific limitation. Of course, based on such a free combination implementation, the number of convolutional layers, number of convolution kernels, convolution kernel size, and activation function type of different processing branches can all be different.

[0046] As an example, processing the size scaling network in the hidden layers of the model can be implemented by Spatial Pyramid Pooling.

[0047] Based on the structure of the processing model described in the above embodiment, in a feasible embodiment of the present application, processing the second plankton image by the processing model includes: determining a processing branch corresponding to the size of the second plankton image, and transmitting the second plankton image to the processing branch corresponding to the size of the second plankton image through an input layer of the processing model; wherein the processing branch is located in a hidden layer of the processing model; extracting a feature map of the second plankton image through a processing branch corresponding to the size of the second plankton image in a hidden layer of the processing model, scaling and fully connecting the feature map, and then transmitting the feature map to an output layer of the processing model; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; The feature map is processed by the output layer of the processing model to output the processing result of the second plankton image.

[0048] In a feasible embodiment of the present application, extracting a feature map of the second plankton image by processing a processing branch corresponding to the size of the second plankton image in a hidden layer of the processing model includes: Setting feature sharing between a first processing branch and a second processing branch through a feature sharing module; wherein the first processing branch is a processing branch corresponding to the size of the second plankton image, the second processing branch is a processing branch that does not correspond to the size of the second plankton image, the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer; When the second plankton image passes through the first convolutional layer, a first feature tensor of the second plankton image is extracted using the first convolution kernel, and a second feature tensor of the second plankton image is extracted using the second convolution kernel; wherein the first convolution kernel is a convolution kernel from the first convolutional layer, and the second convolution kernel is a convolution kernel from both the first and second convolutional layers; Fusing the first eigentensor and the second eigentensor to obtain a third eigentensor; A feature map of the second plankton image is determined according to the third feature tensor.

[0049] Specifically, continue to refer to Figure 2A feature sharing module is also provided in the hidden layer of the processing model. The feature sharing module is used to set feature sharing between the first processing branch and at least one second processing branch. The first processing branch is a processing branch corresponding to the standard size of the currently input second plankton image, and the second processing branch is a processing branch that does not correspond to the standard size of the currently input second plankton image. It can be seen that there can be multiple second processing branches.

[0050] At the same time, since each processing branch of the processing model can be used as the first processing branch to set a feature sharing module, it can be seen that there can be multiple feature sharing modules.

[0051] Specifically, the feature sharing module sets a first processing branch and at least one second processing branch to share features, the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer.

[0052] For example, the second plankton picture currently input is processed in processing branch A1, and the feature sharing module sets processing branches A2, A3 and A4 to share features with A1, then A1 is the first processing branch, and A2, A3 and A4 are all second processing branches.

[0053] After the feature sharing module sets feature sharing between the first processing branch and the second processing branch, when the second plankton image passes through the first convolution layer, the first feature tensor of the second plankton image is extracted by the first convolution kernel, and the first convolution kernel is the convolution kernel from the first convolution layer; at the same time, when the second plankton image passes through the first convolution layer, the second feature tensor of the second plankton image is extracted by the second convolution kernel, and the second convolution kernel is the convolution kernel from both the first convolution layer and the second convolution layer.

[0054] It should be noted that when the feature sharing module is set to perform feature sharing between the first processing branch and the second processing branch, the first processing branch may include multiple first convolutional layers, and feature sharing is performed when passing through each first convolutional layer.

[0055] After extracting the first feature tensor and the second feature tensor of the second plankton image, the first processing branch fuses the first feature tensor and the second feature tensor to obtain a third feature tensor of the second plankton image, and transmits the third feature tensor to the lower convolution layer of the first convolution layer for further processing until it is transmitted to the end of the first processing branch, and a feature map of the second plankton image is obtained from the third feature tensor.

[0056] If the first convolutional layer is located at the beginning of the first processing branch, the input of the first convolutional layer is the second plankton image, and the output feature tensor is output to the lower convolutional layer of the first convolutional layer, and continues to be processed in the first processing branch; if the first convolutional layer is located at the end of the first processing branch, the input of the first convolutional layer is the feature tensor output by the upper convolutional layer of the first convolutional layer, and the output feature map is output to the size scaling network; if the first convolutional layer is located in the middle of the first processing branch, the input of the first convolutional layer is the feature tensor output by the upper convolutional layer of the first convolutional layer, and the output feature tensor is output to the lower convolutional layer of the first convolutional layer, and continues to be processed in the first processing branch.

[0057] For example, refer to Figure 3 The Tensor (the original feature tensor) is processed in the first processing branch. When the Tensor passes through the first convolutional layer, it is processed by both the first and second convolutional kernels. The first convolutional kernel comes only from the first convolutional layer, while the second convolutional kernel comes from both the first and second convolutional layers. After processing by the first convolutional kernel, the Tensor outputs Tensor1 (the first feature tensor). After processing by the second convolutional kernel, the Tensor outputs Tensor2 (the second feature tensor). Tensor1 and Tensor2 are fused to obtain Tensor3 (the third feature tensor). After the Tensor is processed in the first convolutional layer, it continues to be transmitted to the other convolutional layers of the first processing branch until the feature map of the second plankton image corresponding to the Tensor is output at the end of the first processing branch.

[0058] It can be understood that since the second plankton image is processed by the first convolution kernel and the second convolution kernel at the same time, the second plankton image can refer to the features from the processing branch corresponding to its own standard size, and can also refer to the features from the processing branches corresponding to other standard sizes. Setting feature sharing between processing branches can ensure that the common features between similar images in different size categories can be shared between different processing branches, which is conducive to improving the classification effect of the processing model on plankton images.

[0059] In a feasible embodiment of the present application, setting feature sharing between the first processing branch and the second processing branch through the feature sharing module includes: Determining a first convolutional layer in the first processing branch and determining a second convolutional layer in the second processing branch; wherein a size of a convolution kernel included in the first convolutional layer is consistent with a size of a convolution kernel included in the second convolutional layer; Mark shared convolution kernels in the first convolution layer and the second convolution layer; wherein the number of shared convolution kernels is not greater than the total number of convolution kernels included in the first convolution layer, and is not greater than the total number of convolution kernels included in the second convolution layer; In the first convolutional layer, all convolution kernels except the shared convolution kernel are marked as non-shared convolution kernels. The non-shared convolution kernel is determined as the first convolution kernel, and the shared convolution kernel is determined as the second convolution kernel.

[0060] Specifically, the feature sharing module first determines the first convolution layer in the first processing branch and determines the second convolution layer in the second processing branch. The size of the convolution kernel included in the first convolution layer needs to be consistent with the size of the convolution kernel included in the second convolution layer.

[0061] After determining the first and second convolutional layers, the feature sharing module marks the shared convolution kernels in the first and second convolutional layers. The shared convolution kernels are any number of convolution kernels in the first convolutional layer and any number of convolution kernels in the second convolutional layer. The feature sharing module also marks all convolution kernels in the first convolutional layer other than the shared convolution kernels as non-shared convolution kernels.

[0062] It can be seen that the non-shared convolution kernel in the first convolution layer is the first convolution kernel, and the other convolution kernels except the shared convolution kernel in the first convolution layer and the other convolution kernels except the shared convolution kernel in the second convolution layer are the second convolution kernel.

[0063] S4: determining a classification result of the second plankton picture according to the processing result of the processing model, thereby determining a classification result of the first plankton picture corresponding to the second plankton picture; Specifically, the processing result output by the output layer of the processing model represents the classification result of the second plankton picture. After determining the classification result of the second plankton picture, the classification result of the first plankton picture corresponding to the second plankton picture can be determined, thereby completing the classification of the first plankton picture.

[0064] Based on the technical solutions provided in the various embodiments described above, first, the size category to which the size of the first plankton image belongs is identified, and image processing is performed based on the size category, thereby avoiding large-scale scaling of the first plankton image. Second, a processing model is configured to include multiple processing branches, so that the second plankton image is adaptively processed through the corresponding processing branches. In summary, the technical solutions provided in the embodiments of this application, based on the reasonable transformation of plankton images and the adaptive setting of the processing model, enhance the processing model's ability to process multi-sized plankton images, thereby improving the classification effect of multi-sized plankton images.

[0065] In a feasible embodiment of the present application, the processing model is trained before S1. The training process of the processing model specifically includes: Acquire multiple third plankton images; wherein the third plankton images are marked with classification results corresponding to the third plankton images; Dividing the plurality of third plankton images into a plurality of training data sets according to the sizes of the plurality of third plankton images; wherein one training data set includes a plurality of third plankton images of inconsistent sizes; Build an initial processing model; The initial processing model is trained multiple times using the training data set to obtain a processing model.

[0066] Specifically, a plurality of third plankton pictures are obtained, and the corresponding classification results have been marked on the third plankton pictures. The classification of the third plankton pictures can be performed manually by a technician.

[0067] A plurality of training data sets are determined, wherein one training data set includes a plurality of third plankton images with inconsistent sizes.

[0068] In a feasible embodiment of the present application, the third plankton pictures can be clustered according to the size of the third plankton pictures through a clustering algorithm to determine multiple clusters, where any cluster includes multiple third plankton pictures, and the third plankton pictures in the same cluster belong to the same size category; then the size of the third plankton pictures in a cluster is uniformly converted into a standard size corresponding to the size category corresponding to the current cluster; finally, multiple third plankton pictures of different standard sizes are selected to form a training set.

[0069] In one feasible embodiment of the present application, the clustering results of the third plankton image can be used as a basis for classifying the first plankton image into size categories. Specifically, by clustering the third plankton image, multiple clusters are determined, and these clusters serve as size categories. To determine the size category of the first plankton image, the distance from the first plankton image to the centroids of these clusters is simply calculated. When the distance from the first plankton image to the centroid of a cluster is the smallest, the first plankton image belongs to the size category corresponding to that cluster.

[0070] To ensure data balance, each training dataset is subjected to data balancing. For example, data balancing can include assigning specific weights to the third plankton images in all training datasets to be balanced, so that the total weights of all balanced training datasets remain consistent. Alternatively, data balancing can include duplicating the third plankton images in training datasets to be balanced that have fewer third plankton images, so that the total number of images in all balanced training datasets remains consistent.

[0071] To ensure training efficiency, the third plankton image was normalized and standardized, limiting statistical quantities such as the maximum, minimum, mean, and variance of the image pixels to a certain range. As an example, normalization shifted the pixel values ​​from the range [0, 255] to the range [-1, 1]. Standardization transformed the data into a distribution with a mean of 0 and a standard deviation of 1.

[0072] Set the parameters of the initial processing model and build the initial processing model. As an example, setting the parameters of the initial processing model includes, but is not limited to, setting the stride, padding, learning rate, momentum, loss function, optimization algorithm, mini-batch size, number of training epochs, and whether to use regularization technique.

[0073] After the initial processing model is constructed, it is trained multiple times using multiple training data sets to obtain a processing model.

[0074] The processing model derived from the initial processing model requires not only training but also validation and testing. Therefore, it is necessary to construct a test dataset and a validation dataset, and divide the third plankton images converted to a standard size into the test dataset and validation dataset.

[0075] In a feasible embodiment of the present application, after dividing the plurality of third plankton images into a plurality of training data sets according to the sizes of the plurality of third plankton images, the method further includes: According to the preset division ratio, several training data sets are determined as validation data sets, and several training data sets are determined as test data sets.

[0076] In some specific examples, the preset split ratio is set by the technician. For example, if the split ratio is currently set to 20% and 10%, 20% of the training dataset will be used as a validation dataset to verify the trained initial processing model, and 10% of the training dataset will be used as a test dataset to test the trained initial processing model.

[0077] In a feasible embodiment of the present application, the initial processing model includes N initial processing branches, the training data set is divided into N sub-training data sets, and the sub-training data sets include multiple third plankton images of uniform size, where N is an integer greater than or equal to 2. The initial processing model is trained multiple times using the training data sets, including: During one training, N sub-training data sets in one training data set are simultaneously input into N initial processing branches to train the initial processing branches into processing branches; The initial processing model is trained multiple times using multiple training data sets to obtain a processing model.

[0078] Specifically, the initial processing model is repeatedly trained multiple times to obtain a processing model. The initial processing model includes N initial processing branches, and the training dataset is divided into N sub-training datasets. When the initial processing model is trained once, the N sub-training datasets in one training dataset are simultaneously input into the N initial processing branches to train the initial processing branches into processing branches.

[0079] It can be seen that when the initial processing model is trained once, the total number of third plankton images M that need to be input to the initial processing model = the size of the sub-training dataset mini-batch size × the number of initial processing branches N.

[0080] When training the initial processing model, different initial processing branches are trained through different sub-training data sets, and one initial processing branch is trained through a third plankton image of the same standard size. In this way, the initial processing branch can extract the features of the standard-sized image of the currently input third plankton image, and the processing branch can extract the feature map of the second plankton image of the standard size.

[0081] The structure of the initial processing model is consistent with the processing model, and it also has a feature sharing module. Therefore, when the initial processing model is trained once, since the feature sharing module sets the feature sharing between the initial processing branches, each initial processing branch can understand some features of the initial processing branches used to process other standard sizes during training, so that the processing model after training can realize feature sharing between processing branches.

[0082] Figure 4 This is a schematic diagram of a device for classifying multi-size plankton images provided in an embodiment of the present application. Corresponding to the above method embodiment, the present application embodiment also provides a device for classifying multi-size plankton images, referring to Figure 4 , the device specifically includes: A first acquisition module 401 is used to acquire a first plankton picture; A first conversion module 402 is configured to convert the first plankton picture into a second plankton picture according to a size category to which the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; a processing module 403 for inputting the second plankton image into a processing model and processing the second plankton image using the processing model; wherein the processing model includes a plurality of processing branches, each of which is used to process second plankton images of different sizes; The determination module 404 is configured to determine the classification result of the second plankton picture according to the processing result of the processing model, thereby determining the classification result of the first plankton picture corresponding to the second plankton picture.

[0083] In a feasible embodiment of the present application, the first conversion module 402 includes: a determining unit, configured to determine a standard size corresponding to a size category to which the size of the first plankton image belongs; wherein the standard size corresponding to the size category to which the size of the first plankton image belongs is an average size or a maximum size of each first plankton image in the size category to which the size of the first plankton image belongs; a first conversion unit, configured to compress or stretch the first plankton picture to convert the first plankton picture into a second plankton picture; or The second conversion unit is configured to add pixels with a pixel value of 0 to the first plankton picture to convert the first plankton picture into a second plankton picture.

[0084] In a feasible embodiment of the present application, the processing module 403 includes: a first processing unit configured to determine a processing branch corresponding to a size of the second plankton image, and transmit the second plankton image to the processing branch corresponding to the size of the second plankton image through an input layer of the processing model; wherein the processing branch is located in a hidden layer of the processing model; a second processing unit, configured to extract a feature map of the second plankton image through a processing branch corresponding to a size of the second plankton image in a hidden layer of the processing model, and scale and fully connect the feature map before transmitting it to an output layer of the processing model; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; The third processing unit is configured to process the feature map through an output layer of the processing model and output a processing result of the second plankton image.

[0085] In a feasible embodiment of the present application, a feature sharing module is set in the hidden layer of the processing model, and the second processing unit includes: a setting subunit, configured to set feature sharing between a first processing branch and a second processing branch through a feature sharing module; wherein the first processing branch is a processing branch corresponding to a size of the second plankton image, the second processing branch is a processing branch not corresponding to the size of the second plankton image, the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer; an extraction subunit, configured to extract, when the second plankton image passes through the first convolutional layer, a first feature tensor of the second plankton image using a first convolutional kernel, and extract a second feature tensor of the second plankton image using a second convolutional kernel; wherein the first convolutional kernel is a convolutional kernel from the first convolutional layer, and the second convolutional kernel is a convolutional kernel from both the first and second convolutional layers; a fusion subunit, configured to fuse the first feature tensor and the second feature tensor to obtain a third feature tensor; The determining subunit is configured to determine a feature map of the second plankton image according to the third feature tensor.

[0086] In a feasible embodiment of the present application, a subunit is provided, including: a first determining unit, configured to determine a first convolutional layer in the first processing branch and a second convolutional layer in the second processing branch; wherein a size of a convolution kernel included in the first convolutional layer is consistent with a size of a convolution kernel included in the second convolutional layer; a first marking unit, configured to mark shared convolution kernels in the first convolution layer and the second convolution layer; wherein the number of shared convolution kernels is no greater than the total number of convolution kernels included in the first convolution layer, and no greater than the total number of convolution kernels included in the second convolution layer; The second marking unit is used to mark the convolution kernels other than the shared convolution kernels in the first convolution layer as non-shared convolution kernels; The second determination unit is used to determine the non-shared convolution kernel as the first convolution kernel, and determine the shared convolution kernel as the second convolution kernel.

[0087] In a feasible embodiment of the present application, the device further includes: A second acquisition module is configured to acquire a plurality of third plankton images, wherein the third plankton images are marked with classification results corresponding to the third plankton images; a division module, configured to divide the plurality of third plankton images into a plurality of training data sets according to the sizes of the plurality of third plankton images; wherein one training data set includes a plurality of third plankton images of inconsistent sizes; A building module for constructing an initial processing model; The training module is used to train the initial processing model multiple times using the training data set to obtain the processing model.

[0088] In a feasible embodiment of the present application, the initial processing model includes N initial processing branches, the training data set is divided into N sub-training data sets, and the sub-training data sets include multiple third plankton images of uniform size, where N is an integer greater than or equal to 2. The training module includes: An input unit, configured to simultaneously input N sub-training data sets in a training data set into N initial processing branches during one training, so as to train the initial processing branches into processing branches; The training unit is used to train the initial processing model multiple times using multiple training data sets to obtain a processing model.

[0089] like Figure 5 As shown, an embodiment of the present application provides an electronic device, including a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. Memory 503, used for storing computer programs; In one embodiment of the present application, the processor 501 is configured to execute a program stored in the memory 503 to implement a method for classifying multi-sized plankton images provided by any of the aforementioned method embodiments, for example, including: Get the first plankton picture; converting the first plankton picture into a second plankton picture according to the size category to which the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; Inputting the second plankton image into the processing model, and processing the second plankton image by the processing model; wherein the processing model includes a plurality of processing branches, and the processing branches are respectively used to process the second plankton images of different sizes; The classification result of the second plankton picture is determined according to the processing result of the processing model, thereby determining the classification result of the first plankton picture corresponding to the second plankton picture.

[0090] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a method for classifying multi-sized plankton images provided in any of the aforementioned method embodiments are implemented.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0092] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a general hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0093] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0094] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A method for classifying multi-size plankton images, characterized in that: The method comprises: Get the first plankton picture; converting the first plankton picture into a second plankton picture according to the size category to which the size of the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; Inputting the second plankton picture into a processing model, and processing the second plankton picture by the processing model; wherein the processing model includes a plurality of processing branches, and the processing branches are respectively used to process the second plankton pictures of different sizes; The classification result of the second plankton picture is determined according to the processing result of the processing model, thereby determining the classification result of the first plankton picture corresponding to the second plankton picture.

2. The method according to claim 1, characterized in that Converting the first plankton picture into a second plankton picture according to a size category to which a size of the first plankton picture belongs includes: Determine the standard size corresponding to the size category to which the size of the first plankton picture belongs; wherein the standard size corresponding to the size category to which the size of the first plankton picture belongs is the average size or maximum size of each of the first plankton pictures in the size category to which the size of the first plankton picture belongs; compressing or stretching the first plankton picture to convert the first plankton picture into the second plankton picture; or, Add pixels with a pixel value of 0 to the first plankton picture to convert the first plankton picture into the second plankton picture.

3. The method according to claim 1, characterized in that Processing the second plankton picture by the processing model includes: Determine the processing branch corresponding to the size of the second plankton picture, and transmit the second plankton picture to the processing branch corresponding to the size of the second plankton picture through the input layer of the processing model; wherein the processing branch is located in the hidden layer of the processing model; Extracting a feature map of the second plankton image through the processing branch corresponding to the size of the second plankton image in the hidden layer of the processing model, and transmitting the feature map to the output layer of the processing model after scaling and fully connecting the feature map; wherein the size of the feature map transmitted to the output layer of the processing model is a preset feature map size; The feature map is processed by the output layer of the processing model, and the processing result of the second plankton picture is output.

4. The method according to claim 3, characterized in that A feature sharing module is set in the hidden layer of the processing model, and a feature map of the second plankton picture is extracted through the processing branch corresponding to the size of the second plankton picture in the hidden layer of the processing model, including: Setting feature sharing between a first processing branch and a second processing branch through the feature sharing module; wherein the first processing branch is the processing branch corresponding to the size of the second plankton picture, the second processing branch is the processing branch not corresponding to the size of the second plankton picture, the first processing branch includes a first convolutional layer, and the second processing branch includes a second convolutional layer; When the second plankton picture passes through the first convolution layer, a first feature tensor of the second plankton picture is extracted by a first convolution kernel, and a second feature tensor of the second plankton picture is extracted by a second convolution kernel; wherein the first convolution kernel is a convolution kernel from the first convolution layer, and the second convolution kernel is a convolution kernel from both the first convolution layer and the second convolution layer; fusing the first feature tensor and the second feature tensor to obtain a third feature tensor; The feature map of the second plankton picture is determined according to the third feature tensor.

5. The method according to claim 4, characterized in that Setting feature sharing between the first processing branch and the second processing branch through the feature sharing module includes: Determining the first convolutional layer in the first processing branch, and determining the second convolutional layer in the second processing branch; wherein the size of the convolutional kernel included in the first convolutional layer is consistent with the size of the convolutional kernel included in the second convolutional layer; Marking shared convolution kernels in the first convolution layer and the second convolution layer; wherein the number of the shared convolution kernels is not greater than the total number of convolution kernels included in the first convolution layer, and is not greater than the total number of convolution kernels included in the second convolution layer; Marking convolution kernels other than the shared convolution kernel in the first convolution layer as non-shared convolution kernels; The non-shared convolution kernel is determined to be the first convolution kernel, and the shared convolution kernel is determined to be the second convolution kernel.

6. The method according to claim 1, characterized in that Before acquiring the first plankton picture, the method further includes: Acquire a plurality of third plankton pictures; wherein the third plankton pictures are marked with the classification results corresponding to the third plankton pictures; Dividing the plurality of third plankton pictures into a plurality of training data sets according to the sizes of the plurality of third plankton pictures; wherein one of the training data sets includes a plurality of third plankton pictures of inconsistent sizes; constructing an initial processing model; The initial processing model is trained multiple times using the training data set to obtain the processing model.

7. The method according to claim 6, characterized in that The initial processing model includes N initial processing branches, each of the training data sets is divided into N sub-training data sets, and the sub-training data sets include multiple third plankton pictures with the same size, N is an integer greater than or equal to 2, and the initial processing model is trained multiple times through the training data sets, including: During one training, N sub-training data sets in one training data set are simultaneously input into N initial processing branches to train the initial processing branches into the processing branches; The initial processing model is trained multiple times using multiple training data sets to obtain the processing model.

8. A device for classifying multi-size plankton images, characterized in that: The device comprises: A first acquisition module, used to acquire a first plankton picture; A first conversion module, configured to convert the first plankton picture into a second plankton picture according to a size category to which the size of the first plankton picture belongs; wherein the size of the second plankton picture is a standard size corresponding to the size category of the first plankton picture; a processing module, used for inputting the second plankton picture into a processing model, and processing the second plankton picture by the processing model; wherein the processing model includes a plurality of processing branches, and the processing branches are respectively used for processing the second plankton pictures of different sizes; A determination module is used to determine the classification result of the second plankton picture according to the processing result of the processing model, so as to determine the classification result of the first plankton picture corresponding to the second plankton picture.

9. An electronic device, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to implement the method for classifying multi-sized plankton images as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the classification method of multi-size plankton pictures as described in any one of claims 1-7.

Images