Deep learning-based shrimp larvae counting method, device, and storage medium

Abstract

This invention provides a deep learning-based method, apparatus, and storage medium for counting shrimp larvae. The method includes: S1, acquiring shrimp larvae images; S2, integrating, cropping, and / or overlaying shrimp larvae images at different magnifications and inputting them into a deep learning model, performing target recognition of shrimp larvae in the images through multi-feature fusion, and outputting the identified initial shrimp larvae targets; S3, identifying the contextual relationship features between shrimp larvae targets based on the original target features of the initial shrimp larvae targets, and fusing the original target features of the initial shrimp larvae targets and the identified contextual relationship features between shrimp larvae targets to obtain shrimp larvae fusion features; S4, inputting the shrimp larvae fusion features into an SVM classification and position regression model for processing to obtain the final shrimp larvae targets for counting. Using the above technical solution, relatively accurate automatic counting of a large number of small, easily clustered, and overlapping individual shrimp larvae can be achieved.

Description

Technical Field

[0001] This invention relates to the field of automatic counting, and in particular to a shrimp larvae counting method, apparatus, and storage medium based on deep learning. Background Technology

[0002] King Kong shrimp larvae are tiny and slender, and when there are many of them, they are difficult to distinguish and count with the naked eye. Manual counting methods have significant errors. Therefore, accurately estimating the number of King Kong shrimp larvae is an important issue in the shrimp farming industry.

[0003] Currently, there are methods on the market that use OpenCV machine vision to identify the number of black dots in shrimp larvae images for counting and statistical purposes. This method has acceptable accuracy when measuring fewer than 5000 Litopenaeus vannamei larvae, but its accuracy drops when the number exceeds 5000 or when counting king prawn larvae. The low accuracy is due to the small size of king prawn larvae, their tendency to cluster and overlap, and the increased overlap as their numbers increase, resulting in lines or clumps in the image. OpenCV machine vision cannot accurately identify the number of black dots, therefore, current shrimp larvae counting solutions cannot accurately count king prawn larvae and cannot meet the actual needs of hatcheries. Summary of the Invention

[0004] The embodiments of the present invention provide a deep learning-based shrimp larvae counting method, device and storage medium to improve the accuracy of automatic counting of shrimp larvae such as king prawn larvae that are small, elongated, numerous and easily aggregated.

[0005] On the one hand, a deep learning-based shrimp larvae counting method is provided, including:

[0006] S1, Get shrimp larvae images;

[0007] S2, integrate, crop and / or overlay shrimp larvae images at different magnifications and input them into a deep learning model, perform target recognition of shrimp larvae in the shrimp larvae images through multi-feature fusion, and output the initial shrimp larvae target identified;

[0008] S3, Identify the contextual relationship features between shrimp seedling targets based on the original target features of the initial shrimp seedling targets, and fuse the original target features of the initial shrimp seedling targets and the identified contextual relationship features between shrimp seedling targets to obtain shrimp seedling fusion features;

[0009] S4. The shrimp larvae fusion features are input into the SVM classification and position regression model for processing to obtain the final shrimp larvae target for counting.

[0010] Preferably, the shrimp larvae counting method further includes, after obtaining the shrimp larvae image, the step of obtaining shrimp larvae feature maps at five scales (4, 8, 16, 32, and 64 times downsampled from the shrimp larvae image).

[0011] In step S2, the feature maps of shrimp larvae at five different scales are input into the YOLOv5 model for initial target recognition.

[0012] Preferably, the shrimp larvae counting method further includes a step of enhancing the shrimp larvae feature maps at the five scales before inputting them into the YOLOv5 model;

[0013] The enhancements include one or more of the following:

[0014] (1) The feature maps of shrimp larvae at five scales are enhanced using pixel-level methods, including Gaussian blur, aliasing, splicing, cropping, rotation, inversion and / or aspect ratio.

[0015] (2) Enhance the feature images of shrimp larvae at five different scales in terms of supplemental lighting, including hue, saturation and / or exposure.

[0016] Preferably, in the shrimp larvae counting method, wherein,

[0017] The original characteristics of the target shrimp larvae include the target's external shape and / or geometric features; among which, external shape features include size, head color depth and / or head orientation; geometric features include position and / or length.

[0018] Preferably, in the shrimp larvae counting method, the contextual relationship features between shrimp larvae targets include: for a selected shrimp larvae target, the positional relationship between other shrimp larvae targets related to the position of the selected shrimp larvae target and the selected shrimp larvae target, and one or more original target features of other shrimp larvae targets.

[0019] Preferably, in the shrimp larvae counting method, the number of convolutional layers and parameters of the deep learning model are reduced to a predetermined number; wherein the deep learning model uses 5s as the main network model, and in terms of network depth, the first CSP1 uses 1 residual component, and the second CSP2 uses 2×X=2×1=2 convolutions, where X=1.

[0020] Preferably, in the shrimp larvae counting method, step S2 uses the CIOU_LOSS function as the loss function for the bounding box.

[0021] Preferably, in the shrimp larvae counting method, the pre-trained model is input through the input terminal FPN+PAN structure, and an automatically labeled file is generated and automatically labeled.

[0022] On the other hand, a deep learning-based shrimp larvae counting device is provided, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement any of the methods described above.

[0023] In another aspect, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement any of the methods described above.

[0024] The above technical solution has the following technical effects:

[0025] The technical solution of this invention uses a deep learning model to perform multi-feature fusion analysis on shrimp larvae images and considers the contextual association features between shrimp larvae targets. It solves the problem of shrimp larvae being clustered or overlapping in lines or clumps and cannot be accurately identified and located. It realizes automatic counting of a large number of small and slender shrimp larvae, such as king prawn larvae, with high accuracy and fast counting speed, and is ready for market promotion. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the deep learning-based shrimp larvae counting method according to an embodiment of the present invention.

[0027] Figure 2 This is a flowchart illustrating a deep learning-based shrimp larvae counting method according to another embodiment of the present invention.

[0028] Figure 3 Here is an example of a shrimp larvae image obtained by the shrimp larvae counting method in this embodiment of the invention;

[0029] Figure 4 This is a schematic diagram illustrating the principle of target detection in a shrimp larvae counting method according to an embodiment of the present invention, which combines contextual association features between shrimp larvae targets.

[0030] Figure 5 This is a schematic diagram of the parameters of the YOLOv5 model used in a shrimp larvae counting method according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of the CSP parameters of the YOLOv5 model used in a shrimp larvae counting method according to an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of the shrimp seedling counting device according to an embodiment of the present invention. Detailed Implementation

[0033] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0034] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0035] Example 1:

[0036] Figure 1 This is a flowchart illustrating a deep learning-based shrimp larvae counting method according to an embodiment of the present invention. Figure 1 The shrimp larvae counting method in this embodiment includes:

[0037] S1, Get shrimp larvae images;

[0038] S2, integrate, crop and / or overlay shrimp larvae images at different magnifications and input them into a deep learning model, perform target recognition of shrimp larvae in the shrimp larvae images through multi-feature fusion, and output the initial shrimp larvae target identified;

[0039] S3, Identify the contextual relationship features between shrimp seedling targets based on the original features of the initial shrimp seedling targets, and fuse the original features of the initial shrimp seedling targets and the identified contextual relationship features between the shrimp seedling targets to obtain the fused shrimp seedling fusion features.

[0040] S4. The shrimp larvae fusion features are input into the SVM classification and position regression model for processing to obtain the final shrimp larvae target for counting.

[0041] Preferably, the original characteristics of the shrimp larvae target are pre-selected original characteristics of the target; for example, the original characteristics of the target are characteristics that the target itself has, such as: the target's shape characteristics and / or geometric characteristics; wherein, the above-mentioned shape characteristics include size, head color depth and / or head orientation; the above-mentioned geometric characteristics include position and / or length.

[0042] Preferably, when determining the contextual relationship between shrimp larvae, for a selected shrimp larvae target, the contextual relationship between the selected shrimp larvae target and its related shrimp larvae targets is determined; typically, the relationship here is a positional relationship, such as a predetermined range or area around or near the selected shrimp larvae target; preferably, the contextual relationship features between shrimp larvae targets include: the positional relationship between the selected shrimp larvae target and other shrimp larvae targets whose positions are related to the selected shrimp larvae target, and one or more original target features of the aforementioned other shrimp larvae targets.

[0043] Compared with existing technologies, the shrimp larvae counting method of this invention does not directly use the initial shrimp larvae targets identified by the deep learning model for counting. Instead, it further identifies the contextual relationships of the shrimp larvae targets based on the identification of the initial shrimp larvae targets. This allows for the extraction of more subtle features of the shrimp larvae targets, resulting in more accurate feature information about the targets. This avoids the coarseness of existing counting methods, such as avoiding misidentification of multiple stacked or clustered shrimp larvae as the same shrimp larvae. It also improves the accuracy of counting large numbers of small, easily clustered shrimp larvae, such as king prawn larvae, which are difficult to accurately identify and locate due to their clustering into lines or clumps.

[0044] Example 2:

[0045] In this embodiment, a high-resolution camera is used to capture a frame of King Kong shrimp seedling image and upload it to a sample database. A deep learning model is used to identify the features of the King Kong shrimp seedling. The addition of a YOLO convolutional neural network instead of a traditional convolutional neural network can better extract the features of the King Kong shrimp seedling, while also reducing the number of model parameters. Figure 2 A flowchart illustrating the deep learning-based shrimp larvae counting method of this embodiment of the invention is shown. Figure 2 The counting method in this embodiment specifically includes the following steps:

[0046] Step 1: Collect images of king shrimp larvae to obtain king shrimp larvae pictures;

[0047] Step 2, optimization for YOLOv5; This embodiment uses the YOLOv5 model for target recognition. To solve the technical problem of this invention, this embodiment makes the following optimizations based on the characteristics of YOLOv5:

[0048] (1) Preprocessing and enhancing images of shrimp larvae, including:

[0049] a) Pixel-level enhancements, such as using Gaussian blur, aliasing, stitching, cropping, rotation, and aspect ratio reversal to enhance the feature images of king prawn larvae;

[0050] b. Enhancement in supplementary lighting, such as using methods related to supplementary lighting, such as hue, saturation, and exposure, to enhance the feature images of shrimp larvae can significantly improve the texture features of king shrimp larvae. When the number of king shrimp larvae increases, the ability to read and distinguish this feature can be improved.

[0051] (2) Design convolutional layers and use the YOLOv5 algorithm with a multi-feature fusion scheme to identify shrimp larvae targets in shrimp larvae images. The multi-feature fusion scheme integrates, crops and superimposes images of king shrimp larvae at different magnifications and inputs them into the algorithm for recognition and analysis. By fusing and analyzing multiple king shrimp larvae features, the target localization can be achieved, which can improve the counting accuracy.

[0052] Specifically, the typical YOLOv5 model uses feature maps downsampled by 8 times to detect small targets. This means that if the pixel size of a juvenile king shrimp image is smaller than 8×8, it will be difficult to detect, and the localization performance for densely packed, small, and elongated juvenile king shrimp is not ideal. Therefore, an improved feature fusion scheme is proposed. The path aggregation network in YOLOv5 is adjusted, taking five juvenile king shrimp feature maps downsampled by different factors, such as 4, 8, 16, 32, and 64 times, as input nodes instead of the original single feature map input node. This allows for the fusion of more useful juvenile king shrimp feature information without significantly increasing computational cost. By repeatedly stacking and learning juvenile king shrimp to achieve deep fusion analysis of shrimp features, the recognition and localization accuracy when juvenile king shrimp are clustered can be improved. In the specific implementation, shrimp features include: shrimp head and / or shrimp body texture features; others may include shrimp head and / or shrimp body shape features and / or geometric features; among which, shape features include size, head color depth, and / or head orientation; geometric features include position and / or length.

[0053] (3) The context-related information small target detection method is adopted, which can effectively handle shrimp larvae targets in crowded situations and improve the ability to extract shrimp larvae feature information:

[0054] Traditional shrimp larvae counting methods detect each target individually, neglecting the correlation between shrimp larvae. This embodiment employs a method to extract the correlation between different king shrimp larvae. Specifically, each king shrimp larvae feature is divided into shape features such as size, head color depth, and head orientation, as well as geometric features such as position and length. This step uses 2-3 features from all foreground targets (i.e., selected shrimp larvae targets) as input nodes to obtain the relationship features between different shrimp larvae. These features are then concatenated and fused with the original feature information of the shrimp larvae targets to obtain the shrimp larvae fusion feature. This fusion feature is used as the final feature for king shrimp larvae detection, thereby enhancing the feature information of king shrimp larvae and improving the accuracy of shrimp larvae identification and counting. Preferably, the contextual correlation includes the positional relationship between shrimp larvae, etc.

[0055] The following example illustrates how this step is implemented. Figure 3 Here is an example image of shrimp larvae, where the black areas represent shrimp larvae. Figure 4This is a schematic diagram illustrating the principle of target detection in a shrimp larvae counting method according to an embodiment of the present invention, which combines contextual features between shrimp larvae targets. Figure 5 and Figure 6 These are the parameters of the YOLOv5 model used in the embodiments of this invention. Figure 3 Taking the two shrimp larvae, A and B, as shown in the middle as an example, combined with Figure 4 This section explains how to detect small targets using contextually relevant information and obtain fused features. Specifically:

[0056] In this example, the head orientation is selected as the target original feature, that is, the original feature of the shrimp larvae. Specifically, shrimp A's head faces the upper right (T1), and shrimp B's head faces the lower left (T2).

[0057] In this example, there is a small horizontal shrimp larva directly above shrimp larva A. For shrimp larva A, the contextual relationship feature (G1) is "there is a small horizontal shrimp larva directly above shrimp larva A". Below shrimp larva B, there is a shrimp larva with its head facing upward to the right. For shrimp larva B, the contextual relationship feature (G2) is "there is a shrimp larva with its head facing upward to the right below shrimp larva B". In specific implementation, the contextual relationship features can be obtained through existing contextual association information small target detection methods; and / or, the shrimp larva target recognition results, i.e., target recognition images, output by machine learning models such as target detection models, can be manually summarized based on the contextual relationships, such as positional relationships, between shrimp larvae shown in the target results.

[0058] For example, for a given shrimp larvae target image, i.e., the shrimp larvae target result output by a machine learning model such as an object detection model, ROI regions of 1.0× and 1.5× are selected, such as... Figure 4 As shown in the upper and lower branches, the target original features (T1) and target context features (G1) are extracted respectively, which can fuse detailed information (conv3) and semantic information (conv5). The target original features (T1) and target context features (G1) are fused through the HRBM model to obtain the final feature (T1+G1) used for detection. This final feature fuses the original features of the shrimp larva and the contextual relationship features with the surrounding shrimp larvae. This final feature is helpful to distinguish the selected target, such as shrimp larva A, from the surrounding shrimp larvae with similar original features, which can avoid misidentification in the case of shrimp larvae overlapping or clustering. Then, the original and context-fused features (T1+G1) are input into the SVM classification and position regression models respectively to realize target category prediction and bounding box position correction, thereby improving the accuracy of small target detection and recognition.

[0059] (4) Reduce the number of convolutional layers and parameters to lower computational costs:

[0060] A 5s main network model was adopted. In terms of network depth, the first CSP1 used one residual component, and the second CSP2 used 2×X = 2×1 = 2 convolutions. Since X = 1, only one set of convolutions was used. This method reduces the computational load of the algorithm and improves the recognition and counting speed.

[0061] S5. Optimize the loss function: Use CIOU_LOSS as the loss function for the bounding box to optimize the object detection process. CIOU_LOSS considers the scale information of the aspect ratio of the bounding box, which significantly improves the accuracy of object detection and recognition.

[0062] CIOU_LOSS considers the aspect ratio of bounding boxes in addition to the overlapping area between detection boxes and ground truth boxes, the problem of non-overlapping bounding boxes, and the distance information of bounding box center points. It performs CIOU matching on the same target regression box pairs in adjacent shrimp seedling image frames. Targets with high similarity are identified as the same target, thus greatly improving the accuracy of target detection and recognition, and improving the accuracy of King Kong shrimp seedling recognition and counting.

[0063] S6. Automatic annotation:

[0064] To address the challenges of small, slender, and numerous male shrimp larvae that tend to cluster and overlap, and the time-consuming and inaccurate nature of manual labeling, this paper proposes an input-side FPN+PAN structure that aggregates parameters based on predicted feature maps and strong localization feature maps to complete the pre-trained model input, generating an automatically labeled file and completing the automatic labeling process. This step also helps improve the accuracy of shrimp larvae identification and counting.

[0065] Step 3: Train using the optimized YOLOv5;

[0066] Step 4: Use the target recognition results to count, and further analyze the counting results.

[0067] Once the model is trained, it can be used to automatically count shrimp larvae in real-world scenarios, such as king prawn larvae.

[0068] Example 3:

[0069] This invention also provides a shrimp larvae counting device based on deep learning, such as... Figure 7 As shown, the device includes a processor 701, a memory 702, a bus 703, and a computer program stored in the memory 702 and executable on the processor 701. The processor 701 includes one or more processing cores. The memory 702 is connected to the processor 701 via the bus 703. The memory 702 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above-described method embodiment of Embodiment 1 of the present invention.

[0070] Furthermore, as an executable solution, the shrimp larvae counting device can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0071] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0072] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, SmartMediaCard (SMC), Secure Digital (SD) card, FlashCard, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0073] Example 4:

[0074] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the embodiments of the present invention.

[0075] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0076] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A deep learning-based method for counting shrimp larvae, characterized in that, include: S1, Get shrimp larvae image; S2, integrate, crop and / or overlay the shrimp larvae images at different magnifications and input them into a deep learning model, perform target recognition of the shrimp larvae in the shrimp larvae images through multi-feature fusion, and output the recognized initial shrimp larvae target; S3, based on the original target features of the initial shrimp larvae target, identify the contextual relationship features between shrimp larvae targets, and fuse the original target features of the initial shrimp larvae target and the identified contextual relationship features between the shrimp larvae targets to obtain shrimp larvae fusion features; the original target features of the shrimp larvae target include the target's shape features and / or geometric features; wherein, the shape features include size, head color depth and / or head orientation; the geometric features include position and / or length; the contextual relationship features between shrimp larvae targets include: for a selected shrimp larvae target, the positional relationship between other shrimp larvae targets related to the position of the selected shrimp larvae target and the selected shrimp larvae target, and one or more original target features of the other shrimp larvae targets; S4. The shrimp larvae fusion features are input into the SVM classification and position regression model for processing to obtain the final shrimp larvae target for counting.

2. The shrimp larvae counting method according to claim 1, characterized in that, After obtaining the shrimp larvae image, the process also includes obtaining shrimp larvae feature maps at five scales (4, 8, 16, 32, and 64 times downsampled from the shrimp larvae image). In step S2, the feature maps of the five scales of shrimp larvae are input into the YOLOv5 model for initial target recognition.

3. The shrimp larvae counting method according to claim 2, characterized in that, Before inputting the five scales of shrimp larvae feature maps into the YOLOv5 model, the process also includes a step of enhancing the five scales of shrimp larvae feature maps; The enhancements include one or more of the following: (1) The feature maps of the five scales of shrimp larvae are enhanced using pixel-level methods, including Gaussian blur, aliasing, stitching, cropping, rotation, inversion, and / or aspect ratio; (2) Enhance the five scales of shrimp larvae feature images in terms of supplemental lighting, including hue, saturation and / or exposure.

4. The shrimp larvae counting method according to claim 1, characterized in that, The number of convolutional layers and parameters in the deep learning model is reduced to a predetermined number; wherein, the deep learning model uses 5s as the main network model, and in terms of network depth, the first CSP1 uses 1 residual component, and the second CSP2 uses 2×X=2×1=2 convolutions, where X=1.

5. The shrimp larvae counting method according to claim 1, characterized in that, In step S2, the CIOU_LOSS function is used as the loss function for the bounding box.

6. The shrimp larvae counting method according to claim 1, characterized in that, The pre-trained model is input through the FPN+PAN structure at the input end, and an automatically labeled file is generated and automatically labeled.

7. A device for counting shrimp larvae based on deep learning, characterized in that, It includes a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is executed by a processor to implement the method as described in any one of claims 1 to 6.

Images