Aquatic organism multi-target detection method and device based on YOLO, terminal and storage medium

Abstract

This application provides a YOLO-based method, apparatus, terminal, and storage medium for multi-target detection of aquatic organisms. The method includes: acquiring an image of an aquatic organism to be detected; inputting the aquatic organism image into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by multiple groups of predicted bounding boxes in the first detection result; performing non-maximum suppression processing on the first detection result to extract the optimal group of predicted bounding boxes from the multiple groups of predicted bounding boxes to obtain a second detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by a single optimal group of predicted bounding boxes in the second detection result; wherein the group of predicted bounding boxes consists of predicted bounding boxes, and each predicted bounding box corresponds one-to-one with a part of the aquatic organism. This application can accurately and efficiently detect each part of the aquatic organism, and the selected parts always possess completeness and uniqueness, facilitating in-depth research on the living habits and behaviors of aquatic organisms.

Description

Technical Field

[0001] This application relates to the field of target detection, and in particular to a method, device, terminal and storage medium for multi-target detection of aquatic organisms based on YOLO. Background Technology

[0002] YOLO is one of the most commonly used anchor-bound object detection algorithms capable of performing object localization and classification in one step. It boasts excellent detection efficiency and recognition accuracy, and has a wide range of applications. Conventional YOLO algorithms can only select rectangular boxes for the target object. Therefore, many tilted-box detection algorithms have been proposed, which obtain the target's angle simultaneously with detection, allowing for more accurate target selection. The main YOLO model framework uses CSPDarknet as the backbone to extract image features. Through Pannet and the detection module, it outputs the target's bounding box coordinates confidence and classification information, enabling detection of targets of different sizes across three feature maps. While conventional YOLO outputs the coordinates of a regular rectangular target box, the rotated-box YOLO model using circular smooth labels (CSL) outputs a target box with rotation angle information, including the target's center point coordinates, the lengths of the long and short sides, and the rotation angle. Circular smooth labeling divides angles into 180 classes per degree, using classification to find the target angle within the model; its essence is a classification-based approach rather than regression. The advantages of the YOLO model for rotating bounding boxes are more accurate target selection, especially for tilted targets, and smaller bounding area to reduce the error of the intersection-over-union (IoU) ratio of two rectangles in non-maximum suppression (NMS).

[0003] However, current YOLO algorithms can generally only output the bounding box information of a single target. For objects composed of multiple targets with combined features, YOLO can only output the recognition results multiple times, unable to simultaneously select different parts of the object in a single detection. For example, to detect the head, body, and tail of a fish, multiple detection tasks need to be constructed separately. This detection method often results in poor performance, for example... Figure 1 As shown, when the detection confidence of some targets is low, the complete target may not be output. This may also be due to false detections or inappropriate parameter settings. Figure 2The detection results show multiple partial targets. Therefore, the existing YOLO algorithm can only identify the fish as a whole in the image to be detected, but cannot identify the more specific relationships between the head, body, and tail of the fish to further study the living habits and behaviors of aquatic organisms. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a YOLO-based method, device, terminal and storage medium for multi-target detection of aquatic organisms, in order to solve the problems that the YOLO algorithm in the prior art cannot perform accurate multi-target detection of aquatic organisms, and that the detection results may be incomplete or duplicated.

[0005] To achieve the above and other related objectives, a first aspect of this application provides a YOLO-based method for multi-target detection of aquatic organisms, comprising: acquiring an image of an aquatic organism to be detected; inputting the aquatic organism image into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein each aquatic organism in the image is bounded by multiple prediction box groups in the first detection result; performing non-maximum suppression processing on the first detection result to extract the optimal prediction box group from the multiple prediction box groups to obtain a second detection result of the aquatic organism image, wherein each aquatic organism in the image is bounded by a single optimal prediction box group in the second detection result; wherein the prediction box group is composed of prediction boxes, and each prediction box corresponds one-to-one with a part of the aquatic organism.

[0006] In some embodiments of the first aspect of this application, the pre-trained neural network is obtained by improving the YOLOv5 model; the improvement includes: expanding the output layer of the neural network prediction module in the YOLOv5 model to n times the size before the improvement, so that the output dimension of the prediction module is correspondingly expanded to n times the size before the improvement; where n represents the number of prediction boxes in a single prediction box group.

[0007] In some embodiments of the first aspect of this application, the pre-trained neural network is obtained based on the following method: acquiring a set of aquatic organism images, the set of aquatic organism images including multiple labeled aquatic organism images; wherein, each part of each aquatic organism in the labeled aquatic organism images is labeled with a corresponding part label, and the part labels of the aquatic organisms are used as a label group; inputting the set of aquatic organism images into a convolutional neural network, constructing a loss function based on the difference between the output value of each part of the aquatic organism and the true value of the convolutional neural network, and training the convolutional neural network until convergence to obtain the pre-trained neural network.

[0008] In some embodiments of the first aspect of this application, the loss function is the sum of a localization loss function, a classification loss function, an angle loss function, and a bounding loss function.

[0009] In some embodiments of the first aspect of this application, before performing non-maximum suppression processing on the first detection result, the method further includes: deleting the detection results in the first detection result that do not completely frame the various parts of the aquatic organism.

[0010] In some embodiments of the first aspect of this application, a method for performing non-maximum suppression processing on the first detection result includes: obtaining a first prediction box group and a second prediction box group of the aquatic organism in the first detection result; calculating the cross-union ratio (CUP) of the first prediction box group and the second prediction box group; and deleting the prediction box group with lower confidence from the first prediction box group and the second prediction box group when the CUP is greater than a preset threshold.

[0011] In some embodiments of the first aspect of this application, a data augmentation operation is included before the aquatic organism image set is input into the convolutional neural network; wherein the data augmentation operation uses a single label group as the basic operation unit.

[0012] To achieve the above and other related objectives, a second aspect of this application provides a YOLO-based multi-target detection device for aquatic organisms, comprising: an image acquisition module for acquiring an image of an aquatic organism to be detected; a first detection module for inputting the aquatic organism image into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by multiple prediction box groups in the first detection result; and a second detection module for performing non-maximum suppression processing on the first detection result to extract the optimal prediction box group from the multiple prediction box groups to obtain a second detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by a single optimal prediction box group in the second detection result; wherein the prediction box group is composed of prediction boxes, and each prediction box corresponds one-to-one with a part of the aquatic organism.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides an electronic terminal, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the terminal to perform the method.

[0015] As described above, the YOLO-based multi-target detection method, device, terminal, and storage medium of this application have the following beneficial effects:

[0016] The method provided in this application uses the combination of the head, body, and tail of a fish as the basis for model training and detection. It can effectively detect the relationships between different body parts of a fish in a target image, and the output for these relationships is always complete, outputting the head, body, and tail of a single fish together. Furthermore, the output for these relationships is always unique; that is, for the head, body, and tail of a single fish, only a single, accurate prediction box is output. This improved method provides more reasonable output for multi-target combinations and reduces post-processing operations, enabling multi-target detection of aquatic organisms and facilitating subsequent research on the living habits and behaviors of fish. Attached Figure Description

[0017] Figure 1 The image shown is a schematic diagram illustrating the detection results of existing technologies, which can only detect some body parts.

[0018] Figure 2 The diagram shows a detection of multiple identical body parts in a single fish in a prior art detection image.

[0019] Figure 3 The diagram shown is a flowchart of a YOLO-based multi-target detection method for aquatic organisms in one embodiment of this application.

[0020] Figure 4 This is a schematic diagram of the YOLOv5 model structure in the prior art.

[0021] Figure 5 The diagram shown illustrates the crossover ratio (CRR) calculation method in one embodiment of this application.

[0022] Figure 6 The diagram shown is a schematic of the improved model detection process in one embodiment of this application.

[0023] Figure 7 The diagram shows the detection results of the improved model for multiple aquatic organisms in one embodiment of this application.

[0024] Figure 8 The diagram shown is a schematic representation of the detection results of the improved model for a single aquatic organism in one embodiment of this application.

[0025] Figure 9 The diagram shown is a schematic representation of the improved model training process in one embodiment of this application.

[0026] Figure 10The diagram shown is a schematic diagram of a YOLO-based multi-target aquatic organism detection terminal in one embodiment of this application.

[0027] Figure 11 The diagram shown is a schematic diagram of a YOLO-based multi-target aquatic organism detection device in one embodiment of this application. Detailed Implementation

[0028] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0029] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0030] To address the problems described in the background section, this invention provides a method, apparatus, terminal, and storage medium for multi-target detection of aquatic organisms based on YOLO, aiming to solve the problem that the existing YOLO algorithm cannot perform multi-target detection of aquatic organisms. Meanwhile, to make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0031] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0032] <1> Deep learning is a new research direction in the field of machine learning. It refers to learning the inherent patterns and representation levels of sample data. By designing and building an appropriate number of neural computing nodes and a multi-layered computational hierarchy, and through network learning and optimization, it establishes a functional relationship from input to output, thereby approximating the real-world correlation as closely as possible.

[0033] <2> Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computation and has a deep structure. They are one of the representative algorithms of deep learning. CNNs have representation learning capabilities and can perform supervised and unsupervised learning. They have the characteristics of shared convolutional kernel parameters within hidden layers and sparse inter-layer connections. Therefore, they can learn gridded features such as pixels and audio with relatively small computational cost and have stable learning performance.

[0034] <3> YOLO (You Only Look Once) network: It is a convolutional neural network that can predict the location and category of multiple rectangular regions containing objects at once. It can find specific targets in input images, better separate targets from background regions, and achieve target detection and recognition.

[0035] <4> Non-maximum suppression (NMS) is an algorithm that removes non-maximum values ​​and is commonly used in computer vision for edge detection, object recognition, face detection, and target detection (DPM, YOLO, SSD, Faster R-CNN). Its core idea is to search for local maxima and suppress non-maximum values.

[0036] <5> Intersection over Union (IOU) is a metric that measures the accuracy of detecting corresponding objects in a given dataset. IOU is a simple metric that can be used to measure any task that derives a prediction bounding box in the output.

[0037] This invention provides a YOLO-based method for multi-target detection of aquatic organisms, a system for implementing the YOLO-based method for multi-target detection of aquatic organisms, and a storage medium storing an executable program for implementing the YOLO-based method for multi-target detection of aquatic organisms. Regarding the implementation of the YOLO-based method for multi-target detection of aquatic organisms, this invention will describe exemplary implementation scenarios of YOLO-based multi-target detection of aquatic organisms.

[0038] like Figure 3The diagram illustrates a flowchart of a YOLO-based multi-target detection method for aquatic organisms according to an embodiment of the present invention. The YOLO-based multi-target detection method for aquatic organisms in this embodiment mainly includes the following steps:

[0039] Step S11: Obtain images of the aquatic organisms to be detected.

[0040] In some examples, the image of the aquatic organism to be detected can be an existing image, such as a fish image taken by someone else and obtained from the Internet. Alternatively, the image of the aquatic organism to be detected can also be a fish image taken by a camera device, including but not limited to: a camera, a video camera, a camera module integrating an optical system or a CCD chip, a camera module integrating an optical system and a CMOS chip, etc. In a common application scenario of this application, the image of the aquatic organism to be detected is usually an image of several fish in a fish tank taken by a camera device.

[0041] Step S12: Input the aquatic organism image into the pre-trained neural network to obtain the first detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by multiple prediction boxes in the first detection result.

[0042] Specifically, after acquiring the corresponding image to be detected, the image is input into a pre-trained neural network. This neural network is an improvement on the existing YOLOv5 algorithm. YOLOv5 is one of the most advanced object detection algorithms currently available, with performance comparable to YOLOv4, but the model is nearly 90% smaller than Darknet, at only about 27MB. YOLOv5 proposed four network models: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. Among them, YOLOv5s has the smallest network, the fastest speed, and the lowest accuracy. The other three networks are built upon this foundation, continuously deepening and widening the network, resulting in some improvement in accuracy, but also a decrease in speed. The basic framework of YOLOv5s is as follows: Figure 4 As shown.

[0043] However, when applied to the identification of aquatic organisms, the existing YOLOv5 model can only identify the entire fish body for classification and localization, and cannot perform multi-target detection of different body parts such as the head, body, and tail. For example, it can only detect the head, body, and tail of the fish in the image separately in multiple tasks. This essentially treats and observes each body part of the fish differently during its activity, inevitably losing a lot of information (such as the angle and displacement relationship between the head, body, and tail), which is not conducive to more in-depth research on the living habits and behaviors of fish.

[0044] To address the aforementioned issues, this application improves the convolutional neural network (CNN) in YOLOv5 by expanding its output dimension by a certain factor to meet the requirements for multi-target detection of aquatic organisms. The factor by which the output dimension of the CNN is expanded can be varied based on actual needs. For example, to detect more body parts of aquatic organisms, the expansion factor of the output layer is increased; to detect fewer body parts, the expansion factor is decreased.

[0045] After inputting the acquired image of the aquatic organism to be detected into a pre-trained and improved neural network, the first detection result output by the neural network can be obtained. In this first detection result, when selecting various body parts of a single fish, there are multiple prediction box groups. Each prediction box group consists of several prediction boxes. All multiple prediction box groups are the detection output of the neural network for the fish, and the difference between them lies in the difference in the confidence level of each prediction box group.

[0046] For example, in a scenario involving multi-target detection of a fish, the neural network outputs four prediction box groups, each with a different confidence level. Furthermore, each prediction box group consists of three prediction boxes: the first predictive box selects the fish's head, the second selects the body, and the third selects the tail, thus achieving multi-target detection of the fish. It should be noted that the above example is only for illustrating the principle of this application; the number of prediction box groups is not fixed and can be set according to the confidence level requirements. Similarly, the number of prediction boxes within each prediction box group is not fixed and can be set differently depending on the detection target and purpose. For example, in addition to detecting the head, body, and tail of the target fish, other body parts such as fins can also be detected, thus adjusting the number of prediction boxes in each prediction box group as needed.

[0047] In some implementations of this embodiment, the pre-trained neural network is improved based on the YOLOv5 model; the improvement includes: expanding the output layer of the neural network prediction module in the YOLOv5 model to n times the original size, so that the output dimension of the prediction module is correspondingly expanded to n times the original size; where n represents the number of prediction boxes in a single prediction box group.

[0048] Specifically, to address the limitation of existing YOLOv5 models in multi-target detection of aquatic organisms, this method expands the output layer of the neural network prediction module in the YOLOv5 model to n times its original size. This correspondingly expands the output dimension to n times its original size, where n is the number of prediction boxes in each prediction box group, representing the number of parts of a single fish body to be detected using this method. For example, if the current multi-target detection task involves selecting the head, body, and tail of a target fish body, then at least three prediction boxes are needed for each individual fish body to select the head, body, and tail respectively. In this case, n = 3, and the output layer of the neural network's prediction module is expanded to 3 times its original size.

[0049] In some optimized implementations, the output dimension of the neural network's prediction module is 3×3×(5+180+number of categories), where the first 3 represents the number of anchor boxes in the current downsampled feature map; the second 3 represents the number of fish body parts detected, namely head, body, and tail; 5 represents the output of five dimensions: the x-coordinate of the prediction box center point, the y-coordinate of the prediction box center point, the length of the long side of the prediction box, the length of the short side of the prediction box, and the confidence score; 180 represents the angle classification output from 1 to 180 degrees, used to determine the tilt angle of the prediction box; and the number of categories is set as needed according to detection and training requirements. Based on the above improvements, the function of the prediction module can be divided into three parts to perform corresponding training and detection for the head, body, and tail of the fish respectively.

[0050] Based on the above improvements, the improved prediction module has better interpretability, with the three parts focusing more on their respective corresponding parts, thus enabling more accurate identification of the target fish's body parts during training and detection.

[0051] In some implementations of this embodiment, the pre-trained neural network is obtained based on the following method: acquiring an aquatic organism image set, the aquatic organism image set including multiple labeled aquatic organism images; wherein, each part of each aquatic organism in the labeled aquatic organism image is labeled with a corresponding part label, and the part labels of the aquatic organism are used as a label group; inputting the aquatic organism image set into a convolutional neural network, constructing a loss function based on the difference between the output value of each part of the aquatic organism and the true value of the convolutional neural network, and training the convolutional neural network until convergence to obtain the pre-trained neural network.

[0052] Specifically, the purpose of this step is to train the improved neural network to achieve better multi-target detection performance for aquatic organisms. Before training the neural network, a dataset needs to be created, primarily the training set. First, a certain number of aquatic organism images need to be collected. These images can be self-taken or existing images obtained from public sources. After collecting the images, the aquatic organisms in the images need to be labeled accordingly. Each part of each aquatic organism is labeled with its corresponding part, and the labels for each part of the aquatic organism form a label group. For example, in any image in the training set, the label for each aquatic organism must completely include the head, body, and tail, and the labels for the head, body, and tail of a single aquatic organism form a label group.

[0053] Based on this improvement, it is ensured that the label group read during model training is a complete set containing information about the head, body, and tail. This allows multiple targets on the target fish to be simultaneously bound into the model as training targets for calculating the model loss. As a result, the model takes the combined whole as the smallest unit for both input and output, allowing the model to train and detect the best combination of targets.

[0054] After the aforementioned training set is completed, it is input into the convolutional neural network in batches. The convolutional neural network will output the results of the head, body and tail of the target fish, and will perform loss calculation with the corresponding training target that also contains the head, body and tail. Finally, the sum of the losses of the three parts is used as the parameters when backpropagating to optimize the model.

[0055] In some examples, the loss function expression for this convolutional neural network is as follows:

[0056] L total =SUM(L head +L body +L tail ); Formula 1

[0057] In the formula, L head For the predicted loss of the fish's head; L body For predicting the loss of the fish's body; L tail This is for predicting losses to the fish's tail.

[0058] In this implementation process, the loss function is the sum of the localization loss function, the classification loss function, the angle loss function, and the bounding loss function.

[0059] Specifically, the loss functions for individual body parts within a fish include localization loss function, classification loss function, angle loss function, and bounding loss function, the expressions of which are as follows:

[0060] The localization loss function, classification loss function, and angle loss function are all BECLogits Loss functions, and their loss function expressions are all:

[0061] L n =-w n [y n log(sigmoid(x n ))+(1-y n log(1-sigmoid(x)) n ))]; Formula 2

[0062] Among them, y n x is the true value. n These are predicted values.

[0063] It should be noted that, in order to make x n The value of x is in the interval [0, 1], therefore the BEC Logits Loss function is based on the BCE loss function with respect to x. n Perform a sigmoid function operation in advance.

[0064] The expression for the frame loss function is as follows:

[0065]

[0066]

[0067]

[0068] In the formula, ρ 2 (b,b gt ) represent the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, respectively, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted bounding box and the ground truth bounding box.

[0069] With L head For example, the expression for its loss function is:

[0070] L head =L obj +L cls +L ang +L CIoU ; Formula 6

[0071] In the formula, L obj For the localization loss function; L cls L is the classification loss function; ang Let L be the angle loss function; CIoU Define the loss function.

[0072] In this implementation process, a data augmentation operation is included before the labeled aquatic organism image is input into the convolutional neural network; wherein the data augmentation operation uses a single label group as the basic operation unit.

[0073] Specifically, when performing data augmentation operations, the images in the training set are processed in groups. The data augmentation operations in this application include, but are not limited to, Mosaic, Mix-up, Flip, Rotation, and Copy Paste. It should also be noted that all data augmentation operations in this application use a label group as the basic unit for data augmentation, rather than performing individual augmentation operations on individual labels within a label group. Furthermore, during data augmentation operations, if a portion of a target cannot be detected, the entire label group of that target is removed and not included in the training data.

[0074] Based on the above improvements, it is ensured that the data is augmented simultaneously according to multiple objectives, so that the training and detection of the model can be more suitable for the purpose of detecting multiple body parts of aquatic organisms in this application.

[0075] Step S13: Perform non-maximum suppression processing on the first detection result to extract the optimal prediction box group among the multiple prediction box groups, and obtain the second detection result of the aquatic organism image. In the second detection result, each aquatic organism in the image is selected by a single optimal prediction box group.

[0076] The purpose of this step is to remove redundant prediction boxes from multiple prediction boxes in the first detection result, ultimately retaining only one prediction box group for a single aquatic organism, thus making the model output a more intuitive and accurate detection result. The non-maximum suppression algorithm sorts all detection results output by the convolutional neural network (e.g., multiple prediction boxes for a single fish) according to their confidence level. By calculating the intersection-union ratio (IUU) between each prediction box group and gradually removing redundant prediction boxes based on the calculation results, only one prediction box group is ultimately retained for a single fish. The prediction boxes in this group are the output after processing by the non-maximum suppression algorithm.

[0077] In some implementations of this embodiment, before performing non-maximum suppression processing on the first detection result, the method further includes deleting the detection results in the first detection result that do not completely select each part of the aquatic organism.

[0078] Specifically, before performing non-maximum suppression processing on the first detection result, a preprocessing step is also included, which is to obtain the detection results in the first detection result that do not completely select each part of the aquatic organism (for example, in the detection result of a certain fish, only the predicted boxes for the head and body are selected, but the predicted boxes for the tail are missing), and directly delete these incomplete detection results.

[0079] The purpose of this step is to quickly eliminate some low-quality detection results that only predict part of the body parts, so that the nonmaximum suppression algorithm can more quickly select the best target fish body parts and eliminate the interference of low-quality detection results.

[0080] In some implementations of this embodiment, the method for performing non-maximum suppression processing on the first detection result includes: obtaining a first prediction box group and a second prediction box group of the aquatic organisms in the first detection result; calculating the cross-union ratio (CUP) of the first prediction box group and the second prediction box group; and deleting the prediction box group with lower confidence from the first prediction box group and the second prediction box group when the CUP is greater than a preset threshold.

[0081] Specifically, this step describes how to use a non-maximum suppression algorithm to delete one of the two predicted bounding box groups, retaining the better predicted bounding box group. First, obtain the first and second predicted bounding box groups for the target fish, and calculate the cross-union ratio (CURPR) between the corresponding predicted bounding boxes in the first and second predicted bounding box groups. The CURPR calculation formula is as follows:

[0082]

[0083] In the formula, IOU is the intersection-union ratio of the two predicted boxes, A∩B is the intersection of the two predicted boxes, and A∪B is the union of the two predicted boxes.

[0084] like Figure 5 The figure shows a graphical representation of the intersection-union ratio calculation method.

[0085] In this formula, the Cross-Union Ratio (CIRR) is directly proportional to the detection result; that is, the larger the CIRR, the closer the predicted bounding box in the actual detection result is to the ground truth bounding box of the object. A higher degree of overlap between the ground truth bounding box and the predicted bounding box indicates that the algorithm can accurately identify the object in the bounding box diagram, resulting in a higher labeling accuracy and better algorithm performance.

[0086] For a single fish body, the cross-union ratio (CUP) of the corresponding prediction boxes in the first and second prediction box groups is calculated separately. The CUP of the head, body, and tail of the fish body is summed to obtain the CUP score between the first and second prediction box groups. This CUP score is then compared with a pre-set threshold. If the CUP score is greater than the pre-set threshold, the results with lower confidence in the first and second prediction box groups are considered duplicate detection results and are excluded. Accordingly, when there are more than two prediction box groups, two of them are selected sequentially to perform the non-maximum suppression algorithm operation as described above, until only one prediction box group is retained to select each body part of the fish body. The last retained result is used as the second detection result, which is the final output of the model.

[0087] Based on the above improvements, the first detection result containing multiple prediction box groups is processed by the non-maximum suppression algorithm, and the optimal prediction box group among the multiple prediction box groups is extracted and retained, thereby obtaining the second detection result that selects each aquatic organism using only a single prediction box group. The second detection result is used as the final output of the model, so that each body part of each aquatic organism in the image can be intuitively and accurately represented, which is convenient for further research on the living habits and behaviors of aquatic organisms.

[0088] In steps S11 to S13 above, the prediction box group consists of prediction boxes, each corresponding to a different part of the aquatic organism. Specifically, the prediction boxes are used to select different body parts of the target fish. For example, the first prediction box selects the head of the fish, the second prediction box selects the body of the fish, and the third prediction box selects the tail of the fish, thereby outputting multi-target detection results based on a single fish.

[0089] To facilitate understanding of the implications of this application, an example is provided here to illustrate the flowchart of the model's detection of the original image.

[0090] like Figure 6 The diagram shown is a flowchart of the model detection process in one embodiment of this application.

[0091] In this example, the image of the aquatic organism to be detected is compressed to a size of 682*384 before entering the model, ensuring it is divisible by 32 to meet the input requirements of the improved model. The prediction module of the improved model outputs a 3*7 dimensional detection result. Here, 3 represents the head, body, and tail, and the 7 dimensions are the x-coordinate of the center point of the prediction box, the y-coordinate of the center point of the prediction box, the length of the longer side of the prediction box, the length of the shorter side of the prediction box, the angular classification of the prediction box, the confidence score, and the target classification. After processing with a non-maximum suppression algorithm, the prediction module retains the best detection result and outputs as shown below. Figure 8 The results shown include the complete detection of the head, body, and tail. Figure 7This will be displayed as a schematic diagram of the detection results for multiple aquatic organisms in one scenario of this application.

[0092] like Figure 9 The diagram shown is a flowchart of the model training process in one embodiment of this application.

[0093] Before being fed into the model, labeled aquatic organism images in the training set undergo resizing and grayscale padding to normalize them to a size of 640*640 pixels. When the normalized images are input into the model, the three body parts of a single fish are grouped together as a single label and then split into a 3*187 array. The 187 (180+7) dimension includes the number of predicted bounding boxes, the x-coordinate of the predicted bounding box center point, the y-coordinate of the predicted bounding box center point, the length of the long side of the predicted bounding box, the length of the short side of the predicted bounding box, the predicted bounding box tilt angle, the target classification, and a 180-dimensional circular smooth label angle classification.

[0094] After processing the training images by the improved model, three prediction outputs with different feature map sizes (80*80, 40*40, 20*20) are output, representing downsampling by 8, 16, and 32 times, respectively. The 3x dimension output of the improved prediction module represents the head, body, and tail of the fish. In the figure, the second dimension 3 indicates the number of anchor boxes in the current downsampling, and 200 (180+15+4+1) includes four dimensions of positional information (x-coordinate of the predicted box center point, y-coordinate of the predicted box center point, length of the long side of the predicted box, and length of the short side of the predicted box), confidence score, 15-class classification information, and a 180-dimensional circular smooth label angle prediction. After obtaining the detection results, the prediction output is compared with the training target, and the sum of the losses for the head, body, and tail is used for backpropagation optimization of the model. The loss calculation includes localization loss, bounding box loss, classification loss, and angle classification loss.

[0095] It should be emphasized that existing YOLO or other similar models can only locate and classify aquatic organisms as a whole, and cannot perform multi-target detection of different parts of the aquatic organism's body. Therefore, they are not suitable for studying the living habits and behavioral characteristics of aquatic organisms (such as tail swing amplitude, head turning angle during turning, and the relationship between the tail and body). However, this application, based on the aforementioned technical solution, can effectively overcome these technical problems and achieve multi-target detection of different parts of the aquatic organism's body, facilitating further scientific research.

[0096] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.

[0097] The YOLO-based multi-target aquatic organism detection method provided in this invention can be implemented on the terminal side or the server side. For the hardware structure of the YOLO-based multi-target aquatic organism detection terminal, please refer to [link to relevant documentation]. Figure 10 This is a schematic diagram of an optional hardware structure of a YOLO-based multi-target aquatic organism detection terminal 1000 provided in an embodiment of the present invention. The terminal 1000 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The YOLO-based multi-target aquatic organism detection terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 1004, and a user interface 1006. The various components in the device are coupled together through a bus system 1005. It is understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general will label all buses as bus systems.

[0098] The user interface 1006 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0099] It is understood that memory 1002 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0100] In this embodiment of the invention, the memory 1002 is used to store various types of data to support the operation of the YOLO-based aquatic organism multi-target detection terminal 1000. Examples of this data include any executable program for operation on the YOLO-based aquatic organism multi-target detection terminal 1000, such as the operating system 10021 and application program 10022; the operating system 10021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 10022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the YOLO-based aquatic organism multi-target detection method provided in this embodiment of the invention can be included in the application program 10022.

[0101] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor 1001 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0102] In an exemplary embodiment, the YOLO-based multi-target aquatic organism detection terminal 1000 can be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0103] like Figure 11The diagram illustrates a schematic of a YOLO-based multi-target aquatic organism detection device according to an embodiment of the present invention. In this embodiment, the YOLO-based multi-target aquatic organism detection device 1100 includes: an image acquisition module 1101 for acquiring an image of an aquatic organism to be detected; a first detection module 1102 for inputting the aquatic organism image into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein the aquatic organism is selected by multiple groups of predicted bounding boxes in the first detection result; and a second detection module 1103 for performing non-maximum suppression processing on the first detection result to obtain a second detection result of the aquatic organism image, wherein the aquatic organism is selected by a single group of predicted bounding boxes in the second detection result; wherein the group of predicted bounding boxes consists of predicted bounding boxes, and each predicted bounding box corresponds one-to-one with a different part of the aquatic organism.

[0104] It should be noted that the YOLO-based multi-target aquatic organism detection device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing YOLO-based multi-target aquatic organism detection. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the YOLO-based multi-target aquatic organism detection device and the YOLO-based multi-target aquatic organism detection method embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0105] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0106] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0107] In summary, this application provides a method, apparatus, terminal, and medium for multi-target detection of aquatic organisms based on YOLO. This invention provides a method to improve the efficiency of multi-target detection of aquatic organisms based on YOLO. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0108] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A YOLO-based method for multi-target detection of aquatic organisms, characterized in that, include: Acquire images of the aquatic organisms to be detected; The aquatic organism image is input into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by multiple prediction boxes in the first detection result. Delete the test results in the first test results that do not completely select each part of the aquatic organism; Non-maximum suppression is applied to the remaining first detection results to extract the optimal prediction box group from the plurality of prediction box groups, thereby obtaining the second detection result of the aquatic organism image. In the second detection result, each aquatic organism in the image is selected by a single optimal prediction box group. The prediction box group is composed of prediction boxes, and each prediction box corresponds one-to-one with each part of the aquatic organism. The pre-trained neural network is obtained by improving the YOLOv5 model; the improvement includes: expanding the output layer of the neural network prediction module in the YOLOv5 model to n times the original size, so that the output dimension of the prediction module is correspondingly expanded to n times the original size; where n represents the number of prediction boxes in a single prediction box group; The pre-trained neural network is obtained based on the following method: A set of aquatic organism images is acquired, comprising multiple labeled aquatic organism images; wherein each part of each aquatic organism in the labeled aquatic organism images is labeled with a corresponding part label, and the labels of each part of the aquatic organism are grouped into a single label group; the aquatic organism image set is input into a convolutional neural network, and a loss function is constructed based on the difference between the output value and the true value of each part of the aquatic organism by the convolutional neural network, to train the convolutional neural network until convergence, thus obtaining the pre-trained neural network; wherein the loss function is the sum of the losses of each part of the aquatic organism, and the loss function for each part includes a localization loss function, a classification loss function, an angle loss function, and a bounding loss function.

2. The YOLO-based multi-target detection method for aquatic organisms according to claim 1, characterized in that, A method for performing non-maximum suppression processing on the first detection result includes: Obtain the first and second prediction box groups of the aquatic organisms in the first detection result; Calculate the intersection-union ratio (IU / U) of the first prediction box group and the second prediction box group; When the crossover ratio is greater than a preset threshold, delete the prediction box groups with lower confidence levels in the first and second prediction box groups.

3. The YOLO-based multi-target detection method for aquatic organisms according to claim 1, characterized in that, Before inputting the aquatic organism image set into the convolutional neural network, a data augmentation operation is also included; wherein the data augmentation operation uses a single label group as the basic operation unit.

4. A YOLO-based multi-target aquatic organism detection device, characterized in that, include: Image acquisition module: used to acquire images of the aquatic organisms to be detected; First detection module: used to input the aquatic organism image into a pre-trained neural network to obtain a first detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by multiple prediction box groups in the first detection result; The second detection module is used to delete the detection results in the first detection results that do not completely select each part of the aquatic organism, and to perform non-maximum suppression processing on the remaining first detection results to extract the optimal prediction box group among the multiple prediction box groups, so as to obtain the second detection result of the aquatic organism image, wherein each aquatic organism in the image is selected by a single optimal prediction box group in the second detection result. The prediction box group is composed of prediction boxes, and each prediction box corresponds one-to-one with each part of the aquatic organism. The pre-trained neural network is obtained by improving the YOLOv5 model; the improvement includes: expanding the output layer of the neural network prediction module in the YOLOv5 model to n times the original size, so that the output dimension of the prediction module is correspondingly expanded to n times the original size; where n represents the number of prediction boxes in a single prediction box group; The pre-trained neural network is obtained based on the following method: A set of aquatic organism images is acquired, comprising multiple labeled aquatic organism images; wherein each part of each aquatic organism in the labeled aquatic organism images is labeled with a corresponding part label, and the labels of each part of the aquatic organism are grouped into a single label group; the aquatic organism image set is input into a convolutional neural network, and a loss function is constructed based on the difference between the output value and the true value of each part of the aquatic organism by the convolutional neural network, to train the convolutional neural network until convergence, thus obtaining the pre-trained neural network; wherein the loss function is the sum of the losses of each part of the aquatic organism, and the loss function for each part includes a localization loss function, a classification loss function, an angle loss function, and a bounding loss function.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 3.

6. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory to cause the terminal to perform the method as described in any one of claims 1 to 3.

Images