A method and system for dense detection of larvae and juveniles based on disentanglement and task alignment

Through the method of unwrap and task alignment, the sample allocation and loss function of dense object detectors are optimized, and the classification and positioning tasks are coordinated, which solves the problems of inaccurate label allocation and insufficient task interaction in the existing technology. Improves detection accuracy and reliability.

CN118691912BActive Publication Date: 2024-11-05SHANDONG MARINE RESOURCE AND ENVIRONMENT RESEARCH INSTITUTE (SHANDONG MARINE ENVIRONMENTAL MONITORING CENTER SHANDONG AQUATIC PRODUCTS QUALITY INSPECTION CENTER)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411186504.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-05
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

When detecting young fish, existing dense target detectors have problems such as inaccurate label allocation, lack of interaction between classification and positioning tasks, and spatial misalignment, resulting in limited performance.

Method used

Using a method based on unwrap and task alignment, through the dewrap mechanism and task alignment, the sample allocation and loss function are optimized, object classification and positioning tasks are coordinated, and detection accuracy is improved.

Benefits of technology

It effectively solves the occlusion and overlap problems in dense object detection, improves the accuracy and reliability of detection results, reduces classification errors and positioning deviations, and improves the ability to identify different states of fish.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691912B_ABST
    Figure CN118691912B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of dense detection technology, and in particular to a method and system for dense detection of fry and juvenile fish based on disentanglement and task alignment. The method comprises obtaining a multi-scale image information dataset of fry and juvenile fish, and preprocessing the dataset; performing label assignment disentanglement on the preprocessed image information; performing feature extraction on the image information using a dense target detection model, and performing spatial feature disentanglement on the extracted features; and by designing a sample allocation scheme and task alignment loss, explicitly pulling the best anchor points of the two tasks closer during training, better coordinating the object classification and positioning tasks, and improving the accuracy of dense detection of the quality and health status of fish at different stages.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dense detection, and in particular to a method and system for dense detection of larvae and juveniles based on disentanglement and task alignment. Background Art

[0002] At present, the development of fisheries is very rapid, and attention to the developmental stages of fish is crucial. Protecting and maintaining the developmental stages of fish is essential to maintaining the health of marine and freshwater ecosystems. The quality and health of fish at different stages, such as eggs, larvae and juveniles, are also crucial for subsequent development. Whether it is fish eggs, larvae or juveniles, there are a large number of them. It would be very time-consuming to manually detect the developmental status of each single individual. Therefore, the use of dense object detection methods based on computer vision can effectively and in real time distinguish fish at different developmental stages and the health of fish. Dense object detection methods have developed very rapidly in recent years, so that their use in the field of fisheries has also increased, but this method also has certain disadvantages. The training pipeline of most dense detectors is affected by many possible invalid phases, and is usually achieved by optimizing two subtasks: object classification and localization, using a head with two parallel branches, which may cause a certain degree of spatial misalignment in the prediction between the two tasks. This limits the performance of the object detector.

[0003] At present, dense detectors for larvae and juveniles have the following main defects in dense detection tasks: 1. In dense object detection, too many larvae and overlaps may lead to inaccurate label assignment. 2. When detecting overlapping larvae, classification requires areas with rich semantic information, while regression tends to focus on the edge parts. Therefore, the same receptive field in the same space cannot guarantee optimal performance. 3. Due to the existence of the pyramid structure, there is an imbalance problem between detection boxes at different levels. 4. Dense detectors are usually implemented by optimizing two subtasks: object classification and localization, using a head with two parallel branches. Such a dual-branch design may lead to a lack of interaction between the two tasks, resulting in a certain degree of spatial misalignment in the predictions between the two tasks, limiting the performance of the object detector. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides a method and system for dense detection of larvae and juveniles based on disentanglement and task alignment. Through an effective disentanglement mechanism, the constraints of dense object detectors are untied, and a novel task alignment head is proposed. By designing a sample allocation scheme and task alignment loss, the optimal anchor points of the two tasks are explicitly brought closer during training, which better coordinates the object classification and positioning tasks, and improves the accuracy of dense detection of the quality and health status of fish at different stages.

[0005] In a first aspect, the present invention provides a method for dense detection of larvae and juveniles based on disentanglement and task alignment, which adopts the following technical solution:

[0006] A method for dense detection of larvae and juveniles based on disentanglement and task alignment, comprising:

[0007] Obtain a multi-scale image information dataset of larvae and juveniles, and preprocess the dataset;

[0008] Perform label assignment and disentanglement on the preprocessed image information;

[0009] The dense object detection model is used to extract features from image information and perform spatial feature disentanglement on the extracted features;

[0010] The dense object detection model is used to perform feature pyramid supervised disentanglement on the data features after spatial feature disentanglement;

[0011] The task alignment method is used to supervise the disentanglement of the feature pyramid data and align the tasks to achieve model optimization;

[0012] The optimized model is used to detect fish larvae and juveniles in real time.

[0013] Furthermore, the data set is preprocessed, including feature calibration and classification calibration of the collected image information of fry and juvenile fish at different stages, and random scaling, cropping, rotation and image enhancement operations are performed on the calibrated image information.

[0014] Furthermore, the label assignment and disentanglement of the preprocessed image information includes inputting the preprocessed image information into a network model to generate a dense label map for each pixel, introducing a local consistency constraint, and assigning the generated dense label map to different object instances, so that the data features of the image information can be better captured and extracted.

[0015] Furthermore, the extracted features are spatially disentangled, including using a feature disentanglement network to disentangle the extracted features into different object instances, and then jointly optimizing the feature disentanglement network with the object detection and segmentation network to improve the detection and segmentation performance.

[0016] Furthermore, the data features after spatial feature disentanglement are subjected to feature pyramid supervised disentanglement, including obtaining a feature map after processing image information using a feature pyramid network, and generating a supervisory signal using label information of corresponding scales from different levels of the feature pyramid network.

[0017] Furthermore, the feature pyramid supervised disentanglement of the data features after spatial feature disentanglement also includes introducing a supervised connection module at the level of each feature pyramid network, connecting the supervisory signal and the corresponding feature pyramid network level, and defining the corresponding object detection loss function for the feature pyramid network level to measure the difference between the model prediction and the supervisory signal.

[0018] Furthermore, the task alignment method is used to supervise the data feature processing and task alignment after disentanglement of the feature pyramid to achieve model optimization, including using the task head T-head to process the features of the feature pyramid network to obtain classification probability and positioning prediction respectively, and then using task alignment learning TAL to balance the alignment degree of classification and positioning prediction to improve the generalization ability of the model.

[0019] In the second aspect, a system for dense detection of larvae and juveniles based on disentanglement and task alignment includes:

[0020] A data acquisition module is configured to acquire a multi-scale image information dataset of fry and juvenile fish and pre-process the dataset;

[0021] A preprocessing module is configured to perform label assignment disentanglement on the preprocessed image information;

[0022] The disentanglement module is configured to extract features from image information using a dense target detection model, and perform spatial feature disentanglement on the extracted features; and perform feature pyramid supervised disentanglement on data features after spatial feature disentanglement using the dense target detection model;

[0023] The task alignment module is configured to optimize the model by processing the data features and aligning the tasks after the feature pyramid supervision disentanglement using the task alignment method;

[0024] The detection module is configured to perform real-time detection of fry and juvenile fish using the optimized model.

[0025] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the described method.

[0026] In a fourth aspect, the present invention provides a terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the described method.

[0027] In summary, the present invention has the following beneficial technical effects:

[0028] This paper designs an effective disentanglement mechanism to untie the constraints of dense object detectors, and proposes a novel task alignment head, which explicitly pulls the optimal anchor points of the two tasks closer during training by designing a sample distribution scheme and task alignment loss, better coordinates object classification and localization tasks, and improves the accuracy of dense detection of the quality and health of fish at different stages.

[0029] Through the disentanglement mechanism, the present invention solves the problem of mutual occlusion and overlap of objects in dense object detection, so that the detector can more accurately identify and classify different states of fish. The task alignment head optimizes the selection of anchor points through the sample allocation scheme, so that the classification and localization tasks can work together more effectively based on shared features. The task alignment loss further strengthens the linkage between classification and localization tasks, reduces classification errors and localization deviations, and improves the consistency and reliability of detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 4 is a schematic diagram of a method for densely detecting larvae and juveniles based on disentanglement and task alignment according to Example 1 of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0032] Example 1

[0033] Reference Figure 1 , a method for dense detection of larvae and juveniles based on disentanglement and task alignment in this embodiment includes:

[0034] Obtain a multi-scale image information dataset of larvae and juveniles, and preprocess the dataset;

[0035] Perform label assignment and disentanglement on the preprocessed image information;

[0036] The dense object detection model is used to extract features from image information and perform spatial feature disentanglement on the extracted features;

[0037] The dense object detection model is used to perform feature pyramid supervised disentanglement on the data features after spatial feature disentanglement;

[0038] The task alignment method is used to supervise the disentanglement of the feature pyramid data and align the tasks to achieve model optimization;

[0039] The optimized model is used to detect fish larvae and juveniles in real time.

[0040] Specifically, the following steps are included:

[0041] S1: Data acquisition: Obtain fish larvae and juveniles data set, collect multi-scale image information of fish larvae and juveniles at different stages, perform feature calibration and classification calibration on the collected image information of fish larvae and juveniles at different stages, and finally divide and preprocess the calibrated data.

[0042] The fish larvae and juveniles data set includes a category label data set and a dense target selection data set of fish larvae and juveniles, and labelimg is used to perform target selection on the data set. The data preprocessing includes random scaling, cropping, rotation, and image enhancement operations on the data.

[0043] S2: Label assignment disentanglement: After the dataset is input into the network model, a dense label map for each pixel in the input image is generated, local consistency constraints are introduced, and the generated labels are assigned to different object instances.

[0044] The label assignment disentanglement of S2 generates a dense label map for each pixel in the input image. The dataset image is input into the model, using a dense label map , the size is the same as the input image size, . Indicates the image Line In the dense label generation stage, an initial label is assigned to each pixel based on the object boundary information in the input image.

[0045] The label assignment in S2 introduces local consistency constraints in the generated input image during disentanglement, and assigns the generated labels to different object instances. Define a label assignment function , the pixel position Mapped to the label of an object instance, and then define a local consistency function , measure pixels and In the label assignment disentanglement stage, the final label of each pixel is determined by analyzing the information of the pixel and its surrounding pixels, and the local consistency constraint is added to the label assignment process to keep the labels of adjacent pixels consistent. It can be expressed by the following formula:

[0046]

[0047] in Represents pixels Context information, including pixel values, gradient information, etc. Indicates that given pixel context information, the pixel Belongs to object instance probability. Represents pixels Neighborhood of It is a hyperparameter that weighs the local consistency constraint. The label assignment problem is solved by calculating the probability that each pixel belongs to each object instance and selecting the object instance with the highest probability as the final label.

[0048] S3: Spatial feature disentanglement: Extract the features of image data in the input dataset, propose a feature disentanglement network and introduce spatial consistency constraints, use the feature disentanglement network to disentangle the features in the feature map into different object instances, and then jointly optimize the feature disentanglement network with the object detection and segmentation network, and then use the feature pyramid to process the data features.

[0049] Among them, in the spatial feature disentanglement of S3, a feature disentanglement network is proposed and a spatial consistency constraint is introduced. In the spatial feature disentanglement method, a feature disentanglement network is used:

[0050]

[0051] in, is the feature map after disentanglement, which contains the feature representation of each pixel belonging to different object instances. In order to enhance the accuracy of disentanglement, spatial consistency constraints are introduced. Adjacent feature maps have similar feature representations. It can be expressed by the following formula:

[0052]

[0053] in, represents the first Line The feature representation of the column is, Represents pixels in the feature map Neighborhood of is a function that measures the consistency between adjacent feature representations. is a hyperparameter that weighs the spatial consistency constraint.

[0054] In the spatial feature disentanglement of S3, the feature disentanglement network is used to disentangle the features in the feature map into different object instances, and then the feature disentanglement network is jointly optimized with the object detection and segmentation network. It is usually jointly optimized with object detection and segmentation networks. These networks learn together to maximize the detection and segmentation performance. The objective function of joint optimization can be expressed as:

[0055]

[0056] in It is a disentangled network Parameters, are the parameters of the detection and segmentation networks, and Represent the loss functions of detection and segmentation respectively, Represents the loss function between the disentangled features and the original features. The performance of dense object detection is improved through disentangled networks, spatial consistency constraints and joint optimization.

[0057] S4: Feature pyramid supervised disentanglement: Generate supervision signals from different feature pyramid levels using label information of corresponding scales, then connect the supervision signals with the corresponding feature pyramid levels, and finally define the corresponding loss function for each feature pyramid level.

[0058] Among them, in the feature pyramid supervised disentanglement of S4, the supervision signal is generated from the label information of the corresponding scale at different feature pyramid levels, and the input image , after being processed by the feature pyramid network (FPN), the feature map is obtained ,in Represents the number of layers of the feature pyramid. For each feature pyramid level, the corresponding scale information is used to generate a supervisory signal. It can be expressed as:

[0059]

[0060] in is related to the feature pyramid level The relevant supervisory signal, is a function that generates a supervisory signal, whose input is an image and scale information .

[0061] In the feature pyramid supervised disentanglement of S4, the supervision signal is connected with the corresponding feature pyramid level. A supervised connection module is introduced to convert the supervisory signal and the corresponding feature pyramid level , to connect. It can be expressed as:

[0062]

[0063] in , is the enhanced feature obtained through supervised connection, is a supervised connection module that converts the feature pyramid level and supervisory signal to connect.

[0064] The feature pyramid supervised disentanglement of S4 defines a corresponding loss function for each feature pyramid level, and uses a supervised connection method to define a loss function for each feature pyramid level. Define the corresponding object detection loss function. This can be expressed as:

[0065]

[0066] in is related to the feature pyramid level The related object detection loss, is the object detector, is the loss function for the object detection task, which is used to measure the difference between the model prediction and the supervisory signal. The supervised connection method in FPN improves the accuracy and robustness of dense object detection by introducing supervisory signals at each feature pyramid level, enabling the model to better utilize feature information of different scales.

[0067] S5: Task Alignment: Using T-head (task head) and TAL (task alignment learning) to work together, T-head is used to process the features of the feature pyramid network and then perform classification and location prediction; TAL uses a new task alignment metric to calculate the task alignment signal to measure the degree of alignment between classification and location predictions. Using TAL for back propagation, T-head automatically adjusts its classification probability and location prediction using the task alignment signal calculated from TAL.

[0068] Among them, in the task alignment of S5, T-head is used to process the features of the feature pyramid network, and then classification and positioning prediction are performed. T-head first processes the FPN features, and then classification and positioning prediction are performed. The formula is:

[0069]

[0070]

[0071] Among them, the classification prediction is , the positioning prediction is ,in is a vector containing the probability of object categories, is a vector containing the bounding box position predictions. is the feature from the feature pyramid network, and They are classification and localization sub-networks respectively.

[0072] In the task alignment of S5, TAL uses a new task alignment metric to calculate the task alignment signal to measure the degree of alignment between classification and positioning predictions. The formula can be expressed as:

[0073]

[0074] in, is the task alignment signal, indicating the degree of alignment between classification and localization predictions. The task alignment metric is , used to calculate the task alignment signal input.

[0075] In the task alignment of S5, TAL is used for back propagation, and T-head automatically adjusts its classification probability and positioning prediction using the task alignment signal calculated from TAL. During the back propagation process, T-head adjusts its classification probability and positioning prediction according to the task alignment signal λ calculated from TAL. It can be expressed by the following formula:

[0076]

[0077]

[0078] in, is the adjusted classification and positioning prediction, and is an adjusted weight parameter used to balance the impact of task alignment signals on prediction results. T-head and TAL work together to improve the consistency between classification and localization predictions through task alignment metrics, improve the performance of the model in object detection tasks, and improve the generalization ability of the model.

[0079] S6: Larvae and fry detection: Use the trained dense target detection model to perform real-time detection of larvae and fry at different growth stages to obtain health status information of larvae and fry at each current development stage.

[0080] The fish fry detection in S6 obtains the health status information of the fish fry at each current developmental stage, and the model uses the trained weight parameter file to perform intensive target detection on the fish fry at different growth stages, and reflects the health status information of the fish fry through the real-time labels of the model.

[0081] Example 2

[0082] This embodiment provides a system for dense detection of larvae and juveniles based on disentanglement and task alignment, including:

[0083] A data acquisition module is configured to acquire a multi-scale image information dataset of fry and juvenile fish and pre-process the dataset;

[0084] A preprocessing module is configured to perform label assignment disentanglement on the preprocessed image information;

[0085] The disentanglement module is configured to extract features from image information using a dense target detection model, and perform spatial feature disentanglement on the extracted features; and perform feature pyramid supervised disentanglement on data features after spatial feature disentanglement using the dense target detection model;

[0086] The task alignment module is configured to optimize the model by processing the data features and aligning the tasks after the feature pyramid supervision disentanglement using the task alignment method;

[0087] The detection module is configured to perform real-time detection of fry and juvenile fish using the optimized model.

[0088] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the method described.

[0089] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the described method.

[0090] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for dense detection of larvae and juveniles based on disentanglement and task alignment, It is characterized in that include: Obtain a multi-scale image information dataset of larvae and juveniles, and preprocess the dataset; Perform label assignment and disentanglement on the preprocessed image information; Among them, in the label assignment disentanglement stage, the final label of each pixel is determined by analyzing the information of the pixel and its surrounding pixels, and the local consistency constraint is added to the label assignment process to keep the labels of adjacent pixels consistent; The dense object detection model is used to extract features from image information and perform spatial feature disentanglement on the extracted features; Among them, spatial feature disentanglement includes using a feature disentanglement network to disentangle the features in the feature map into different object instances, and then jointly optimizing the feature disentanglement network with the object detection and segmentation network; The dense target detection model is used to perform feature pyramid supervised disentanglement on the data features after spatial feature disentanglement; wherein, the feature pyramid supervised disentanglement includes generating supervision signals from different feature pyramid levels using label information of corresponding scales, then connecting the supervision signals with the corresponding feature pyramid levels, and finally defining the corresponding loss function for each feature pyramid level; The task alignment method is used to supervise the disentanglement of the data features and align the tasks of the feature pyramid to achieve the optimization of the model. Specifically, the task head T-head is used to process the features of the feature pyramid network, and then the classification and positioning prediction are performed. The task alignment learning TAL is used to calculate the task alignment signal to measure the degree of alignment between the classification and positioning predictions. The T-head uses the task alignment signal calculated from TAL to automatically adjust its classification probability and positioning prediction. The optimized model is used to detect fish larvae and juveniles in real time.

2. A method for dense detection of larvae and juveniles based on disentanglement and task alignment according to claim 1, It is characterized in that The data set is preprocessed, including feature calibration and classification calibration of the collected image information of fry and juvenile fish at different stages, and random scaling, cropping, rotation and image enhancement operations are performed on the calibrated image information.

3. A method for dense detection of larvae and juveniles based on disentanglement and task alignment according to claim 2, It is characterized in that The label assignment and disentanglement of the preprocessed image information includes inputting the preprocessed image information into a network model to generate a dense label map for each pixel, introducing a local consistency constraint, and assigning the generated dense label map to different object instances, so that the data features of the image information can be better captured and extracted.

4. A method for dense detection of larvae and juveniles based on disentanglement and task alignment according to claim 3, It is characterized in that The data features after spatial feature disentanglement are subjected to feature pyramid supervised disentanglement, including obtaining a feature map after processing image information using a feature pyramid network, and generating a supervisory signal using label information of corresponding scales from different levels of the feature pyramid network.

5. A method for dense detection of larvae and juveniles based on disentanglement and task alignment according to claim 4, It is characterized in that The feature pyramid supervised disentanglement of the data features after spatial feature disentanglement also includes introducing a supervised connection module at the level of each feature pyramid network, connecting the supervisory signal and the corresponding feature pyramid network level, and defining the corresponding object detection loss function for the feature pyramid network level to measure the difference between the model prediction and the supervisory signal.

6. A larval and juvenile dense detection system based on disentanglement and task alignment, It is characterized in that include: A data acquisition module is configured to acquire a multi-scale image information dataset of fry and juvenile fish and pre-process the dataset; A preprocessing module is configured to perform label assignment disentanglement on the preprocessed image information; Among them, in the label assignment disentanglement stage, the final label of each pixel is determined by analyzing the information of the pixel and its surrounding pixels, and the local consistency constraint is added to the label assignment process to keep the labels of adjacent pixels consistent; The disentanglement module is configured to extract features from image information using a dense target detection model, and perform spatial feature disentanglement on the extracted features; perform feature pyramid supervised disentanglement on the data features after spatial feature disentanglement using the dense target detection model; wherein the spatial feature disentanglement includes using a feature disentanglement network to disentangle features in a feature map into different object instances, and then jointly optimizing the feature disentanglement network with an object detection and segmentation network; wherein the feature pyramid supervised disentanglement includes generating a supervisory signal from label information of corresponding scales in different feature pyramid levels, and then connecting the supervisory signal with the corresponding feature pyramid level, and finally defining a corresponding loss function for each feature pyramid level; The task alignment module is configured to use the task alignment method to supervise the disentanglement of the feature pyramid data and align the tasks to achieve model optimization; specifically, the task head T-head is used to process the features of the feature pyramid network, and then to perform classification and positioning prediction, and the task alignment signal is calculated by task alignment learning TAL to measure the degree of alignment between classification and positioning prediction. T-head automatically adjusts its classification probability and positioning prediction using the task alignment signal calculated from TAL; The detection module is configured to perform real-time detection of fry and juvenile fish using the optimized model.

7. A computer-readable storage medium having a plurality of instructions stored therein, It is characterized in that The instructions are suitable for being loaded by a processor of a terminal device and executed by the method for dense detection of larvae and juveniles based on disentanglement and task alignment as claimed in claim 1.

8. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions. It is characterized in that The instructions are suitable for being loaded by a processor and executing the method for dense detection of larvae and juveniles based on disentanglement and task alignment as claimed in claim 1.