Fish key point detection method based on double attention enhanced HRNet
By combining the HRNet fish keypoint detection method with global-local feature fusion and dual attention enhancement, and the distribution perception method, the accuracy and robustness issues of fish keypoint detection in complex underwater environments are solved, and efficient and accurate fish keypoint detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HENAN UNIVERSITY
- Filing Date
- 2026-02-24
- Publication Date
- 2026-05-29
Smart Images
Figure CN122116106A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a method for detecting key points in fish bodies based on dual attention-enhanced HRNet. Background Technology
[0002] In the field of intensive aquaculture, the key points of fish body are the core biological indicators for measuring fish body size, analyzing behavioral status, estimating posture, classifying species, and monitoring the aquaculture water environment. The accuracy of their detection directly determines the level of intelligence and economic benefits of aquaculture production management.
[0003] Traditional fish body key point detection technologies have significant drawbacks: First, the technical approach relies heavily on manually designed features, which not only requires a large investment of manpower for feature debugging and detection operations, resulting in low detection efficiency, but also has weak generalization ability, making it difficult to adapt to diverse environmental conditions such as water quality changes and light fluctuations in intensive aquaculture scenarios; Second, the detection process requires manual sampling and contact with the fish, which can easily cause stress reactions, physical damage, or even death in the fish, which is seriously contrary to the development requirements of modern aquaculture for "high efficiency and low damage".
[0004] To address the bottlenecks of traditional technologies, researchers have been exploring deep learning-based fish keypoint detection schemes in recent years, forming two main technical branches: (1) Direct coordinate regression method: By establishing the mapping relationship between CNN features and keypoint coordinates through fully connected layers, the keypoint coordinate results are directly output. However, this method does not make full use of the spatial context information of the image, resulting in low keypoint localization accuracy. Moreover, model training relies on a large-scale labeled dataset, which limits its applicability when labeled data is scarce in aquaculture scenarios. (2) Heatmap regression method: By generating a probability distribution heatmap for each keypoint, the position with the highest heat value in the heatmap is used as the keypoint prediction result, which effectively makes up for the shortcomings of the direct coordinate regression method in the use of spatial information and significantly improves the detection robustness. However, existing heatmap regression techniques generally have the inherent problem of "heatmap resolution being lower than the input image", which easily introduces quantization errors. Although some scholars have proposed improvement schemes such as improving the accuracy of heatmap generation and optimizing the logic of heatmap coordinate derivation, the balance between "improving localization accuracy and optimizing computational efficiency" has not been completely solved, making it difficult to meet the real-time detection needs in aquaculture scenarios.
[0005] Existing fish body key point detection technologies still have significant shortcomings in terms of detection accuracy, environmental robustness, real-time performance, and adaptability to aquaculture. There is an urgent need to propose a fish body key point detection solution that can adapt to complex aquaculture environments and balance accuracy and efficiency. Summary of the Invention
[0006] This invention proposes a fish body keypoint detection method based on dual attention enhancement HRNet. Specifically, it constructs a deep learning network that integrates global-local feature fusion and dual attention enhancement, and combines it with a distribution-aware coordinate balance representation method to comprehensively solve the problems of scarce fish body keypoint datasets and low keypoint detection accuracy.
[0007] This invention provides a method for detecting key points in fish bodies based on dual-attention enhanced HRNet, the method comprising: S1. Obtain underwater fish images, annotate key points, and construct a fish key point dataset; S2, Construct the initial feature extraction module for global-local feature fusion; S3, a fish keypoint detection model is built based on the HRNet network including the initial feature extraction module, and a dual attention mechanism module is embedded; S4. In the fish key point detection model, a distribution-aware method is used to alleviate the quantization bias of coordinate regression. S5, based on quantization bias and fish keypoint dataset, detects keypoints and restores them to the original image space, outputting the fish body keypoint detection results.
[0008] Optionally, the acquired underwater fish images include single-fish image data and multi-fish image data; the key points are labeled by using LabelMe to select 7 key points on the fish image as fish body characteristics for labeling, including eyes, mouth, dorsal fin, anal fin, pectoral fin, pelvic fin and tail.
[0009] Optionally, the initial feature extraction module for constructing global-local feature fusion includes: The initial feature extraction module includes Conv branch, Sobel branch, and Transformer branch; The Conv branch is used to extract local feature information. The local feature extraction process is expressed by the following formula: in, Indicates input, express Convolution kernel, LN denotes layer normalization, This represents the activation function. This indicates max pooling. express Convolution kernel; The Sobel branch is used to extract gradient information, and the edge feature extraction process is expressed by the following formula: in, Indicates input, This represents the gradient operator in the horizontal direction. This represents the gradient operator in the vertical direction. Indicates taking the absolute value; The Transformer branch is used to extract global long-range dependencies. The global feature extraction process is expressed by the following formula: in, Representation layer normalization, This indicates the bulls' self-attention. Indicates a feedforward network; This indicates patch embedding processing. Indicates stacking Layer Transformer Block, Indicates stacking Layer Transformer Block; The global-local feature fusion module fuses the features extracted from the three branches. The fusion process can be expressed by the following formula: in, This indicates the output of the Conv branch. This indicates the output of the Sobel branch. This indicates the output of the Transformer branch. This indicates the output of the global-local feature fusion module; Concat indicates feature fusion.
[0010] Optionally, the fish keypoint detection model constructed based on the HRNet network including the initial feature extraction module, and the embedded dual attention mechanism module, includes: The dual attention mechanism module consists of a positional attention mechanism and a channel attention mechanism; The channel attention mechanism can be expressed by the following formula: The positional attention mechanism can be expressed by the following formula: in, Representing feature channels , Representing feature channels , Indicates the feature size, with the number of channels being... , height is , width is , Indicates the scale parameter. It is obtained by deforming the input feature map. Representation of features passage , Representation of features passage , This represents the scaling parameter.
[0011] Optionally, the use of a distribution-aware method in the fish keypoint detection model to mitigate coordinate regression quantization bias includes: The distributed sensing method predicts the offset of the true key point position relative to the maximum point on the heatmap during the decoding process using a second-order Taylor expansion. The calculation formula is as follows: in, It predicts the location of a pixel in a heatmap. This represents the Gaussian mean of the joint positions to be estimated. Represents the transpose and covariance of a matrix. It is a diagonal matrix, with the same coordinate encoding.
[0012] Optionally, the step of restoring the detected key points to the original image space based on quantization bias and fish key point dataset, and outputting the fish body key point detection results, includes: Based on distribution perception The formula for predicting the coordinates in the original image space is: in, Indicates the resolution reduction rate. This represents the corresponding coordinate point in the image space after resolution reduction. Represents the coordinates of a point in the original image space.
[0013] The technical solution provided by this invention comprehensively considers the problems of insufficient datasets and insufficient lighting and noise interference in underwater images when training fish keypoint detection networks. The constructed network has strong generalization ability and can accurately detect underwater fish keypoints. Its beneficial effects include at least the following: First, this invention constructs a targeted dataset of key points on fish bodies, which are universal in fish. This dataset makes it possible to establish a public dataset similar to human pose estimation.
[0014] Second, this invention proposes a high-resolution keypoint detection network that integrates global-local feature fusion and dual attention enhancement. This significantly enhances feature extraction capabilities in complex environments, enabling accurate capture of discriminative features of key points on fish bodies in complex underwater environments such as turbid water and fluctuating lighting.
[0015] Third, this invention employs a distributed sensing method to effectively alleviate the quantization bias problem in traditional coordinate regression. This provides high-precision data support for subsequent applications such as body shape measurement and behavioral analysis. Attached Figure Description
[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a flowchart of a fish body key point detection method based on dual attention enhancement HRNet according to the present invention; Figure 2 This is an overall flowchart of the high-resolution keypoint detection network with global-local feature fusion and dual attention enhancement of the present invention; Figure 3 This is an example diagram showing the annotation of key points on the fish body according to the present invention; Figure 4 This is a detailed architectural diagram of the global-local feature fusion module of the present invention; Figure 5 This is a diagram illustrating the overall architecture of the high-resolution keypoint detection network with global-local feature fusion and dual attention enhancement according to the present invention. Figure 6 This is a detailed diagram of the architecture of the key point detection network of the present invention, which embeds a dual attention module. Detailed Implementation
[0018] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.
[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0020] refer to Figure 1 This document illustrates the flowchart of some embodiments of a fish body keypoint detection method based on dual-attention enhanced HRNet according to the present invention. The overall flowchart of global-local feature fusion embedding a high-resolution keypoint detection network with dual-attention enhancement can be as follows: Figure 2As shown.
[0021] Specifically, a method for detecting key points in fish bodies based on dual attention-enhanced HRNet includes the following steps: Step S1: Obtain underwater fish images, annotate key points, and construct a fish key point dataset.
[0022] In some embodiments, underwater fish images can be acquired, key points can be labeled, and a fish key point dataset can be constructed.
[0023] The acquired underwater fish images include single-fish and multi-fish images. Keypoint annotation was performed using LabelMe to select seven key points on the fish images as fish characteristic features: eyes, mouth, dorsal fin, anal fin, pectoral fin, pelvic fin, and tail. Some of the fish images in this embodiment were provided by an agricultural university and captured using an underwater camera system in intensive aquaculture ponds; the other part was collected and compiled from the internet.
[0024] Constructing a targeted fish key point dataset can include: prioritizing on-site image collection of spotted sea bream in aquaculture ponds, guided by the core scenarios of intensive aquaculture, to ensure the data is highly compatible with the actual aquaculture environment; simultaneously, to supplement data diversity and expand the coverage of species and scenarios, selectively selecting fish image datasets containing complete fish bodies and key point annotations through the internet. Ultimately, this results in three clearly defined basic datasets: first, a single-fish image dataset with a resolution of 720×1280; second, a single-fish image dataset with a resolution of 1242×2208; and third, a multi-fish image dataset with a resolution of 1080×1920.
[0025] The LabelMe annotation tool was used for pixel-level keypoint annotation. Combining fish biological characteristics and the needs of intensive aquaculture applications, seven core keypoints were identified as fish body characterization features, achieving a precise correspondence between the annotated points and actual application requirements. Specifically, the LabelMe software was used for keypoint annotation, selecting seven key points on the fish body as body shape characteristics for annotation. These could include: the eyes (for determining fish orientation) and mouth (starting point for body shape measurement) on the head; the starting point of the dorsal fin (key to posture analysis) and the starting point of the anal fin (aiding posture judgment) on the trunk; the pectoral fins (for behavioral status recognition) and pelvic fins (for balance posture analysis) on the fins; and the tail (ending point for body shape measurement) on the tail. The actual keypoint annotation locations could be as follows: Figure 3 As shown, the labeled content is then organized and converted into the COCO dataset format for subsequent input into the keypoint detection network model training.
[0026] To address the issues of limited basic dataset size and insufficient model generalization ability, a multi-dimensional data augmentation strategy was designed for variables such as fish swimming direction, posture, lighting conditions, and water disturbance in aquaculture scenarios. This strategy involves performing random horizontal flipping, random rotation, brightness adjustment, and Gaussian denoising on the original images. These operations effectively expand the dataset size and improve the model's adaptability to complex aquaculture environments.
[0027] Step S2: Construct the initial feature extraction module for global-local feature fusion.
[0028] In some embodiments, an initial feature extraction module for global-local feature fusion can be constructed, denoted as the global-local feature fusion module. The architectural details of the global-local feature fusion module can be as follows: Figure 4 As shown.
[0029] It should be noted that the initial feature extraction module consists of three parts: the Conv branch, the Sobel branch, and the Transformer branch, which are used to extract the initial features of the input image.
[0030] The Conv branch is used to extract local feature information. The Conv branch first... Convolution adjusts the number of channels in the feature map to 64 without changing the feature map size, then normalizes it using an LN layer, and finally increases the network's non-linear expressive power using the ReLU activation function. The extracted features are then processed... Convolution and max pooling operations are performed, followed by concat to merge feature maps, and finally... The number of channels in convolution is adjusted. The mathematical model of convolution branches can be expressed by the following formula: in, Indicates input, express Convolution kernel, LN denotes layer normalization, This represents the activation function. This indicates max pooling. express Convolution kernel.
[0031] The Sobel branch is used to extract gradient information and enhance image edge response. The Sobel branch first... The number of channels is adjusted by convolution, and then the horizontal and vertical information is processed separately through Sobel filtering. The absolute values of the results are taken and summed, and finally... Convolution adjusts the number of channels. The Sobel branch mathematical model can be expressed by the following formula: in, Indicates input, This represents the gradient operator in the horizontal direction. This represents the gradient operator in the vertical direction. This indicates taking the absolute value.
[0032] The Transformer branch is used to extract global information, improving the model's ability to handle long-range dependencies. The Transformer branch first processes the input image through patch embedding, transforming the input image's 3D tensor into a sequence containing all patch embeddings, which is then input into the Transformer Block to extract global information. The mathematical model of the Transformer branch can be expressed by the following formula: in, Representation layer normalization, This indicates the bulls' self-attention. This represents a feedforward network. This indicates patch embedding processing. Indicates stacking Layer Transformer Block, This indicates that three layers of Transformer Blocks are stacked.
[0033] The global-local feature fusion module fuses the features extracted from the three branches. The fusion process can be represented by the following formula: in, This indicates the output of the Conv branch. This indicates the output of the Sobel branch. This indicates the output of the Transformer branch. This indicates the output of the global-local feature fusion module; Concat indicates feature fusion.
[0034] Step S3: Construct a fish keypoint detection model based on the HRNet network including the initial feature extraction module, and embed a dual attention mechanism module.
[0035] In some embodiments, a fish keypoint detection model can be built based on the HRNet network, embedding a dual attention mechanism module.
[0036] The overall architecture of a high-resolution keypoint detection network that integrates global-local feature fusion and dual attention enhancement can be described as follows: Figure 5As shown. The overall structure of the dual attention mechanism module can be as follows: Figure 6 As shown, it consists of a positional attention mechanism and a channel attention mechanism.
[0037] Channel attention selectively emphasizes interdependent channel maps by integrating the correlation properties among all channel maps. In the channel attention mechanism, firstly... Remodeling Dimension, and again Multiplication is performed on the transpose of the matrix. Then, the channel attention map is derived using a softmax layer. Next, The transpose matrix and the original features Perform matrix multiplication and reshape the result back to its original state. Dimensions. The final output is weighted by a scaling factor and then element-wise added to the initial features to generate the final output. The mathematical model of the channel attention mechanism can be expressed by the following formula: in, Representing feature channels , Representing feature channels , Indicates the feature size, with the number of channels being... , height is , width is , Indicates the scale parameter.
[0038] Location attention selectively aggregates features at each location by weighted summing of features across all locations. In the location attention mechanism, the original features... As input, the data is first processed through a convolutional layer to generate two additional feature maps. and .right and After transposing, matrix multiplication is performed, and the spatial attention map is calculated through a softmax layer. At the same time, features New feature maps are generated after additional convolutional transformation. Subsequently, on and Perform matrix multiplication on the transpose of the matrix, and reshape the result into... Size. Ultimately, this is achieved through interaction with features. The element-wise addition operation, multiplied by the parameter. Obtain output features The mathematical model of the positional attention mechanism can be expressed by the following formula: in, It is obtained by deforming the input feature map. Representation of features passage , Representation of features passage , This represents the scaling parameter.
[0039] To fully extract long-range contextual information, the outputs of the two modules are first processed through convolutional layers and then feature fusion is performed to obtain relatively effective feature information.
[0040] Step S4: In the fish key point detection model, a distribution-aware method is used to alleviate the quantization bias of coordinate regression.
[0041] In some embodiments, a distributed sensing method can be used to mitigate coordinate regression quantization bias.
[0042] Distribution-aware relocalization finely adjusts the coordinates of keypoint detection. Distribution-aware assumes that both the predicted and labeled heatmaps follow a Gaussian distribution. During decoding, a second-order Taylor expansion is used to predict the offset of the true keypoint position relative to the maximum point on the heatmap. The calculation formula is as follows: in, It predicts the location of a pixel in a heatmap. This represents the Gaussian mean of the joint positions to be estimated. Represents the transpose and covariance of a matrix. It is a diagonal matrix, identical to the coordinate encoding. Then, a logarithmic transformation is performed on the above formula to obtain: Combining the following three formulas After the transformation, we can obtain Q(⋅) Maximum value: Step S5: Based on the quantization bias and fish keypoint dataset, detect keypoints and restore them to the original image space, then output the fish body keypoint detection results.
[0043] In some embodiments, key points can be detected and restored to the original image space, outputting the key point detection results for the fish body. This is based on distribution-aware... , Representing the horizontal component of the coordinate system, and calculating... Similarly, by calculating the vertical component, we can obtain the two-dimensional coordinates of the point, and then predict the coordinates of the original image space using the following formula: in, Indicates the resolution reduction rate. This represents the corresponding coordinate point in the image space after resolution reduction. Represents the coordinates of a point in the original image space.
[0044] After processing by a deep learning-based keypoint detection network and a distribution-aware coordinate equalization algorithm, the problems of low accuracy and poor robustness in underwater fish keypoint detection can be effectively solved, and accurate keypoint detection results can be obtained.
[0045] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for detecting key points in fish bodies based on dual-attention enhanced HRNet, characterized in that, Includes the following steps: S1. Obtain underwater fish images, annotate key points, and construct a fish key point dataset; S2, Construct the initial feature extraction module for global-local feature fusion; S3, a fish keypoint detection model is built based on the HRNet network including the initial feature extraction module, and a dual attention mechanism module is embedded; S4. In the fish key point detection model, a distribution-aware method is used to alleviate the quantization bias of coordinate regression. S5, based on quantization bias and fish keypoint dataset, detects keypoints and restores them to the original image space, outputting the fish body keypoint detection results.
2. The method for detecting key points in a fish body based on dual-attention enhanced HRNet according to claim 1, characterized in that, The acquired underwater fish images include single-fish image data and multi-fish image data; the key points were labeled using LabelMe to select 7 key points on the fish images as fish body characteristics, including eyes, mouth, dorsal fin, anal fin, pectoral fin, pelvic fin and tail.
3. The method for detecting key points in a fish body based on dual-attention enhanced HRNet according to claim 1, characterized in that, The initial feature extraction module for constructing global-local feature fusion includes: The initial feature extraction module includes Conv branch, Sobel branch, and Transformer branch; The Conv branch is used to extract local feature information. The local feature extraction process is expressed by the following formula: in, Indicates input, express Convolution kernel, LN denotes layer normalization, This represents the activation function. This indicates max pooling. express Convolution kernel; The Sobel branch is used to extract gradient information, and the edge feature extraction process is expressed by the following formula: in, Indicates input, This represents the gradient operator in the horizontal direction. This represents the gradient operator in the vertical direction. Indicates taking the absolute value; The Transformer branch is used to extract global long-range dependencies. The global feature extraction process is expressed by the following formula: in, Representation layer normalization, This indicates the bulls' self-attention. Indicates a feedforward network; This indicates patch embedding processing. Indicates stacking Layer Transformer Block, Indicates stacking Layer Transformer Block; The global-local feature fusion module fuses the features extracted from the three branches. The fusion process can be expressed by the following formula: in, This indicates the output of the Conv branch. This indicates the output of the Sobel branch. This indicates the output of the Transformer branch. This indicates the output of the global-local feature fusion module; Concat indicates feature fusion.
4. The method for detecting key points in a fish body based on dual-attention enhanced HRNet according to claim 1, characterized in that, The fish keypoint detection model built on the HRNet network, including an initial feature extraction module, and embedding a dual attention mechanism module, includes: The dual attention mechanism module consists of a positional attention mechanism and a channel attention mechanism; The channel attention mechanism can be expressed by the following formula: The positional attention mechanism can be expressed by the following formula: in, Representing feature channels , Representing feature channels , Indicates the feature size, with the number of channels being... , height is , width is , Indicates the scale parameter. It is obtained by deforming the input feature map. Representation of features passage , Representation of features passage , This represents the scaling parameter.
5. The method for detecting key points in a fish body based on dual-attention enhanced HRNet according to claim 1, characterized in that, The use of a distribution-aware method in the fish keypoint detection model to mitigate coordinate regression quantization bias includes: The distributed sensing method predicts the offset of the true key point position relative to the maximum point on the heatmap during the decoding process using a second-order Taylor expansion. The calculation formula is as follows: in, It predicts the location of a pixel in a heatmap. This represents the Gaussian mean of the joint positions to be estimated. Represents the transpose and covariance of a matrix. It is a diagonal matrix, with the same coordinate encoding.
6. The method for detecting key points in a fish body based on dual-attention enhanced HRNet according to claim 1, characterized in that, The method, based on quantization bias and fish keypoint dataset, detects keypoints and restores them to the original image space, outputting fish body keypoint detection results, including: Based on distribution perception The formula for predicting the coordinates in the original image space is: in, Indicates the resolution reduction rate. This represents the corresponding coordinate point in the image space after resolution reduction. This represents the coordinates of a point in the original image space.