A computer vision-based key point positioning method and system for snakehead

Abstract

The application discloses a kind of based on computer vision's key point positioning method and system of Ophiocephalus, by obtaining Ophiocephalus side image and normalization and data enhancement, extract initial feature map;Multi-resolution branch network is constructed, and global and local features are extracted using double-flow Mamba module and are fused by dynamic gating;Cross-branch interaction and residual fusion are carried out through fully connected multi-resolution attention fusion module, to generate multi-scale feature map;Based on the key point attention mechanism of mask guide generates heat map perception mask, strengthens key point response, suppresses noise, and obtains optimized key point heat map feature;Again, pixel coordinates are extracted by decoding heat map and Gaussian processing, to realize key point visual positioning.The method of the application has high positioning accuracy and strong robustness, and is suitable for Ophiocephalus posture analysis and intelligent fishery monitoring.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and smart agriculture technology, and in particular to a method and system for locating key points of snakehead fish based on computer vision. Background Technology

[0002] Key points on the body surface of snakehead fish reflect an individual's size, shape, weight, and movement behavior, serving as crucial evidence for assessing its growth status, health level, and ecological behavior. Currently, traditional methods for locating key points in snakehead fish primarily rely on manual annotation and measurement. While intuitive, these methods suffer from low efficiency, high cost, and significant susceptibility to human factors, making them unsuitable for large-scale aquaculture environments and real-time monitoring needs. Traditional image processing methods based on morphological or geometric features exhibit poor stability and generalization ability when faced with different individuals, different growth stages, and complex environments such as changes in light and water flow.

[0003] In recent years, with the rapid development of computer vision and deep learning technologies, automated keypoint localization methods for snakehead fish based on convolutional neural networks have gradually become a research hotspot. These methods borrow from human pose estimation models (such as Hourglass, CPN, HRNet, etc.), training the model with a large amount of sample data to automatically learn snakehead image features, thereby achieving high-precision keypoint detection. However, due to the large variations in snakehead body size, complex poses, uneven surface texture, and variable underwater lighting conditions, directly applying existing human keypoint detection techniques is difficult to achieve ideal results.

[0004] Therefore, in response to the problems of the snakehead's varied morphology, large differences in posture, and interference from environmental light and shadow, there is an urgent need to propose a key point localization method and system based on computer vision that is more consistent with the characteristics of snakehead images. This would enable high-precision automatic identification of key points of snakehead, improve the robustness and stability of localization, and provide technical support for intelligent fisheries and ecological monitoring. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this invention proposes a computer vision-based method and system for locating key points in snakehead fish.

[0006] The first aspect of this invention provides a computer vision-based method for locating key points in snakehead fish, comprising: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

[0007] In this scheme, the acquisition of snakehead side-view image data, the normalization and data augmentation preprocessing of the image data, the convolution operation of the preprocessed image data, and the extraction of basic feature information of the preprocessed image to obtain an initial feature map are specifically as follows: The image data of the snakehead was obtained by acquiring side view image data using high-definition camera equipment. The image data was then processed by pixel normalization and enhanced by random rotation and brightness adjustment of the image direction to obtain preprocessed image data. Obtain the accuracy requirement information for key point localization of snakehead, and determine the resolution for key point heatmap construction based on the accuracy requirement information. The preprocessed image data is input into an initial convolutional module consisting of two 3×3 convolutional layers. The convolution operation reduces the image resolution to the resolution required for constructing the keypoint heatmap, while simultaneously expanding the number of channels in the preprocessed image data to 32, thereby generating an initial feature map. The specific formula is as follows: , in, This indicates preprocessed image data. This represents a 3×3 convolution operation.

[0008] In this scheme, a multi-resolution branch network is constructed based on the initial feature map. A dual-stream Mamba enhancement module is used to extract features from each resolution branch to obtain global long-range dependency features and local detail features. The global long-range dependency features and local detail features are then fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. Specifically: The initial feature map is constructed using a transition layer to create multi-resolution branches. This transition layer employs a convolution operation with a stride of 2 to downsample the initial feature map, reducing its resolution to half that of the original image. Simultaneously, a 1×1 convolution doubles the number of channels, generating low-resolution branches. At the same time, it preserves the original resolution features as a high-resolution branch. ; The resulting low-resolution branch The downsampling operation is performed iteratively until the resolution of the latest low-resolution branch reaches the preset value. All obtained resolution branches are then integrated to form a multi-resolution branch network. In each resolution branch, dual-stream Mamba enhancement modules are stacked sequentially for feature extraction. The dual-stream Mamba enhancement modules include a global Mamba stream, a local residual stream, and a dynamic gating module. In the global Mamba stream, features from each resolution branch in the input multi-resolution branch network are processed. Depthwise separable convolution is performed, and the processed result is added to the corresponding resolution branch features before convolution through residual connections. This result is then input into the state space model to model long-range spatial dependencies, capturing the global spatial dependencies between key points on the fish body, resulting in global long-range dependency features. These global long-range dependency features include the spatial relationship features between the fish head and tail, and the symmetry constraint features between the dorsal and ventral fins, specifically: , Where Mamba represents sequence modeling operation based on state space model, and DWConv represents depthwise separable convolution operation; In the local residual flow, an improved ShuffleBlock structure is adopted, which enhances feature interaction through channel shuffling operations. Combined with depthwise separable convolution, high-resolution local detail features, including scale texture and eye contour, are extracted. The extraction process is as follows: , ShuffleBlock includes channel shuffling and depthwise separable convolution operations; In the dynamic gating module, global long-range dependency features and local detail features are concatenated, and adaptive fusion weights are generated through a multilayer perceptron to obtain the fusion features of each resolution branch. The specific fusion formula is as follows: , , Where σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, and Concat represents the feature concatenation operation.

[0009] In this scheme, the fully connected multi-resolution attention fusion module performs cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. It transmits feature information between any branches through grouped cross-attention and maintains the stability of feature fusion using residual connections, resulting in a fused multi-scale feature map. Specifically: The fused features from each resolution branch are uniformly aligned to the high-resolution branch through upsampling or downsampling operations. The resolution scale is used to obtain the aligned low-resolution branch features. Eliminate spatial misalignment between multi-scale features and construct a unified representation space; The bidirectional information between any two branches is fused based on the grouped cross-attention mechanism, and the features of each branch are used as the query subject. Actively aggregate keys from other branches Sum Semantic information, where the cross-branch attention weight matrix is ​​calculated using the following formula: , , in, The dimension of the key vector is denoted by _i_, and the Softmax function is used to normalize the attention weights. _i_ and _j_ represent the indices of different branches, where _i_ ≠ _j_. , , The weight matrix is ​​a learnable matrix; The fused features After adjusting the channel dimensions using 1×1 convolution, it is compared with the original high-resolution branch features. Perform residual connections to output a stable fused multi-scale feature map. The specific formula is as follows: , Where Conv1×1 represents a 1×1 convolution operation.

[0010] In this scheme, the mask-guided keypoint attention mechanism dynamically generates a heatmap-aware mask on the multi-scale feature map. Based on the heatmap-aware mask, the response of keypoint regions is enhanced and background noise interference is suppressed to generate optimized keypoint heatmap features. Specifically: The fused multi-scale feature map is input into a mask-guided keypoint attention module. This module compresses the channel dimension of the multi-scale feature map using a lightweight 1×1 convolutional layer, and then generates an attention mask M with a correlation greater than a preset correlation to the spatial distribution of keypoints using a Sigmoid activation function. The specific formula is as follows: , The generated attention mask M is combined with the multi-scale feature map Perform element-wise multiplication to generate optimized key point heatmap features. The specific formula is as follows: , Here, ⊙ represents element-wise multiplication.

[0011] In this scheme, the decoding of the key point heatmap features to output a probability distribution heatmap of the key points of the snakehead fish, and the Gaussian decoding of the probability distribution heatmap to extract the pixel coordinates of the key points, specifically involves: The key point heatmap features are input into the output module, which consists of a 1×1 convolutional layer. The convolution operation maps the number of channels to the number of key points K, generating a probability distribution heatmap of the key points of the snakehead. The formula is expressed as: , Gaussian decoding is performed on the probability distribution heatmap. For each key point K, the position of the maximum response value of the key point in the probability distribution heatmap is located. A local window is constructed with the position of the maximum response value according to a preset step size. The response distribution within the local window is fitted by a Gaussian function, and the sub-pixel level coordinate offset is calculated. The pixel coordinates of each key point are determined based on the coordinate offset.

[0012] In this solution, the visualization operation of the key points of the snakehead based on the pixel coordinates of the key points to generate the key point localization result is specifically as follows: The pixel coordinates of the key points are superimposed on the original side view image of the snakehead, and a marker symbol of a preset shape is drawn at the pixel coordinate position of each key point to generate a visual image with position markers. The visualized image is displayed as the output result on the user's front end.

[0013] A second aspect of the present invention also provides a computer vision-based key point localization system for snakehead fish. The system includes a memory and a processor. The memory includes a computer vision-based key point localization method program for snakehead fish. When the processor executes the computer vision-based key point localization method program, it performs the following steps: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

[0014] This invention discloses a computer vision-based method and system for keypoint localization in snakehead fish. The method involves acquiring a side-view image of the snakehead, normalizing and augmenting it, and extracting an initial feature map. A multi-resolution branch network is constructed, and a dual-stream Mamba module is used to extract global and local features, which are then dynamically gated and fused. A fully connected multi-resolution attention fusion module is used for cross-branch interaction and residual fusion to generate multi-scale feature maps. A mask-guided keypoint attention mechanism is used to generate a heatmap-aware mask, enhancing keypoint response and suppressing noise, resulting in optimized keypoint heatmap features. The heatmap is then decoded and Gaussian processed to extract pixel coordinates, achieving visualized keypoint localization. This method offers high localization accuracy and robustness, making it suitable for snakehead posture analysis and intelligent fisheries monitoring. Attached Figure Description

[0015] Figure 1 A flowchart of a computer vision-based key point localization method for snakehead fish according to the present invention is shown; Figure 2 A flowchart illustrating the branch feature fusion of the dual-stream Mamba enhancement module of the present invention is shown; Figure 3 A flowchart of the fully connected multi-resolution attention fusion method of the present invention is shown; Figure 4 A block diagram of a computer vision-based key point localization system for snakehead fish according to the present invention is shown. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0018] Figure 1 The flowchart of a computer vision-based key point localization method for snakehead fish according to the present invention is shown.

[0019] like Figure 1 As shown, the first aspect of the present invention provides a computer vision-based method for locating key points of snakehead fish, comprising: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

[0020] It should be noted that by normalizing and data augmenting the acquired snakehead side-view images, the model's adaptability to complex underwater scenes and different lighting conditions is effectively improved. Subsequently, basic feature information is extracted through convolutional operations to construct an over-resolution branch network. After constructing the multi-resolution branch network, a dual-stream Mamba enhancement module is used to extract features from each resolution branch, which can simultaneously capture global long-range dependencies across parts of the fish, such as the head and tail, as well as local fine details such as scale texture and eye contours. The dynamic gating mechanism adaptively fuses global and local information, giving the model a stronger deformation adaptability when facing complex postures such as fish bending and occlusion. The fully connected multi-resolution attention fusion module is used for cross-branch interaction, breaking the limitation of traditional methods that are limited to interaction between adjacent branches. Bidirectional information transfer between branches of any resolution is achieved through grouped cross-attention. This mechanism, combined with residual connections, significantly improves the model's modeling accuracy of the correlation between multi-scale key points while ensuring the stability of feature fusion. A mask-guided keypoint attention mechanism can dynamically generate heatmap-aware masks strongly correlated with the spatial distribution of keypoints on multi-scale feature maps. Through pixel-level weight allocation, it effectively enhances the response of key regions such as the fish eye and fin joints, while suppressing background noise interference. Among them, *Channa argus* includes *Channa argus* (… Channa argus ), spotted snakehead ( Channa maculata ), hybrid snakehead ( Channa argus × Channa maculata ).

[0021] According to an embodiment of the present invention, the step of acquiring the side-view image data of the snakehead fish involves normalizing and performing data augmentation preprocessing on the image data, convolving the preprocessed image data, and extracting the basic feature information of the preprocessed image to obtain an initial feature map. Specifically, this process includes: The image data of the snakehead was obtained by acquiring side view image data using high-definition camera equipment. The image data was then processed by pixel normalization and enhanced by random rotation and brightness adjustment of the image direction to obtain preprocessed image data. Obtain the accuracy requirement information for key point localization of snakehead, and determine the resolution for key point heatmap construction based on the accuracy requirement information. The preprocessed image data is input into an initial convolutional module consisting of two 3×3 convolutional layers. The convolution operation reduces the image resolution to the resolution required for constructing the keypoint heatmap, while simultaneously expanding the number of channels in the preprocessed image data to 32, thereby generating an initial feature map. The specific formula is as follows: , in, This indicates preprocessed image data. This represents a 3×3 convolution operation.

[0022] It should be noted that the snakehead side view image data can also be snakehead side view image data uploaded by the user's front end; assuming the input snakehead image (resolution H×W×3, for example 256×192×3) is processed by the initial convolution module, the image resolution is reduced to the heatmap resolution, assuming it is 64×48, which is 1 / 4 of the original image resolution, while the number of channels is increased to 32. The output initial feature map is denoted as... .

[0023] According to an embodiment of the present invention, the multi-resolution branch network is constructed based on the initial feature map, and a dual-stream Mamba enhancement module is used to extract features from each resolution branch to obtain global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are then fused using a dynamic gating mechanism to obtain the fused features of each resolution branch. Specifically: The initial feature map is constructed using a transition layer to create multi-resolution branches. This transition layer employs a convolution operation with a stride of 2 to downsample the initial feature map, reducing its resolution to half that of the original image. Simultaneously, a 1×1 convolution doubles the number of channels, generating low-resolution branches. At the same time, it preserves the original resolution features as a high-resolution branch. ; It should be noted that after extracting the initial features, branches with different resolutions need to be constructed for feature processing, and the transition layer... Perform downsampling to generate low-resolution branches .

[0024] The resulting low-resolution branch The downsampling operation is performed iteratively until the resolution of the latest low-resolution branch reaches the preset value. All obtained resolution branches are then integrated to form a multi-resolution branch network. In each resolution branch, dual-stream Mamba enhancement modules are stacked sequentially for feature extraction. The dual-stream Mamba enhancement modules include a global Mamba stream, a local residual stream, and a dynamic gating module. In the global Mamba stream, features from each resolution branch in the input multi-resolution branch network are processed. Depthwise separable convolution is performed, and the processed result is added to the corresponding resolution branch features before convolution through residual connections. This result is then input into the state space model to model long-range spatial dependencies, capturing the global spatial dependencies between key points on the fish body, resulting in global long-range dependency features. These global long-range dependency features include the spatial relationship features between the fish head and tail, and the symmetry constraint features between the dorsal and ventral fins, specifically: , Where Mamba represents sequence modeling operation based on state space model, and DWConv represents depthwise separable convolution operation; In the local residual flow, an improved ShuffleBlock structure is adopted, which enhances feature interaction through channel shuffling operations. Combined with depthwise separable convolution, high-resolution local detail features, including scale texture and eye contour, are extracted. The extraction process is as follows: , ShuffleBlock includes channel shuffling and depthwise separable convolution operations; In the dynamic gating module, global long-range dependency features and local detail features are concatenated, and adaptive fusion weights are generated through a multilayer perceptron to obtain the fusion features of each resolution branch. The specific fusion formula is as follows: , , Where σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, and Concat represents the feature concatenation operation.

[0025] It should be noted that the global Mamba stream transforms the features of a 2D fish image into a sequential model through a unidirectional spatial scanning mechanism (such as row-first scanning), capturing long-distance dependencies across the entire image domain (such as the spatial relationship between the fish head and tail, and the symmetry constraints between the dorsal and ventral fins). The local residual stream retains the improved Shuffle Block module, which efficiently extracts high-resolution detail features such as scale texture and eye contours through channel shuffling and depthwise separable convolution. The dynamic gating module concatenates the features of the global and local streams, and generates dynamic fusion weights through MLP to achieve intelligent allocation that emphasizes global structure in deformable regions (such as a curved fish body) and local features in detailed regions (such as fin edges). The multi-resolution branching system is gradually expanded through staged processing. In subsequent stages, transition layers are used to further downsample the current lowest-resolution branch, generating even lower-resolution branch features, such as 16×12 and 8×6. Assuming that the resolution of the latest low-resolution branch reaches a preset value of 8×6, the resulting resolution branch network obtained through the staged iterative processing of the transition layers includes: 64×48, 32×24, 16×12, and 8×6. Simultaneously, the resolution difference between branches is maintained at a factor of 2. All newly added branches undergo feature enhancement processing using the same dual-stream Mamba enhancement module, forming a complete pyramid-shaped multi-scale feature extraction network architecture.

[0026] According to an embodiment of the present invention, the method of using a fully connected multi-resolution attention fusion module to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch, transferring feature information between any branches through grouped cross-attention, and maintaining the stability of feature fusion through residual connections, to obtain a fused multi-scale feature map, specifically: The fused features from each resolution branch are uniformly aligned to the high-resolution branch through upsampling or downsampling operations. The resolution scale is used to obtain the aligned low-resolution branch features. Eliminate spatial misalignment between multi-scale features and construct a unified representation space; The bidirectional information between any two branches is fused based on the grouped cross-attention mechanism, and the features of each branch are used as the query subject. Actively aggregate keys from other branches Sum Semantic information, where the cross-branch attention weight matrix is ​​calculated using the following formula: , ,

[0027] in, The dimension of the key vector is denoted by _i_, and the Softmax function is used to normalize the attention weights. _i_ and _j_ represent the indices of different branches, where _i_ ≠ _j_. , , The weight matrix is ​​a learnable matrix; The fused features After adjusting the channel dimensions using 1×1 convolution, it is compared with the original high-resolution branch features. Perform residual connections to output a stable fused multi-scale feature map. The specific formula is as follows: , Where Conv1×1 represents a 1×1 convolution operation.

[0028] It should be noted that a fully connected multi-resolution attention fusion method is used to break down resolution barriers, establish semantic bridges between arbitrary branches, and ensure a balance between attention gain and original feature fidelity using residual structures. By aligning branches of different resolutions (64×64, 32×32, 16×16, 8×8) to 64×64 resolution through upsampling / downsampling, spatial misalignment between multi-scale features is eliminated, providing a unified representation space for cross-branch interactions. Bidirectional information fusion between any two branches is achieved through Grouped Cross-Attention. Each branch can serve as a query subject (…). Actively aggregate other branches ( , The semantic information of ) breaks through the limitation of traditional HRNet, which only allows local interactions between adjacent branches. The fusion result ( After adjustment by 1×1 convolution, it is compared with the original branch features ( Addition output To avoid the attention mechanism perturbing the original discriminative features and improve training stability, the multi-resolution branch construction, processing, and interaction process expands progressively at different stages of the network, generating lower-resolution branches. Through the same grouping cross-attention mechanism, all branches are fused together to form a feature tensor that integrates multi-scale information. .

[0029] According to an embodiment of the present invention, the mask-guided keypoint attention mechanism dynamically generates a heatmap-aware mask on the multi-scale feature map, enhances the keypoint region response and suppresses background noise interference based on the heatmap-aware mask, and generates optimized keypoint heatmap features, specifically as follows: The fused multi-scale feature map is input into a mask-guided keypoint attention module. This module compresses the channel dimension of the multi-scale feature map using a lightweight 1×1 convolutional layer, and then generates an attention mask M with a correlation greater than a preset correlation to the spatial distribution of keypoints using a Sigmoid activation function. The specific formula is as follows: , The generated attention mask M is combined with the multi-scale feature map Perform element-wise multiplication to generate optimized key point heatmap features. The specific formula is as follows: , Here, ⊙ represents element-wise multiplication.

[0030] It should be noted that a lightweight 1×1 convolutional layer and a sigmoid activation function are used to generate an attention mask strongly correlated with the spatial distribution of keypoints from the fused features. The mask produces high response values ​​in potential keypoint regions (such as fish eyes and fin joints) and low response values ​​in background regions, achieving pixel-level feature weight allocation. By suppressing background interference and enhancing the feature responses in keypoint regions, the signal-to-noise ratio of the feature map is improved.

[0031] According to an embodiment of the present invention, the step of decoding the key point heatmap features to output a probability distribution heatmap of the key points of the snakehead, and performing Gaussian decoding on the probability distribution heatmap to extract the pixel coordinates of the key points, specifically includes: The key point heatmap features are input into the output module, which consists of a 1×1 convolutional layer. The convolution operation maps the number of channels to the number of key points K, generating a probability distribution heatmap of the key points of the snakehead. The formula is expressed as: , Gaussian decoding is performed on the probability distribution heatmap. For each key point K, the position of the maximum response value of the key point in the probability distribution heatmap is located. A local window is constructed with the position of the maximum response value according to a preset step size. The response distribution within the local window is fitted by a Gaussian function, and the sub-pixel level coordinate offset is calculated. The pixel coordinates of each key point are determined based on the coordinate offset.

[0032] It should be noted that the number of key points K is the key point heatmap feature. The number of channels is 32 in this embodiment. By inputting the keypoint heatmap features into the output module consisting of a 1×1 convolutional layer, the number of channels is mapped to the number of keypoints, generating a probability distribution heatmap of the snakehead keypoints. Subsequently, Gaussian decoding is performed on the probability distribution heatmap to locate the position of the maximum response value of each keypoint in the heatmap. A local window is constructed with this position as the center and a preset step size. The response distribution within the window is fitted by a Gaussian function, and the sub-pixel level coordinate offset is calculated to finally determine the precise pixel coordinates of each keypoint. This effectively overcomes the quantization error problem caused by the resolution limitation of the heatmap in traditional methods. By refining the modeling of the local response distribution, the keypoint positioning accuracy is improved from the pixel level to the sub-pixel level. Especially when dealing with scenes such as curved fish bodies and complex scale textures, Gaussian decoding can adaptively smooth the response fluctuations in local areas, reducing the impact of noise interference on the positioning results, thereby ensuring the continuity and reliability of the keypoint coordinate output.

[0033] According to an embodiment of the present invention, the step of visualizing the key points of the snakehead fish based on the pixel coordinates of the key points to generate the key point localization result of the snakehead fish specifically involves: The pixel coordinates of the key points are superimposed on the original side view image of the snakehead, and a marker symbol of a preset shape is drawn at the pixel coordinate position of each key point to generate a visual image with position markers. The visualized image is displayed as the output result on the user's front end.

[0034] It should be noted that this invention decomposes the input image into a pyramid-shaped feature representation from high-resolution details to low-resolution semantics, generating a "far-view image" (low-resolution branch) showing the overall outline and a "close-up image" (high-resolution branch) showing the scales. Subsequently, a dual-stream enhancement module performs parallel feature processing. The global stream uses a state-space model to capture long-distance dependencies in the fish body, while the local stream preserves fine features through lightweight convolution operations. An adaptive fusion of the two types of features is achieved through a dynamic gating mechanism. Furthermore, a fully connected attention fusion module enables deep interaction across resolution branches, breaking the limitations of traditional adjacent branch interactions and establishing semantic associations between features at any scale. Finally, a mask-guided attention mechanism enhances the keypoint region response and suppresses background interference. A Gaussian decoding algorithm then transforms the heatmap response into sub-pixel-level coordinate output. The model does not simply provide a vague approximate location. Within the "hotspot region" focused in the previous step, it calculates the most accurate sub-pixel-level coordinates of the keypoints using a refined Gaussian fitting algorithm, achieving precise localization.

[0035] According to an embodiment of the present invention, it further includes: Peak detection is performed on the probability distribution heatmap of the key points to locate the maximum response point of each key point, and a local sensing area is constructed with each initial maximum response point as the center. Calculate the response distribution entropy value within each of the local sensing regions, and determine whether there is a risk of overlap or occlusion of the corresponding key points based on the response distribution entropy value. Wherein, an entropy value higher than a preset entropy value indicates that the response distribution is dispersed and has multi-peak characteristics, indicating a high probability of key point overlap or occlusion. Based on the risk assessment results, an anti-suppression mask is dynamically generated. For local sensing regions that are identified as high-risk, the anti-suppression mask retains potential weak response characteristics by expanding the range of the local sensing region and reducing the suppression intensity within the region. The anti-suppression mask is weighted and fused with the attention mask M to obtain an improved attention mask M2, wherein the strong suppression characteristics of the original mask M are maintained in low-risk regions, while the protection characteristics provided by the anti-suppression mask are enhanced in high-risk regions. Utilizing the improved attention mask M2 and multi-scale feature maps Element-wise multiplication is performed to generate an enhanced post-keypoint heatmap feature that is more robust to occlusion and overlap of keypoints. .

[0036] According to an embodiment of the present invention, the step of calculating the response distribution entropy value within each local sensing region and determining whether there is a risk of overlap or occlusion of the corresponding key points based on the response distribution entropy value specifically involves: For the local sensing region of each key point, it is divided into multiple non-overlapping sub-windows, and the mean value of the heat map response in each sub-window is calculated to form the response distribution sequence of the local sensing region. The information entropy is calculated based on the response distribution sequence, and the information entropy is judged using a preset entropy threshold. When the information entropy is greater than the entropy threshold, it is determined that there is a risk of overlap or occlusion in the key point area. The dynamic generation of the anti-suppression mask based on the risk assessment result is specifically as follows: for a local sensing region that is determined to be high-risk, an expansion coefficient is calculated based on its entropy value and the response intensity of its initial maximum response point. The boundary of the local sensing region is then proportionally expanded based on the expansion coefficient to form an expanded sensing region. Within the extended sensing region, a weight map is generated using a Gaussian kernel function. This weight map has the highest weight in the central region and gradually decreases towards the edges. This weight map is used as an anti-suppression mask for the region to ensure that potential weak responses within the extended region are preserved rather than being erased by the global suppression mechanism.

[0037] It's important to note that in real-world, complex aquaculture scenarios, key points on fish bodies often overlap or partially obscure due to varying postures or environmental factors, such as densely intertwined fins or the fish's body being obscured by underwater debris. In such cases, standard heatmap-aware masking mechanisms tend to binarize feature maps based on global thresholds, easily misclassifying crucial but weakly responding overlapping or obscured key point features as background noise and suppressing them. This results in key point loss or severely insufficient response in the final probability distribution heatmap. Therefore, by introducing a risk assessment mechanism based on local response distribution entropy, the system can intelligently identify which key point regions are at risk of abnormally dispersed responses due to overlap or obscuration, thus transforming the "one-size-fits-all" suppression strategy into an adaptive, refined feature preservation strategy. Specifically, for these high-risk areas, the method cleverly balances strengthening the main key point response and preserving potentially weak response cues by dynamically expanding the perception range and applying protective weights. Ultimately, the key point heatmap features generated by this method not only retain the accurate localization capability of clear key points, but also significantly improve the recall rate and localization robustness for occluded or overlapping key points.

[0038] Figure 4 A block diagram of a computer vision-based key point localization system for snakehead fish according to the present invention is shown.

[0039] A second aspect of the present invention also provides a computer vision-based key point localization system 4 for snakehead fish. The system includes a memory 41 and a processor 42. The memory includes a computer vision-based key point localization method program for snakehead fish. When the processor executes the computer vision-based key point localization method program, it performs the following steps: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

[0040] This invention discloses a computer vision-based method and system for keypoint localization in snakehead fish. The method involves acquiring a side-view image of the snakehead, normalizing and augmenting it, and extracting an initial feature map. A multi-resolution branch network is constructed, and a dual-stream Mamba module is used to extract global and local features, which are then dynamically gated and fused. A fully connected multi-resolution attention fusion module is used for cross-branch interaction and residual fusion to generate multi-scale feature maps. A mask-guided keypoint attention mechanism is used to generate a heatmap-aware mask, enhancing keypoint response and suppressing noise, resulting in optimized keypoint heatmap features. The heatmap is then decoded and Gaussian processed to extract pixel coordinates, achieving visualized keypoint localization. This method offers high localization accuracy and robustness, making it suitable for snakehead posture analysis and intelligent fisheries monitoring.

[0041] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0042] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0043] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0044] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0045] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0046] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for locating key points of snakehead fish based on computer vision, characterized in that, Includes the following steps: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

2. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The process involves acquiring side-view images of the snakehead fish, performing normalization and data augmentation preprocessing on the image data, convolving the preprocessed image data, and extracting basic feature information to obtain an initial feature map. Specifically: The image data of the snakehead was obtained by acquiring side view image data using high-definition camera equipment. The image data was then processed by pixel normalization and enhanced by random rotation and brightness adjustment of the image direction to obtain preprocessed image data. Obtain the accuracy requirement information for key point localization of snakehead, and determine the resolution for key point heatmap construction based on the accuracy requirement information. The preprocessed image data is input into an initial convolutional module consisting of two 3×3 convolutional layers. The convolution operation reduces the image resolution to the resolution required for constructing the keypoint heatmap, while simultaneously expanding the number of channels in the preprocessed image data to 32, thereby generating an initial feature map. The specific formula is as follows: ， in, This indicates preprocessed image data. This represents a 3×3 convolution operation.

3. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The process involves constructing a multi-resolution branch network based on the initial feature map, extracting features from each resolution branch using a dual-stream Mamba enhancement module to obtain global long-range dependency features and local detail features. A dynamic gating mechanism is then used to fuse these global long-range dependency features and local detail features to obtain the fused features for each resolution branch. Specifically: The initial feature map is constructed using a transition layer to create multi-resolution branches. This transition layer employs a convolution operation with a stride of 2 to downsample the initial feature map, reducing its resolution to half that of the original image. Simultaneously, a 1×1 convolution doubles the number of channels, generating low-resolution branches. At the same time, it preserves the original resolution features as a high-resolution branch. ; The resulting low-resolution branch The downsampling operation is performed iteratively until the resolution of the latest low-resolution branch reaches the preset value. All obtained resolution branches are then integrated to form a multi-resolution branch network. In each resolution branch, dual-stream Mamba enhancement modules are stacked sequentially for feature extraction. The dual-stream Mamba enhancement modules include a global Mamba stream, a local residual stream, and a dynamic gating module. In the global Mamba stream, features from each resolution branch in the input multi-resolution branch network are processed. Depthwise separable convolution is performed, and the processed result is added to the corresponding resolution branch features before convolution through residual connections. This result is then input into the state space model to model long-range spatial dependencies, capturing the global spatial dependencies between key points on the fish body, resulting in global long-range dependency features. These global long-range dependency features include the spatial relationship features between the fish head and tail, and the symmetry constraint features between the dorsal and ventral fins, specifically: ， Where Mamba represents sequence modeling operation based on state space model, and DWConv represents depthwise separable convolution operation; In the local residual flow, an improved ShuffleBlock structure is adopted, which enhances feature interaction through channel shuffling operations. Combined with depthwise separable convolution, high-resolution local detail features, including scale texture and eye contour, are extracted. The extraction process is as follows: ， ShuffleBlock includes channel shuffling and depthwise separable convolution operations; In the dynamic gating module, global long-range dependency features and local detail features are concatenated, and adaptive fusion weights are generated through a multilayer perceptron to obtain the fusion features of each resolution branch. The specific fusion formula is as follows: ， ， Where σ represents the Sigmoid activation function, MLP represents the multilayer perceptron, and Concat represents the feature concatenation operation.

4. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The fully connected multi-resolution attention fusion module performs cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. It transmits feature information between any branches through grouped cross-attention and maintains the stability of feature fusion using residual connections, resulting in a fused multi-scale feature map. Specifically: The fused features from each resolution branch are uniformly aligned to the high-resolution branch through upsampling or downsampling operations. The resolution scale is used to obtain the aligned low-resolution branch features. Eliminate spatial misalignment between multi-scale features and construct a unified representation space; The bidirectional information between any two branches is fused based on the grouped cross-attention mechanism, and the features of each branch are used as the query subject. Actively aggregate keys from other branches Sum Semantic information, where the cross-branch attention weight matrix is ​​calculated using the following formula: ， ， in, The dimension of the key vector is denoted by _i_, and the Softmax function is used to normalize the attention weights. _i_ and _j_ represent the indices of different branches, where _i_ ≠ _j_. , , The weight matrix is ​​a learnable matrix; The fused features After adjusting the channel dimensions using 1×1 convolution, it is compared with the original high-resolution branch features. Perform residual connections to output a stable fused multi-scale feature map. The specific formula is as follows: ， Where Conv1×1 represents a 1×1 convolution operation.

5. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The mask-guided keypoint attention mechanism dynamically generates a heatmap-aware mask on the multi-scale feature map. Based on this heatmap-aware mask, it enhances the response of keypoint regions and suppresses background noise interference, generating optimized keypoint heatmap features. Specifically: The fused multi-scale feature map is input into a mask-guided keypoint attention module. This module compresses the channel dimension of the multi-scale feature map using a lightweight 1×1 convolutional layer, and then generates an attention mask M with a correlation greater than a preset correlation to the spatial distribution of keypoints using a Sigmoid activation function. The specific formula is as follows: ， The generated attention mask M is combined with the multi-scale feature map Perform element-wise multiplication to generate optimized key point heatmap features. The specific formula is as follows: ， Here, ⊙ represents element-wise multiplication.

6. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The process of decoding the key point heatmap features to output a probability distribution heatmap of the key points of the snakehead fish, and then performing Gaussian decoding on the probability distribution heatmap to extract the pixel coordinates of the key points, specifically involves: The key point heatmap features are input into the output module, which consists of a 1×1 convolutional layer. The convolution operation maps the number of channels to the number of key points K, generating a probability distribution heatmap of the key points of the snakehead. The formula is expressed as: ， Gaussian decoding is performed on the probability distribution heatmap. For each key point K, the position of the maximum response value of the key point in the probability distribution heatmap is located. A local window is constructed with the position of the maximum response value according to a preset step size. The response distribution within the local window is fitted by a Gaussian function, and the sub-pixel level coordinate offset is calculated. The pixel coordinates of each key point are determined based on the coordinate offset.

7. The method for locating key points of snakehead fish based on computer vision according to claim 1, characterized in that, The step of visualizing the key points of the snakehead based on their pixel coordinates to generate key point localization results is as follows: The pixel coordinates of the key points are superimposed on the original side view image of the snakehead, and a marker symbol of a preset shape is drawn at the pixel coordinate position of each key point to generate a visual image with position markers. The visualized image is displayed as the output result on the user's front end.

8. A computer vision-based key point localization system for snakehead fish, characterized in that, The computer vision-based snakehead keypoint localization system includes a storage device and a processor. The storage device includes a computer vision-based snakehead keypoint localization method program. When the computer vision-based snakehead keypoint localization method program is executed by the processor, it performs the following steps: Acquire side-view image data of snakehead fish, perform normalization and data augmentation preprocessing on the image data, perform convolution operation on the preprocessed image data, extract the basic feature information of the preprocessed image, and obtain the initial feature map; A multi-resolution branch network is constructed based on the initial feature map. The dual-stream Mamba enhancement module is used to extract the features of each resolution branch to obtain the global long-range dependency features and local detail features of the image. The global long-range dependency features and local detail features are fused through a dynamic gating mechanism to obtain the fused features of each resolution branch. The fully connected multi-resolution attention fusion module is used to perform cross-branch interaction and bidirectional fusion of the fusion features of each resolution branch. The feature information between any branches is transmitted through grouped cross attention, and the stability of feature fusion is maintained by residual connection, resulting in a fused multi-scale feature map. A mask-guided key point attention mechanism dynamically generates a heat map sensing mask on the multi-scale feature map. Based on the heat map sensing mask, the response of key point regions is enhanced and background noise interference is suppressed to generate optimized key point heat map features. The key point heatmap features are decoded to output a probability distribution heatmap of the key points of the snakehead, and the pixel coordinates of the key points are extracted by Gaussian decoding of the probability distribution heatmap. Based on the pixel coordinates of the key points, the key points of the snakehead are visualized to generate the snakehead key point positioning results.

Images