A manta ray submersible key point detection method based on soft weighting fusion

Abstract

The application relates to a manta ray submarine key point detection method based on soft weighting fusion. An underwater manta ray submarine RGB image is collected, the image is input into an improved U-Net semantic segmentation model and an improved HRNet key point detection model in parallel, pixel-level classification of an underwater scene is completed, an initial heat map of key points is predicted in the whole image range, and initial key point coarse positioning is completed; a soft weighting fusion mechanism is adopted, regional enhancement and suppression of each key point HM are carried out by using a semantic segmentation mask of a corresponding part, multi-part constraint guided by semantic segmentation is realized; maximum value index operation is carried out on the HM after soft weighting fusion constraint, and finally, the key point coordinates of each part are obtained. The beneficial effect relies on the prior accurate division of an effective body region by semantic segmentation, and the key point positioning precision in a complex underwater environment is significantly improved. The soft weighting fusion replaces the traditional hard mask multiplication and feature splicing mode, and solves the problems of rigid constraint and poor adaptability of the traditional method.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision technology and underwater biomimetic robot technology, and relates to a key point detection method for manta ray-inspired submersibles based on soft weighted fusion, which is applicable to the accurate attitude perception of flexible underwater robots such as manta ray-inspired submersibles. Background Technology

[0002] In the field of biomimetic underwater robot technology, as manta ray-inspired submersibles develop towards intelligence, their application in complex dynamic scenarios such as swarm collaboration and autonomous operation is becoming increasingly widespread. A highly robust attitude perception system is the core prerequisite for its stable operation. Among them, key point detection is the core link of attitude perception for manta ray-inspired submersibles, and its positioning accuracy directly determines the reliability of the overall attitude detection and operational stability.

[0003] Currently, existing technologies for key point detection of manta ray-inspired submersibles still have several limitations: traditional underwater sensing methods often combine inertial sensors or basic vision algorithms, which are hardware-intensive, have poor scalability, and are difficult to accurately constrain key points in multiple parts; mainstream key point detection models rely on full-image feature regression prediction, which is easily affected by uneven underwater lighting, water turbidity, and bubble interference, causing key points of flexible structures such as left and right fins to easily shift; localization methods that combine semantic information are currently limited to hard fusion methods for general scenarios, relying on mask multiplication and shallow feature splicing to complete the combination, lacking dynamic adaptive adjustment capabilities, and cannot adapt to the large deformation and movement characteristics of the flexible body of manta rays, making it difficult for the overall detection effect to meet the needs of complex underwater working conditions.

[0004] Therefore, for attitude detection tasks of targets with symmetrical structures and clearly defined components, such as manta ray-inspired submersibles, there is an urgent need for a semantic segmentation-guided key point detection method based on soft-weighted fusion, which can provide a new technical approach to improve attitude perception performance in complex underwater environments. Summary of the Invention

[0005] Technical problems to be solved To avoid the shortcomings of existing technologies, this invention proposes a key point detection method for manta ray-inspired submersibles based on soft-weighted fusion. By relying on semantic segmentation prior and soft-weighted fusion mechanism, it achieves refined constraints on the multiple flexible parts of the manta ray-inspired submersible, thereby improving the attitude perception accuracy and robustness of the manta ray-inspired submersible in complex underwater environments.

[0006] Technical solution A key point detection method for a manta ray-inspired submersible based on soft-weighted fusion, characterized by the following steps: Step 1: Acquire N RGB images of the underwater manta ray-inspired submersible, and scale the images twice: scale to the U-Net input size. Scale to HRNet input size

[0007] The resolution of the RGB image is H×W; The RGB images of the underwater manta ray-inspired submersible include the manta ray-inspired submersible's posture under different lighting and turbidity conditions; Step 2: Input the image into a dual-branch synchronous feature extraction network, a U-Net semantic segmentation model, and an HRNet keypoint detection model for feature extraction; 1. Semantic segmentation mask extraction: [This section likely refers to a process involving] size... The image is input into the U-Net semantic segmentation model. Through the already trained U-Net semantic segmentation model, the pixel-level classification of the RGB image of the manta ray submersible is completed, and the semantic segmentation mask of the classification of the RGB image of the manta ray submersible is obtained. Training the U-Net semantic segmentation model: Input the training data into the U-Net semantic segmentation model and train the U-Net semantic segmentation model using the weighted multi-class cross-entropy loss function; The weighted multi-class cross-entropy loss function is:

[0008] in, This represents the semantic segmentation loss function; This represents the total number of pixels in the image; Indicates the total number of segmentation categories. Indicates pixel position; Indicates the category number; Indicate category The corresponding loss weights; Represents pixels In category The real labels on it; Indicates the model predicts pixels Category The probability of; The training data includes N RGB images of an underwater manta ray-like submersible and their corresponding segmentation mask data for a set classification. The segmentation mask data corresponding to the classification: Each underwater manta ray-like submersible RGB image is segmented according to the defined M categories to form segmentation mask data corresponding to each RGB image; 2. Extraction of original key point heatmap: [The text abruptly ends here, likely due to an incomplete sentence or The image is input into the HRNet keypoint detection model. Using the already trained HRNet keypoint detection model, the initial heatmap of N keypoints is predicted in the entire image, thus completing the initial coarse localization of the keypoints. Training the HRNet keypoint detection model: Input the training data into the HRNet keypoint detection model and use the mean squared error loss function with keypoint weights to train the HRNet keypoint detection model. The mean squared error loss function with keypoint weights is:

[0009] in, This represents the loss function for keypoint detection; Indicates the total number of key points. Indicates the key point number; Indicates the first Heatmap of key prediction points; Indicates the first Gaussian heatmaps corresponding to each real key point; Indicates the first Visibility or weighting coefficient of each key point; The training data includes N RGB images of an underwater manta ray-like submersible and their corresponding N key point annotations. The N key point annotation data: N key points in each underwater manta ray-simulating submersible RGB image are annotated to form key point annotation data corresponding to each RGB image; Step 3: Perform closure operation to repair and align the semantic segmentation mask extracted by the U-Net semantic segmentation model, and then use a soft weighted fusion mechanism with the original key point heatmap extracted by the HRNet key point detection model to perform region enhancement and suppression on each key point HM using the semantic segmentation mask of the corresponding part, so as to realize multi-part constraint guided by semantic segmentation and obtain the fused heatmap HM. The calculation of the heatmap after soft-weighted fusion:

[0010] Step 4: Perform indexing on the maximum value on HM to obtain the final key point coordinates for each part.

[0011] The U-Net semantic segmentation model is an improved U-Net semantic segmentation model. Its model structure includes an encoder, a decoder, a skip connection module, and an output convolution module. The encoder includes multi-level convolutional blocks and a downsampling module, and the decoder includes multi-level upsampling modules and a convolutional refinement module. The shallow spatial detail features of the encoder at the same scale are concatenated with the deep semantic features of the decoder through skip connections. After refinement by convolutional blocks, the maximum value index is taken after passing through the output convolution module to obtain M pixel-level classification results.

[0012] Each convolutional block in the improved U-Net semantic segmentation model contains two 3×3 convolutions, batch normalization, and ReLU activation. The downsampling module uses 2×2 max pooling to progressively reduce the feature map resolution to expand the receptive field and extract deep semantic information. The upsampling module uses 2×2 transposed convolution to restore spatial resolution. The output convolutional module uses 1×1 convolution to map the features into a 5-channel segmentation result, and then performs a maximum value indexing operation to obtain M pixel-level classification results.

[0013] The U-Net semantic segmentation model is trained for 100 epochs with a batch size of 2 and an initial learning rate of 100%. The RMSprop optimizer is used with a momentum of 0.9 and a weight decay of . The learning rate is adaptively adjusted based on the Dice coefficients of the validation set.

[0014] When training the U-Net semantic segmentation model and the HRNet keypoint detection model, 9 / 10 of the training data is used for training, and 1 / 10 is used for testing after training.

[0015] The semantic segmentation mask repair and alignment calculation is performed as follows: M binary masks corresponding to each category are extracted from the semantic segmentation mask output by the U-Net semantic segmentation model. A morphological closing operation with a kernel size of 9×9 and 2 iterations is applied to the M binary masks to obtain the repaired masks. The semantic segmentation mask is then updated based on the repaired masks. Next, corresponding part masks are extracted based on the mapping relationship between keypoints and mask categories, and these part masks are scaled to the size of the HRNet keypoint heatmap, achieving spatial alignment between the semantic segmentation mask and the original keypoint heatmap (HM).

[0016] The HRNet keypoint detection model is an improved HRNet keypoint detection model. Its model structure is based on the HRNet-W32 high-resolution feature preservation network, including a Stem module connecting a multi-stage multi-resolution feature fusion module and a detection head module. The Stem module includes two 3×3 convolutional layers with a stride of 2 for initial feature extraction and downsampling. The multi-stage multi-resolution feature fusion module includes a first-stage single-branch feature extraction, a second-stage high-low dual-branch fusion, a third-stage three-branch fusion, and a fourth-stage four-branch fusion, used to fuse multi-scale semantic information while preserving high-resolution features. The detection head module uses 1×1 convolution to output N-channel keypoint heatmaps, and the N-channel keypoint heatmaps correspond to N predefined keypoints of the manta ray-inspired submersible.

[0017] The soft-weighted fusion module that completes the soft-weighted fusion mechanism includes a heatmap input terminal, a mask input terminal, a Hadamard product calculation unit, a weight scaling unit, and a fusion output terminal; the soft-weighted fused heatmap output by the fusion output terminal satisfies: Where H is the original key point heatmap, M is the corresponding part mask, and α is the soft constraint strength.

[0018] Step 4 involves performing a maximum value index operation on the HM to obtain the final key point coordinates for each part, which is then used to fuse the heatmap. Perform channel-by-channel maximum value positioning and use sub-pixel offset to improve accuracy: First, find the integer coordinates.

[0019] Then, the sub-pixel offset is estimated using a local second-order Taylor expansion:

[0020]

[0021] in Prevent division by zero; The final heatmap spatial coordinates are:

[0022] Mapping back to the original image resolution via scaling transformation:

[0023] Output the precise image coordinates of N key points to achieve high-precision key point detection guided by semantics.

[0024] The underwater manta ray-inspired submersible RGB image is defined by M categories: background, left fin, right fin, tail fin, and body. Each underwater manta ray-inspired submersible RGB image contains N key points, 10 of which are from the five categories. The left fin has 4 key points, numbered 0 to 3, where points 0, 1, and 2 are located on the outer edge of the left fin to describe its outline and curvature, and point 3 is located in the center of the left fin to characterize its overall position. The right fin has 4 symmetrically placed key points, numbered 4 to 7, where points 4, 5, and 6 are on the outer edge of the right fin, and point 7 is the center point of the right fin. The body has 1 key point, numbered 8, located at the geometric center of the submersible's main body. The tail fin has 1 key point, numbered 9, located at the end of the tail fin.

[0025] Beneficial effects This invention proposes a keypoint detection method for a manta ray-inspired submersible based on soft-weighted fusion. It acquires RGB images of the underwater manta ray-inspired submersible and inputs these images in parallel into an improved U-Net semantic segmentation model and an improved HRNet keypoint detection model, achieving simultaneous feature extraction via dual branches. The U-Net semantic segmentation model performs pixel-level classification of the underwater scene, while the HRNet keypoint detection model predicts an initial heatmap of keypoints across the entire image, achieving initial coarse localization. A soft-weighted fusion mechanism is employed, utilizing semantic segmentation masks of corresponding parts to enhance and suppress the HM (Hypertext Marker) of each keypoint, achieving multi-part constraints guided by semantic segmentation. Finally, the maximum value index is calculated on the HM after soft-weighted fusion constraints to obtain the final keypoint coordinates for each part.

[0026] Compared with existing technologies, the beneficial effects are: (1) This method relies on semantic segmentation prior to accurately divide the effective body region, suppress noise such as uneven underwater illumination, water turbidity, and bubble interference, greatly reduce key point drift, and significantly improve the key point positioning accuracy in complex underwater environments. (2) This method uses soft weighted fusion to replace the traditional hard mask multiplication and feature splicing method, which can adaptively adjust the region constraint weights and effectively adapt to the large deformation characteristics of flexible parts such as the left and right fins of the manta ray-inspired submersible, solving the problems of rigid constraints and poor adaptability of traditional methods. Attached Figure Description

[0027] Figure 1 A schematic diagram showing the semantic segmentation components for a manta ray-inspired submersible. Figure 2 A schematic diagram showing the key points of the manta ray-inspired submersible. Figure 3 This is a diagram of the U-Net module network structure. Figure 4 Here is a diagram of the HRNet module network structure; Figure 5 This is a schematic diagram of the soft weighted fusion module; Figure 6 This is a flowchart illustrating the operation of an example of the present invention; Figure 7 This is a comparison chart of key point detection accuracy under conditions with and without constraints. Detailed Implementation

[0028] The present invention will now be further described in conjunction with the embodiments and accompanying drawings: This embodiment uses data collected by a manta ray-inspired submersible in pool and lake environments as an example to illustrate the specific implementation process of the present invention.

[0029] Step 1: Acquire RGB images of the underwater manta ray simulant as input. The image resolution is denoted as H×W (H=1080, W=1920 in this example), including the manta ray simulant's posture under different lighting and turbidity conditions. To adapt to subsequent network input, the images are scaled twice: scaled to the U-Net input size. (e.g., 384×672); scaled to HRNet input size (e.g., 448×320).

[0030] The example collected a large number of RGB images of the underwater manta ray-inspired submersible, with 1,000 images actually used in the training.

[0031] Data acquisition method: The camera captures video of the manta ray-inspired submersible's movement, and then frames are extracted to obtain RGB images.

[0032] Use ISAT (software name) to perform semantic segmentation annotation on each image.

[0033] Use LabelMe (the software name) to label key points on each image.

[0034] Figure 1 This diagram illustrates the semantic segmentation components for a manta ray-inspired submersible. The invention divides the image into five semantic regions: background, left fin, right fin, tail fin, and body, corresponding to categories 0, 1, 2, 3, and 4, respectively. This component-level semantic segmentation distinguishes the main structural regions of the manta ray-inspired submersible and provides corresponding mask constraints for subsequent keypoint detection, enabling the left fin, right fin, tail fin, and body keypoints to be located within semantically reasonable regions.

[0035] Figure 2 This invention establishes a schematic diagram of key points for a manta ray-inspired submersible. Based on the structural features, kinematic characteristics, and visibility in underwater images of the manta ray-inspired submersible, 10 key points are established. Specifically, four key points (numbered 0-3) are established on the left fin; points 0, 1, and 2 are located on the outer edge of the left fin, describing its outline and curvature; point 3 is located in the middle of the left fin, representing its overall position. Four key points (numbered 4-7) are symmetrically established on the right fin; points 4, 5, and 6 are located on the outer edge of the right fin, and point 7 is located in the middle of the right fin. One key point (numbered 8) is established on the body, indicating the main body position of the manta ray-inspired submersible. One key point (numbered 9) is established on the tail fin, indicating its tail posture. These key points cover the main moving parts and main structure of the manta ray-inspired submersible, providing spatial constraints for subsequent attitude perception.

[0036] Step 2: Input the image into a dual-branch synchronous feature extraction network, a U-Net semantic segmentation model, and an HRNet keypoint detection model for feature extraction; An improved U-Net semantic segmentation model is constructed, whose model structure includes an encoder, a decoder, a skip connection module, and an output convolution module. The encoder includes multi-level convolutional blocks and a downsampling module, and the decoder includes multi-level upsampling modules and a convolutional refinement module. The shallow spatial detail features of the encoder at the same scale are concatenated with the deep semantic features of the decoder through skip connections, and then refined by convolutional blocks. After passing through the output convolution module, the maximum value index is taken to obtain M pixel-level classification results.

[0037] Each convolutional block in the improved U-Net semantic segmentation model contains two 3×3 convolutions, batch normalization, and ReLU activation. The downsampling module uses 2×2 max pooling to progressively reduce the feature map resolution to expand the receptive field and extract deep semantic information. The upsampling module uses 2×2 transposed convolution to restore spatial resolution. The output convolutional module uses 1×1 convolution to map the features into a 5-channel segmentation result, and then performs a maximum value indexing operation to obtain M pixel-level classification results.

[0038] The improved HRNet keypoint detection model is based on the HRNet-W32 high-resolution feature preservation network. It includes a Stem module connecting a multi-stage multi-resolution feature fusion module and a detection head module. The Stem module comprises two 3×3 convolutional layers with a stride of 2 for initial feature extraction and downsampling. The multi-stage multi-resolution feature fusion module sequentially includes a first-stage single-branch feature extraction, a second-stage high-low dual-branch fusion, a third-stage three-branch fusion, and a fourth-stage four-branch fusion, used to fuse multi-scale semantic information while preserving high-resolution features. The detection head module uses 1×1 convolutions to output N-channel keypoint heatmaps, which correspond to N predefined keypoints of a manta ray-inspired submersible.

[0039] Step 3: Train the dual-branch network: The data for the improved U-Net semantic segmentation model includes: 1) RGB images of underwater manta ray sculptors; 2) segmentation mask annotations for each underwater manta ray sculptor RGB image consisting of 5 categories. The five categories are as follows: the left and right fins are the main locomotor organs that generate thrust. Their movement patterns are independent and they are usually visible as separate in underwater images, so they are classified as category 1 and category 2, respectively; the body is a rigid entity that provides structural reference and is classified as category 4; the caudal fin, although small, controls pitch and yaw and needs to be segmented separately to obtain accurate tail information and is classified as category 3; the background is an area that does not need to be of interest and is classified as category 0.

[0040] The improved U-Net semantic segmentation model uses 9 / 10 of the above two datasets as training data. This model employs an encoder-decoder structure. The encoder extracts image features through convolutional blocks. Each convolutional block contains two 3×3 convolutions, batch normalization, and Rectified Linear Unit (ReLU) activations to enhance the stability of feature representation. Subsequently, it undergoes 2×2 max pooling for progressive downsampling to expand the receptive field and extract deep semantic information. The decoder uses 2×2 transposed convolutions to restore spatial resolution and concatenates shallow spatial detail features of the encoder with deep semantic features of the decoder through skip connections. After further refinement through convolutional blocks, a 1×1 convolution outputs a 5-channel segmentation result. The pixel-level classification result, i.e., the result of 5 categories, is obtained by taking the maximum value index operation.

[0041] Compared to the ordinary U-Net, which mainly performs foreground / background segmentation, the improved U-Net of this invention sets up five component-level semantic outputs for the manta ray-inspired submersible: background, left fin, right fin, tail fin, and body. This allows the network to learn the independent morphological features of the left and right fins, tail fin, and body separately, providing part-level geometric priors for subsequent keypoint heatmap constraints. During training, the acquired images and their corresponding five-class segmentation mask data are used to form training samples. The images are scaled and normalized proportionally, and the masks use nearest neighbor interpolation to maintain the class labels. The training set and validation set can be divided in approximately a 9:1 ratio.

[0042] In this embodiment, the U-Net has 3 input channels, 5 output categories, 100 training epochs, a batch size of 2, and an initial learning rate of 100%. The RMSprop optimizer is used with a momentum of 0.9 and a weight decay of . The learning rate is adaptively adjusted based on the Dice coefficients of the validation set. To address the issue of a high proportion of background pixels and small areas of target parts such as the tail fin, this embodiment uses a weighted multi-class cross-entropy loss function to train the improved U-Net model, with the following expression:

[0043] in, This represents the semantic segmentation loss function; This represents the total number of pixels in the image; This represents the total number of segmentation categories, in this embodiment. ; Indicates pixel position; Indicates the category number; Indicate category The corresponding loss weights; Represents pixels In category The real labels on it; Indicates the model predicts pixels Category The probability. In this embodiment, the background category weight can be set to... The weights for target components such as the left fin, right fin, caudal fin, and body can be set to [values ​​to be filled in]. To reduce the dominant role of background categories in model training and enhance the network's ability to learn small-area component regions; after each training round, multi-class Dice coefficients are calculated on the validation set, and model parameters with better segmentation performance are saved.

[0044] The improved HRNet keypoint detection model data includes N underwater manta ray-inspired submersible RGB images and their corresponding N keypoint annotation data; The N key point annotations are as follows: This invention sets 10 key points, selected based on the kinematic analysis of the manta ray-inspired submersible and the actual visibility of underwater cameras. Four key points are set on the left fin, numbered 0 to 3. Points 0, 1, and 2 are located on the outer edge of the left fin to describe its outline and curvature, while point 3 is located in the center of the left fin to characterize the overall position of the fin surface. Four key points are symmetrically set on the right fin, numbered 4 to 7. Points 4, 5, and 6 are three points on the outer edge of the right fin, and point 7 is the midpoint of the right fin. One key point, numbered 8, is set on the body, located at the geometric center of the submersible's main body. One key point, numbered 9, is set on the tail fin, located at its tip. This layout covers the main moving parts and avoids ambiguous annotations.

[0045] Define the mapping relationship between keypoints and mask categories. Based on prior part affiliation, the left fin keypoint falls within the left fin region (category 1), the right fin keypoint falls within the right fin region (category 2), the body keypoint falls within the body region (category 4), and the caudal fin keypoint falls within the caudal fin region (category 3). Based on this, the mapping function map(k) from keypoints to mask categories is defined as follows:

[0046] The training data input improves the HRNet keypoint detection model. The improved HRNet used in this invention is based on the HRNet-W32 high-resolution feature preservation network, and is adapted to the keypoint detection task of the manta ray-inspired submersible by adjusting the input size, number of output channels, and training strategy. The Stem module of this model consists of two steps with a step size of... of The convolutional layers are used to perform initial feature extraction and downsampling; the first stage includes... The first stage uses residual blocks to extract basic attitude features; the second stage forms two branches, high and low resolution, and achieves bidirectional feature fusion through cross-branch information exchange; the third stage further expands to three resolution branches to enhance multi-scale semantic expression capabilities; the fourth stage expands to four resolution branches and continuously performs feature fusion between different scales. By consistently retaining high-resolution branches in the network and repeatedly fusing multi-scale features, this model can acquire the overall attitude semantics of the manta ray-inspired submersible while maintaining the spatial positioning information of small structures such as fin tips, fin edges, and tail.

[0047] Compared to the existing HRNet human pose estimation network, this embodiment replaces its human keypoint detection output head with a 10-channel thermal image output head suitable for manta ray-inspired submersibles. The 10 channels correspond to 10 predefined keypoints, including the left fin... One key point, the right fin includes One key point is that the body includes... One key point is that the caudal fin includes... One key point. To adapt to the original... Regarding the aspect ratio of the image, this embodiment sets the network input size to... The output key point heatmap size is set to Each output channel corresponds to a keypoint heatmap, representing the confidence level of each pixel location in the image belonging to the corresponding keypoint. During model training, the manta ray-inspired submersible image and its corresponding 10 keypoint annotations are compiled into keypoint detection training samples. The training and validation sets are divided at approximately a 9:1 ratio, employing a top-down keypoint detection process. Specifically, the image undergoes center point and scale normalization and affine transformation based on the target region to ensure consistency in scale and position of the input target. Subsequently, the manta ray-inspired submersible image and its corresponding 10 keypoint annotations are compiled into keypoint detection training sample images and input into the HRNet keypoint detection model for training. The input images can be standardized using mean and standard deviation, and pre-trained HRNet-W32 weights can be loaded as initialization parameters to improve model convergence speed and feature extraction stability.

[0048] Keypoint monitoring labels are generated using a Gaussian heatmap format, which generates a two-dimensional Gaussian distribution heatmap centered on the coordinates of each real keypoint on the corresponding channel. In this embodiment, the Gaussian kernel standard deviation is set to... The training epochs were set to 100, the batch size to 8, the Adam optimizer was used, and the initial learning rate was set to... A linear learning rate warm-up strategy was adopted in the early stage of training to avoid drastic fluctuations in model parameters in the initial stage; then, the learning rate was reduced to 0.1 times the original value in the 70th and 90th rounds, respectively, to improve the stability and convergence accuracy of later training.

[0049] To enhance the model's adaptability to changes in posture, scale, and viewpoint, data augmentation techniques such as random horizontal flipping, random half-body region sampling, random bounding box perturbation, and affine transformation can be employed during the training phase. Specifically, during horizontal flipping, the keypoint labels corresponding to the left and right fins are simultaneously swapped based on the symmetry of the manta ray-inspired submersible's left and right fins to ensure consistency between the training labels and the flipped image.

[0050] The model is trained using a mean squared error loss function with keypoint weights, the expression of which is:

[0051] in, This represents the loss function for keypoint detection; This represents the total number of key points, in this embodiment. ; Indicates the key point number; Indicates the first Heatmap of key prediction points; Indicates the first Gaussian heatmaps corresponding to each real key point; Indicates the first The visibility or weight coefficients of each keypoint are used to adjust the loss contribution of different keypoints during training. After training, HRNet outputs 10 keypoint heatmaps, each heatmap representing the confidence distribution of the corresponding keypoint in the heatmap space.

[0052]

[0053] The distribution of K=10 key points is the same as in step 4: Left fin: Key points 0, 1, 2 (three points on the edge of the fin) and key point 3 (the middle point of the fin); Right fin: Key points 4, 5, 6 (three points on the edge of the fin) and key point 7 (the middle point of the fin); Body: Key Point 8; Tail fin: Key point 9.

[0054] Each heatmap This indicates the confidence level that the pixel belongs to key point k.

[0055] Step 4: Input the data image from the example into the dual-branch synchronous feature extraction network, and perform feature extraction using the improved U-Net semantic segmentation model and the improved HRNet keypoint detection model; Step 5: Perform closure operation to repair and align the semantic segmentation mask extracted by the U-Net semantic segmentation model, and then use a soft weighted fusion mechanism with the original key point heatmap extracted by the HRNet key point detection model to perform region enhancement and suppression on each key point HM using the semantic segmentation mask of the corresponding part, so as to realize multi-part constraint guided by semantic segmentation and obtain the fused heatmap HM. The semantic segmentation mask extracted from the U-Net semantic segmentation model is repaired and aligned by performing a closing operation: To repair potential tail fin breakage that may occur during underwater imaging, a morphological closing operation is performed on the tail fin category (category 3) in the segmentation image to obtain the tail fin mask after the closing operation. :

[0056] The kernel size is 9×9, and the iteration is performed twice. The final segmentation map M is obtained, defined as follows:

[0057] This segmentation map provides a priori geometric information for the affected areas, which is then scaled to the heatmap spatial dimensions using bilinear interpolation. Obtain the part mask .

[0058] The method employs a soft-weighted fusion mechanism, utilizing the semantic segmentation mask of the corresponding part to enhance and suppress the region of each key point HM, thereby achieving multi-part constraint guided by semantic segmentation and obtaining a fused heatmap HM.

[0059] Specifically, the heatmap of each key point is soft-weighted modulated using the obtained part mask. For the k-th key point, the mapping relationship between key points and masks is used... Take the corresponding part of the mask (in ), and the mask is changed from the original size of the segmentation map. Zoom to Record The heatmap calculation after soft-weighted fusion is as follows:

[0060] Or equivalently written as:

[0061] in For soft constraint strength, this embodiment takes... This mechanism keeps the heatmap values ​​inside the mask unchanged, while the heatmap values ​​outside the mask decay to their original values. This approach preserves global information while guiding peak values ​​to converge towards semantically appropriate regions. Compared to hard truncation (setting external values ​​to zero), soft weighting avoids complete information loss and is particularly effective for weakly visible underwater regions.

[0062] Step 6: Process the merged heatmap Perform channel-by-channel maximum value positioning and use sub-pixel offset to improve accuracy. First, find the integer coordinates:

[0063] Then, the sub-pixel offset is estimated using a local second-order Taylor expansion:

[0064]

[0065] in To prevent division by zero, the final heatmap spatial coordinates are:

[0066] Mapping back to the original image resolution via scaling transformation:

[0067] Output the precise image coordinates of 10 key points to achieve high-precision key point detection guided by semantics.

[0068] Figure 3 The diagram shows the network structure of the U-Net module. The improved U-Net semantic segmentation model adopts an encoder-decoder structure, including an encoder, a decoder, a skip connection module, and an output convolutional module. The encoder consists of multi-level convolutional blocks and a 2×2 max-pooling downsampling module, used to extract semantic features of the image step by step; the decoder consists of multi-level 2×2 transposed convolutional upsampling modules and convolutional thinning modules, used to restore the spatial resolution of the feature map; the skip connection module concatenates shallow features of the encoder at the same scale with deep features of the decoder to supplement boundary and spatial detail information; the output convolutional module uses 1×1 convolution to output 5-channel segmentation results, corresponding to five semantic regions: background, left fin, right fin, tail fin, and body.

[0069] Figure 4This is a network structure diagram of the HRNet module. The improved HRNet keypoint detection model is based on the HRNet-W32 high-resolution feature preservation network, including the Stem module, a multi-stage multi-resolution feature fusion module, and a detection head module. The Stem module is used to complete the initial feature extraction and downsampling; the multi-stage multi-resolution feature fusion module includes a first-stage single-branch feature extraction, a second-stage two-branch fusion, a third-stage three-branch fusion, and a fourth-stage four-branch fusion, which are used to fuse multi-scale semantic information while preserving high-resolution features; the detection head module uses 1×1 convolution to output a 10-channel keypoint heatmap, with each channel corresponding to a predefined confidence distribution of a keypoint.

[0070] Figure 5 This is a schematic diagram of the soft-weighted fusion module. The soft-weighted fusion module takes the original keypoint heatmap H output by HRNet and the corresponding part mask M output by U-Net as input, and calculates the mask-constrained heatmap response through Hadamard product. Then, weight scaling is applied to the original heatmap H and the mask-constrained heatmap H⊙M respectively to obtain... and The final output is a soft-weighted heatmap. This module can enhance the response within semantically reasonable regions while preserving the global heatmap response and suppressing interference from regions outside the mask.

[0071] Figure 6 This is a flowchart illustrating the operation of an example of the present invention. First, RGB images of the underwater manta ray-inspired submersible are acquired and scaled to the input size of both the U-Net semantic segmentation model and the HRNet keypoint detection model. Then, five semantic segmentation masks (background, left fin, right fin, tail fin, and body) are obtained using the U-Net model, and the original heatmaps of 10 keypoints are obtained using the HRNet model. Subsequently, based on the correspondence between keypoints and semantic parts, the corresponding part masks and keypoint heatmaps are input into a soft-weighted fusion module to obtain a fused keypoint heatmap. Finally, the fused heatmap is subjected to maximum value localization and coordinate mapping, outputting the final image coordinates of each keypoint of the manta ray-inspired submersible.

[0072] Figure 7 The image shows a comparison of keypoint detection accuracy with and without constraints. Without semantic segmentation constraints, HRNet keypoint detection results mainly rely on the full-image texture and heatmap peak responses, making them susceptible to uneven illumination, background noise, and deformation of the left and right fins, causing some keypoints to shift. After adding semantic segmentation mask constraints, the heatmaps of each keypoint are guided by the corresponding part mask, and the keypoint response peaks are more likely to be concentrated in the semantically reasonable area, thereby reducing false detections across parts and background interference, and improving the accuracy and stability of keypoint localization for the manta ray-inspired submersible.

Claims

1. A method for detecting key points of a manta ray-inspired submersible based on soft-weighted fusion, characterized in that... The steps are as follows: Step 1: Acquire N RGB images of the underwater manta ray-inspired submersible, and scale the images twice: scale to the U-Net input size. Scale to HRNet input size The resolution of the RGB image is H×W; The RGB images of the underwater manta ray-inspired submersible include the manta ray-inspired submersible's posture under different lighting and turbidity conditions; Step 2: Input the image into a dual-branch synchronous feature extraction network, a U-Net semantic segmentation model, and an HRNet keypoint detection model for feature extraction; 1) Semantic segmentation mask extraction: [This part is incomplete and requires further context to translate accurately.] The image is input into the U-Net semantic segmentation model. Through the already trained U-Net semantic segmentation model, the pixel-level classification of the RGB image of the manta ray submersible is completed, and the semantic segmentation mask of the classification of the RGB image of the manta ray submersible is obtained. Training the U-Net semantic segmentation model: Input the training data into the U-Net semantic segmentation model and train the U-Net semantic segmentation model using the weighted multi-class cross-entropy loss function; The weighted multi-class cross-entropy loss function is: in, This represents the semantic segmentation loss function; This represents the total number of pixels in the image; Indicates the total number of segmentation categories. Indicates pixel position; Indicates the category number; Indicate category The corresponding loss weights; Represents pixels In category The real labels on it; Indicates the model predicts pixels Category The probability of; The training data includes N RGB images of an underwater manta ray-like submersible and their corresponding segmentation mask data for a set classification. The segmentation mask data corresponding to the classification: Each underwater manta ray-like submersible RGB image is segmented according to the defined M categories to form segmentation mask data corresponding to each RGB image; 2) Extraction of original key point heatmap: [This involves extracting the dimensions...] The image is input into the HRNet keypoint detection model. Using the already trained HRNet keypoint detection model, the initial heatmap of N keypoints is predicted in the entire image, thus completing the initial coarse localization of the keypoints. Training the HRNet keypoint detection model: Input the training data into the HRNet keypoint detection model and use the mean squared error loss function with keypoint weights to train the HRNet keypoint detection model. The mean squared error loss function with keypoint weights is: in, This represents the loss function for keypoint detection; Indicates the total number of key points. Indicates the key point number; Indicates the first Heatmap of key prediction points; Indicates the first Gaussian heatmaps corresponding to each real key point; Indicates the first Visibility or weighting coefficient of each key point; The training data includes N RGB images of an underwater manta ray-like submersible and their corresponding N key point annotations. The N key point annotation data: N key points in each underwater manta ray-simulating submersible RGB image are annotated to form key point annotation data corresponding to each RGB image; Step 3: Perform closure operation to repair and align the semantic segmentation mask extracted by the U-Net semantic segmentation model, and then use a soft weighted fusion mechanism with the original key point heatmap extracted by the HRNet key point detection model to perform region enhancement and suppression on each key point HM using the semantic segmentation mask of the corresponding part, so as to realize multi-part constraint guided by semantic segmentation and obtain the fused heatmap HM. The calculation of the heatmap after soft-weighted fusion: Step 4: Perform indexing on the maximum value on HM to obtain the final key point coordinates for each part.

2. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The U-Net semantic segmentation model is an improved U-Net semantic segmentation model. Its model structure includes an encoder, a decoder, a skip connection module, and an output convolution module. The encoder includes multi-level convolutional blocks and a downsampling module, and the decoder includes multi-level upsampling modules and a convolutional refinement module. The shallow spatial detail features of the encoder at the same scale are concatenated with the deep semantic features of the decoder through skip connections. After refinement by convolutional blocks, the maximum value index is taken after passing through the output convolution module to obtain M pixel-level classification results.

3. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 2, characterized in that: Each convolutional block in the improved U-Net semantic segmentation model contains two 3×3 convolutions, batch normalization, and ReLU activation. The downsampling module uses 2×2 max pooling to progressively reduce the feature map resolution to expand the receptive field and extract deep semantic information. The upsampling module uses 2×2 transposed convolution to restore spatial resolution. The output convolutional module uses 1×1 convolution to map the features into a 5-channel segmentation result, and then performs a maximum value indexing operation to obtain M pixel-level classification results.

4. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The U-Net semantic segmentation model is trained for 100 epochs with a batch size of 2 and an initial learning rate of 100%. The RMSprop optimizer is used with a momentum of 0.9 and a weight decay of . The learning rate is adaptively adjusted based on the Dice coefficients of the validation set.

5. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: When training the U-Net semantic segmentation model and the HRNet keypoint detection model, 9 / 10 of the training data is used for training, and 1 / 10 is used for testing after training.

6. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The semantic segmentation mask repair and alignment calculation is performed as follows: M binary masks corresponding to each category are extracted from the semantic segmentation mask output by the U-Net semantic segmentation model. A morphological closing operation with a kernel size of 9×9 and 2 iterations is applied to the M binary masks to obtain the repaired masks. The semantic segmentation mask is then updated based on the repaired masks. Next, corresponding part masks are extracted based on the mapping relationship between keypoints and mask categories, and these part masks are scaled to the size of the HRNet keypoint heatmap, achieving spatial alignment between the semantic segmentation mask and the original keypoint heatmap (HM).

7. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The HRNet keypoint detection model is an improved HRNet keypoint detection model. Its model structure is based on the HRNet-W32 high-resolution feature preservation network, including a Stem module connecting a multi-stage multi-resolution feature fusion module and a detection head module. The Stem module includes two 3×3 convolutional layers with a stride of 2 for initial feature extraction and downsampling. The multi-stage multi-resolution feature fusion module includes a first-stage single-branch feature extraction, a second-stage high-low dual-branch fusion, a third-stage three-branch fusion, and a fourth-stage four-branch fusion, used to fuse multi-scale semantic information while preserving high-resolution features. The detection head module uses 1×1 convolution to output N-channel keypoint heatmaps, and the N-channel keypoint heatmaps correspond to N predefined keypoints of the manta ray-inspired submersible.

8. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The soft-weighted fusion module that completes the soft-weighted fusion mechanism includes a heatmap input terminal, a mask input terminal, a Hadamard product calculation unit, a weight scaling unit, and a fusion output terminal; the soft-weighted fused heatmap output by the fusion output terminal satisfies: Where H is the original key point heatmap, M is the corresponding part mask, and α is the soft constraint strength.

9. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: Step 4 involves performing a maximum value index operation on the HM to obtain the final key point coordinates for each part, which is then used to fuse the heatmap. Perform channel-by-channel maximum value positioning and use sub-pixel offset to improve accuracy: First, find the integer coordinates. Then, the sub-pixel offset is estimated using a local second-order Taylor expansion: in Prevent division by zero; The final heatmap spatial coordinates are: Mapping back to the original image resolution via scaling transformation: Output the precise image coordinates of N key points to achieve high-precision key point detection guided by semantics.

10. The key point detection method for manta ray-inspired submersibles based on soft-weighted fusion according to claim 1, characterized in that: The underwater manta ray-inspired submersible RGB image is defined by M categories: background, left fin, right fin, tail fin, and body. Each underwater manta ray-inspired submersible RGB image contains N key points, 10 of which are from the five categories. The left fin has 4 key points, numbered 0 to 3, where points 0, 1, and 2 are located on the outer edge of the left fin to describe its outline and curvature, and point 3 is located in the center of the left fin to characterize its overall position. The right fin has 4 symmetrically placed key points, numbered 4 to 7, where points 4, 5, and 6 are on the outer edge of the right fin, and point 7 is the center point of the right fin. The body has 1 key point, numbered 8, located at the geometric center of the submersible's main body. The tail fin has 1 key point, numbered 9, located at the end of the tail fin.

Images