Water surface floating object detection method and device based on hierarchical semantic tree and medium

Abstract

The present application relates to a water surface floating object detection method, device and medium based on a hierarchical semantic tree, and belongs to the technical field of computer vision and target detection. A water surface scene image is obtained and preprocessed, a feature extraction network extracts features and performs encoding and decoding to generate a matching query matrix; the matching query matrix is simultaneously input into a hierarchical semantic tree classification network, a target positioning network and an existence discrimination network, and parallel calculation is performed to obtain floating object classification probability, frame coordinates and existence original scores respectively; through a Hungarian matching algorithm, the classification probability and frame coordinates are matched with the real frame labeled by artificial labeling to determine the matching query vector corresponding to the known object and the background; the hierarchical semantic tree guides re-labeling, and according to the Hungarian matching result and the re-labeling result, a training target distribution q of the existence discrimination network is generated, and loss calculation and model updating are performed. The present application is suitable for various water surface scenes and various light conditions, has high detection accuracy, low missed detection rate and strong robustness.

Description

Technical Field

[0001] This invention relates to a method, device, and medium for detecting floating objects on the water surface based on a hierarchical semantic tree. It belongs to the fields of computer vision and target detection technology and is applicable to scenarios such as water environment monitoring, water surface garbage cleanup, and water quality protection. It can effectively detect known floating objects and automatically identify unlabeled unknown floating objects, thereby improving the accuracy and completeness of floating object detection on the water surface. Background Technology

[0002] With rapid industrialization and urbanization, pollution from floating debris (such as plastic waste, plant remains, and household items) on water surfaces has become increasingly prominent. This pollution not only disrupts the ecological balance of the aquatic environment and affects the aesthetics of water bodies, but also poses a threat to aquatic life, and may even clog water conservancy facilities, creating safety hazards. Therefore, achieving efficient and accurate detection of floating debris is of significant practical importance for water environment management and protection.

[0003] Currently, deep learning-based object detection technology has been widely applied in the field of floating object detection on water surfaces. Mainstream methods primarily rely on anchored or unanchored frames, extracting image features through convolutional neural networks, and then using classification and regression heads to predict the object category and location, respectively. However, existing technologies have significant limitations in real-world scenarios: 1. Existing methods are mostly trained on known categories of floating objects, and have a weak ability to identify unlabeled unknown floating objects. They are prone to misclassifying unknown floating objects not in the training dataset as background, resulting in a high false negative rate. 2. Floating objects have a natural hierarchical structure (e.g., "floating objects - plastics - bottles" or "floating objects - plants - branches"). Existing methods do not fully utilize this hierarchical relationship, leading to inconsistent classification logic and unreasonable situations such as "high confidence in subclasses and low confidence in parent classes". 3. In object detection, the Softmax activation function is usually used to process classification probabilities, which results in an overly dispersed probability distribution, making it difficult to effectively distinguish between objects and backgrounds. This is especially true in scenes with complex backgrounds on water surfaces (such as ripples, reflections, and shadows), where the accuracy of object identification is insufficient. To address the aforementioned issues, there is an urgent need for a method and system for detecting floating objects on the water surface that can fully utilize the hierarchical relationships between floating object categories, accurately distinguish objects from the background, and efficiently identify unknown floating objects, in order to improve detection performance and meet the needs of actual water environment monitoring. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a method for detecting floating objects on the water surface based on a hierarchical semantic tree. This method is applicable to various water surface scenarios (rivers, lakes, reservoirs, etc.) and various lighting conditions, and has high detection accuracy, low false negative rate, and strong robustness.

[0005] The technical solution adopted in this invention is as follows: The method for detecting floating objects on the water surface based on hierarchical semantic trees includes the following steps: S1. Acquire water surface scene images and preprocess them, input them into a feature extraction network to extract features, and encode and decode them to generate a matching query matrix of potential floating objects; S2. Input the matching query matrix into the hierarchical semantic tree classification network, the target localization network, and the existence discrimination network simultaneously, and calculate in parallel to obtain the floating object classification probability, bounding box coordinates, and original existence score; S3. Using the Hungarian matching algorithm, the classification probability and bounding box coordinates are matched with the manually labeled ground truth bounding boxes to determine the matching query vectors corresponding to the known objects and the background; S4. Call the hierarchical semantic tree structure preset in step S2, clarify the parent-child semantic relationship of each category, and select the parent class probability higher than the preset threshold from the matching query vectors marked as background after Hungarian matching as relabeling candidate matching query vectors. Further filter the relabeling candidate matching query vectors to exclude obvious background areas, and relabel the candidate matching query vectors that meet the conditions from the background as unknown floating objects. S5. Based on the Hungarian matching results and relabeling results, generate the training target distribution q of the existence discriminant network, which is used for subsequent existence loss calculation to guide the existence discriminant network to learn to distinguish between objects and background; S6. Loss Calculation and Model Update: Calculate the hierarchical semantic tree classification loss, target localization loss, and existence loss respectively, and calculate the total loss by weighting. Update the parameters of the entire network through backpropagation. S7. Input the water surface image to be detected into the trained model, and output the category, location and target presence score of the floating object to complete the detection task.

[0006] In the above method, the feature extraction network used in step S1 is preferably a residual feature extraction network, and more preferably ResNet-50; the encoding adopts a Transformer Encoder structure, which captures long-distance dependencies between features through a self-attention mechanism. The preferred encoding structure consists of 6 encoder layers connected in series, each layer containing multi-head self-attention (8 attention heads) and a feedforward neural network, supplemented by layer normalization and residual connections; the decoding adopts a Transformer Decoder structure, which adds a cross-attention layer on the basis of the encoder to generate matching query vectors using global feature maps. The preferred decoding structure consists of 6 decoder layers connected in series, each layer containing multi-head self-attention, cross-attention, and a feedforward neural network, also equipped with layer normalization and residual connections.

[0007] In step S2, the hierarchical semantic tree classification network consists of two linear layers and a sigmoid activation function. The first linear layer maps the input d-dimensional matching query vector e to C-dimensional dimensions, where C is the total number of categories, and includes the root node, parent node, and leaf node of the hierarchical semantic tree. The second linear layer further optimizes the features, and the sigmoid activation function maps the output category scores to the [0,1] interval to obtain the probability of each category, i.e., the classification probability matrix.

[0008] The target localization network consists of two linear layers. The first linear layer maps the input d-dimensional matching query vector to 4-dimensional vectors, and the second linear layer further optimizes the 4-dimensional features and outputs the predicted values ​​of the bounding box coordinates. Without the need for an activation function, it directly outputs continuous coordinate values.

[0009] The existence discrimination network consists of a linear layer and a Sparsemax activation function. The linear layer maps the input d-dimensional matching query vector to 1-dimensional and outputs the original existence score z. The Sparsemax activation function maps the original score to the interval [0,1] and achieves sparsity, so that the matching query vector score corresponding to the background is 0 and the matching query vector score corresponding to the floating object is not 0.

[0010] The specific matching process in step S3 is as follows: First, manually labeled ground truth bounding boxes are obtained. Then, the matching degree between each predicted bounding box and the ground truth bounding box is calculated. Subsequently, a matching degree matrix is ​​constructed, and the Hungarian matching algorithm is used to solve for the best combination. A matching degree threshold is set to distinguish the matching results. Finally, three types of results are output: those that are successfully matched and have a matching degree higher than the threshold are known floating object positive samples; those matching query vectors with matching degrees lower than the threshold with all ground truth bounding boxes are temporarily judged as background negative samples; and ground truth bounding boxes that do not match any predicted bounding boxes are considered as not detected.

[0011] The matching degree is obtained by weighting the positional similarity index IoU and the category similarity index classification confidence, with a weighting coefficient α=0.5.

[0012] In step S4, the preset threshold is a dynamic threshold, and the calculation formula is: Dynamic threshold = Minimum parent class probability of all matching query vectors marked as positive samples, where the parent class probability is the probability value of the corresponding parent class in the classification probability matrix.

[0013] In step S5, the calculation process for the target distribution q is as follows: (1) Count the number of positive samples: Count the total number of positive samples obtained by Hungarian matching and relabeling, denoted as K, K = the number of successfully matched query vectors + the number of relabeled matched query vectors; (2) Calculate the probability of positive samples: The probability value of each positive sample is 1 / K. Since all positive samples share the probability of 1.0 equally, the probability of each positive sample is equal, ensuring that the sum of the probabilities is 1. (3) Generate q vector: Initialize a zero vector with dimension N×1, where N represents the total number of matching query vectors, i.e. the maximum number of detection targets preset by the model. Each vector has a fixed row number (such as row 1, row 2, ... row N), which is its position. Then, according to the position of the positive sample, assign the corresponding element to 1 / K, and keep the corresponding element of the background matching query vector as 0 to obtain the target distribution q.

[0014] In step S6, the hierarchical semantic tree classification loss is calculated using cross-entropy, and the formula is as follows: , in: (1) The matrix represents the true category labels, with dimensions N×C. For a successfully matched query vector, the label for its corresponding true category, all parent categories, and the root node is 1, while the labels for other categories are 0. For a relabeled query vector, the label for its corresponding parent category and the root node is 1, while the label for the leaf nodes is 0. For a background matching query vector, all category labels are 0. (2) The classification probability matrix output by the hierarchical semantic tree classification network ; (3) The second term is the hierarchical semantic penalty term, where λ is the penalty coefficient. The predicted probability for the subclass. It is the predicted probability of the parent class, used to punish the unreasonable situation of "high probability of child class and low probability of parent class", and to force the probability of child class to depend on the probability of parent class.

[0015] The formula for calculating the target localization loss in step S6 is: , in: (1) The bounding box coordinate prediction matrix output by the target localization network; (2) This is the coordinate matrix of the manually labeled true bounding boxes. For successfully matched query vectors, The coordinates of the matched ground truth bounding box are used; for unmatched query vectors, they are not included in the calculation of target localization loss. (3) The definition is: when |x|≤1, When |x|>1, .

[0016] The formula for calculating the existence loss in step S6 is as follows: , in: q represents the target distribution to be generated, with a dimension of N×1; z is the original existence score output by the existence discriminant network, with a dimension of N×1, and is not activated by Sparsemax. S(z) is the set of matching query vectors with a probability greater than 0 after Sparsemax activation, i.e., positive sample matching query vectors; The threshold for Sparsemax activation is used to determine which matching query vectors are positive samples; It is the L2 norm of q, used to stabilize the training process and avoid gradient explosion.

[0017] A water surface floating object detection device based on hierarchical semantic tree includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the water surface floating object detection method based on hierarchical semantic tree as described above.

[0018] A computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the water surface floating object detection method based on a hierarchical semantic tree as described above.

[0019] The beneficial effects of this invention are: (1) By using the hierarchical semantic-guided relabeling strategy, it can automatically identify unknown floating objects that have not been labeled in the dataset and relabel the background matching query vector with high parent class confidence as unknown objects. Although it cannot accurately identify what the floating objects are, it can capture the major categories of unknown floating objects based on semantic information, effectively reducing the false negative rate and improving the completeness of detection. (2) Make full use of the category relationship of the hierarchical semantic tree, adopt cross-entropy loss, force the probability of subclass to depend on the probability of parent class, avoid the problem of inconsistent classification logic, and improve the classification accuracy of the hierarchical semantic tree classification network; (3) The existence discrimination network is constructed by using the Sparsemax activation function to achieve the sparsity of the target existence probability, which can effectively distinguish between objects and backgrounds. It is especially suitable for scenes with complex water backgrounds and few objects, thus improving the accuracy of target existence judgment. (4) No manual labeling of unknown floating objects is required during training, and no modification of the label file is required. The model guides relabeling through its own classification features, which reduces the cost of manual labeling and improves the practicality and generalization ability of the model. (5) The modules of the device of the present invention have clear division of labor and efficient collaboration. It can be deployed in actual scenarios such as UAV water monitoring and fixed shore monitoring. It is suitable for various water surface scenarios (rivers, lakes, reservoirs, etc.) and various lighting conditions. It has high detection accuracy, low false negative rate and strong robustness, and can meet the needs of various water environment monitoring and garbage cleaning. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to specific embodiments.

[0022] Example 1: A method for detecting floating objects on the water surface based on a hierarchical semantic tree, comprising the following steps: S1. Acquire water surface scene images and preprocess them, input them into a feature extraction network to extract features, and encode and decode them to generate a matching query matrix of potential floating objects.

[0023] Image Input and Preprocessing: Images of water surfaces such as rivers, lakes, and reservoirs are acquired using cameras, drones, and other devices, covering various lighting conditions including sunny, cloudy, and rainy days to ensure data diversity. Images are in RGB three-channel format, and resolution can be adjusted according to the actual scene. Preprocessing involves resizing, normalizing, removing noise, and performing optional data augmentation on the acquired raw images, outputting standardized images. The preprocessing stage mainly includes: resizing to 640×640 (using bilinear interpolation), normalizing pixel values ​​to [0,1], and using a 3×3 Gaussian filter to remove noise. During training, optional data augmentation strategies such as random flipping, cropping, and brightness adjustment can be used to prevent overfitting, ultimately outputting standardized images.

[0024] The manually labeled content of the acquired images includes the bounding box coordinates and categories of known floating objects. The labeled categories are based on a preset hierarchical semantic tree, which has the following structure: root node (floating objects), parent node (plastic, plant, and miscellaneous objects), and leaf node (plastic objects include bottles, foam, and plastic bags; plant objects include branches and leaves; and miscellaneous objects include aluminum cans and paper towels).

[0025] This study utilizes existing mature feature extraction, encoding, and decoding models. Feature extraction: The preprocessed image is input into a residual feature extraction network (ResNet-50). This network alleviates the vanishing and exploding gradient problems in deep networks through residual connections, effectively extracting multi-scale features. Structurally, it includes an initial convolutional layer (Conv1) and four residual block groups (Conv2 to Conv5), with the number of output channels being 64, 128, 256, and 512 respectively, progressively increasing the feature dimension and reducing the spatial size. Finally, a global average pooling layer is used to obtain a 512-dimensional global feature vector, fusing local details of floating objects on the water surface with global contextual information.

[0026] Global Attention Encoding: The aforementioned global feature vectors are input into a global attention encoding network. This network employs a Transformer Encoder structure, capturing long-range dependencies between features through a self-attention mechanism. Specifically, it consists of six encoder layers connected in series, each layer containing multi-head self-attention (eight attention heads) and a feedforward neural network, supplemented by layer normalization and residual connections. Before encoding, sine and cosine positional encodings are added to the input features to inject spatial location information. After processing by the encoding network, the output is a global feature map with a dimension of 512, further fusing multi-scale and global contextual information.

[0027] Target-guided decoding: The global feature map output from the encoding network is input into the target-guided decoding network. This network is based on the Transformer Decoder structure, adding a cross-attention layer on top of the encoder to generate matching query vectors using the global feature map. The network consists of 6 decoder layers concatenated, each containing multi-head self-attention, cross-attention, and feedforward neural networks, and also equipped with layer normalization and residual connections. First, N initial matching query vectors are generated. It is input into the decoding network along with the global feature map, and then passed through each decoder layer in sequence to finally output... The matching query matrix. Each query vector corresponds to a potential floating object target, which is used for subsequent parallel module computation.

[0028] S2. Input the matching query matrix into the hierarchical semantic tree classification network, the target localization network, and the existence discrimination network simultaneously, and calculate in parallel the floating object classification probability, bounding box coordinates, and original existence score.

[0029] The output of step S1 The matching query matrix is ​​processed by three parallel networks (a hierarchical semantic tree classification network, an object localization network, and an existence discrimination network) to calculate the classification probability, bounding box coordinates, and original existence score for each matching query vector e. This provides the foundation for subsequent Hungarian matching, relabeling, and loss calculation. The parallel computation of the three networks, without any specific order, improves computational efficiency, ensures consistency of input features across the three networks, and enhances detection accuracy.

[0030] The specific process is as follows: (1) Hierarchical semantic tree classification network The core purpose of the hierarchical semantic tree classification network is to predict the category probability corresponding to each matching query vector e. The category includes the root node, parent node and leaf node of the pre-defined hierarchical semantic tree, covering all known floating object categories and their hierarchical semantic relationships. It is used to determine which category of floating object (known) or whether the target corresponding to each matching query vector e belongs to (determined by the parent category probability).

[0031] Hierarchical Semantic Tree Classification Network Structure: The hierarchical semantic tree classification network consists of two linear layers and a sigmoid activation function. The first linear layer maps the input d-dimensional matching query vector e to C dimensions (C is the total number of categories, including the root node, parent node, and leaf node of the hierarchical semantic tree). The second linear layer further optimizes the features, and the sigmoid activation function maps the output category scores to the [0,1] interval to obtain the probability of each category, i.e., the classification probability matrix. The output dimension of the hierarchical semantic tree classification network is N×C, which represents the probability of each matching query vector e corresponding to C categories.

[0032] The hierarchical semantic tree classification network uses the Sigmoid activation function instead of the Softmax activation function because floating objects on the water surface may have multiple labels (such as an object that belongs to both the plastic and bottle categories). The Sigmoid function can make independent judgments for each category and allows multiple categories to be activated at the same time, while the Softmax function requires the sum of the probabilities of all categories to be 1 and can only select one category, which does not meet the needs of real-world scenarios.

[0033] (2) Target localization network The core objective of the target localization network is to predict the coordinates of the floating bounding box corresponding to each matching query vector e, which is used to locate the position of the floating object in the image. Each matching query vector e corresponds to a bounding box coordinate, which is used for subsequent matching with the ground truth bounding box.

[0034] The target localization network consists of two linear layers. The first linear layer maps the input d-dimensional matching query vector to 4-dimensionality. The second linear layer further optimizes the 4-dimensional features and outputs the predicted bounding box coordinates. No activation function is required; the output is a continuous set of coordinate values. To ensure the reasonableness of the bounding box coordinates, the predicted coordinates will be cropped to ensure that the coordinate values ​​are within the image size range.

[0035] (3) Existence discrimination network The core purpose of the existence discrimination network is to determine whether the region corresponding to each matching query vector e is an object (including known floating objects and unknown floating objects), output the original existence score, and then perform sparsification processing through the Sparsemax activation function to obtain the target existence probability, which is used to distinguish objects from the background.

[0036] The existence discrimination network consists of a linear layer and a Sparsemax activation function. The linear layer maps the input d-dimensional matching query vector to 1-dimensional space and outputs the original existence score z. The Sparsemax activation function maps the original score to the interval [0,1] and achieves sparsity, making the matching query vector score corresponding to the background 0, while the matching query vector score corresponding to the floating object is not 0. Softmax causes all matching query vectors to have a probability greater than 0, resulting in an overly dispersed distribution that makes it difficult to effectively distinguish between objects and the background. Sparsemax, on the other hand, achieves sparsity by directly compressing the probability of the matching query vector corresponding to the background to 0, retaining only the probability of the matching query vector corresponding to the object. This better meets the scenario requirements of "few objects, many background" in water surface floating object detection, thus improving the accuracy of target existence judgment.

[0037] Finally, three outputs are obtained: classification probability matrix y (N×C), bounding box coordinate prediction matrix (N×4), and target existence probability matrix (N×1). These three results are output in parallel for subsequent steps.

[0038] S3. Using the Hungarian matching algorithm, the classification probability and bounding box coordinates are matched with the manually labeled ground truth bounding boxes to determine the matching query vectors corresponding to the known objects and the background.

[0039] The core objective of this step is to match the classification probabilities output by the hierarchical semantic tree classification network and the bounding box coordinates output by the object localization network with manually labeled ground truth bounding boxes. This determines which matching query vectors correspond to known floating objects (positive samples) and which correspond to the background (negative samples), providing a foundation for subsequent relabeling and object distribution generation. The core of Hungarian matching is finding the optimal pairing of "predicted bounding boxes and ground truth bounding boxes," ensuring that each ground truth bounding box corresponds to a unique matching query vector, and each matching query vector corresponds to at most one ground truth bounding box.

[0040] The specific process is as follows: First, manually labeled ground truth bounding boxes are obtained (the category number corresponds to the hierarchical semantic leaf node, and the coordinate format is consistent with the output of the target localization network); then, the matching degree between each predicted box and the ground truth bounding box is calculated (weighted by the position similarity index IoU and the category similarity index classification confidence, with a weight coefficient α=0.5); then, an N×M matching degree matrix is ​​constructed (M is the number of ground truth bounding boxes), and the Hungarian matching algorithm is used to solve for the best combination. A matching degree threshold is set to distinguish the matching results; finally, three types of results are output: those that are successfully matched and have a matching degree higher than the threshold are known floating object positive samples; the matching query vectors with matching degrees lower than the threshold with all ground truth bounding boxes are temporarily judged as background negative samples; and ground truth bounding boxes that do not match any predicted boxes are considered as not detected.

[0041] This step clarifies the matching query vectors between known floating objects and the background, providing a clear initial sample classification for screening potential unknown floating objects in the subsequent relabeling process. It also provides accurate positive and negative sample basis for model loss calculation, ensuring the effectiveness of model training.

[0042] S4. Call the preset hierarchical semantic tree structure in step S2 to clarify the parent-child semantic relationship of each category. From the matching query vectors marked as background after Hungarian matching, select those with a parent class probability higher than a preset threshold as relabeling candidate matching query vectors. Further filter the relabeling candidate matching query vectors to exclude obvious background areas and relabel the candidate matching query vectors that meet the conditions from the background as unknown floating objects.

[0043] The core objective of this step is to utilize the category probabilities output by the hierarchical semantic tree classification network, combined with the preset hierarchical semantic tree structure, to mine matching query vectors that may be unknown floating objects but did not match the ground truth bounding boxes. These vectors are then relabeled from the background (negative samples) as unknown objects (positive samples), thereby achieving automatic mining of unknown floating objects and improving the completeness of detection in the real world.

[0044] The specific implementation process is as follows: (1) Preset hierarchical semantic tree structure: Call the hierarchical semantic tree structure preset in step S2 to clarify the parent-child semantic relationship of each category (e.g., the parent of bottle is plastic, and the parent of plastic is floating objects). The relabeling process mainly focuses on the probability of the parent (non-leaf node) because unknown floating objects may not be labeled as specific leaf node categories, but may have the semantic features of the parent (e.g., new plastic waste is not labeled as bottle or foam, but has the semantic features of plastic).

[0045] (2) Determine candidate matching query vectors for relabeling: From the matching query vectors marked as background after Hungarian matching, select those with a parent class probability higher than a preset threshold as candidate matching query vectors for relabeling. The parent class probability is the probability value of the corresponding parent class (plastic, plant, miscellaneous) in the classification probability matrix y. The preset threshold is a dynamic threshold, calculated as: Dynamic threshold = Minimum parent class probability of all matching query vectors marked as positive samples. For example, if a matching query vector x is marked as a positive sample, and this vector corresponds to a bottle in the image, then its parent class (plastic) probability value is taken. Use this method to query all matching query vectors marked as positive samples, and take the minimum parent class probability as the threshold. The purpose of setting a dynamic threshold is to ensure that the candidate matching query vectors for relabeling have parent class semantic features equivalent to known objects, avoiding mislabeling the background as unknown objects.

[0046] (3) Re-annotation execution: Further filter the candidate matching query vectors for re-annotation to exclude obvious background areas (such as water ripples and reflective areas). The filtering rules are: the IoU between the predicted box of the candidate matching query vector and all real boxes is less than 0.1 (to avoid overlapping with known objects), and the parent class probability of the candidate matching query vector is consistently higher than the dynamic threshold.

[0047] After screening, the candidate matching query vectors that meet the conditions are relabeled from the background (negative samples) as unknown floating objects (positive samples). These matching query vectors e correspond to unlabeled unknown floating objects. Although they do not have specific leaf node categories, they have clear parent class semantic features and need to be identified as floating objects rather than the background by the existence discrimination network.

[0048] (4) Relabeling results output: After relabeling is completed, update the identity label of the matching query vector. At this time, the positive sample includes two parts: the matching query vector that is successfully matched (known object) and the relabeled matching query vector (unknown object). The negative sample is the matching query vector that is not matched and has not been relabeled (the real background).

[0049] S5. Based on the Hungarian matching results and relabeling results, generate the training target distribution q of the existence discriminant network, which is used for subsequent existence loss calculation to guide the existence discriminant network to learn to distinguish between objects and background.

[0050] The core objective of this step is to generate the training target distribution q for the existence discriminant network based on the Hungarian matching results and the hierarchical semantic-guided relabeling results. q is an N×1 vector that serves as the "standard answer" for the existence discriminant network, telling it which matching query vectors are floating objects (known + unknown) and which are background. This is used for subsequent existence loss calculations and guides the existence discriminant network to learn to distinguish between objects and background.

[0051] The core rule for generating q is: all positive samples (successfully matched query vectors + re-annotated query vectors) share a probability of 1.0, while the probability of background matching query vectors is 0, ensuring that the sum of the probabilities of all positive samples is 1, which meets the requirements of the probability distribution.

[0052] The specific process is as follows: (1) Count the number of positive samples: Count the total number of positive samples obtained by Hungarian matching and relabeling, denoted as K, K = the number of successfully matched query vectors + the number of relabeled matched query vectors; (2) Calculate the probability of positive samples: The probability value of each positive sample is 1 / K. Since all positive samples share the probability of 1.0 equally, the probability of each positive sample is equal, ensuring that the sum of the probabilities is 1. (3) Generate q vector: Initialize a zero vector with dimension N×1, and then assign the corresponding element to 1 / K according to the position of the positive sample. Keep the element corresponding to the background matching query vector as 0 to obtain the target distribution q.

[0053] Example: Matching query vectors Matching query vectors Matching query vectors Since it is a positive sample, q

[12] = 0.333, q

[57] = 0.333, q

[66] = 0.333, and the remaining N-3 elements are all 0. The dimension of q is N×1, which is consistent with the dimension of the target existence probability matrix output by the existence discrimination network, which is convenient for subsequent loss calculation.

[0054] S6. Loss Calculation and Model Update: Calculate the hierarchical semantic tree classification loss, object localization loss, and existence loss respectively, and calculate the total loss by weighting. Update the parameters of the entire network through backpropagation.

[0055] The core purpose of this step is to calculate the classification loss, target localization loss, and existence loss separately, and then sum the three losses by weight to obtain the total loss. The entire network is then updated through the backpropagation algorithm, enabling the model to continuously learn and improve detection accuracy and the ability to discover unknown objects.

[0056] The specific process is as follows: (a) Classification loss calculation The classification loss is used to measure the difference between the classification probability matrix y output by the hierarchical semantic tree classification network and the true class labels, guiding the hierarchical semantic tree classification network to learn accurate class predictions, especially hierarchical semantic relationships. Since the classification uses the Sigmoid activation function and has a hierarchical semantic structure, this invention uses cross-entropy to calculate the loss. This loss can utilize the class relationships of the hierarchical semantic tree to force the subclass probability to depend on the parent class probability, avoiding inconsistencies in classification logic.

[0057] The formula for calculating classification loss is: , in: (1) The matrix represents the true category labels, with dimensions N×C. For a successfully matched query vector, the labels for its corresponding true category (leaf node) and all its parent and root nodes are 1, while the labels for other categories are 0. For a relabeled query vector, the labels for its corresponding parent and root nodes are 1, while the labels for its leaf nodes are 0. For a background query vector, all category labels are 0. (2) The classification probability matrix output by the hierarchical semantic tree classification network ; (3) The second term is the hierarchical semantic penalty term, where λ is the penalty coefficient (with a value of 0.1). The predicted probability for the subclass. The predicted probability of the parent class is used to punish the unreasonable situation of "high probability of child class and low probability of parent class", and to force the probability of child class to depend on the probability of parent class.

[0058] (II) Calculation of Target Location Loss The target localization loss is used to measure the difference between the bounding box coordinates output by the target localization network and the true bounding box coordinates, guiding the network to learn accurate bounding box localization. This invention employs a smoothed L1 loss function, which is insensitive to outliers and can effectively mitigate the interference of extreme values ​​in bounding box coordinate prediction, thereby improving regression accuracy.

[0059] The formula for calculating target localization loss is: , in: (1) The bounding box coordinate prediction matrix output by the target localization network; (2) This is the coordinate matrix of the manually labeled true bounding boxes. For successfully matched query vectors, The coordinates of the matched ground truth bounding box are used; for unmatched query vectors (background and relabeled unknown objects), they are not included in the target localization loss calculation. (3) The definition is: when |x|≤1, When |x|>1, .

[0060] (III) Calculation of Existence Loss The existence loss measures the difference between the target existence probability output by the existence discrimination network and the generated target distribution q, guiding the existence discrimination network to learn to distinguish between objects (known + unknown) and the background. The loss function used is L_obj, and its calculation formula is as follows: , in: (1) q is the target distribution generated in step S5, with a dimension of N×1; (2) z is the original existence score output by the existence discrimination network, with a dimension of N×1 (not activated by Sparsemax). (3) S(z) is the set of matching query vectors with a probability greater than 0 after Sparsemax activation, i.e., positive sample matching query vectors; (4) The threshold for Sparsemax activation is used to determine which matching query vectors are positive samples; (5) It is the L2 norm of q, used to stabilize the training process and avoid gradient explosion;

[0061] The core function of this loss function is to make the original score z output by the existence discrimination network as close as possible to the target distribution q. At the same time, the second term forces the probability sparsification after Sparsemax activation, so that the probability of the background matching the query vector is 0, while the probability of the object matching the query vector is not 0. The third term is used to stabilize the training process.

[0062] (iv) Total Loss and Model Update The total loss is a weighted sum of classification loss, target localization loss, and existence loss, calculated using the following formula: , Here, β and γ are weighting coefficients used to balance the weights of the three losses, ensuring that the model simultaneously optimizes the three tasks of classification, object localization, and existence determination. After the total loss is calculated, the gradient descent optimization algorithm is used to update the parameters of the entire network through backpropagation to minimize the total loss.

[0063] S7. Input the water surface image to be detected into the trained model, and output the category, location and target presence score of the floating object to complete the detection task.

[0064] Example 2: A water surface floating object detection device based on a hierarchical semantic tree includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the water surface floating object detection method based on a hierarchical semantic tree as described in Example 1.

[0065] A computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the water surface floating object detection method based on a hierarchical semantic tree as described in Embodiment 1.

[0066] The above is a further description of the present invention in conjunction with specific embodiments, and the scope of protection of the present invention is not limited thereto.

Claims

1. A method for detecting floating objects on the water surface based on hierarchical semantic trees, characterized by: The steps include the following: S1. Acquire water surface scene images and preprocess them, input them into a feature extraction network to extract features, and encode and decode them to generate a matching query matrix of potential floating objects; S2. Input the matching query matrix into the hierarchical semantic tree classification network, the target localization network, and the existence discrimination network simultaneously, and calculate in parallel to obtain the floating object classification probability, bounding box coordinates, and original existence score; S3. Using the Hungarian matching algorithm, the classification probability and bounding box coordinates are matched with the manually labeled ground truth bounding boxes to determine the matching query vectors corresponding to the known objects and the background; S4. Call the hierarchical semantic tree structure preset in step S2, clarify the parent-child semantic relationship of each category, and select the parent class probability higher than the preset threshold from the matching query vectors marked as background after Hungarian matching as relabeling candidate matching query vectors. Further filter the relabeling candidate matching query vectors to exclude obvious background areas, and relabel the candidate matching query vectors that meet the conditions from the background as unknown floating objects. S5. Based on the Hungarian matching results and relabeling results, generate the training target distribution q of the existence discriminant network, which is used for subsequent existence loss calculation to guide the existence discriminant network to learn to distinguish between objects and background; S6. Loss Calculation and Model Update: Calculate the hierarchical semantic tree classification loss, target localization loss, and existence loss respectively, and calculate the total loss by weighting. Update the parameters of the entire network through backpropagation. S7. Input the water surface image to be detected into the trained model, and output the category, location and target presence score of the floating object to complete the detection task.

2. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, The hierarchical semantic tree classification network described in step S2 consists of two linear layers and a sigmoid activation function. The first linear layer maps the input d-dimensional matching query vector e to C-dimensional dimensions, where C is the total number of categories, and includes the root node, parent node, and leaf node of the hierarchical semantic tree. The second linear layer further optimizes the features, and the sigmoid activation function maps the output category scores to the [0,1] interval to obtain the probability of each category, i.e., the classification probability matrix.

3. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, The target localization network consists of two linear layers. The first linear layer maps the input d-dimensional matching query vector to 4-dimensional vectors, and the second linear layer further optimizes the 4-dimensional features and outputs the predicted values ​​of the bounding box coordinates. Without the need for an activation function, it directly outputs continuous coordinate values.

4. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, The existence discrimination network consists of a linear layer and a Sparsemax activation function. The linear layer maps the input d-dimensional matching query vector to 1-dimensional and outputs the original existence score z. The Sparsemax activation function maps the original score to the interval [0,1] and achieves sparsity, so that the matching query vector score corresponding to the background is 0 and the matching query vector score corresponding to the floating object is not 0.

5. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, The specific matching process in step S3 is as follows: First, manually labeled ground truth bounding boxes are obtained. Then, the matching degree between each predicted bounding box and the ground truth bounding box is calculated. Subsequently, a matching degree matrix is ​​constructed, and the Hungarian matching algorithm is used to solve for the best combination. A matching degree threshold is set to distinguish the matching results. Finally, three types of results are output: those that are successfully matched and have a matching degree higher than the threshold are known floating object positive samples; those matching query vectors with matching degrees lower than the threshold with all ground truth bounding boxes are temporarily judged as background negative samples; and ground truth bounding boxes that do not match any predicted bounding boxes are considered as not detected.

6. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, In step S4, the preset threshold is a dynamic threshold, and the calculation formula is: Dynamic threshold = Minimum parent class probability of all matching query vectors marked as positive samples, where the parent class probability is the probability value of the corresponding parent class in the classification probability matrix.

7. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, In step S5, the calculation process of the target distribution q is as follows: (1) Count the number of positive samples: Count the total number of positive samples obtained by Hungarian matching and relabeling, denoted as K, K = number of successfully matched query vectors + number of relabeled matching query vectors; (2) Calculate the probability of positive samples: The probability value of each positive sample is 1 / K. Since all positive samples share the probability of 1.0 equally, the probability of each positive sample is equal, ensuring that the sum of the probabilities is 1. (3) Generate q vector: Initialize a zero vector with dimension N×1, where N represents the total number of matching query vectors, i.e. the maximum number of detection targets preset by the model. Each vector has a fixed row number, which is its position. Then, according to the position of the positive sample, assign the corresponding element to 1 / K. Keep the element corresponding to the background matching query vector as 0 to obtain the target distribution q.

8. The method for detecting floating objects on the water surface based on a hierarchical semantic tree according to claim 1, characterized in that, In step S6, the hierarchical semantic tree classification loss is calculated using cross-entropy, and the calculation formula is as follows: ， in: The matrix represents the true category labels, with dimensions N×C. For a successfully matched query vector, the label for its corresponding true category, all parent categories, and the root node is 1, while the labels for other categories are 0. For a relabeled query vector, the label for its corresponding parent category and the root node is 1, while the label for the leaf nodes is 0. For a background matching query vector, all category labels are 0. The classification probability matrix output by the hierarchical semantic tree classification network ; The second term is the hierarchical semantic penalty term, where λ is the penalty coefficient. The predicted probability for the subclass. The predicted probability of the parent class is used to punish the unreasonable situation of "high probability of child class and low probability of parent class", and to force the probability of child class to depend on the probability of parent class. The formula for calculating target localization loss is: ， in: The bounding box coordinate prediction matrix output by the target localization network; This is the coordinate matrix of the manually labeled true bounding boxes. For successfully matched query vectors, The coordinates of the matched ground truth bounding box are used; for unmatched query vectors, they are not included in the target localization loss calculation. The definition is: when |x|≤1, When |x|>1, ; The formula for calculating the existence loss is: ， in: q represents the generated target distribution with dimension N×1; z is the original existence score output by the existence discriminant network, with a dimension of N×1, and is not activated by Sparsemax. S(z) is the set of matching query vectors with a probability greater than 0 after Sparsemax activation, i.e., positive sample matching query vectors; The threshold for Sparsemax activation is used to determine which matching query vectors are positive samples; It is the L2 norm of q, used to stabilize the training process and avoid gradient explosion.

9. A water surface floating object detection device based on a hierarchical semantic tree, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the water surface floating object detection method based on any one of claims 1-8.

10. A computer-readable storage medium on which a program is stored, characterized in that, When the program is executed by the processor, it implements the steps of the water surface floating object detection method based on a hierarchical semantic tree as described in any one of claims 1-8.

Images