A water surface multi-target detection and automatic labeling method based on threshold segmentation

Abstract

The present application relates to a kind of water surface multi-target detection and automatic labeling method based on threshold segmentation, belong to water surface target detection and deep learning field, solve the problem of low efficiency, unstable precision that exists in traditional artificial data labeling method.It includes obtaining water surface channel visible light monitoring video and pre-processing, obtaining the multi-frame binary image of water surface target and background separation;Each frame binary image is analyzed to obtain a plurality of target mark frame set;Multiple target mark frame is recycled and screened to obtain the mark frame of each real water surface target;Real water surface target mark frame in each frame is tracked and predicted, and the predicted water surface target position coordinate and size information are obtained;Based on water surface target position coordinate and size information, water surface target image cutting, water surface target position coordinate normalization conversion and automatic labeling label are carried out, and the labeled data set is obtained.The water surface target sample data and automatic labeling label are realized automatically.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of water surface target detection and deep learning, and particularly relates to a water surface multi-target detection and automatic labeling method based on threshold segmentation. BACKGROUND

[0002] With the increasing number of navigable vessels, the complexity and potential risks of water transportation are gradually rising. The problem of ship-bridge collision has gradually become a serious safety hazard, especially in densely navigable areas such as urban rivers and lakes. In order to effectively prevent ship-bridge collision accidents, it is urgent to develop more intelligent monitoring means to monitor and evaluate water transportation conditions in real time. This demand has promoted the research on new monitoring systems and intelligent algorithms, aiming to improve the accuracy and efficiency of monitoring.

[0003] In the field of deep learning, the production of data sets is crucial for model training. Traditionally, many application scenarios rely on manual generation of data sets. However, this method has many drawbacks. First, manual annotation usually requires a lot of time and human resources, especially in complex scenarios, the accuracy and consistency of annotation are often difficult to guarantee. Second, manual annotation is easily affected by subjective factors, leading to uneven quality of data sets, which affects the performance of the model. In addition, in dynamic environments such as changes in water surface targets and unpredictable movements, manual annotation is difficult to adapt to real-time changes, reducing the timeliness and applicability of data sets.

[0004] Under this background, computer intelligent data generation methods have emerged as the times require, becoming an important means to solve the above problems. Existing computer-aided data generation methods mainly include synthetic data generation, automatic labeling and semi-supervised learning. These methods use advanced computer vision and deep learning technologies to greatly improve the efficiency and accuracy of data generation. For example, synthetic data generation can simulate the environment and objects to achieve a large number of high-quality training samples. Automatic labeling technology uses existing models for preliminary labeling, reducing human intervention, thereby speeding up the data set production process.

[0005] These intelligent data generation methods still have certain limitations in practical application. In the field of image target detection, common automatic labeling methods each have their advantages and disadvantages. The automatic labeling based on weakly supervised learning uses image-level labels to train the model to generate pseudo bounding box labels, which can reduce the amount of manual labeling, but the positioning accuracy may be limited, and the accuracy of the generated pseudo labels needs to be improved; semi-supervised learning automatic labeling combines a small amount of manual labeling with a large amount of unlabeled data, which can effectively utilize data resources to improve model performance, but the quality of pseudo labels depends on the initial model, and if the initial model performance is poor, it may lead to error propagation; the automatic labeling based on active learning can assist in focusing on key samples to reduce the number of labels and improve labeling efficiency, but it relies on manual labeling of algorithm-selected samples, which is not completely automated; the automatic labeling based on generative adversarial network (GAN) can generate pseudo samples and label auxiliary training, but its training process is complex, and the generated pseudo labels may deviate from the real data; the combination of crowdsourcing and automatic screening can quickly obtain labeled data with the help of a large number of labelers, and through automatic screening, a certain quality is ensured, but the quality of crowdsourcing labeling is uneven, the automatic screening algorithm may also misjudge, and manual review and correction are still needed to ensure the reliability of the labeling. SUMMARY

[0006] In view of the above analysis, the embodiments of the present application aim to provide a water surface multi-target detection and automatic labeling method based on threshold segmentation, to solve the technical problems of low efficiency and unstable precision of traditional manual data labeling methods.

[0007] The main purpose of the present application is achieved by the following technical solutions:

[0008] The present application provides a water surface multi-target detection and automatic labeling method based on threshold segmentation, comprising:

[0009] Obtaining waterway visible light monitoring video and preprocessing to obtain a plurality of binary images with water surface target and background separated;

[0010] Performing connected component analysis on each binary image to obtain a plurality of target bounding box sets; performing loop merging and screening on the plurality of target bounding boxes to obtain the bounding box of each real water surface target; performing target tracking prediction on the real water surface target bounding box in each frame to obtain the predicted water surface target position coordinates and size information;

[0011] Based on the water surface target position coordinates and size information, performing water surface target image cropping, water surface target position coordinate normalization conversion and automatic labeling label to obtain the labeled water surface target dataset.

[0012] Further, the preprocessing of the waterway visible light monitoring video comprises:

[0013] The total number of frames of the video is calculated based on the total length and frame rate of the visible light monitoring video of the waterway;

[0014] Based on the set frame extraction interval, the visible light monitoring video of the waterway is extracted to obtain a plurality of corresponding color RGB images;

[0015] Each frame of the color RGB image is converted into a grayscale image;

[0016] The grayscale image is subjected to inverse color processing to obtain an inverse color-processed grayscale image;

[0017] Based on the perspective-optimized binary mask image, the inverse color-processed grayscale image is subjected to threshold segmentation to obtain a plurality of binary images in which the water surface target and the background are separated.

[0018] Further, the perspective-optimized binary mask image includes:

[0019] Based on the installation position and perspective of the bridge camera for shooting the visible light monitoring video, the waterway range is calibrated;

[0020] A binary matrix mask image with the same size as the color RGB image is created, and the pixel points within the waterway range are assigned a value of 1, and the pixel points outside the waterway range are assigned a value of 0.

[0021] Further, the water surface target image is automatically labeled, including:

[0022] A small amount of water surface target images are subjected to HOG feature extraction to obtain corresponding feature vectors and manually annotated water surface target labels to form a pre-training sample set;

[0023] The pre-training sample set is used to train the SVM model;

[0024] When the detection accuracy and recall rate of the SVM model meet the requirements, a pre-trained SVM model is obtained;

[0025] The feature vectors extracted from the unlabeled water surface target images are input into the pre-trained SVM model to obtain the detection results of the corresponding water surface target images as the class labels of the water surface target images;

[0026] The class labels of the water surface target images, the water surface target image coordinates and size information, and the frame number of the water surface target image are generated to form a YOLO format labeled label.

[0027] Further, the inverse color-processed grayscale image is subjected to threshold segmentation, including:

[0028] Set a target background segmentation threshold T;

[0029] When the grayscale value in the anti-color processed grayscale image is higher than T and the element in the corresponding binary mask image is 0, mark 1, indicating a detected water surface target; otherwise, mark 0, indicating a detected background. The segmentation process of the water surface target and the background is as follows:

[0030]

[0031] Where, I binary (x,y) is the binary image of the water surface target and the background after threshold segmentation, and M(x,y) is the binary mask image.

[0032] Further, a plurality of water surface target marking box sets are obtained by performing connected domain analysis on the binary image, including:

[0033] The connected domain analysis is performed on the binary image to extract all target connected regions, and the minimum circumscribed rectangle BBox of the target region is obtained. For each connected region C i , the following is represented:

[0034]

[0035] Where, N is the number of detected connected domains, (x min ,y min ), (x max ,y max ) are the top-left corner and bottom-right corner coordinates of BBox, respectively;

[0036] Based on C i , the area S i of BBox is calculated; if S i is less than a preset area threshold S th , it is determined to be noise or a non-water surface target and is removed;

[0037] After the removal processing, a plurality of water surface target marking box sets are obtained.

[0038] Further, the plurality of target marking boxes are cyclically combined and screened to obtain a marking box of each real water surface target, including:

[0039] The intersection and union areas of any two target marking boxes B a and B b of the plurality of water surface target marking box sets are calculated to obtain the intersection and union ratio IoU(B a ,B b ) of any two marking boxes;

[0040] If the intersection and union ratio of the two target marking boxes is greater than an intersection and union ratio threshold IoUthreshold , then the two target marking frames are determined to belong to the same target, and the smaller area of ​​the two target marking frames is merged into the larger area marking frame; multiple merged BBoxes are obtained;

[0041] After the merging is completed, if the area of ​​the merged BBox is less than the preset area threshold S m , it is determined to be a noise or interference area and removed to obtain the marking box BBox of each real surface target.

[0042] Furthermore, target tracking prediction is performed on the marking frame of the real surface target to obtain the predicted surface target position coordinates and size information, including:

[0043] The set of real surface target frames of the current frame t is B t , The target frame set of the previous frame t-1 is B t-1，

[0044] Calculate the Euclidean distance d(B) of each frame between the current frame and the previous frame t i ,B j t-1 );

[0045] like Satisfy the distance threshold d th ,choose The smallest B t i As Matching objects;

[0046] Based on Euclidean distance and a time interval between two frames, predicting the next frame speed and acceleration of the surface target corresponding to the matching object; and calculating the speed and acceleration in the horizontal direction and the vertical direction based on the speed and acceleration;

[0047] Based on the horizontal and vertical speeds and accelerations of the surface target, the position of the surface target in the next frame t+1 and the width and height size data estimation results of the target box BBox are predicted.

[0048] Furthermore, based on the position coordinates and size information of the water surface target, the water surface target image having an area larger than a preset area S is cut out to obtain the water surface target image;

[0049] The label annotation of each surface target image satisfies the YOLO format, and the label of the surface target image is <frame number> <class><x center ><y center > <width> <height>;

[0050] wherein, frame number is the frame sequence number corresponding to the water surface target image; class is the category label of the water surface icon image; (x center ,y center ) is the center point coordinate of the water surface target bounding box BBox; width and height are the normalized width and height of BBox, respectively.

[0051] Further, the water surface target data set comprises the multi-frame color RGB images extracted by frame extraction and the water surface target images, and the corresponding category labels.

[0052] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0053] 1. The water surface multi-target detection and labeling method disclosed by the present application realizes effective background suppression and target extraction by combining threshold segmentation and mask technology, significantly improves the accuracy and robustness of water surface target monitoring, and is particularly suitable for dynamic water surface environment;

[0054] 2. The present application automatically labels the labels of water surface targets through a pre-trained SVM model, greatly reduces the workload and time of manual labeling, and improves the labeling efficiency;

[0055] 3. The present application can monitor and evaluate water traffic conditions in real time, adapt to different environments and target changes, and has stronger adaptability and practicality;

[0056] 4. The present application can generate high-quality water surface target sample data set, avoid errors and omissions that may occur in manual labeling, improve the integrity and accuracy of the water surface sample data set, and provide more reliable data support for subsequent model training and analysis; and provide effective technical support for the intelligent development of bridge anti-collision warning system.

[0057] In the present application, the above-mentioned technical solutions can be combined with each other to realize more preferred combination schemes. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification, or will be understood by implementing the present application. The purpose and other advantages of the present application can be achieved and obtained by the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not considered as limiting the present application, and throughout the drawings, the same reference signs represent the same parts.

[0059] Figure 1 It is a flow chart of a water surface multi-target detection and automatic labeling method based on threshold segmentation in the embodiments of the present application.

[0060] Figure 2 A water surface channel visible light monitoring video schematic diagram taken by a monitoring device deployed for a certain bridge in an embodiment of the present application;

[0061] Figure 3 An extraction result schematic diagram with a video interval k of 4 seconds in an embodiment of the present application;

[0062] Figure 4 A certain bridge color RGB image to grayscale image schematic diagram in an embodiment of the present application;

[0063] Figure 5 A certain bridge monitoring screen mask diagram design schematic diagram in an embodiment of the present application;

[0064] Figure 6 An anti-color processing image result schematic diagram in an embodiment of the present application;

[0065] Figure 7 A binary image schematic diagram after threshold segmentation in an embodiment of the present application;

[0066] Figure 8 A certain bridge monitoring screen binary image to obtain a target minimum bounding rectangle frame result schematic diagram in an embodiment of the present application;

[0067] Figure 9 A cyclic merging and deleting BBox result schematic diagram in an embodiment of the present application;

[0068] Figure 10 A BBox representation schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present application will be described in detail below with reference to the accompanying drawings, which form a part of this application. The accompanying drawings and the embodiments together illustrate the principles of the present application, but are not intended to limit the scope of the present application.

[0070] To solve the above problems, the present application proposes a water surface multi-target detection and tracking auxiliary labeling method based on threshold segmentation. The method takes the monitoring video of the water surface channel taken by the monitoring device deployed on the bridge as input, and realizes automatic labeling of water surface target samples by using the method in the present application. Manual frame-by-frame target calibration and label generation are no longer needed, and a high-quality water surface target data set is generated.

[0071] With the increasing demand for water traffic safety, using intelligent data generation methods is the key to improving the effectiveness and safety of monitoring systems. The method of water surface multi-target detection and automatic labeling based on threshold segmentation not only provides a new idea for solving the ship-bridge conflict problem, but also provides a reference for related field water surface target dataset making, promoting the progress of intelligent monitoring technology.

[0072] Exemplarily, the application utilizes visible light video monitoring of multiple navigable bridges in Guangdong Province, and through the combination of image processing technology and target detection and tracking algorithm, intelligent water surface target dataset making is carried out, which can not only improve the sample labeling efficiency, but also ensure the accuracy of sample labeling, and meet the high-quality demand of deep learning model for water surface target dataset.

[0073] The water surface multi-target detection and tracking auxiliary automatic labeling method disclosed by the application first slices the monitoring video, uses threshold segmentation technology to preliminarily process the water surface, and extracts the candidate area of the target ship. Then, the target detection algorithm is used to accurately identify the candidate area, the position and motion trajectory information of the water surface target ship are obtained, and the water surface target tracking algorithm is used to realize the tracking of multiple targets. Through coordinate conversion and automatic label generation, the detected water surface target information is converted into labeled data conforming to the YOLO model format. Finally, the labeled data constructed is combined with the original monitoring video to form a high-quality water surface target dataset, which can not only effectively support the training of subsequent deep learning models, but also significantly shorten the time of dataset making.

[0074] One specific embodiment of the application discloses a water surface multi-target detection and automatic labeling method based on threshold segmentation, as shown in Figure 1 The steps include:

[0075] Step S1, obtaining a water surface channel visible light monitoring video and preprocessing to obtain a plurality of frames of binary images with water surface targets and background separated;

[0076] Step S2, performing connected domain analysis on each frame of binary image to obtain a plurality of target marking box sets; performing loop merging and screening on the plurality of target marking boxes to obtain a marking box of each real water surface target; performing target tracking prediction on the real water surface target marking box in each frame to obtain predicted water surface target position coordinates and size information;

[0077] Step S3, based on the water surface target position coordinates and size information, performing water surface target image cropping, water surface target position coordinate normalization conversion and automatic labeling label to obtain a labeled water surface target dataset.

[0078] Step S1 is divided into steps S11-S12.

[0079] Step S11, obtain a visible light monitoring video of a waterway.

[0080] Based on the monitoring equipment deployed on the bridge, the visible light monitoring video of the waterway is obtained. Exemplarily, the present application selects a real bridge monitoring video as a data source, which is the water surface monitoring video of a certain bridge from July to September 2024.

[0081] As shown in Figure 2 , the visible light video of the waterway monitored by a certain bridge on June 4, 2024, has rich water surface ship targets and complex environmental background, which can provide strong support for the test, verification and data set generation of the method of the present application.

[0082] The selection of video data fully considers the diversity and representativeness in actual application scenarios, especially the complex factors commonly seen in water surface target detection, such as light changes, water surface fluctuations, and debris interference.

[0083] Step S2, pre-processing the obtained visible light monitoring video of the waterway to obtain a plurality of binary images with water surface targets and background separated.

[0084] The pre-processing of the visible light monitoring video of the waterway includes:

[0085] Based on the total length and frequency of the visible light monitoring video of the waterway, the total number of video frames is calculated;

[0086] Based on the set frame extraction interval, the visible light monitoring video of the waterway is extracted to obtain a plurality of corresponding color RGB images;

[0087] Each frame of the color RGB image is converted into a grayscale image;

[0088] Each frame of the grayscale image is subjected to inverse color processing to obtain an inverse color processed grayscale image;

[0089] Based on the binary mask image designed according to the perspective optimization, each frame of the inverse color processed grayscale image is subjected to threshold segmentation to obtain a plurality of binary images with water surface targets and background separated.

[0090] The pre-processing of the visible light monitoring video of the waterway is to ensure the efficiency and accuracy of the subsequent water surface target detection process. Specifically as follows:

[0091] (1) Calculate the total number of frames of each visible light monitoring video of the waterway.

[0092] In the image target detection and data set generation tasks of the waterway visible light monitoring video, the original monitoring video data is converted into single frame image data. In order to ensure the effectiveness of subsequent image processing, the frame number of the video is first calculated accurately. For example, the total length of the monitoring video is T (unit: second), and the total frame number N of the waterway visible light monitoring video is calculated as follows:

[0093] N = T x f Formula (1)

[0094] Wherein, f is the frame rate of the video, unit: frame / second.

[0095] (2) Based on the total frame number of the video, set the frame extraction interval k, and extract the waterway visible light monitoring video to obtain the corresponding multi-frame color RGB image.

[0096] When processing the waterway visible light monitoring video, in order to ensure the continuity of the image sequence and avoid overfitting, a suitable frame extraction interval k is set to improve the processing efficiency under the premise of ensuring the integrity of the information.

[0097] The purpose of frame extraction is to reduce the redundant information between adjacent frames, especially when the position of the water surface target object (such as a ship) changes little in adjacent frames, and the difference is not significant. Frame-by-frame extraction will lead to redundant calculation and overfitting of the model.

[0098] The monitoring video is extracted to obtain the image format suitable for processing, such as png, jpg image format, and the size is 2048*2048.

[0099] In order to ensure the continuity of the image while avoiding overfitting when processing the monitoring video, a suitable frame extraction interval k is set based on specific requirements. The selection of frame extraction interval k is adjusted flexibly according to the training requirements and the characteristics of the target motion, especially considering the speed of the water surface target object. Through this interval, the required frame number M can be effectively extracted from the total frame number, as shown below:

[0100]

[0101] In practical application, the selection of k is adjusted according to the training requirements and the speed of the target motion to balance the integrity of the information and the efficiency of the processing. For example, assuming that the frame rate of the video is 25 frames / second, and the ship moves slowly in the video, it may take about 5 minutes from the ship entering the field of view to completely disappearing. If the image is extracted frame by frame, it may lead to a large amount of redundant information and increase the computational burden. For example, k is taken as 4 seconds; by setting a suitable frame extraction interval, the processing frame number can be effectively reduced, while ensuring that each frame of image has sufficient dynamic change information.

[0102] After frame extraction, the obtained multi-frame color RGB image sequence is subjected to subsequent image analysis and target detection. As shown in Figure 3 The image is a monitoring picture with a video interval k of 4 seconds in the same time period, and it can be seen that the ship movement in the lower right corner has changed. Not only does it ensure the continuity of the image sequence, but it also improves the algorithm processing efficiency by reducing the extraction of redundant frames and avoids overfitting in the training process.

[0103] After frame extraction, a plurality of color RGB images corresponding to the visible light monitoring video of each waterway are obtained.

[0104] (3) The color RGB images are subjected to gray scale conversion to obtain corresponding gray scale images.

[0105] The purpose of the gray scale conversion process is to convert the color information in the color RGB image into a single luminance value to facilitate subsequent image analysis and target detection. The gray scale conversion process is completed through the following weighting formula:

[0106] I gray (x,y) = 0.2989 R(x,y) + 0.5870 G(x,y) + 0.1140 B(x,y) Formula (3)

[0107] where (x,y) is the point coordinate of the pixel point in the image; I gray (x,y) is the gray scale value of the point (x,y) in the 1-channel gray scale image; R(x,y) is the gray scale value of the R-channel point (x,y), G(x,y) is the gray scale value of the G-channel point (x,y), and B(x,y) is the gray scale value of the B-channel point (x,y).

[0108] The color RGB images obtained by frame extraction are converted into gray scale images. Through this gray scale conversion process, the redundant color information in the original color RGB image is compressed into luminance information, significantly reducing the computational complexity and helping to improve the efficiency and robustness of the target detection algorithm. The gray scale image is not only more concise in vision, but also effectively retains the shape and contour features of the target, which is beneficial to subsequent connected component analysis and water surface target detection. As shown in Figure 4 After image gray scale conversion, a gray scale image of a certain bridge monitoring image is obtained.

[0109] (4) Design a binary mask based on the viewing angle optimization.

[0110] To achieve accurate water surface target detection and avoid the interference of background objects such as wharfs and shores on the water surface target detection results, the present application proposes an image region screening method based on mask technology. A binary mask is designed to highlight the regions of interest in the image and suppress the influence of other irrelevant regions.

[0111] Design principle of binary mask image: the region of interest is represented by black (gray value 0) in the mask image, and the non-interest region is represented by white (gray value 1).

[0112] Optimize the design of binary mask image based on the view angle, including:

[0113] Based on the installation position and view angle of the bridge camera for shooting the visible light monitoring video, the waterway range is calibrated.

[0114] Create a binary matrix mask image with the same size as the color RGB image, and assign the pixel points in the waterway range to 1 and the pixel points outside the waterway range to 0.

[0115] Identify and extract the waterway region in the grayscale image, and mark it as 1 in the mask image to ensure that this region is fully focused in the subsequent processing process. The wharf, shore and other non-target regions are marked as 0, effectively suppressing these interference regions from the image.

[0116] Through this mask design, the non-target region of the image is effectively excluded, thereby improving the accuracy and robustness of the subsequent water surface target detection algorithm. This method can significantly reduce background interference and improve the accuracy of water surface target extraction.

[0117] Based on the monitoring equipment deployed on the bridge, such as the installation position and view angle of the monitoring camera, manually calibrate the waterway range (such as the curved river region of a certain bridge waterway).

[0118] In the calibration software (such as LabelMe), use the polygon tool to accurately outline the waterway boundary and generate a vector coordinate file.

[0119] The binary mask image M is represented as follows:

[0120]

[0121] According to the waterway vector coordinates, create a binary matrix with the same size as the grayscale image:

[0122] The pixel points in the waterway are assigned to 1, and the pixel points outside the waterway are assigned to 0. As shown in Figure 5 The monitoring screen of a certain bridge and the mask image design are shown, in which the wharf, riverbank and non-ship waterway positions are set as non-interest regions to prevent affecting the detection of water surface targets.

[0123] Customizing the design of the binary mask image makes the subsequent processing only focus on the region identified by the mask image, optimizing the operation efficiency of the algorithm, and can be adjusted according to the actual situation of different bridges or monitoring view angles, ensuring the adaptability and accuracy of the mask image.

[0124] (5) performing reverse color processing on each frame of the gray-scale image to obtain a reverse color processed gray-scale image.

[0125] In the target detection process, since the gray value of the water surface ship is usually lower than the gray value of the background (such as the river surface), in order to facilitate subsequent binaryzation processing, a reverse color processing method is adopted. The main purpose of the reverse color processing is to reverse the gray value in the image, so that the gray value of the water surface ship target changes from low to high, and the gray value of the background area changes from high to low, thereby improving the contrast between the water surface target and the background, and laying a foundation for subsequent threshold segmentation operation. For the gray value of each pixel I gray (x,y), the reverse color processing is calculated as follows:

[0126] I invert (x,y) = 255 - I gray (x,y) Formula (5)

[0127] Wherein, I gray (x,y) is the gray value of the position (x, y) in the original gray-scale image, I invert (x,y) is the gray value after reverse color processing, and 255 is the maximum value of the gray-scale image (i.e. the gray value of white).

[0128] As Figure 6 shown, after reverse color processing, the gray value of the water surface ship target will become relatively high, and the gray value of the background area will be relatively low.

[0129] (6) performing threshold segmentation on each frame of the reverse color processed gray-scale image to obtain a plurality of frames of binaryzation images in which the water surface target and the background are separated.

[0130] Performing threshold segmentation on the reverse color processed gray-scale image comprises:

[0131] Setting a target background segmentation threshold T;

[0132] When the gray value in the reverse color processed gray-scale image is higher than T and the corresponding element in the binary mask image is 0, marking 1, indicating a detected water surface target; otherwise, marking 0, indicating a detected background, and the segmentation process of the water surface target and the background is as follows:

[0133]

[0134] Wherein, I binary (x,y) is the binaryzation image in which the water surface target and the background are separated after threshold segmentation, and M(x,y) is a binary mask image.

[0135] The image is divided into target region and background region by threshold segmentation technology. A suitable target-background segmentation threshold T is set, and the region with a gray value higher than T in the image is marked as a ship water surface target, while the region with a gray value lower than T is marked as background. Exemplarily, for an 8-bit image (gray value interval 0-255), the target-background threshold T is set to 120.

[0136] The value of 0 indicates that the background has been effectively suppressed. The gray image is threshold segmented to separate the water surface target from the background, suppress the interference background while retaining the water surface target information, thereby improving the detection accuracy.

[0137] As shown in Figure 7 , after background suppression and threshold segmentation, a background-suppressed image is obtained. At this time, the background is all black (gray value 0), and the target is a white region (gray value 255).

[0138] Step S1 separates the water surface target from the background by preprocessing the water surface channel visual monitoring video, and generates multiple binary images, providing a basis for subsequent target detection and tracking.

[0139] Step S2 is divided into steps S21-S23.

[0140] Based on the multiple binary image information obtained in step S1, the water surface target is detected and tracked.

[0141] Step S21 performs connected domain analysis on each binary image to obtain a plurality of target bounding box sets.

[0142] The connected domain analysis on the binary image obtains a plurality of water surface target bounding box sets, comprising:

[0143] The connected domain analysis on the binary image extracts all target connected regions to obtain the minimum bounding rectangle BBox of the target region. For each connected region C i , it is expressed as follows:

[0144]

[0145] Wherein, N is the number of detected connected domains, (x min ,y min ), (x max ,y max ) are the top-left and bottom-right coordinates of BBox, respectively;

[0146] Based on C i , the area S i of BBox is calculated; if S i is less than a preset area threshold S th , it is determined as noise or non-water surface target and is removed;

[0147] After the removal processing, a plurality of water surface target marking box sets are obtained.

[0148] The binary image I binary (x, y), and the target region is composed of connected white pixels. The I binary (x, y) is subjected to connected component analysis (Connected Components Labeling, CCL), all target regions are extracted, and the minimum circumscribed rectangle BBox is calculated.

[0149] For each connected region C i , the BBox thereof is determined by the upper left corner coordinates (x min , y min ) and the lower right corner coordinates (x max , y max ).

[0150] The area of the BBox is calculated:

[0151] S i = (x max -x min ) × (y max -y min ) Formula (8)

[0152] Exemplarily, the preset area threshold S th is set to 3.

[0153] After the above processing, a plurality of water surface target marking box geometries are obtained. As Figure 8 shown is the target minimum circumscribed rectangular box result obtained after the above-mentioned step processing of a certain bridge monitoring picture. It can be seen that there are mainly three targets in the image, the existing ship structure is labeled BBox, and the whole ship needs to be detected, so unnecessary BBox needs to be merged.

[0154] Step S22, the plurality of target marking boxes are circularly merged and screened to obtain the marking box of each real water surface target.

[0155] In order to optimize the generation of target box and reduce the number of redundant boxes, IoU (Intersection over Union) is used to realize. IoU can effectively measure the degree of overlap between two target boxes, thereby providing a basis for the merging and elimination of targets.

[0156] The plurality of target marking boxes are circularly merged and screened to obtain the marking box of each real water surface target, comprising:

[0157] Calculate the area of any two target marking boxes B a and B b The intersection and union area of ​​any two marked boxes are obtained by IoU (B a ,B b );

[0158] If the intersection-over-union ratio of the two target marker boxes is greater than the intersection-over-union ratio threshold IoU threshold , then the two target marking frames are determined to belong to the same target, and the smaller area of ​​the two target marking frames is merged into the larger area marking frame; multiple merged BBoxes are obtained;

[0159] After the merging is completed, if the area of ​​the merged BBox is less than the preset area threshold S m , it is determined to be a noise or interference area and removed to obtain the marking box BBox of each real surface target.

[0160] For any two marked boxes B a and B b The intersection over union (IoU) a ,B b ), as shown below:

[0161]

[0162] Among them, B a ∩B b 、B a ∪B b Mark box B a and B b The intersection and union of .

[0163] The intersection area |Ba∩Bb| and the union area |Ba∪Bb| are calculated as follows:

[0164]

[0165] in, Mark box B a The maximum and minimum values ​​on the horizontal and vertical axes; Mark box B b The maximum and minimum values ​​on the horizontal and vertical axes.

[0166] |B a ∪B b |=|B a |+|B b |-|B a ∩B b | Formula (10)

[0167] This step obtains the intersection-over-union ratio of any two marked boxes.

[0168] For example, the intersection-over-union threshold IoU threshold Set to 0.5; if the intersection over union (IoU) of any two marked boxes is a ,B b ) is greater than IoU threshold , then it is determined that the two marked boxes represent the same target and merge them into a larger marked box B merged :

[0169] B merged =B a ∪B b Formula (11)

[0170] If the intersection of two BBoxes is large, it is determined that the spatial overlap of the two BBoxes is high, and the IoU (B large ,B small )>IoU threshold , then it is more likely to belong to the same target, then B small Merge into B large , the smaller box B small It is directly eliminated and merged into the larger box B large .

[0171] After the merging is completed, all BBoxes are screened to remove the false detection areas. The screening rule is: if the area of ​​a BBox is less than the preset area threshold S m And it cannot be merged with other BBox (i.e. it fails to meet the IoU (B a ,B b ) is greater than IoU threshold merging conditions), it is determined to be a noise or water surface interference area and is removed. m Set it to 100 pixels area.

[0172] The merging step ensures that surface ship targets are not fragmented into multiple BBoxes due to surface fluctuations or noise. The filtering step eliminates isolated small connected domains to avoid false alarms. The resulting output of the ship boundary BBox is more stable and accurately describes the ship's boundaries, providing high-quality detection results for subsequent target tracking and behavior analysis.

[0173] Step S22 effectively reduces redundant frames, improves the accuracy of target detection, and ensures that each target is represented by only one marked frame, such as Figure 9 As shown in , the problem of repeated counting of surface targets is avoided. This process iterates between all BBoxes until there are no more BBoxes that meet the conditions and need to be merged. Figure 9 This is a schematic diagram of the result of cyclic merging and filtering BBox. The non-target components on the water surface are deleted, and the BBox of the hull itself and surrounding structures are retained and merged. A total of 3 surface ship targets are obtained in the image.

[0174] like Figure 10 The figure shows a schematic diagram of BBox representation. The number of rows indicates the number of surface targets, and the number of columns indicates the BBox information, which are the coordinates of the upper left corner (horizontal coordinate, vertical coordinate) and the height and width of the BBox, in pixels.

[0175] In step S22, multiple BBoxes containing the complete position information of surface ship targets are obtained by merging and eliminating marked boxes. Initial BBoxes are generated through connected domain extraction. However, due to surface noise, the target may be segmented into multiple fragmented regions, or false detections may occur. To this end, a multi-loop merging strategy is adopted to filter and fuse BBoxes based on their size, shape, and intersection-over-union (IoU) ratio, ensuring that each true target corresponds to a complete BBox and eliminating false detection areas.

[0176] The input of this process is a binary image, and the output is an optimized BBox set. The physical significance of this process is to reconstruct the morphology of water targets, improve detection integrity, reduce false detections, and provide accurate data for subsequent target tracking and behavior analysis.

[0177] Step S23: perform target tracking prediction on the real water surface target mark frame in each frame to obtain the predicted water surface target position coordinates and size information.

[0178] After the detection of real water targets is completed, continuous tracking of the water targets is the key. Since the movement of ship targets in the water environment is usually relatively stable, a matching algorithm between multiple frames is used to achieve real-time tracking of the targets.

[0179] Performing target tracking prediction on the marked frame of the real surface target to obtain predicted surface target position coordinates and size information, including:

[0180] The set of real surface target frames of the current frame t is B t , The target frame set of the previous frame t-1 is B t-1，

[0181] Calculate the Euclidean distance of each box between the current frame and the previous frame

[0182] like Satisfy the distance threshold d th ,choose The smallest B t i As Matching object

[0183] Based on the Euclidean distance and the time interval of two frames, the next frame speed and acceleration of the water surface target corresponding to the matching object are predicted; and the horizontal direction and vertical direction speed and acceleration are calculated based on the speed and acceleration;

[0184] Based on the horizontal direction and vertical direction speed and acceleration of the water surface target, the position of the water surface target in the next frame t+1 and the size data estimation results of the width and height of the target box BBox are predicted.

[0185] The Euclidean distance of each box between the current frame and the previous frame is calculated as follows:

[0186]

[0187] Wherein, i and j are the serial number i of the target in the current frame image and the serial number j in the previous frame image, and the maximum value is the number of water surface targets in the current image (i.e. the number of BBox after the above steps), and are the center point coordinates of the marked box B i and B j , which are calculated as follows:

[0188]

[0189] The distance threshold d th is usually twice the average moving distance of the water surface target, and exemplarily, d th is 8 pixels.

[0190] After matching is completed, the motion state of the water surface target is analyzed. By comparing the changes of the box positions between the current frame and the previous frame, the speed v and the acceleration a of the target are calculated as follows:

[0191]

[0192] Wherein, Δt is the time interval between the current frame and the previous frame, v t and v t-1 are the speeds of the current frame and the previous frame; v x , v y are the moving speed of the target on the horizontal direction of the image and the speed on the vertical direction, v x , v y are the acceleration of the target on the horizontal direction of the image and the acceleration on the vertical direction, and θ is the included angle between the direction of the speed and the acceleration and the horizontal direction.

[0193] Based on the calculation results of the speed and the acceleration, the position of the target in the next frame t+1 (x pred , y pred ) and the size of the box:

[0194]

[0195] where, is the center coordinate of the current frame, (x pred , y pred ) is the center coordinate of the predicted next frame target bbox.

[0196] In addition to the change in position, the size of the target box may also change due to the change in distance between the ship and the monitoring device camera. Therefore, the width and height of the target box at frame t+1 need to be predicted using velocity extrapolation, which is calculated as follows:

[0197]

[0198] where, w pred and h pred are the width and height of the predicted next frame t+1, w i , h i are the width and height of the target in the current frame, w t -w t-1 and h t -h t-1 represent the change rate of the target box width and height between frame t-1 and t. This method can adapt to the size scaling caused by the change in viewing angle of the ship. For example, when the ship gradually moves away from the camera, its width and height will decrease accordingly, thus predicting the width w pred and height h pred of the next frame t+1.

[0199] Through this prediction mechanism, the system can adjust the position and size of the target box in real time, thereby maintaining continuous tracking of the ship target.

[0200] Based on the kinematic principle, by calculating the historical motion trend of the target, the position and size of the target in the next frame are predicted. The position prediction predicts the center position of the next frame based on the motion of the target in the previous two frames, ensuring that the target is not lost due to temporary occlusion or detection error during tracking. The size prediction is based on the change trend of the target box to adjust the size of the target box to adapt to the change in distance of the target, preventing tracking failure due to scale changes.

[0201] The function of step S2 is to detect and track the position and size changes of the water surface target based on the binary image through connected component analysis, BBox merging screening, and target tracking prediction.

[0202] Step S3, in particular.

[0203] After the prediction calculation based on the speed and size change trend, the estimation results of the target position and target size of frame t+1 are obtained. Then, the water surface target data set required by the YOLO (You Only Look Once) intelligent network is converted, and the data set includes multiple water surface target image data and corresponding labels.

[0204] The water surface target image is automatically labeled, and the label includes:

[0205] The HOG feature of a small amount of water surface target image is extracted to obtain the corresponding feature vector and the corresponding artificial water surface target label, and a pre-training sample set is formed.

[0206] The SVM model is trained by using the pre-training sample set.

[0207] When the detection accuracy and recall rate of the SVM model meet the requirements, a pre-trained SVM model is obtained.

[0208] The feature vector extracted by the HOG feature of the unlabeled water surface target image is input into the pre-trained SVM model to obtain the detection result of the corresponding water surface target image as the class label of the water surface target image.

[0209] The class label of the water surface target image, the water surface target image coordinate and size information, and the frame number of the water surface target image are used to generate a YOLO format label.

[0210] Exemplarily, the SVM model is realized by using the scikit-learn library of Python.

[0211] When the SVM model is pre-trained, a small amount of water surface target image is selected. Exemplarily, 100-200 water surface target images are selected.

[0212] Exemplarily, the label of the water surface target image is cargo ship, passenger ship, fishing boat, yacht, etc. The definition is made according to the specific requirements.

[0213] Based on the water surface target position coordinate and size information, the water surface image with an area greater than a preset area S is cut out as a water surface target image.

[0214] The label of each water surface target image meets the YOLO format, and the label of the water surface target image is <frame number> <class><x center ><y center > <width> <height>;

[0215] wherein, frame number is the frame sequence number corresponding to the water surface target image; class is the class label of the water surface icon image; (x center ,y center ) is the center point coordinate of the water surface target bounding box BBox; width and height are the normalized width and height of BBox, respectively.

[0216] Exemplarily, the preset area S is set to 200 pixel areas, and the morphology of the water surface target can be clearly seen.

[0217] In order to avoid the storage space of the water surface target dataset being too large, the storage upper limit needs to be set to limit the number of water surface target ship images, and the maximum of N max images is set. Exemplarily, N max is set to 5.

[0218] In the YOLO target detection algorithm, the training sample data adopts a standardized annotation format. In the folder of each water surface target ship ID, an `annotations.txt` file is generated to store the position information and frame number of each image of the water surface target ship in the monitoring video.

[0219] width and height are normalized to [0, 1], rather than using pixel units. For each water surface target B i , the normalized coordinates of the center point and the normalized dimensions of width and height are calculated as follows:

[0220]

[0221] wherein, W and H are the width and height of the water surface target image, is the normalized width; is the normalized height, and (x ,y

[0222] ) are the coordinates of the top-left corner and the bottom-right corner, respectively.

[0223] In order to facilitate management and classification, two main folders will be generated:

[0224] ① Frame extraction image folder: this folder contains all the image frames extracted from the monitoring video.

[0225] ② Ship information folder: this folder is used to store the detailed information of each ship.

[0226] Label storage: The label of each water surface target image is recorded in the annotations.txt file, which meets the data labeling requirements of the YOLO format.

[0227] In the ship information folder, a subfolder is created for each detected ship, named according to the custom ship ID, for example, the first ship in the monitoring video is 0001, and the folder name is "0001". The structure of this folder is as follows:

[0228] / ship information folder

[0229] ├──0001 /

[0230] │ ├──images / #subfolder, storing water surface target images

[0231] │ ├──annotations.txt #text file, storing YOLO format position information

[0232] ├──0002 /

[0233] │ ├──images /

[0234] │ ├──annotations.txt

[0235] └── ...

[0237] This step builds a high-quality water surface target image dataset, that is, all detected targets are initially assigned the same class label to form a unified dataset. At the same time, in order to further improve the usability and accuracy of the data, the system records different time frames of the same target during target tracking and stores its large-size shape information in key frames. For each stably tracked target, n representative frames are selected from its complete motion trajectory, and the tracked target is cropped and stored in an independent folder.

[0238] In the data sorting stage, each folder corresponds to an independent water surface target instance, containing its image samples at different time points. These samples not only reflect the shape changes of the target, but also provide visual features under different environmental conditions (such as light, angle, etc.), thereby enhancing the robustness of the dataset and providing higher quality input data for deep learning model training.

[0239] The function of step S3 is to generate annotated data sets that meet the YOLO format based on the predicted water surface target position and size information, providing high-quality input data for deep learning model training.

[0240] In summary, the water surface multi-target detection and automatic labeling method based on threshold segmentation has the following beneficial effects:

[0241] 1、The water surface multi-target detection and labeling method disclosed by the application realizes effective background suppression and target extraction by combining threshold segmentation and mask technology, significantly improves the accuracy and robustness of water surface target monitoring, and is particularly suitable for dynamic water surface environment;

[0242] 2、The application automatically labels the labels of water surface targets through the pre-trained SVM model, greatly reduces the workload and time of manual labeling, and improves the labeling efficiency;

[0243] 3、The application can monitor and evaluate water traffic conditions in real time, adapt to different environments and target changes, and has stronger adaptability and practicality;

[0244] 4、The application can generate high-quality water surface target sample data set, avoid errors and omissions that may occur in manual labeling, improve the integrity and accuracy of the water surface sample data set, and provide more reliable data support for subsequent model training and analysis; and provide effective technical support for the intelligent development of the bridge anti-collision warning system.

[0245] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. Among them, the computer readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0246] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the application, which should be covered within the protection scope of the application.< / height> < / width> < / class> < / height> < / width> < / class>

Claims

1. A water surface multi-target detection and automatic labeling method based on threshold segmentation, characterized in that, The method comprises the following steps: Obtain waterway visible light monitoring video and preprocess it to obtain a plurality of binary images in which water surface targets are separated from backgrounds; Perform connected domain analysis on each binary image to obtain a plurality of target bounding box sets; Merge and screen the plurality of target bounding boxes to obtain a bounding box of each real water surface target; Perform target tracking prediction on the real water surface target bounding box in each frame to obtain predicted water surface target position coordinates and size information; Based on the water surface target position coordinates and size information, perform water surface target image cropping, water surface target position coordinate normalization conversion and automatic labeling to obtain a labeled water surface target dataset; Preprocess the waterway visible light monitoring video, including: Calculate the total number of frames of the video based on the total length and frame rate of the waterway visible light monitoring video; Extract a plurality of color RGB images corresponding to the waterway visible light monitoring video based on a set frame extraction interval; Convert each of the color RGB images into a grayscale image; Perform color inversion processing on each of the grayscale images to obtain a grayscale image after color inversion processing; Perform threshold segmentation on each of the grayscale images after color inversion processing based on a binary mask image designed according to an optimized viewing angle to obtain a plurality of binary images in which water surface targets are separated from backgrounds; Design the binary mask image based on an optimized viewing angle, including: Calibrate the range of the waterway based on the installation position and viewing angle of the bridge camera that captures the visible light monitoring video; Create a binary mask image with the same size as the color RGB image, in which the pixel points within the range of the waterway are assigned a value of 1 and the pixel points outside the range of the waterway are assigned a value of 0; Perform target tracking prediction on the bounding box of the real water surface target to obtain predicted water surface target position coordinates and size information, including: The real water surface target frame set of the current frame t is B t , The target frame set of the previous frame t-1 is B t-1， calculating the Euclidean distance d(B t i ,B j t-1 ); If d(B t i ,B j t-1 ) satisfies the distance threshold d th , select B t i with the smallest d(B j t-1 ) as the matching object; based on the Euclidean distance d(B t i ,B j t-1 ) and the time interval of two frames, the next frame speed and acceleration of the water surface target corresponding to the matching object are predicted; the horizontal and vertical speed and acceleration are calculated based on the speed and acceleration. Based on the speed and acceleration of the water surface target in the horizontal direction and the vertical direction, estimate the position of the water surface target in the next frame t+1 and the size data of the width and height of the target box BBox.

2. The method of claim 1, wherein, Automatically label the water surface target image, including: Extract HOG features from a small number of water surface target images to obtain corresponding feature vectors and corresponding manually annotated water surface target labels to form a pre-training sample set; Train an SVM model using the pre-training sample set; When the detection accuracy and recall rate of the SVM model meet the requirements, a pre-trained SVM model is obtained; Input the feature vectors extracted from the unlabeled water surface target images into the pre-trained SVM model to obtain the detection results of the corresponding water surface target images as the class labels of the water surface target images; Generate a YOLO format label based on the class labels of the water surface target images, the water surface target image coordinates and size information, and the frame number of the water surface target image.

3. The method of claim 2, wherein, Perform threshold segmentation on the grayscale image after color inversion processing, including: Set a target background segmentation threshold T; When the gray value in the inverse color-processed gray image is higher than T and the element in the corresponding binary mask image is 0, it is marked as 1, indicating a detected water surface target; otherwise, it is marked as 0, indicating a detected background, and the segmentation process of the water surface target and the background is as follows: where I binary (x, y) is the binary image of the water surface target and background separation after threshold segmentation, and M(x, y) is the binary mask image.

4. The method of claim 1, wherein, The binary image is subjected to connected domain analysis to obtain a plurality of water surface target marking box sets, including: Performing connected domain analysis on the binary image, extracting all target connected regions, obtaining the minimum circumscribed rectangle BBox of the target region, for each connected region C i is represented as follows: wherein N is the number of connected domains detected, (x min ,y min ), (x max ,y max ) are the coordinates of the top-left and bottom-right corners of the BBox, respectively; C-based i , calculate the area S of BBox i If S i Smaller than the preset area threshold S th , it is determined to be noise or non-surface target and is removed; After the elimination processing, a plurality of water surface target marking box sets are obtained.

5. The method of claim 4, wherein, The plurality of target marking boxes are subjected to cyclic merging and screening to obtain a marking box of each real water surface target, including: Calculate the intersection and union areas of any two target marking boxes B a and B b in the plurality of water surface target marking box sets to obtain the intersection and union ratio IoU(B a ,B b ) of any two marking boxes. If the intersection-over-union of two target marking boxes is greater than the intersection-over-union threshold IoU threshold determine that the two target marking boxes belong to the same target, and merge the smaller one into the larger one; obtain a plurality of merged BBoxes; After the merging is completed, if the area of the merged BBox is less than a preset area threshold S m , it is determined as a noise or interference region for rejection, and a marked frame BBox of each real water surface target is obtained.

6. The method of claim 1, wherein, Based on the water surface target position coordinates and size information, a water surface image with an area greater than a preset area S is cut out as a water surface target image; The label of each water surface target image meets the YOLO format, and the label of the water surface target image is <frame number> <class><x center ><y center > <width> <height> ；< / height> < / width> < / class> wherein the frame number is a frame sequence number corresponding to the water surface target image; the class is a category label of the water surface icon image; (x center ,y center ) is the center point coordinate of the water surface target bounding box BBox; width and height are respectively the normalized width and height of the BBox.

7. The method according to any one of claims 1 to 6, characterized in that, The water surface target data set includes a plurality of frames of color RGB images obtained by frame extraction and the water surface target image, and a corresponding category label.

Images