Method and device for detecting floating objects on water surface and computer equipment

Abstract

The invention relates to a water surface floating object detection method and device and computer equipment. Comprising the following steps: inputting an obtained initial water surface image into a completely trained water body segmentation network for water body segmentation processing to obtain a water body image carrying a segmentation result; inputting the image of the area where the water surface floating object is located in the water body image into a preset semantic segmentation large model for detection processing, and obtaining a first segmentation probability result corresponding to the area where the water surface floating object is located; performing saliency detection on the image of the area where the water surface floating object is located in the water body image to obtain a second segmentation probability result corresponding to the water surface floating object; and carrying out fusion processing on the first segmentation probability result and the second segmentation probability result to obtain a detection result corresponding to the water surface floating object. By adopting the method, the accuracy of water surface floating object detection can be improved, and the calculation cost can be reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and computer equipment for detecting floating objects on a water surface. Background Art

[0002] With the rapid development of the economy, the continuous advancement of urbanization and industrialization, human activities and natural factors have caused serious pollution to the environment. A large number of floating objects have appeared on the surface of rivers, which has seriously damaged the river landscape and water ecological environment, and directly threatened human survival and development. The monitoring and management of floating objects in rivers has become a key issue in river supervision. To solve this problem, the existing technologies include fixed-position camera monitoring solutions and drone-based automatic inspection solutions. Among them, the drone automatic inspection solution can not only realize the complete monitoring of the entire river, but also realize the real-time positioning of the detected floating objects. However, most of the recognition algorithms are supervised image classification, target detection and semantic segmentation algorithms based on deep learning. Such algorithms require a large number of training data sets. For river floating object data, due to the different styles of waters in various regions, the types of floating objects are different, and the sizes are different. The amount of effective data collected is very limited, and the model training results are often not ideal. Moreover, such algorithms have high pixel restrictions on effective targets in images. The size of river floating objects in drone images is often smaller than the pixel requirements of effective targets, and the actual application effect is poor.

[0003] At present, no effective solution has been proposed for the problems of low accuracy and high application threshold requirements in the existing technology for detecting floating objects on the water surface. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device and computer equipment for detecting floating objects on the water surface to address the above technical problems.

[0005] In a first aspect, the present application provides a method for detecting floating objects on a water surface. The method comprises: The obtained initial water surface image is input into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image with segmentation results; The image of the area where the floating objects on the water surface are located in the water body image is input into a preset semantic segmentation model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating objects on the water surface are located; Performing saliency detection on an image of a region where floating objects on the water surface are located in the water body image to obtain a second segmentation probability result corresponding to the floating objects on the water surface; The first segmentation probability result and the second segmentation probability result are fused to obtain a detection result corresponding to the floating object on the water surface.

[0006] In one embodiment, obtaining an initial water surface image includes: Get the initial image; The initial image is cropped according to a preset cropping size to obtain at least one initial water surface image.

[0007] In one embodiment, the initial image is cropped according to a preset cropping size, including: Obtaining a preset overlapping region clipping rule, wherein the overlapping region clipping rule represents the size of the overlapping region between two initial water surface images at adjacent positions; The initial image is cropped according to a preset cropping size and an overlapping area cropping rule to obtain an initial water surface image, wherein the size of the overlapping area is greater than 0.

[0008] In one embodiment, each initial water surface image corresponds to a clipping index; the acquired initial water surface image is input into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image with a segmentation result, including: The initial water surface image is input into the water body segmentation network for water body segmentation processing to obtain the water body segmentation result, wherein the water body segmentation result is in the form of a probability matrix; The water body segmentation result is restored to the initial water surface image to obtain a water body sub-image carrying the water body segmentation result; The water body sub-images are spliced ​​based on the clipping index, wherein the probability matrix of the water body area included in each water body sub-image is determined according to the water body segmentation result; When the area where the water body segmentation result is located is located in the overlapping area of ​​at least two water body sub-images, the water body segmentation results in the water body sub-images are fused to calculate a water body image carrying the segmentation result.

[0009] In one embodiment, an image of an area where floating objects are located in a water body image is input into a preset semantic segmentation model for detection processing, including: The water body image is cropped based on the preset cropping size and overlapping area cropping rules, and the cropped water body image and the corresponding water body detection instruction template are input into the well-trained target detection model to obtain the target frame corresponding to all floating objects on the water surface; Based on the target frame, the image of the area where the floating objects are located is extracted from the cropped water image, and the image of the area where the floating objects are located is input into the semantic segmentation model for detection and processing.

[0010] In one embodiment, the water body image is cropped based on a preset cropping size, and the cropped water body image and the corresponding water body detection instruction template are input into a well-trained target detection model, and after obtaining the target frames corresponding to all floating objects on the water surface, the method further includes: Extracting an image of the area where the floating objects are located from the water body image, and classifying the image of the area where the floating objects are located to obtain a classification corresponding to the floating objects on the water surface; If the category of the floating object on the water surface belongs to the preset anomaly detection category, the image of the area where the floating object on the water surface is located corresponding to the floating object on the water surface is input into the semantic segmentation model for detection processing.

[0011] In one embodiment, after obtaining the detection result corresponding to the floating object on the water surface, the method further includes: The detection results are restored to the image of the area where the floating objects on the water surface are located in the water body image, and the cropped water body image carrying the detection results is stitched based on the cropping index, cropping size and overlapping area cropping rules; When the area where the detection result is located is located in at least two cropped water body images, the probability matrix of the cropped water body images for the detection result is fused to calculate a water body image carrying a final probability matrix.

[0012] In one embodiment, a saliency detection is performed on an image of a region where floating objects on the water surface are located in a water body image to obtain a second segmentation probability result corresponding to the floating objects on the water surface, including: Perform saliency detection on the image of the area where the floating objects on the water surface are located in the water body image, and obtain the feature value corresponding to each pixel point in the image of the area where the floating objects on the water surface are located; The similarity between the feature value corresponding to each pixel and the feature value corresponding to the adjacent pixel is calculated, and based on the target similarity of the similarities that is less than a preset similarity threshold, a second segmentation probability result is determined.

[0013] In one embodiment, obtaining a water body segmentation network includes: Obtain a preset water body image training set, wherein the water body image training set carries a water body feature label; The water body image training set is input into the preset initial water body segmentation network for training to obtain the training water body segmentation prediction result. The loss function result is calculated based on the training water body segmentation prediction result and the water body feature label, and the gradient of the loss function result is back-transferred to the initial water body segmentation network for iterative training to generate a fully trained water body segmentation network.

[0014] In a second aspect, the present application also provides a device for detecting floating objects on a water surface. The device comprises: An acquisition module is used to input the acquired initial water surface image into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image with segmentation results; The calculation module is used to input the image of the area where the floating objects on the water surface are located in the water body image into the preset semantic segmentation large model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating objects on the water surface are located; perform saliency detection on the image of the area where the floating objects on the water surface are located in the water body image to obtain a second segmentation probability result corresponding to the floating objects on the water surface; The generation module is used to fuse the first segmentation probability result with the second segmentation probability result to obtain a detection result corresponding to the floating object on the water surface.

[0015] The above-mentioned method, device and computer equipment for detecting floating objects on the water surface first input the acquired initial water surface image into a well-trained water segmentation network for water segmentation processing to obtain a water image with segmentation results; input the image of the area where the floating objects on the water surface are located in the water image into a large semantic segmentation model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating objects on the water surface are located; perform significance detection on the image of the area where the floating objects on the water surface are located in the water image to obtain a second segmentation probability result corresponding to the floating objects on the water surface; fuse the first segmentation probability result with the second segmentation probability result to obtain a detection result corresponding to the floating objects on the water surface. Through this application, it is possible to combine a supervised water segmentation network with a large semantic segmentation model that does not require additional training, greatly reduce the required training set size, lower the threshold for use, and further, this application obtains the detection result for floating objects on the water surface by calculating the first segmentation probability result and the second segmentation probability result, which effectively improves the accuracy of processing floating objects on the water surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of a flow chart of a method for detecting floating objects on a water surface in one embodiment; Figure 2 A schematic diagram of a method for detecting floating objects on a water surface in a preferred embodiment; Figure 3 A schematic diagram of a flow chart of abnormal segmentation processing in one embodiment; Figure 4 is an original image captured by a drone in one embodiment; Figure 5 is a water body image processed by water body segmentation in one embodiment; Figure 6 is a detection result obtained after abnormal segmentation processing in one embodiment; Figure 7 A mask image of the area where floating objects on the water surface are located in one embodiment; Figure 8 is a structural block diagram of a device for detecting floating objects on a water surface in one embodiment; Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In one embodiment, Figure 1 As shown, a method for detecting floating objects on a water surface is provided, comprising the following steps: Step S110, inputting the acquired initial water surface image into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image carrying the segmentation result.

[0019] Specifically, the above-mentioned initial water surface image is an image obtained by photographing a water surface such as a river channel or a lake surface, and there may be one or more initial water surface images. The present application does not impose too many restrictions on the method for obtaining the initial water surface image. For example, when only one initial water surface image is obtained, the initial water surface image is input into the above-mentioned water body segmentation network for water body segmentation processing to obtain a segmentation result for the water body area, and the segmentation result is restored to the initial water surface image to obtain the above-mentioned water body image. Similarly, a drone may be used to photograph the river channel to obtain an initial image. Furthermore, in actual applications, considering that the high-resolution image memory taken will take up more space, the initial image taken may be cropped to obtain multiple initial water surface images of smaller area. Furthermore, the above-mentioned initial water surface image may be cropped. The image is input into a well-trained water segmentation network for water segmentation processing, wherein the water segmentation network includes but is not limited to a ViT-Adapter network, and the water segmentation network preferably adopts a fully supervised network. Based on the above water segmentation network, the semantic segmentation of the water area can be completed, and each initial water surface image corresponds to a water segmentation result. The above water segmentation result is in the form of a probability matrix, and different pixel points correspond to their own probability values, such as the water area generally corresponds to 1, and the non-water area corresponds to 0. It can be understood that in actual applications, the probability result of the area where the water area and the non-water area intersect is generally a decimal between 0 and 1. In some preferred embodiments, when the initial water surface image is a plurality of images with smaller areas cropped from the initial image taken by the drone, the segmentation result is first restored to the corresponding initial water surface image, and then all the initial water surface images are spliced ​​according to the cropping rules to obtain the above water image. It can be understood that the segmentation result at this time includes the water segmentation results corresponding to the plurality of initial water surface images.

[0020] Step S120, inputting the image of the area where the floating objects on the water surface are located in the water body image into a preset semantic segmentation model for detection processing, and obtaining a first segmentation probability result corresponding to the area where the floating objects on the water surface are located.

[0021] Specifically, a local image of the area where the floating objects are located in the water body image is obtained, that is, in order to reduce the amount of calculation and improve the calculation efficiency in this embodiment, the area where the floating objects are located in the water body image is first detected and segmented, and the local image of the area where the floating objects are located is segmented from the water body image. The segmentation method includes but is not limited to manually calibrating the above local image, or completing the recognition and segmentation of the local image through a preset algorithm. The above semantic segmentation large model is a large model that does not require additional training, such as SAM (Segment Anything Model). The image of the area where the floating objects are located is detected and segmented by the above semantic segmentation large model, and the first segmentation probability result corresponding to the area where the floating objects are located is output, wherein, in some preferred embodiments, the above first segmentation probability result is in the form of a probability matrix, and different pixel points correspond to corresponding probability values, and the probability value represents the classification result of the corresponding position of the pixel point, such as the probability that the point is a floating object on the water surface, and / or the probability that the point is a water body, etc.

[0022] Step S130, performing saliency detection on the image of the area where the floating objects on the water surface are located in the water body image, and obtaining a second segmentation probability result corresponding to the floating objects on the water surface.

[0023] Specifically, the image of the area where the floating objects are located is subjected to saliency detection, with a focus on the area where the floating objects are located in the image. In some embodiments, the HC algorithm (Histogram-based Contrast) or the RC algorithm (Region-based Contrast) can be used to complete the saliency detection. By performing saliency detection on the image of the area where the floating objects are located, the area where the floating objects are located is distinguished, and the corresponding second segmentation probability result is obtained. Similarly, in some preferred embodiments, the second segmentation probability result is in the form of a probability matrix, and different pixel points thereof correspond to corresponding probability values, which characterize the classification result of the corresponding position of the pixel point.

[0024] In summary, in some embodiments, it is necessary to detect the image of the same area where the floating object is located twice based on two different detection methods, and obtain the first segmentation probability result and the second segmentation probability result accordingly. Although the above two detection probability results are the detection results of the same target, the probability values ​​corresponding to the same pixel points are usually not exactly the same. In some embodiments, the number of detection times and algorithms of the above-mentioned images of the area where the floating object is located are not limited, and the images of the area where the floating object is located can be detected multiple times by different detection algorithms, that is, the above-mentioned semantic segmentation large model can include a variety of large models for detecting floating objects on the water surface, such as PerSAM model, CaR (CLIP as RNN) model, etc., and accordingly, the above-mentioned first segmentation probability result also includes a variety of probability results, each probability result corresponds to each of the above-mentioned semantic segmentation large models. Similarly, the above-mentioned saliency detection algorithm can include a variety of algorithms for detecting the features of floating objects on the water surface, such as HC algorithm, RC algorithm, etc., and accordingly, the above-mentioned second segmentation probability result also includes a variety of probability results, each probability result corresponds to each of the above-mentioned saliency detection algorithms.

[0025] Step S140, fusing the first segmentation probability result and the second segmentation probability result to obtain a detection result corresponding to the floating object on the water surface.

[0026] Specifically, after obtaining all the first segmentation probability results and the second segmentation probability results for the same floating object on the water surface, the first segmentation probability result and the second segmentation probability result are fused, wherein, in some preferred embodiments, the above-mentioned fusion process usually refers to multiplying the probability values ​​corresponding to the same pixel points in the first segmentation probability result and the second segmentation probability result. After the first segmentation probability result and the second segmentation probability result are fused, a unique detection result can be obtained for a floating object on the water surface, and the detection result characterizes the probability that the position of each pixel point in the image is a floating object on the water surface. In some preferred embodiments, after obtaining the detection result in the form of a probability matrix, the probability value in the detection result can be filtered according to a preset probability threshold. If the probability value of detecting a floating object on the water surface is greater than 0.8, the position corresponding to the pixel point is judged as a floating object area on the water surface, so that the final detection result for the floating object on the water surface can be obtained after filtering.

[0027] Through steps S110 to S140, the initial water surface image is input into a well-trained water body segmentation network for water body segmentation processing to obtain the corresponding water body image, and then the image of the area where the floating objects on the water surface are located in the water body image is input into a large semantic segmentation model that does not require additional training for detection processing to obtain a first segmentation probability result, and a significance detection is performed on the image of the area where the floating objects on the water surface are located to obtain a second segmentation probability result, that is, in this application, a supervised semantic segmentation algorithm is combined with a zero-sample detection algorithm for floating objects on the water surface, which reduces the computational cost while ensuring the accuracy of detection, and has lower requirements for the effective pixels of floating objects on the water surface. Furthermore, in this application, the first segmentation probability result and the second segmentation probability result are combined to obtain a detection result for floating objects on the water surface, which effectively improves the accuracy of detecting floating objects on the water surface.

[0028] In some embodiments, obtaining an initial water surface image includes: Get the initial image; The initial image is cropped according to a preset cropping size to obtain at least one initial water surface image.

[0029] Specifically, first obtain an initial image, which is an image that needs to be detected and is captured by a high-resolution camera. In some preferred embodiments, high-resolution drone images can be used. After obtaining the initial image, the initial image is cropped according to a preset cropping size to obtain multiple smaller initial water surface images. In this embodiment, the high-resolution image is cropped in a continuous or overlapping manner according to the cropping method, and the cropping size can be set by relevant technical personnel according to actual needs. Through this embodiment, the acquired initial image can be cropped, avoiding the problem of high memory space occupied by the original high-resolution initial image, thereby greatly improving the computing efficiency in subsequent processing.

[0030] In some embodiments, the initial image is cropped according to a preset cropping size, including: Obtaining a preset overlapping region clipping rule, wherein the overlapping region clipping rule represents the size of the overlapping region between two initial water surface images at adjacent positions; The initial image is cropped according to a preset cropping size and an overlapping area cropping rule to obtain an initial water surface image, wherein the size of the overlapping area is greater than 0.

[0031] Specifically, in this embodiment, the initial image is cropped in an overlapping manner. First, a preset overlapping area cropping rule is obtained, which indicates the size of the overlapping area between two adjacent images obtained by cropping when the initial image is cropped. The size of the above overlapping area cannot be equal to 0. In some preferred embodiments, the size of the overlapping area is mostly set to 50% of the size of the initial water surface image obtained by cropping. According to the preset cropping size and the above overlapping area cropping rule, the cropping process of the initial image is completed to obtain the initial water surface image, wherein the size of the above initial water surface image is preferably set to 512×512. This embodiment provides an overlapping cropping method for the initial image, which is beneficial to improving the accuracy of identifying floating objects on the water surface in the subsequent image stitching.

[0032] In some embodiments, each initial water surface image corresponds to a clipping index; the acquired initial water surface image is input into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image carrying a segmentation result, including: The initial water surface image is input into the water body segmentation network for water body segmentation processing to obtain the water body segmentation result, wherein the water body segmentation result is in the form of a probability matrix; The water body segmentation result is restored to the initial water surface image to obtain a water body sub-image carrying the water body segmentation result; The water body sub-images are spliced ​​based on the clipping index, wherein the probability matrix of the water body area included in each water body sub-image is determined according to the water body segmentation result; When the area where the water body segmentation result is located is located in the overlapping area of ​​at least two water body sub-images, the water body segmentation results in the water body sub-images are fused to calculate a water body image carrying the segmentation result.

[0033] Specifically, the initial water surface images are input into the water body segmentation network for water body segmentation processing, and the water body segmentation result corresponding to each initial water surface image is obtained, and the water body segmentation result is in the form of a probability matrix of the area where the water body is located. Then the water body segmentation result is restored to the initial water surface image to obtain a water body sub-image, which corresponds to the initial water surface image, so the clipping index of the water body sub-image is the same as the clipping index of the initial water surface image, and the water body sub-image carries the corresponding water body segmentation result. It can be understood that the water body segmentation result is a probability matrix for the area where the water body is located in the water body sub-image.

[0034] It can be understood that the above water body segmentation result is in the form of a probability matrix, and each pixel point has a corresponding probability value. Restoring the water body segmentation result to the initial water surface image is to multiply the probability matrix with the initial water surface image pixel point and the corresponding pixel point.

[0035] Then, each water body sub-image is spliced ​​based on the clipping index. When the initial image is clipped, adjacent initial water surface images or adjacent water body sub-images have overlapping areas of a certain size, and the same water body area may be located in multiple water body sub-images at the same time, and the segmentation results obtained by water body segmentation on different initial water surface images may be slightly different. Therefore, when the area where the water body segmentation result is located is located in the overlapping area of ​​at least two water body sub-images, the water body segmentation results for the same water body area in each water body sub-image are fused. The fusion is generally to multiply the corresponding water body segmentation results, that is, the probability matrix point-to-point. In summary, after splicing and fusing each water body sub-image, a water body image carrying the segmentation result is obtained. The water body image includes multiple water body sub-images, and the segmentation result includes the water body segmentation results carried in multiple water body sub-images. In this embodiment, the acquired initial image is cropped into multiple initial water surface images with overlapping areas, which can effectively avoid the problem of high memory space occupation of the original high-resolution initial image. During stitching, the water body segmentation results for the same water body area in adjacent water body sub-images are fused, which can effectively improve the accuracy of water body segmentation and also prevent jagged edges at the graphic boundaries, thereby improving the image quality.

[0036] In some of the embodiments, the image of the area where the floating objects are located in the water body image is input into a preset semantic segmentation model for detection processing, including: The water body image is cropped based on the preset cropping size and overlapping area cropping rules, and the cropped water body image and the corresponding water body detection instruction template are input into the well-trained target detection model to obtain the target frame corresponding to all floating objects on the water surface; Based on the target frame, the image of the area where the floating objects are located is extracted from the cropped water image, and the image of the area where the floating objects are located is input into the semantic segmentation model for detection and processing.

[0037] Specifically in this embodiment, the water body image is obtained by splicing the initial water surface image carrying the water body segmentation result, that is, the difference between the water body image and the initial image is that the water body image carries the segmentation result for the water body area. In this embodiment, when obtaining the target frame corresponding to the floating objects on the water surface, an unsupervised target detection large model can be used to complete it. The target detection large model preferably uses the GroundDINO (Grounding DINO) model. The water body image and the corresponding water body detection instruction template are input into the above-mentioned target detection large model to obtain the target frame containing the floating objects on the water surface, wherein the water body detection instruction template is generally a prompt template containing prompt words, and the template is generally used to guide the model to generate a text or instruction of a specific type, theme or format. By adding keywords, phrases or specific format requirements to the template, the above-mentioned target detection large model can be guided to generate results that meet specific requirements. For example, in the present application, the keywords / prompt words can be set to the floating objects on the water surface that need to be extracted, including but not limited to floats, floating, anomly and other words. Further, the target frame obtained for the floating objects on the water surface can be summarized according to a pre-set threshold. In some preferred embodiments, the threshold can be set to 0.1. In summary, based on the water body detection instruction module and the well-trained target detection large model, a target frame corresponding to the above-mentioned floating objects on the water surface can be obtained.

[0038] After obtaining the target frame corresponding to the required floating object on the water surface, a local image of the area where the floating object on the water surface is located can be extracted from the water body image based on the target frame, and the local image of the area where the floating object on the water surface is located can be input into the above-mentioned semantic segmentation large model for accurate segmentation of the floating object on the water surface. In practical applications, a water body image may include multiple floating objects on the water surface, or a large area of ​​water body image may include only one or more small-volume floating objects on the water surface. Through this embodiment, the local area image where the floating object on the water surface is located in the water body image can be extracted, so that in the subsequent semantic segmentation, there is no need to detect and segment the entire image, and only the extracted small local image needs to be detected, thereby greatly improving the efficiency of image processing and reducing the waste of computing resources.

[0039] In some embodiments, the method further comprises: Extracting an image of the area where the floating objects are located from the water body image, and classifying the image of the area where the floating objects are located to obtain a classification corresponding to the floating objects on the water surface; If the category of the floating object on the water surface belongs to the preset anomaly detection category, the image of the area where the floating object on the water surface is located corresponding to the floating object on the water surface is input into the semantic segmentation model for detection processing.

[0040] Specifically, after completing the extraction of the local image of the area where the floating objects on the water surface are located, the local image of the area where the floating objects on the water surface are located is first classified and processed to obtain the classification results corresponding to the floating objects on the water surface. In practical applications, it is generally necessary to perform subsequent detection work on some floating objects on the water surface to facilitate the relevant personnel to carry out cleaning work, such as garbage floating on the water surface, etc.; but for other types of floating objects on the water surface, such as aquatic plants floating on the water surface, ships traveling on the water surface, etc., there is no need to perform subsequent cleaning work or subsequent detection tasks. Therefore, in this embodiment, the image of the area where the floating objects on the water surface are located can be classified and processed. If the category of the floating objects on the water surface is detected to belong to the preset abnormal detection category, such as surface garbage, the image of the area where the floating objects on the water surface are located is input into the semantic segmentation large model for subsequent detection and processing. If the category of the floating objects on the water surface is detected to not belong to the preset abnormal detection category, there is no need to perform any subsequent processing on it. Through this embodiment, the area where floating objects on the water surface belong to the abnormal detection category is located can be extracted in a targeted manner, and the subsequent required detection steps can be carried out without the need to perform complete detection steps on all detected floating objects on the water surface. This ensures the detection effect while greatly improving the efficiency of image processing.

[0041] In some embodiments, the method further comprises: The detection results are restored to the image of the area where the floating objects on the water surface are located in the water body image, and the cropped water body image carrying the detection results is stitched based on the cropping index, cropping size and overlapping area cropping rules; When the area where the detection result is located is located in at least two cropped water body images, the probability matrix of the cropped water body images for the detection result is fused to calculate a water body image carrying a final probability matrix.

[0042] Specifically, after obtaining the detection results for all floating objects on the water surface, the segmented and cropped images need to be spliced, so that the position relationship of the floating objects on the water surface and the relative position relationship between all floating objects on the water surface can be better and more intuitively obtained.

[0043] When performing stitching, it is necessary to correspond to the detection process for floating objects on the water surface mentioned above, and first restore the final detection result for floating objects on the water surface to the corresponding position of the image of the area where the floating objects are located. The detection result carries the probability matrix for floating objects on the water surface. Then, the restored cropped water body image carrying the detection result is stitched according to the cropping index, cropping size and overlapping area cropping rules. It can be understood that the cropped water body image has the same cropping index as the water body sub-image at the corresponding position, and is also the same as the cropping index of the initial water surface image at the corresponding position, wherein the cropping index is, when the initial image is cropped to obtain multiple initial water surface images, the corresponding cropping index is set according to the position of each initial water surface image in the initial image, and each cropping index includes the position information of the corresponding initial water surface image.

[0044] When stitching the cropped water body images based on the cropping index, since the initial image is cropped based on the preset overlapping area cropping rules, when stitching, if the area where the detection result is located is in the overlapping area of ​​at least two cropped water body images, the detection results carried by the cropped water body images need to be fused. In practical applications, the detection results for the same floating object on the water surface carried in different cropped water body images are usually not exactly the same. Therefore, when stitching, the detection results for the same floating object on the water surface in the cropped water body images need to be fused to obtain a more accurate final detection result for the floating object on the water surface. In practical applications, it can also prevent jagged edges at the graphic boundaries and improve the image quality.

[0045] In some embodiments, a saliency detection is performed on an image of a region where floating objects on the water surface are located in a water body image to obtain a second segmentation probability result corresponding to the floating objects on the water surface, including: Perform saliency detection on the image of the area where the floating objects on the water surface are located in the water body image, and obtain the feature value corresponding to each pixel point in the image of the area where the floating objects on the water surface are located; The similarity between the feature value corresponding to each pixel and the feature value corresponding to the adjacent pixel is calculated, and based on the target similarity of the similarities that is less than a preset similarity threshold, a second segmentation probability result is determined.

[0046] Specifically, the image of the area where the floating objects are located is subjected to saliency detection. In practical applications, the saliency detection method includes but is not limited to using a saliency segmentation algorithm, such as the HC algorithm and the RC algorithm, or using a saliency segmentation model, such as the ITTI model and the SGAN model (Saliency Generative Adversarial In some preferred embodiments, a pre-trained convolutional neural network can also be used to extract image features to obtain a feature value or feature vector corresponding to each pixel. After the feature value corresponding to each pixel is detected, the similarity between the feature value corresponding to each pixel and the feature value corresponding to the adjacent pixel is calculated. If the similarity between the feature value corresponding to a certain pixel and the feature value corresponding to the adjacent pixel is too low, it indicates that the point is likely to be the area where the floating object is located. Similarly, if the similarity between the feature value corresponding to a certain pixel and the feature value corresponding to the adjacent pixel is high, it indicates that the point and its surrounding adjacent pixels are likely not to belong to the area where the floating object is located. The calculation of the similarity between the pixels can be achieved by calculating the cosine similarity distance between the feature points. In summary, all pixels are traversed, and the saliency map of the image of the area where the floating object is located is screened to obtain a second segmentation probability result. The second segmentation probability result is preferably in the form of a probability matrix. In the probability matrix of the second segmentation probability result, different probability values ​​represent the probability of the corresponding pixel belonging to the floating object. Through this embodiment, a saliency map of the image of the area where floating objects on the water surface are located can be calculated, and the calculated second segmentation probability result is merged with the first segmentation probability result calculated above to obtain more accurate detection and segmentation results.

[0047] In some embodiments, obtaining a water body segmentation network includes: Obtain a preset water body image training set, wherein the water body image training set carries a water body feature label; The water body image training set is input into the preset initial water body segmentation network for training to obtain the training water body segmentation prediction result. The loss function result is calculated based on the training water body segmentation prediction result and the water body feature label, and the gradient of the loss function result is back-transferred to the initial water body segmentation network for iterative training to generate a fully trained water body segmentation network.

[0048] Specifically, the water body segmentation network can be any neural network based on deep learning, and in actual use, a suitable network can be selected according to the performance of the device. In some preferred embodiments, a ViT-Adapter network can be used.

[0049] The present application also provides a preferred embodiment of a method for detecting floating objects on a water surface. Figure 2 The present invention is a method for detecting floating objects on the water surface in a preferred embodiment.

[0050] Step S210, obtaining an initial image, and cropping the initial image according to a preset cropping size and the overlapping area cropping rule, to obtain a plurality of initial water surface images. The initial image is preferably an image taken by a high-resolution drone, and the initial image is cropped according to a preset cropping size and overlapping area cropping rule. The cropping size can preferably be set to a target length and width of 512, and the overlapping area cropping rule can be set to preferably crop the initial image with an overlapping area of ​​50%.

[0051] Step S220, perform water body segmentation processing on all initial water surface images respectively to obtain water body segmentation results, and restore the water body segmentation results to the initial water surface image to obtain a water body sub-image, and splice each water body sub-image to obtain a water body image carrying the segmentation results. Among them, the water body segmentation processing can be completed by a water body segmentation network, and the water body segmentation network can be based on any deep learning neural network. In this preferred embodiment, ViT-Adapter is used as an example to illustrate that the network framework is mainly divided into two parts. The first part is the classified Backbone network BEIT, which consists of a patch embedding module and 4 Transformer Blocks; the second part is the Adapter module, which is mainly used for the information interaction between the Adapter branch and the feature map of the ViT network. The module includes a spatial feature extraction module for extracting spatial features from the input initial water surface image, a spatial feature injection module for inputting spatial features into ViT, and a multi-scale feature extraction module for extracting multi-scale spatial information from ViT. In practical applications, the input initial water surface image is first cropped into 16×16 non-overlapping blocks through a Patch Embedding module. Then, these blocks will be flattened into D-dimensional encoding, at which time the feature layer resolution will become 1 / 16 of the original image, and the feature channel will be 16×3. Then, these encoded feature blocks plus the position encoding are sent to the Transformer Block of ViT. In another branch, the input image will be input into the spatial feature extraction module to generate three spatial feature layers with different resolutions of 1 / 8, 1 / 16, and 1 / 32. This feature map is flattened and spliced ​​together as the input of the spatial feature injection module. The ViT network used in this study includes 4 TransformerBlocks, each of which contains L / N Transformer layers. Before each Transformer Block, the spatial feature injection module is first used to inject the spatial feature layer obtained before the previous Block into the ViT network; after the TransformerBlock, the multi-scale feature extraction module is used to extract multi-level spatial features from the output feature layer of the Block. After four feature interactions, a high-quality multi-scale feature layer is obtained, which is then separated to generate three feature layers: 1 / 8, 1 / 16, and 1 / 32. Finally, a 2×2 deconvolution layer is used to upsample the 1 / 8 feature layer to a 1 / 4 feature layer. Finally, the four feature layers obtained above are input into the Decoder network for segmentation prediction.

[0052] Step S230, checking whether all initial water surface images have been traversed, if so, jumping to step S240, if not, jumping to step S220.

[0053] Step S240, the water body image is cropped again according to the above cropping index, and the cropped water body image and the water body detection instruction template are input into the well-trained target detection large model to obtain the target frame corresponding to the floating object on the water surface, and the image of the area where the floating object on the water surface is located is extracted based on the target frame. Among them, firstly, the cropped water body image and the corresponding water body detection instruction template are input into the target detection large model to obtain the target frame of the floating object, wherein the water body detection instruction template includes prompt words, which are the floating objects to be extracted, including but not limited to floats, floating, anomly and other words, and multiple prompt words are input in sequence, and the obtained target frame is summarized according to a preset threshold value (such as 0.1).

[0054] Step S250, performing abnormal segmentation processing on the images of all the areas where floating objects on the water surface are extracted, and fusing the first segmentation probability result and the second segmentation probability result to obtain a detection result, wherein the abnormal segmentation processing includes: inputting the image of the area where floating objects on the water surface are located into a large semantic segmentation model (such as SAM) for segmentation processing to obtain a first segmentation probability result corresponding to the area where the floating objects on the water surface are located; inputting the image of the area where floating objects on the water surface are located into a significant segmentation module for significant segmentation to obtain a second segmentation probability result; multiplying the first segmentation probability result and the second segmentation probability result to obtain a final detection result, and filtering the probability value in the detection result according to a preset probability threshold to obtain a mask map of the area where floating objects on the water surface are located. Figure 3 The figure is a flowchart of abnormal segmentation processing in one embodiment.

[0055] Step S260, whether to traverse the images of the areas where all floating objects on the water surface are located, if so, jump to step S270, if not, jump to step S250.

[0056] Step S270, restore the detection result to the image of the area where the corresponding floating object is located, and corresponding to the above-mentioned cropping size and overlapping area cropping rules, splice the cropped water body image carrying the detection result to obtain the final detection result.

[0057] In summary, Figure 4 This is the original image taken by the drone. Figure 5 For water body images that have been segmented, Figure 6 is the detection result obtained after abnormal segmentation processing. Figure 7 It is a mask map of the area where floating objects are located on the water surface.

[0058] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0059] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting floating objects on the water surface for realizing the above-mentioned method for detecting floating objects on the water surface. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of the embodiments of one or more devices for detecting floating objects on the water surface provided below can refer to the limitations of the method for detecting floating objects on the water surface above, and will not be repeated here.

[0060] In one embodiment, Figure 8 As shown, a device for detecting floating objects on a water surface is provided, comprising: an acquisition module 81, a calculation module 82 and a generation module 83, wherein: An acquisition module 81 is used to input the acquired initial water surface image into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image carrying a water body segmentation result; The calculation module 82 is used to input the image of the area where the floating objects on the water surface are located in the water body image into the preset semantic segmentation large model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating objects on the water surface are located; perform saliency detection on the image of the area where the floating objects on the water surface are located in the water body image to obtain a second segmentation probability result corresponding to the floating objects on the water surface; The generating module 83 is used to fuse the first segmentation probability result with the second segmentation probability result to obtain a detection result corresponding to the floating object on the water surface.

[0061] Each module in the above-mentioned device for detecting floating objects on the water surface can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0062] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to the detection of floating objects on the water surface. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for detecting floating objects on the water surface is implemented.

[0063] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0066] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting floating objects on a water surface, characterized in that: The method comprises: The obtained initial water surface image is input into the well-trained water body segmentation network for water body segmentation processing to obtain a water body image with segmentation results; Inputting the image of the area where the floating object is located in the water body image into a preset semantic segmentation model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating object is located; performing saliency detection on an image of a region where the floating object on the water surface is located in the water body image to obtain a second segmentation probability result corresponding to the floating object on the water surface; The first segmentation probability result and the second segmentation probability result are fused to obtain a detection result corresponding to the water surface floating object.

2. The method according to claim 1, characterized in that Acquiring the initial water surface image includes: Get the initial image; The initial image is cropped according to a preset cropping size to obtain at least one initial water surface image.

3. The method according to claim 2, characterized in that The step of cropping the initial image according to a preset cropping size includes: Obtaining a preset overlapping region clipping rule, wherein the overlapping region clipping rule represents the size of the overlapping region between two adjacent initial water surface images; The initial image is cropped according to a preset cropping size and the overlapping area cropping rule to obtain the initial water surface image, wherein the size of the overlapping area is greater than 0.

4. The method according to claim 3, characterized in that Each of the initial water surface images corresponds to a clipping index; the initial water surface image is input into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image with a segmentation result, including: Inputting the initial water surface image into the water body segmentation network for water body segmentation processing to obtain a water body segmentation result, wherein the water body segmentation result is in the form of a probability matrix; Restoring the water body segmentation result to the initial water surface image to obtain a water body sub-image carrying the water body segmentation result; splicing the water body sub-images based on the clipping index, wherein a probability matrix of the water body region included in each water body sub-image is determined according to the water body segmentation result; When the area where the water body segmentation result is located is located in the overlapping area of ​​at least two water body sub-images, the water body segmentation results in the water body sub-images are fused to calculate the water body image carrying the segmentation result.

5. The method according to claim 1, wherein The step of inputting the image of the area where the floating object is located in the water body image into a preset semantic segmentation model for detection processing includes: The water body image is cropped based on a preset cropping size and overlapping area cropping rules, and the cropped water body image and the corresponding water body detection instruction template are input into a well-trained target detection model to obtain target frames corresponding to all the floating objects on the water surface; Based on the target frame, an image of the area where the floating objects on the water surface are located is extracted from the cropped water body image, and the image of the area where the floating objects on the water surface are located is input into the semantic segmentation model for detection processing.

6. The method according to claim 5, characterized in that The method further comprises: cropping the water body image based on a preset cropping size, inputting the cropped water body image and the corresponding water body detection instruction template into a well-trained target detection model, and obtaining target frames corresponding to all the floating objects on the water surface. Extracting an image of the area where the floating object is located from the water body image, and classifying the image of the area where the floating object is located to obtain a classification corresponding to the floating object; If the category of the floating object on the water surface belongs to the preset abnormality detection category, the image of the area where the floating object on the water surface is located corresponding to the floating object on the water surface is input into the semantic segmentation model for detection processing.

7. The method according to claim 5, characterized in that After obtaining the detection result corresponding to the floating object on the water surface, the method further includes: Restoring the detection result to the image of the area where the floating object is located in the water body image, and splicing the cropped water body image carrying the detection result based on the cropping index, cropping size and overlapping area cropping rules; When the area where the detection result is located is located in at least two of the cropped water body images, the probability matrices of the cropped water body images for the detection results are fused to calculate a water body image carrying a final probability matrix.

8. The method according to claim 1, characterized in that The performing saliency detection on the image of the area where the floating object is located in the water body image to obtain a second segmentation probability result corresponding to the floating object includes: Performing saliency detection on an image of an area where the floating object is located in the water body image to obtain a feature value corresponding to each pixel in the image of the area where the floating object is located; The similarity between the feature value corresponding to each pixel point and the feature value corresponding to the adjacent pixel point is calculated, and the second segmentation probability result is determined based on a target similarity among the similarities that is less than a preset similarity threshold.

9. The method according to claim 1, characterized in that The obtaining of the water body segmentation network includes: Obtaining a preset water body image training set, wherein the water body image training set carries a water body feature label; The water body image training set is input into the preset initial water body segmentation network for training to obtain the training water body segmentation prediction result, the loss function result is calculated based on the training water body segmentation prediction result and the water body feature label, and the gradient of the loss function result is reversely transmitted to the initial water body segmentation network for iterative training to generate the fully trained water body segmentation network.

10. A device for detecting floating objects on a water surface, characterized in that: The device comprises: An acquisition module is used to input the acquired initial water surface image into a well-trained water body segmentation network for water body segmentation processing to obtain a water body image with segmentation results; a calculation module, configured to input the image of the area where the floating object is located in the water body image into a preset semantic segmentation large model for detection processing to obtain a first segmentation probability result corresponding to the area where the floating object is located; and perform saliency detection on the image of the area where the floating object is located in the water body image to obtain a second segmentation probability result corresponding to the floating object; A generation module is used to fuse the first segmentation probability result and the second segmentation probability result to obtain a detection result corresponding to the water surface floating object.

Images