A method for water area image segmentation and training method and medium for segmentation model thereof

Abstract

A training method for a water area image segmentation model includes: obtaining a water area sample image; obtaining a label image corresponding to the water area sample image; obtaining the predicted probability that each pixel in the water area sample image belongs to each classification result; obtaining the classification result of each pixel in the label image; obtaining the obstacle distribution weight of the pixel based on the distance between the pixel and its corresponding waterline position point; obtaining the obstacle distribution weighted loss based on the classification result of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image whose classification result belongs to an obstacle, and the predicted probability that each pixel in the water area sample image belongs to each classification result; adjusting the parameters of the image segmentation model based on at least the obstacle distribution weighted loss. Since the correlation between the distribution pattern of obstacles and the waterline is taken into account, the detection accuracy of small obstacles or blurred targets is improved. The present invention also provides a water area image segmentation method and storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for segmenting a water area image and a training method and medium for its segmentation model. Background Art

[0002] Autonomous water transportation is a future trend, and just like unmanned vehicles on land, autonomous driving can be achieved. For example, autonomous water transportation in ocean scenarios can enable autonomous cruising of unmanned vessels. Ocean scene segmentation, one of the most fundamental unmanned sensing technologies, helps unmanned vessels understand the passage area and obstacle information, ensuring safe obstacle avoidance and navigation.

[0003] Current ocean scene segmentation methods usually implement water area detection and obstacle extraction in two steps. First, edge features or probabilistic graph models are used to extract the water area, and then non-water pixels are extracted as obstacles within the water area constraint range. However, such methods are susceptible to environmental interference and cannot detect obstacles above the waterline. In this regard, there are currently some ocean scene segmentation methods based on deep learning, which use a large amount of training data and convolution operations to learn rich features, making pixel-level ocean scene segmentation possible. However, this deep learning-based ocean scene segmentation method has difficulty detecting small obstacles or blurred obstacles in the distance in the ocean scene, which affects the subsequent planning of the unmanned ship for the passage area.

[0004] Therefore, there is still much room for improvement in ocean scene segmentation methods. Summary of the Invention

[0005] The main technical problem solved by the present invention is how to improve the detection accuracy of small obstacles and fuzzy obstacles in the distance when segmenting water area images.

[0006] According to the first aspect, an embodiment provides a method for training a water area image segmentation model, comprising:

[0007] Acquire water sample images;

[0008] Obtaining a label image corresponding to the water sample image, wherein the label image is an image in which classification results are marked for each pixel on the water sample image, and the classification results include at least water and obstacles;

[0009] Inputting the water sample image into an image segmentation model to obtain the predicted probability of each pixel in the water sample image belonging to each classification result;

[0010] Obtaining classification results for each pixel in the label image;

[0011] For any pixel point in the water area of ​​the labeled image that is classified as an obstacle, obtain its corresponding waterline position point, and obtain the obstacle distribution weight of the pixel point based on the distance between the pixel point and the corresponding waterline position point;

[0012] Obtaining an obstacle distribution weighted loss based at least on the classification result of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image classified as an obstacle, and the predicted probability of each pixel in the water area sample image belonging to each classification result;

[0013] Parameters of the image segmentation model are adjusted at least according to the obstacle distribution weighted loss until the image segmentation model converges to obtain a trained image segmentation model.

[0014] In some embodiments, obtaining the corresponding waterline position point for any pixel point in the water area of ​​the label image that is classified as an obstacle includes:

[0015] For any pixel point in the water area of ​​the label image that is classified as an obstacle, obtain a column of pixels where the pixel point is located;

[0016] Pixel points in the column of pixels whose classification results do not belong to water are obtained from top to bottom until a pixel point belonging to water is obtained, and the pixel point belonging to water is used as the waterline position point corresponding to any pixel point, and the corresponding waterline position point is located above any pixel point.

[0017] In some embodiments, obtaining the obstacle distribution weight of any pixel point based on the distance between the pixel point and the corresponding waterline position point includes:

[0018] Obtaining a corresponding probability density function based on the corresponding waterline position point;

[0019] Calculate the value of the probability density function corresponding to any pixel point based on the distance between the pixel point and the corresponding waterline position point;

[0020] The obstacle distribution weight of any pixel point is obtained according to the value of the probability density function.

[0021] In some embodiments, the probability density function obeys a Gaussian distribution. When calculating the value of the probability density function corresponding to any pixel point, the value is obtained by the following formula:

[0022]

[0023]

[0024] Among them, p i (x i ,yi ) is the coordinate of any pixel point in the water area of ​​the label image that is classified as an obstacle, d i For any pixel point p i (x i ,y i ) and its corresponding waterline position point The distance between them, σ is the standard deviation, is the mean, y all is the total number of rows of pixels in the water sample image.

[0025] In some embodiments, the standard deviation σ is:

[0026]

[0027] In some embodiments, the training method of the water area image segmentation model further includes:

[0028] For any pixel point in the non-water area of ​​the label image that is classified as an obstacle, obtaining its corresponding obstacle distribution weight;

[0029] The obstacle distribution weighted loss is obtained based on at least the classification result of each pixel point in the label image, the obstacle distribution weight of each pixel point whose classification result in the water area of ​​the label image is an obstacle, the obstacle distribution weight of each pixel point whose classification result in the non-water area of ​​the label image is an obstacle, and the predicted probability that each pixel point in the water area sample image belongs to each classification result.

[0030] In some embodiments, the image segmentation model includes a convolutional neural network and a context prior layer, and the method further includes:

[0031] Downsampling the water area sample image through the convolutional neural network layer to obtain a depth feature map of the water area sample image;

[0032] Performing context feature extraction on the depth feature map of the water sample image through the context prior layer to obtain a prior feature map of the water sample image;

[0033] obtaining an ideal affinity map according to the label image;

[0034] determining affinity loss based on the prior feature map and the ideal affinity map;

[0035] Parameters of the image segmentation model are adjusted according to the obstacle distribution weighted loss and the affinity loss.

[0036] In some embodiments, the image segmentation model includes a convolutional neural network and a detail head, and the method further includes:

[0037] Downsampling the water area sample image through the convolutional neural network layer to obtain a detail feature map of the water area sample image;

[0038] Performing Laplacian convolution on the label image with strides of 1, 2, 4, and 8 respectively to obtain four soft edge maps, upsampling some of the soft edge maps, and then merging the four soft edge maps to obtain an edge map;

[0039] Extracting details from the detail feature map of the water sample image through the detail head, and obtaining a detail map with the same size as the edge map;

[0040] determining detail loss according to the edge map and the detail map;

[0041] Parameters of the image segmentation model are adjusted according to the obstacle distribution weighted loss and the detail loss.

[0042] According to the second aspect, an embodiment provides a method for segmenting a water area image, comprising:

[0043] Obtaining the water area image to be segmented;

[0044] The water area image to be segmented is segmented using the image segmentation model trained based on the method described in the first aspect to obtain a segmented water area image.

[0045] According to a third aspect, an embodiment provides a computer-readable storage medium, on which a program is stored. The program can be executed by a processor to implement the method as described in the first aspect or the second aspect.

[0046] According to the training method, image segmentation method, and medium for the water area image segmentation model of the above-described embodiments, any pixel in the water area of ​​the label image that is classified as an obstacle is obtained, and an obstacle distribution weight for that pixel is obtained based on the distance between the pixel and its corresponding waterline location. A weighted obstacle distribution loss is then calculated based on the classification results of each pixel in the label image, the obstacle distribution weight for each pixel in the water area of ​​the label image that is classified as an obstacle, and the predicted probability that each pixel in the water area sample image belongs to each classification result. Finally, the parameters of the image segmentation model are adjusted based on the weighted obstacle distribution loss to obtain a trained image segmentation model. Considering the correlation between obstacle distribution patterns and the waterline, namely, the closer to the waterline in the water area, the greater the probability of an obstacle. Locations close to the waterline are generally also far away in the ocean scene, the smaller the distance between any pixel in the water area that is classified as an obstacle and its corresponding waterline location, the greater the obstacle distribution weight for that pixel. This allows for more sensitive obstacle detection for that pixel, thereby improving the detection accuracy of small obstacles or blurred objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a training method for a water area image segmentation model according to an embodiment;

[0048] Figure 2 A flow chart of a training method for a water area image segmentation model according to another embodiment;

[0049] Figure 3 A flowchart of a training method for a water area image segmentation model according to another embodiment;

[0050] Figure 4 A structural block diagram of an image segmentation model according to an embodiment;

[0051] Figure 5 A structural block diagram of a context prior layer according to an embodiment;

[0052] Figure 6 A flowchart for obtaining an ideal affinity map in one embodiment;

[0053] Figure 7 A flowchart of obtaining detail loss in an embodiment. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present application to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted in different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification. This is to avoid the core portion of the present application being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They will fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0055] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0056] The serial numbers assigned to components herein, such as "first," "second," etc., are used solely to distinguish the objects being described and do not convey any sequential or technical meaning. References to "connection" and "coupling" herein, unless otherwise specified, include both direct and indirect connections (couplings).

[0057] Current ocean scene segmentation methods fail to consider the specific characteristics of aquatic environments, such as open waters like oceans, inland rivers, and lakes. In particular, they fail to account for the unique distribution patterns of obstacles like ships on the water surface within aquatic environments, making it difficult to detect small, distant obstacles in ocean scenes. Furthermore, the climate in aquatic environments is more severe, with rain and fog being common. Current ocean scene segmentation methods also struggle to accurately detect distant, blurred obstacles.

[0058] In an embodiment of the present invention, the relationship between the distribution of obstacles and the waterline is taken into account. Specifically, the closer to the waterline in the water area, the greater the likelihood of an obstacle. Locations close to the waterline are typically also far away in the ocean scene. Therefore, for any pixel in the water area of ​​the labeled image that is classified as an obstacle, an obstacle distribution weight is obtained based on the distance between the pixel and its corresponding waterline location. Then, based on the classification results of each pixel in the labeled image, the obstacle distribution weights of each pixel in the water area of ​​the labeled image that is classified as an obstacle, and the predicted probability of each pixel in the water area sample image belonging to each classification result, an obstacle distribution weighted loss is obtained. Finally, the parameters of the image segmentation model are adjusted based on the obstacle distribution weighted loss. Because the obstacle distribution weighted loss assigns a larger obstacle distribution weight to pixels in the water area that are close to the waterline and are classified as obstacles, the detection accuracy of small and blurred obstacles in the distance of the water environment can be improved.

[0059] Some embodiments provide a method for training a water image segmentation model. This method can input sample water images into the image segmentation model, and then adjust the model's parameters based on the label images corresponding to the sample water images to complete the training of the image segmentation model. The sample water images can be of marine or inland waterways. The trained image segmentation model can segment the corresponding water images, thereby distinguishing between water and non-water portions of the water images. This allows for further collision-free water detection in water images, helping unmanned vessels understand traffic areas and obstacle information, ensuring safe obstacle avoidance and navigation.

[0060] Please refer to Figure 1 , the following is a detailed description of the training method of the water image segmentation model:

[0061] Step 100: Acquire a water area sample image.

[0062] Step 200: Obtain a label image corresponding to the water area sample image, where the label image is an image with classification results marked on each pixel point on the water area sample image, and the classification results include at least water and obstacles.

[0063] In some embodiments, a water sample image is used as input into an image segmentation model to train the image segmentation model. The water sample image can be a water image of an ocean environment or a water image of an inland river environment, and generally includes a water portion and a non-water portion.

[0064] In some embodiments, the label image may also be referred to as a true value label or label or ground truth. The label image is obtained by annotating the water sample image with a true value, that is, labeling each pixel in the water sample image with its category. For example, in a water image, all pixels in the water image are labeled as belonging to two categories: water and non-water, while objects appearing in the water surface area, such as water plants, buoys, bridge piers, other ships, etc., are labeled as non-water categories. In some embodiments, the non-water category may include obstacles and sky. For example, objects appearing in the waterline area can be labeled as obstacles, and objects in the non-water surface area that are not obstacles can be labeled as sky.

[0065] In some embodiments, the target sample image can be annotated manually, or the annotated label image can be directly obtained.

[0066] Step 300: Input the water sample image into an image segmentation model to obtain the predicted probability of each pixel in the water sample image belonging to each classification result.

[0067] Step 400: Obtain the classification result of each pixel in the label image.

[0068] In some embodiments, when segmenting the target sample image, in order to improve the accuracy of the classification results of each pixel in the target sample image, the classification results marked for each pixel in the label image, that is, the true value, and the water sample image in the image segmentation model, the predicted probability that the corresponding pixel belongs to each classification result, that is, the predicted probability value, are obtained, and the true value and predicted probability of the pixel classification can be used to calculate the classification loss of the image segmentation model, so that the image segmentation model can accurately classify each pixel in the target sample image.

[0069] Step 500: For any pixel point in the water area of ​​the label image that is classified as an obstacle, obtain its corresponding waterline position point, and obtain the obstacle distribution weight of the pixel point based on the distance between the any pixel point and its corresponding waterline position point.

[0070] In some embodiments, since the distribution of obstacles on the water surface in water images is not completely random, if the weights of pixels in each classification result are set the same when calculating the classification loss, segmentation accuracy will be insufficient for areas with a high probability of obstacle distribution, and small obstacles or blurred targets may be missed. In some embodiments, when studying the distribution patterns of obstacles on the water surface, it is considered that the closer the area is to the waterline, that is, the smaller the distance between the pixel in the area and the corresponding waterline location, the greater the probability of an obstacle. Therefore, for any pixel in the water area of ​​the labeled image classified as an obstacle, the obstacle distribution weight of the pixel is determined based on the distance between the pixel and its corresponding waterline location. In some embodiments, considering the special characteristics of marine or inland waterways, especially the special distribution patterns of obstacles such as ships, it is necessary to increase detection intensity in areas with a high probability of obstacle distribution. Therefore, the smaller the distance between the pixel and its corresponding waterline location, the greater the obstacle distribution weight of the pixel, thereby enabling more sensitive obstacle detection for the pixel and improving the detection accuracy of small obstacles or blurred targets.

[0071] In some embodiments, the classification results for each pixel in the labeled image can be obtained first. Then, based on the pixels classified as water, the waterline in the labeled image can be determined. This means that the boundary of the water area in the water area image is obtained. The pixels in the waterline that are in the same column as any pixel are the corresponding waterline locations. Therefore, when calculating the distance between any pixel and the corresponding waterline location, the straight-line distance between the two can be calculated.

[0072] Step 600: Obtain an obstacle distribution weighted loss based at least on the classification results of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image whose classification result is an obstacle, and the predicted probability that each pixel in the water area sample image belongs to each classification result.

[0073] Step 700: Adjust the parameters of the image segmentation model at least according to the obstacle distribution weighted loss until the image segmentation model converges to obtain a trained image segmentation model.

[0074] In some embodiments, a classification loss function for the image segmentation model can be designed first, and the obstacle distribution weights of each pixel in the water area of ​​the label image that is classified as an obstacle can be assigned to the corresponding pixel in the classification loss function whose classification result is an obstacle. Then, based on the classification results of each pixel in the label image and the predicted probability that each pixel in the water area sample image belongs to each classification result, the obstacle distribution weighted loss of the image segmentation model is obtained. Since the calculation of the loss function has no effect on the inference speed of the image segmentation model, the parameters of the image segmentation model are adjusted at least according to the obstacle distribution weighted loss, and the convergence of the image segmentation model can be determined by the obstacle distribution weighted loss. For example, during the training process, batch training is used, and when the obstacle distribution weighted loss no longer decreases, the image segmentation model can be considered to have converged, thereby obtaining a trained image segmentation model.

[0075] As can be seen from the above embodiment, before calculating the classification loss of the image segmentation model, based on the obstacle distribution pattern on the water surface, any pixel point in the water area of ​​the label image that is classified as an obstacle is obtained, and the obstacle distribution weight of the pixel point is obtained based on the distance between the any pixel point and its corresponding waterline position point. Then, the obstacle distribution weighted loss is calculated based on the classification result of each pixel point in the label image, the obstacle distribution weight of each pixel point in the water area of ​​the label image that is classified as an obstacle, and the predicted probability of each pixel point in the water area sample image belonging to each classification result. Since the smaller the distance between the any pixel point and its corresponding waterline position point, the greater the obstacle distribution weight of the pixel point, more sensitive obstacle detection can be performed on the pixel point, thereby improving the detection accuracy of small obstacles or blurred targets.

[0076] As can be seen from the above embodiment, this embodiment calculates the obstacle distribution weight based on the correlation between obstacles and the waterline. The distance between pixels in the water area classified as obstacles and the waterline location is used to calculate the obstacle distribution weight. Specifically, the closer the pixel is to the waterline, the greater the obstacle distribution weight for its corresponding classification result of "obstacle," thereby ensuring accurate obstacle detection. In some embodiments, this principle also applies to the classification result of "water." In other words, the obstacle distribution weight can also be set as the classification weight for the water classification result. Specifically, the closer the pixel is to the waterline location and classified as "water," the greater the weight for its corresponding classification result of "water," thereby ensuring accurate detection of both obstacles and water. Specifically, for any pixel in the label image classified as "water," its corresponding waterline location is obtained, and the weight of that pixel is calculated based on the distance between that pixel and its corresponding waterline location. The obstacle distribution weighted loss is then calculated based on the classification results of each pixel in the label image, the obstacle distribution weights for each pixel in the water area classified as "obstacle," the weights for each pixel classified as "water," and the predicted probability of each pixel in the water area sample image belonging to each classification result.

[0077] Please refer to Figure 2 In some embodiments, for any pixel point in the water area of ​​the label image that is classified as an obstacle, the corresponding waterline position point is obtained, and the specific method includes:

[0078] Step 510: For any pixel point in the water area of ​​the label image that is classified as an obstacle, obtain a column of pixels where the pixel point is located.

[0079] Step 520: Obtain pixel points in the column of pixels whose classification results do not belong to water from top to bottom until a pixel point belonging to water is obtained, and use the pixel point belonging to water as the waterline position point corresponding to any pixel point, and the corresponding waterline position point is located above any pixel point.

[0080] In some embodiments, any water area image can be considered to be divided into two parts, upper and lower, by a waterline, where the waterline and the area below the waterline are the water area, and the area above the waterline is the non-water area. Therefore, for any pixel in the water area of ​​the label image that is classified as an obstacle, the column of pixels in which it is located is first acquired, and then the pixels in the column that are not classified as water are acquired from top to bottom. Since the acquisition is performed from top to bottom, the first pixels acquired are all non-water. When the acquired pixel is water, it is considered that the pixel included in the waterline between the non-water area and the water area has been acquired. Therefore, the pixel belonging to water is used as the waterline position corresponding to the pixel. In some embodiments, after the water area image is rotated, the acquired column of pixels can be changed to the acquired row of pixels, and the acquired pixel in the column that is not classified as water from top to bottom can be changed to the acquired column from bottom to top, from left to right, or from right to left, and the corresponding waterline position can be located below, to the right, or to the left of the pixel.

[0081] Please refer to Figure 3 In some embodiments, when obtaining the obstacle distribution weight of any pixel point based on the distance between the pixel point and the corresponding waterline position point, the specific method includes:

[0082] Step 530: Obtain a corresponding probability density function based on the corresponding waterline position point.

[0083] Step 540: Calculate the value of the probability density function corresponding to any pixel point based on the distance between the pixel point and its corresponding waterline position point.

[0084] Step 550: Obtain the obstacle distribution weight of any pixel point according to the value of the probability density function.

[0085] In some embodiments, a probability density function corresponding to the waterline position point is first designed. The probability density function represents the probability of an obstacle appearing at each pixel point when an obstacle appears in the pixel point corresponding to the waterline position point. The closer the area is to the waterline, that is, the smaller the distance between the pixel point in the area and the corresponding waterline position point, the greater the possibility of an obstacle appearing there, and therefore the value of the corresponding probability density function is also larger. Finally, the corresponding obstacle distribution weight is obtained based on the value of the probability density function for the calculation of the distribution loss. In some embodiments, the probability density function can be obtained based on data statistics.

[0086] In some embodiments, the probability density function follows a Gaussian distribution and is obtained by the following formula:

[0087]

[0088] Among them, p i (x i ,y i ) is the coordinate of any pixel point in the water area of ​​the label image that is classified as an obstacle, d i For any pixel point p i (x i ,y i ) and its corresponding waterline position point The distance between them, σ is the standard deviation, is the mean.

[0089] In some embodiments, the standard deviation σ is:

[0090]

[0091] Among them, y all is the total number of rows of pixels in the target sample image.

[0092] The probability density function expression above shows that the locations of obstacles follow a Gaussian distribution based on the waterline location. Therefore, the mean of the probability density function is the row coordinate of the corresponding waterline location. The value of the probability density function is based on the waterline location, so the probability density function changes dynamically based on the waterline location.

[0093] In this example, the row coordinates of pixels in both the label image and the water sample image are calculated from top to bottom, with the row coordinates of the topmost pixel in the first row. Therefore, the standard deviation of the probability density function indicates that most obstacles occur approximately 1 / 8 of the way between the waterline and the bottom of the image, corresponding to the standard deviation. The standard deviation is primarily based on the statistical patterns of the collected dataset and can be adjusted adaptively based on the specific application scenario.

[0094] In some embodiments, when calculating the weighted obstacle distribution loss, only the obstacle distribution weights of each pixel in the water area of ​​the labeled image that is classified as an obstacle need to be obtained. Therefore, all pixels in the water area that are classified as obstacles can be obtained first, and then the obstacle distribution weights of each obtained pixel can be calculated for use in the weighted obstacle distribution loss. Alternatively, the probability density function corresponding to each waterline position point can be obtained first, and then the obstacle distribution weight of each pixel can be obtained. Then, only the pixels in the water area that are classified as obstacles can be obtained, and their obstacle distribution weights can be used to calculate the weighted obstacle distribution loss.

[0095] In some embodiments, for any pixel in the non-water area of ​​the label image that is classified as an obstacle, its corresponding obstacle distribution weight is also obtained. Then, based on at least the classification result of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image that is classified as an obstacle, the obstacle distribution weight of each pixel in the non-water area of ​​the label image that is classified as an obstacle, and the predicted probability that each pixel in the water area sample image belongs to each classification result, a weighted obstacle distribution loss is obtained.

[0096] In this embodiment, for the water area and non-water area of ​​the label image, the obstacle distribution weight of any pixel point whose classification result is an obstacle is obtained respectively, wherein the obstacle distribution weight of any pixel point in the water area is related to the distance from the pixel point to the corresponding waterline position point to improve the detection accuracy of small obstacles or blurred targets, and the obstacle distribution weight of any pixel point in the non-water area can be a constant value or a value related to the waterline position point corresponding to the pixel point to improve the detection accuracy of obstacles above the waterline.

[0097] In some embodiments, when calculating any pixel point p i (x i ,y i ) and its corresponding waterline position point The distance d i When , it is calculated by the following formula:

[0098]

[0099] From the above formula, we can know that when any pixel point is above or on the waterline, the distance d between it and the corresponding waterline position point is i are all 0, at which point the corresponding probability density function reaches its maximum value, and when any pixel point is below the waterline, its distance d from the corresponding waterline position point i is the row coordinate difference between the two, and the closer to the waterline position point, the larger the value of the corresponding probability density function, thereby ensuring that obstacles close to and above the waterline are easier to detect.

[0100] In some embodiments, when obtaining the obstacle distribution weighted loss based on the classification result of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image that is classified as an obstacle, and the predicted probability that each pixel in the water area sample image belongs to each classification result, the obstacle distribution weighted loss is calculated using the following formula:

[0101]

[0102] Among them, y iw ,y io and y is are the label values ​​of water, obstacle and sky for the i-th pixel in the label image, respectively, p iw , p io and p is are respectively the predicted probabilities of water, obstacles and sky for the i-th pixel in the water sample image by the image segmentation model, and w io is the obstacle distribution weight.

[0103] In some embodiments, the obstacle distribution weights are:

[0104]

[0105] From the above formula for calculating the weighted loss of obstacle distribution, we can see that when the i-th pixel is an obstacle, y iw , and y is is 0, and y io When the value is 1, the i-th pixel is classified as an obstacle and the classification loss is assigned an obstacle distribution weight. The closer the i-th pixel is to its corresponding waterline position, the greater its obstacle distribution weight is. This allows for more sensitive obstacle detection of the i-th pixel, thereby improving the detection accuracy of small obstacles or blurred targets.

[0106] Please refer to Figure 4 , which illustrates the framework and principles of the image segmentation model, which is described in detail below.

[0107] In some embodiments, the image segmentation model includes a deep convolutional neural network layer, a detail head, and a context prior layer. The deep convolutional neural network layer is used to extract depth features from the input water sample image and generate a detail feature map and a depth feature map. The detail head uses the detail feature map generated by the deep convolutional neural network layer as input and uses detail loss to guide the model to learn the detail features of the classification edge. The context prior layer uses the depth feature map generated by the deep convolutional neural network layer as input and uses affinity loss to guide the model to learn the contextual association between features and generate a contextual association map. The image segmentation model is described in detail below:

[0108] In some embodiments, the framework used in the design of the deep neural network layer can be a lightweight MobileNetv2 model, which can be used to ensure real-time performance. It can be understood that the framework is not limited to the MobileNet v2 model, and other model frameworks for deep feature extraction are also applicable to the method proposed in this embodiment.

[0109] Please refer to Figure 4 In some embodiments, the deep neural network layer proposed in this embodiment includes two-dimensional convolution and bottleneck layer 1 to bottleneck layer 7. Among them, when downsampling is performed on the water sample image, it is arranged at the forefront of the model. For example, in the initial stage of the network, three downsampling operations can be performed continuously, that is, 8-fold downsampling. The purpose of this is to reduce the loss of detailed features caused by more downsampling operations in the subsequent bottleneck layer, and at the same time provide a sufficiently large receptive field for the context prior layer to capture contextual features within and between classes. However, in some embodiments, the sampling rate and the location where downsampling is performed are not limited to this, such as 16-fold or 32-fold downsampling, and the downsampling location can also be set at bottleneck layer 5 or bottleneck layer 7.

[0110] Please refer to Figure 5 In some embodiments, after the context prior layer inputs the depth feature map generated by the deep convolutional neural network layer, it generates a context prior feature map after a set of completely separable convolutions and 1×1 convolutions. According to the feature map after the context prior feature map and the depth feature map are subjected to completely separable convolutions, the intra-class context feature map and the inter-class context feature map are obtained. By cascading the intra-class context feature map, the inter-class context feature map and the depth feature map on the channel, the output of the context prior layer, that is, the context association feature map, is obtained. The context association feature map is the above-mentioned association feature map. The context prior feature map directly reflects the intra-class similarity and inter-class difference of the features, and as the final output of the context prior layer, the context association feature map applies this context association to the original feature map. The context association feature map is divided into the intra-class context association map P intra and inter-class contextual association graph P inter , and the calculation formulas are as follows:

[0111] P intra =PX

[0112] P inter =(1-P)X

[0113] Where P is the contextual prior feature map, and X is the intermediate feature map obtained by fully separable convolution of the original depth feature map in the contextual prior layer, and resized before matrix multiplication. The final contextual feature map can be obtained by channel-wise concatenation of the intra-class contextual feature map and the inter-class contextual feature map.

[0114] Please refer to Figure 6 , the contextual prior feature map is supervised by the ideal affinity map calculated from the label image, and the calculation steps of the ideal affinity map are as follows:

[0115] Step A: Downsample the label image to obtain a binary matrix G, where the size of the binary matrix G is the same as the size of the context prior feature map (h×w), where h and w are height and width respectively.

[0116] Step B: Based on the label image, the classification results of water (0), obstacle (1), and sky (2) are one-hot encoded and concatenated into a matrix M of size n×3, where n=h×w. The three columns in M ​​correspond to the binary labels of water, obstacle, and sky, respectively.

[0117] Step C: Calculate the ideal affinity graph A using the following formula:

[0118] A=MM T

[0119] It can be seen that each pixel a in the ideal affinity map ij ∈A,i∈[1,n],j∈[1,n] all reflect the i-th element g in the binary matrix G i With the jth element g j The relationship is:

[0120]

[0121] From the above, we can see that the contextual prior feature map P supervised by the ideal affinity map must also contain a lot of contextual information about intra-class similarity and inter-class differences. Specifically, the ideal affinity loss used for supervision The calculation formula is as follows:

[0122]

[0123] in, and Represents the unary term and global term in the ideal affinity loss. is the binary cross entropy loss between the contextual prior feature map and the affinity map and is obtained by the following formula:

[0124]

[0125] Among them, p k ∈P,a k ∈A respectively represent the kth (k∈[1,n 2 ]) elements.

[0126] Global Items It represents a set of binary cross entropy losses including intra-class accuracy, intra-class recall, and inter-class recall, and is obtained by the following formula:

[0127]

[0128] where p ij (i∈[1,n],j∈[1,n]),a ij (i∈[1,n],j∈[1,n]) represents the value of the i-th row and j-th column in the context prior feature map and the ideal affinity map, respectively.

[0129] Please refer to Figure 7 In some embodiments, before calculating the detail loss, a Laplacian convolution with different strides is performed on the label image to obtain four soft edge maps, where the Laplacian convolution kernel is:

[0130]

[0131] It can be seen that Laplace convolution can enhance the edge effect, and the strides of Laplace convolution corresponding to the four soft edge maps are 1, 2, 4, and 8 respectively.

[0132] Then, the soft edge maps with Laplace convolution strides of 1, 2, and 4 are upsampled by 8 times, 4 times, and 2 times, respectively. After the sizes of the four soft edge maps are adjusted to be consistent, the images are merged to generate the final edge map. In this embodiment, by using Laplace convolutions with strides of 1, 2, 4, and 8, especially by using Laplace convolutions with a stride of 8, on the one hand, the scale consistency of the detail feature map and the edge map is ensured, and it also helps the model learn edge information with a large receptive field and improves the performance of contour perception. The larger receptive field and the stride corresponding to the depth feature map can ensure that the features learned by the segmentation model are consistent, and the edge map with a larger receptive field is conducive to the model learning complete contour information rather than simple edge details.

[0133] In some embodiments, the edge map is used to supervise the detail map generated by the detail head. The detail head inputs the detail feature map generated by the deep convolutional neural network layer, and then goes through a 3×3 convolution, a batch normalization layer, a ReLU activation layer, and a 1×1 convolution and then upsamples by 8 times to generate a detail map with the same size as the edge map. The calculation is performed by calculating the binary cross entropy loss of the edge map and the detail map, and is obtained by the following formula:

[0134]

[0135] Among them, e k , d kRepresent the corresponding pixel values ​​in the edge image and detail image respectively, k∈[1,m], m is the total number of pixels in the edge image.

[0136] In the above embodiment, the weighted loss of obstacle distribution is calculated respectively Loss of detail and affinity loss Among them, the obstacle distribution weighted loss corresponds to the final segmentation result, the detail loss is used to supervise the detail feature map, and the ideal affinity loss is used to supervise the intermediate output context prior map of the context prior layer. Since the loss function has no effect on the model inference speed, more reasonable loss functions can be used to improve the performance of the image segmentation model. In this embodiment, the loss function of the image segmentation model consists of three parts, namely, the obstacle distribution weighted loss, Loss of detail and affinity loss The final loss function is the sum of the above three loss functions:

[0137]

[0138] Then according to the loss function The parameters of the image segmentation model are adjusted until convergence, resulting in a trained image segmentation model. During the segmentation result prediction phase, the detail feature map, depth feature map, and contextual feature map are concatenated on a channel-by-channel basis, then upsampled using bilinear interpolation and pixel classification predicted. Finally, a segmentation result consistent with the size of the water sample image is generated.

[0139] As can be seen from the above embodiment, compared to other ocean scene segmentation algorithms, this embodiment addresses the unique characteristics of ocean or lake environments and the special challenges of water transportation. Three effective loss functions are designed based on a lightweight model framework. This improves the model's segmentation accuracy while ensuring real-time performance, providing timely and accurate environmental information for unmanned vessels' navigation and obstacle avoidance. The weighted obstacle distribution loss guides obstacle detection, the detail loss optimizes contour details, and the ideal affinity loss learns changing texture features.

[0140] It can be seen from the above embodiments that the obstacle distribution weighted loss dynamically adjusts the obstacle distribution weight of each pixel point with respect to the classification loss according to the distribution law of obstacles in ocean or inland river scenes, thereby improving the detection accuracy of obstacles, especially the detection of small obstacles and blurred obstacles in the distance.

[0141] It can be seen from the above embodiments that the lightweight model designed in this embodiment does not use a decoder for detail optimization, but instead designs a detail loss based on the edge features of the ground truth (labeled value) in the label image, and uses it to guide the model to learn the detail information of each classification boundary, ensuring more accurate detail segmentation while avoiding the additional computational overhead brought by the use of a decoder.

[0142] As can be seen from the above examples, Laplacian convolutions with strides of 1, 2, 4, and 8 are designed for edge map generation in detail loss to avoid scale inconsistencies between the detail feature map and the edge map, which could weaken detail perception performance. The Laplacian convolution with a stride of 8 ensures scale consistency between the detail feature map and the edge map, while also helping the model learn edge information with a large receptive field, improving contour perception performance.

[0143] Some embodiments provide a method for water area image segmentation, comprising:

[0144] Step 1): Get the water area image to be segmented.

[0145] Step 2): The image segmentation model trained based on any of the methods described in the above embodiments is used to segment the water area image to obtain the segmented water area image.

[0146] Some embodiments provide a computer-readable storage medium having a program stored thereon. The program can be executed by a processor to implement the methods described in the above embodiments.

[0147] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.

[0148] The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A training method for a water area image segmentation model, characterized in that: include: Acquire water sample images; Obtaining a label image corresponding to the water sample image, wherein the label image is an image in which classification results are marked for each pixel on the water sample image, and the classification results include at least water and obstacles; Inputting the water sample image into an image segmentation model to obtain the predicted probability of each pixel in the water sample image belonging to each classification result; Obtaining classification results for each pixel in the label image; For any pixel point in the water area of ​​the labeled image that is classified as an obstacle, obtain its corresponding waterline position point, and obtain the obstacle distribution weight of the pixel point based on the distance between the pixel point and the corresponding waterline position point; Obtaining an obstacle distribution weighted loss based on the classification result of each pixel in the label image, the obstacle distribution weight of each pixel in the water area of ​​the label image that is classified as an obstacle, and the predicted probability of each pixel in the water area sample image belonging to each classification result; Adjusting parameters of the image segmentation model according to the obstacle distribution weighted loss until the image segmentation model converges to obtain a trained image segmentation model; Among them, obtaining the obstacle distribution weight of any pixel point based on the distance between the any pixel point and the corresponding waterline position point includes: obtaining the corresponding probability density function based on the corresponding waterline position point; calculating the value of the probability density function corresponding to any pixel point based on the distance between the any pixel point and the corresponding waterline position point; and obtaining the obstacle distribution weight of any pixel point based on the value of the probability density function.

2. The training method for the water area image segmentation model according to claim 1, wherein: For any pixel point in the water area of ​​the label image that is classified as an obstacle, obtaining the corresponding waterline position point includes: For any pixel point in the water area of ​​the label image that is classified as an obstacle, obtain a column of pixels where the pixel point is located; Pixel points in the column of pixels whose classification results do not belong to water are obtained from top to bottom until a pixel point belonging to water is obtained, and the pixel point belonging to water is used as the waterline position point corresponding to any pixel point, and the corresponding waterline position point is located above any pixel point.

3. The training method for the water area image segmentation model according to claim 1, wherein: The probability density function obeys the Gaussian distribution. When calculating the value of the probability density function corresponding to any pixel point, it is obtained by the following formula: ； ； Among them, p i (x i ,y i ) is the coordinate of any pixel point in the water area of ​​the label image that is classified as an obstacle, d i For any pixel point p i (x i ,y i ) and its corresponding waterline position point The distance between them, σ is the standard deviation, is the mean, y all is the total number of rows of pixels in the water sample image.

4. The training method for the water area image segmentation model according to claim 3, wherein: The standard deviation σ is: 。 5. The training method for the water area image segmentation model according to claim 1, wherein: Also includes: For any pixel point in the non-water area of ​​the label image that is classified as an obstacle, obtaining its corresponding obstacle distribution weight; The obstacle distribution weighted loss is obtained based on the classification results of each pixel in the label image, the obstacle distribution weights of each pixel whose classification results in the water area of ​​the label image are obstacles, the obstacle distribution weights of each pixel whose classification results in the non-water area of ​​the label image are obstacles, and the predicted probability of each pixel in the water area sample image belonging to each classification result.

6. The method for training a water area image segmentation model according to any one of claims 1 to 5, wherein: The image segmentation model includes a convolutional neural network layer and a context prior layer, and the method further includes: Downsampling the water area sample image through the convolutional neural network layer to obtain a depth feature map of the water area sample image; Performing context feature extraction on the depth feature map of the water sample image through the context prior layer to obtain a prior feature map of the water sample image; obtaining an ideal affinity map according to the label image; determining affinity loss based on the prior feature map and the ideal affinity map; Parameters of the image segmentation model are adjusted according to the obstacle distribution weighted loss and the affinity loss.

7. The method for training a water area image segmentation model according to any one of claims 1 to 5, wherein: The image segmentation model includes a convolutional neural network layer and a detail head, and the method further includes: Downsampling the water area sample image through the convolutional neural network layer to obtain a detail feature map of the water area sample image; Performing Laplacian convolution on the label image with strides of 1, 2, 4, and 8 respectively to obtain four soft edge maps, upsampling some of the soft edge maps, and then merging the four soft edge maps to obtain an edge map; Extracting details from the detail feature map of the water sample image through the detail head, and obtaining a detail map with the same size as the edge map; determining detail loss according to the edge map and the detail map; Parameters of the image segmentation model are adjusted according to the obstacle distribution weighted loss and the detail loss.

8. A method for water area image segmentation, characterized in that: include: Obtaining the water area image to be segmented; The water area image to be segmented is segmented using an image segmentation model trained based on the method according to any one of claims 1 to 7 to obtain a segmented water area image.

9. A computer-readable storage medium, characterized in that The medium stores a program, which can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images