Semantic segmentation water level identification method and device based on prior knowledge

Abstract

The invention discloses a semantic segmentation water level identification method based on priori knowledge, and the method comprises the steps: learning the overall characteristics of a segmented object through a pre-trained interactive semantic segmentation model, marking a part of materials to train the segmentation model only when the segmentation effect does not reach the standard, and effectively reducing the training materials; on the basis of priori knowledge that water is located below the image and the water level line is horizontal in water level identification, a sampling point guiding model is automatically generated in the water body part of the image to segment the water body part, water level identification confidence is provided, images with low confidence are screened out, the water level identification effect is effectively evaluated, and the water level identification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to a semantic segmentation water level recognition method and device based on prior knowledge, belonging to the field of deep learning semantic segmentation. Background Art

[0002] In recent years, the construction of water conservancy informatization has continued to advance, and water level monitoring plays an important role in ecological flow supervision, flood prevention and early warning, etc. Traditional automated water level monitoring methods, such as ultrasonic type, have certain restrictions on installation conditions. The latest vision-based method first detects the water gauge position through the target detection algorithm, and then obtains the water gauge value through the image processing algorithm, but this method is not suitable for observation sites that are not equipped with water gauges. Some studies also segment the water surface through image segmentation to obtain the relative water level, but often require a lot of training materials. Therefore, it is urgent to propose a new water level recognition solution to effectively solve the above problems. Summary of the invention

[0003] The present invention aims to provide a semantic segmentation water level recognition method based on prior knowledge. The overall characteristics of the object are learned based on an interactive semantic segmentation model. Combined with the fixed and unique orientation of water and land in the water level recognition image, sampling points are automatically generated in the water part of the image to guide the model to segment the water part, which effectively improves the segmentation accuracy and reduces the image training materials.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a semantic segmentation water level recognition method, the method comprising the following steps:

[0005] Training process:

[0006] Step 1), collect training water level images;

[0007] Step 2), in the lower area of ​​the training water level image, generating training sampling points by a random function, and performing interactive segmentation processing on the training sampling points and the training water level image based on the pre-trained weights of the interactive semantic segmentation model to obtain a training mask image;

[0008] Step 3), segmenting and labeling the training water level image to obtain a labeled water level image; using the training sampling points and the training water level image as training input data, and using the labeled water level image as training output data, training the interactive semantic segmentation model to obtain a trained interactive semantic segmentation model;

[0009] Testing process:

[0010] Step 4), collecting test water level images;

[0011] Step 5), in the lower area of ​​the test water level image, a number of test sampling points are generated by a random function, and the test sampling points and the test water level image are input into the trained interactive semantic segmentation model to obtain a corresponding test mask image;

[0012] Step 6), based on the prior knowledge that the water level line is horizontal, the test mask images with a confidence level lower than a second threshold are screened out, and the average water level value of the retained test mask images is used as the final water level recognition value.

[0013] Each sampling point and a water level image are input into the interactive segmentation model to obtain a mask image, and multiple sampling points and a water level image are input into the segmentation model to obtain corresponding multiple mask images.

[0014] An interactive semantic segmentation model refers to a model that performs semantic segmentation on pixels or areas in an image by combining user input (such as clicking, painting, etc.) with a machine learning algorithm. This type of model uses manual interaction to guide the model to perform more accurate segmentation. However, in image water level recognition applications, manual interaction operations are difficult. The present invention provides a semantic segmentation water level recognition method using prior knowledge, which automatically generates sampling points and interacts with the model based on the prior knowledge of the water and shore positions to solve the above problems. Prior knowledge refers to prior knowledge or experience about the problem domain that is already possessed before learning or reasoning. It is knowledge acquired through previous learning, practice or observation, and can be professional knowledge from domain experts, results of previous studies, rules or common sense knowledge, etc. The prior knowledge used by the water level recognition method provided by the present invention is conventional knowledge, that is, the water is below the water level image, the shore is above the water level image, and the water level line is horizontal.

[0015] The interactive segmentation model is usually pre-trained on a large-scale data set and has a certain generalization ability. In practical applications, the interactive segmentation model is loaded into a computing environment, and a pre-trained weight file is loaded. The weight file usually contains parameters obtained by training the model on a large number of data sets, which can help the model converge faster on new data sets and achieve better performance. The water level recognition method provided by the present invention is applicable to all interactive segmentation models.

[0016] According to the embodiments of the present invention, the present invention can be further optimized. The following is the technical solution formed after optimization:

[0017] In one of the preferred embodiments, the step 1) also includes: intercepting the key area of ​​the training water level image, rotating the training water level image so that the water level line is in a horizontal position; the key area is an area where the color difference between the water and the shore is obvious and there is no interference from weeds; the step 4) also includes: intercepting the key area of ​​the test water level image, rotating the test water level image so that the water level line is in a horizontal position; the key area is an area where the color difference between the water and the shore is obvious and there is no interference from weeds.

[0018] In one of the preferred embodiments, in the step 2), the random function is randint, and two integers are randomly generated in the lower area of ​​the training water level image by the randint function, and the two integers are the first coordinates and the second coordinates of the training sampling points; in the step 5), the random function is randint, and two integers are randomly generated in the lower area of ​​the test water level image by the randint function, and the random generation is performed several times to obtain the first coordinates and the second coordinates of several test sampling points.

[0019] In one of the preferred embodiments, before step 3), it also includes: determining whether the MIoU index of the training mask image is less than a first threshold, if it is less than the first threshold, continuing with step 3); if it is not less than the first threshold, the interactive semantic segmentation model in step 2) can be directly used in the test process, and continuing with step 4).

[0020] In one preferred embodiment, in the step 3), the training water level image is segmented and labeled using the Labelme image annotation tool.

[0021] In one preferred embodiment, the interactive semantic segmentation model is RITM_interactive_segment. As an interactive segmentation model, RITM_interactive_segment has shown powerful performance in the field of computer vision, especially in image segmentation tasks.

[0022] In one preferred embodiment, in step 6), the step of screening out the test mask image with a confidence lower than a second threshold value specifically comprises the following steps:

[0023] According to the mask height value of each column of the test mask image, the height value variance of the test mask image is calculated, and the formula is as follows:

[0024]

[0025] Among them, x iis the number of pixels judged as water in the i-th column of the test mask image, that is, the mask height value of the i-th column; 1≤i≤w, w is the column of the test mask image, that is, the width of the test mask image; is the mean mask height value of the test mask image, s 2 is the mask height value variance of the test mask image;

[0026] The confidence level is related to s 2 Correspondingly, s 2 The smaller it is, the greater the confidence is; according to the second threshold, the test mask image whose confidence is lower than the second threshold is screened out.

[0027] In one of the preferred embodiments, the method further comprises the following steps: Step 7), converting the final water level identification value into an absolute water level, ie, an actual water level height, by interpolation.

[0028] Based on the same concept, the present invention also provides a semantic segmentation water level recognition device, which includes a memory and one or more processors; one or more programs are stored on the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention proposes a semantic segmentation water level recognition method based on prior knowledge, which uses a pre-trained interactive segmentation model to learn the overall characteristics of the object. Only when the segmentation effect does not meet the standard, some materials are marked to train the segmentation model, which effectively reduces the training materials; combined with the prior knowledge such as the fixed and unique orientation of water and land in the water level recognition image, sampling points are automatically generated in the water part of the image to guide the model to segment the water part; confidence is added to each water level recognition sampling point, and mask images with low confidence are screened out, so as to effectively evaluate the water level recognition effect and improve the accuracy of water level recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the semantic segmentation water level recognition method according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of water level recognition according to another embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0034] Semantic segmentation technology can assign each pixel in an image to a specific category, thereby achieving fine division of different areas in the image. Different from traditional target detection technology, semantic segmentation requires not only identifying the target object in the image, but also classifying each pixel to achieve pixel-level classification accuracy.

[0035] Water level recognition refers to the process of accurately measuring and identifying the height of the free surface of a water body through various means (such as sensors, radars, image recognition, etc.). In water level recognition, image recognition technology has attracted much attention due to its intuitiveness and non-contact nature. By identifying the water level area in the image, real-time monitoring and early warning of the water height can be achieved.

[0036] The application of semantic segmentation in water level recognition is to use deep learning algorithms (such as convolutional neural networks (CNNs)) to build a semantic segmentation model. The model input is an image containing water bodies. Through model processing, each pixel in the image is divided into "water bodies" or "non-water bodies". Based on the segmentation results, the height of the water body area (i.e., the water level) is calculated.

[0037] Example 1

[0038] The embodiment of the present invention provides a semantic segmentation water level recognition method based on prior knowledge, wherein the prior knowledge refers to that the water is below the water level image, the bank is above the water level image, and the water level line is horizontal; the water level recognition method comprises the following steps:

[0039] Training process:

[0040] Step 1), collecting training water level images; the training water level images are image materials for which water level measurement is required;

[0041] Step 2), in the lower area of ​​the training water level image, generating training sampling points by a random function, and performing interactive segmentation processing on the training sampling points and the training water level image based on the pre-trained weights of the interactive semantic segmentation model to obtain a training mask image;

[0042] The mask image is a binary image or a multi-category image of the same size as the original image, in which the value of each pixel indicates the category to which the point belongs. In the semantic segmentation task, the mask image is used to represent the segmentation results of different regions or objects in the image.

[0043] Step 3), segmenting and labeling the training water level image to obtain a labeled water level image; using the training sampling points and the training water level image as training input data, and using the labeled water level image as training output data, training the interactive semantic segmentation model to obtain a trained interactive semantic segmentation model;

[0044] Testing process:

[0045] Step 4), collecting test water level images;

[0046] Step 5), in the lower area of ​​the test water level image, generate a number of test sampling points through a random function, input the test sampling points and the test water level image into the trained interactive semantic segmentation model to obtain the corresponding test mask image.

[0047] Each sampling point and a water level image are input into the interactive segmentation model to obtain a mask image. Multiple sampling points and a water level image are input into the segmentation model to obtain corresponding multiple mask images. A sampling point is generated using a random function at the lower upper position of the image. Specifically, a point is randomly sampled in the area 50 pixels below the bottom of the image and input into the segmentation model to obtain the corresponding mask image. To enhance the robustness of the model, multiple sampling points are generated using a random function at different positions in the lower area of ​​the image, and multiple mask images are obtained through the segmentation model. Specifically, multiple points are randomly sampled in the area 50 pixels below the bottom of the image and input into the segmentation model to obtain multiple mask images.

[0048] In order to obtain a better water level recognition effect, the water level image can also be preprocessed before semantic segmentation. Therefore, further, the step 1) also includes: intercepting the key area of ​​the training water level image, rotating the training water level image so that the water level line is in a horizontal position; the key area is an area where the water bank has obvious color difference and is not disturbed by weeds; the step 4) also includes: intercepting the key area of ​​the test water level image, rotating the test water level image so that the water level line is in a horizontal position; the key area is an area where the water bank has obvious color difference and is not disturbed by weeds.

[0049] The MIoU (Mean Intersection over Union) metric is called the mean intersection over union or mean IoU, and is used to measure the similarity between the model's prediction results and the true labels in image segmentation tasks. The MIoU metric comprehensively considers the overlap and accuracy between the prediction results and the true labels, and can comprehensively evaluate the model performance.

[0050] Furthermore, before step 3), it also includes: determining whether the MIoU index of the training mask image is less than a first threshold, if it is less than the first threshold, continuing step 3); if it is not less than the first threshold, the interactive semantic segmentation model in step 2) can be directly used in the test process, and continuing step 4). In this embodiment, the first threshold is 0.9.

[0051] Furthermore, in the step 2), the random function is randint, and two integers are randomly generated in the lower area of ​​the training water level image by the randint function, and the two integers are the first coordinates and the second coordinates of the training sampling points; in the step 5), the random function is randint, and two integers are randomly generated in the lower area of ​​the test water level image by the randint function, and the random generation is performed several times to obtain the first coordinates and the second coordinates of several test sampling points.

[0052] Furthermore, in step 3), the training water level image is segmented and labeled by using the Labelme image labeling tool, with the water portion labeled as water and the other portions labeled as background.

[0053] Interactive segmentation models are usually pre-trained on large-scale data sets and have certain generalization capabilities. In practical applications, the interactive segmentation model is loaded into a computing environment, and a pre-trained weight file is loaded. The weight file usually contains parameters obtained by training the model on a large number of data sets, which can help the model converge faster and achieve better performance on new data sets. The water level recognition method provided in this embodiment is applicable to all interactive segmentation models. RITM_interactive_segment, as an interactive segmentation model, has shown strong performance in the field of computer vision, especially in image segmentation tasks.

[0054] Therefore, further, the interactive semantic segmentation model is RITM_interactive_segment.

[0055] Step 6), based on the prior knowledge that the water level is horizontal, the image with an uneven water level is screened out, specifically, the test mask image with a confidence level lower than a second threshold is screened out, comprising the following steps:

[0056] According to the mask height value of each column of the test mask image, the height value variance of the test mask image is calculated, and the formula is as follows:

[0057]

[0058] Among them, x iis the number of pixels judged as water in the i-th column of the test mask image, that is, the mask height value of the i-th column; 1≤i≤w, w is the column of the test mask image, that is, the width of the test mask image; is the mean mask height value of the test mask image, s 2 is the mask height value variance of the test mask image.

[0059] Will s 2 As a measure of the unevenness of the horizontal line, it is also a measure of the recognition confidence, that is, s 2 The smaller the value, the greater the confidence level; according to the second threshold, the test mask image with the confidence level lower than the second threshold is screened out. In practical applications, s 2 The mask image is greater than 30. The average water level of the retained test mask image is used as the final water level recognition value.

[0060] An interactive semantic segmentation model refers to a model that performs semantic segmentation on pixels or areas in an image by combining user input (such as clicking, painting, etc.) with a machine learning algorithm. This type of model uses manual interaction to guide the model to perform more accurate segmentation. However, in image water level recognition applications, manual interaction operations are difficult. An embodiment of the present invention provides a semantic segmentation water level recognition method using prior knowledge, which automatically generates sampling points and interacts with the model based on the prior knowledge of the water bank position to solve the above problems. Prior knowledge refers to prior knowledge or experience about the problem domain that is already possessed before learning or reasoning. It is knowledge acquired from previous learning, practice or observation, and can be professional knowledge from domain experts, results of previous research, rules or common sense knowledge, etc. The prior knowledge used by the method provided in the embodiment of the present invention is conventional knowledge.

[0061] Step 7), the obtained relative water level, i.e. the final water level identification value, is mapped to the absolute water level through a virtual water gauge. Specifically, it can be converted to the absolute water level, i.e. the actual water level height, through interpolation.

[0062] Example 2

[0063] The present invention also provides a semantic segmentation water level recognition device, which includes a memory and one or more processors; one or more programs are stored in the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the method described in Example 1 of the present invention.

[0064] Example 3

[0065] The semantic segmentation water level recognition method based on prior knowledge provided in Example 1 of the present invention has been verified in engineering practice. The schematic diagram of water level recognition is shown in the attached figure. Figure 2 As shown, the khaki part is the bank, the blue part is the segmented water body, and the green dots are sampling points automatically generated by the program based on prior knowledge. After comparison with the actual situation, the water body part has been accurately segmented.

[0066] The contents explained in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the embodiments made by those skilled in the art fall within the scope defined by the claims attached to the present invention.

Claims

1. A semantic segmentation water level recognition method, characterized in that: The method comprises the following steps: Training process: Step 1), collect training water level images; Step 2), in the lower area of ​​the training water level image, generating training sampling points by a random function, and performing interactive segmentation processing on the training sampling points and the training water level image based on the pre-trained weights of the interactive semantic segmentation model to obtain a training mask image; Step 3), segmenting and labeling the training water level image to obtain a labeled water level image; using the training sampling points and the training water level image as training input data, and using the labeled water level image as training output data, training the interactive semantic segmentation model to obtain a trained interactive semantic segmentation model; Testing process: Step 4), collecting test water level images; Step 5), in the lower area of ​​the test water level image, a number of test sampling points are generated by a random function, and the test sampling points and the test water level image are input into the trained interactive semantic segmentation model to obtain a corresponding test mask image; Step 6), based on the prior knowledge that the water level line is horizontal, the test mask images with a confidence level lower than a second threshold are screened out, and the average water level value of the retained test mask images is used as the final water level recognition value.

2. The semantic segmentation water level recognition method according to claim 1, characterized in that: The step 1) further comprises: Intercept the key area of ​​the training water level image, and rotate the training water level image so that the water level line is in a horizontal position; the key area is an area where the water bank has obvious color difference and is not disturbed by weeds; The step 4) further comprises: The key area of ​​the test water level image is intercepted, and the test water level image is rotated so that the water level line is in a horizontal position; the key area is an area where the water bank has obvious color difference and there is no weed interference.

3. The semantic segmentation water level recognition method according to claim 1, characterized in that: In the step 2), the random function is randint, and two integers are randomly generated in the lower area of ​​the training water level image by the randint function, and the two integers are the first coordinates and the second coordinates of the training sampling point; In the step 5), the random function is randint, and two integers are randomly generated in the lower area of ​​the test water level image by the randint function. The random generation is performed several times to obtain the first coordinates and the second coordinates of several test sampling points.

4. The semantic segmentation water level recognition method according to claim 1, characterized in that: Before step 3), the method further includes: determining whether the MIoU index of the training mask image is less than a first threshold value, If it is less than the first threshold, continue with step 3); If it is not less than the first threshold, the interactive semantic segmentation model in step 2) can be directly used in the test process to continue with step 4).

5. The semantic segmentation water level recognition method according to claim 1, characterized in that: In the step 3), the training water level image is segmented and labeled using the Labelme image annotation tool.

6. The semantic segmentation water level recognition method according to claim 1, characterized in that: The interactive semantic segmentation model is RITM_interactive_segment.

7. The semantic segmentation water level recognition method according to claim 1, characterized in that: In step 6), the step of screening out the test mask image with a confidence level lower than a second threshold value specifically includes the following steps: According to the mask height value of each column of the test mask image, the height value variance of the test mask image is calculated, and the formula is as follows: Among them, x i is the number of pixels judged as water in the i-th column of the test mask image, that is, the mask height value of the i-th column; 1≤i≤w, w is the column of the test mask image, that is, the width of the test mask image; is the mean mask height value of the test mask image, s 2 is the mask height value variance of the test mask image; The confidence level is related to s 2 Correspondingly, s 2 The smaller it is, the greater the confidence is; according to the second threshold, the test mask image whose confidence is lower than the second threshold is screened out.

8. The semantic segmentation water level recognition method according to claim 1, characterized in that: The method further comprises the following steps: Step 7), converting the final water level identification value into an absolute water level by interpolation.

9. A semantic segmentation water level recognition device, characterized in that: The device includes a memory and one or more processors; one or more programs are stored in the memory, and when the one or more programs are executed by the one or more processors, the one or more processors implement the steps of any one of the methods described in claims 1 to 8.

Images