A method for real-time image recognition and distribution display of submerged plants

Abstract

The application discloses a submerged plant image real-time recognition and distribution display method and belongs to the technical field of water ecological monitoring. The method comprises the following steps: acquiring a map of a water area to be measured and calibrating the map; synchronously receiving a submerged plant image shot by a terminal camera, GPS data of a position where the terminal is located and water transparency data measured by a terminal sensor; performing real-time semantic segmentation processing on the received submerged plant image; performing real-time recognition on the processed image; displaying the recognition result on the map in real time; displaying the GPS data of the current position and the water transparency data in a text form; viewing a latest submerged plant image shot at a current position of the terminal and the recognition result; performing statistics and arrangement on the submerged plant image and the recognition result and saving the submerged plant image and the recognition result in the forms of an image and a text respectively; and generating a plant distribution map according to the recognition result for viewing. The application realizes efficient submerged plant image real-time recognition and distribution display.

Description

Technical Field

[0001] This invention relates to the field of aquatic ecological monitoring technology, and more specifically to a method for real-time image recognition and distribution display of submerged plants. Background Technology

[0002] Submerged plants play a crucial role in maintaining aquatic biodiversity. They are important primary producers in aquatic ecosystems, providing habitats and spawning grounds for many aquatic animals, and contributing to water clarity. However, due to various environmental pressures, such as eutrophication and climate change, certain species, such as invasive submerged plants, may overgrow in some areas, while others may decline or even disappear. Preventing the formation of monocultures by overgrown submerged plants like *Potamogeton crispus* and *Hydrilla verticillata*, and improving the biodiversity of submerged plant communities, is a challenge and a major need in the development of lake ecological restoration technologies in my country. The regulation of plant community structure has become a core technical approach for aquatic ecological restoration. Achieving coordinated growth of submerged plant species, high community diversity, and rapid ecosystem stability during the restoration of submerged vegetation, and thus improving biodiversity, is a recognized challenge in the field of aquatic ecological restoration.

[0003] Currently, there are several patents for plant identification and area measurement. For example, CN112509030A, "Plant Measurement Method, System, Computer Equipment and Medium for Multi-Target Detection," only involves plant area and size measurement technology and does not involve the identification of submerged plants. CN113128405A, "A Plant Identification and Model Construction Method Combining Semantic Segmentation and Point Cloud Processing," only classifies plants into three categories for identification: grass, shrubs, and trees, all of which are terrestrial plants. Application 202310066041.X, "An Intelligent Monitoring Method for Submerged Plant Coverage Area," only involves an intelligent monitoring method for submerged plant coverage area, obtaining the coverage area of ​​submerged plants, but does not involve the identification of specific submerged plants. Due to the phenomena of reflection, refraction, scattering, and polarization of light sources in water bodies, accurate identification of submerged plants is an urgent need in the field of aquatic ecological restoration. How to provide an efficient real-time image recognition and distribution display method for submerged plants is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a method for real-time identification and distribution display of submerged plant images to solve the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for real-time image recognition and distribution display of submerged plants includes the following steps:

[0007] S100: Acquire a map of the water area to be measured and perform calibration;

[0008] S200: Simultaneously receive images of submerged plants captured by the terminal camera, GPS data of the terminal's location, and water transparency data measured by the terminal's sensors;

[0009] S300: Performs real-time semantic segmentation processing on the received submerged plant images;

[0010] S400: Performs real-time recognition on the processed image;

[0011] S500: Displays recognition results on the map in real time;

[0012] S600: Displays current location GPS data and water transparency data in text format;

[0013] S700: View the latest image of a submerged plant captured at the current location of the terminal and the recognition result;

[0014] S800: Statistically analyzes and organizes images and recognition results of submerged plants, and saves them in both image and text formats.

[0015] S900: A plant distribution map is generated based on the identification results and is available for viewing.

[0016] Preferably, the acquisition and calibration of the map of the water area to be measured specifically includes: obtaining coarse GPS data of the water area by calling the map API and inputting the name of the water area; selecting the current GPS information icon in the coarse GPS data to obtain fine GPS data of the center point of the water area; inputting this fine GPS data in the text box of the UI interface on the display; and calling the map API again by clicking the "Get Water Area Map" button to obtain a map image of a certain range centered on this fine GPS data. Then, clicking the "Calibrate Map" button will use a method similar to "binary search" to obtain the latitude and longitude range of the map. Specifically, the method involves first counting the number of pixels of the icon representing the terminal's location on the map, and then continuously fine-tuning the latitude and longitude of the icon until the number of pixels displayed on the map for the icon is exactly half the number of pixels of the complete icon.

[0017] Preferably, the synchronous reception of submerged plant images captured by the terminal camera, GPS data of the terminal's location, and water transparency data measured by the terminal's sensors specifically includes: the UI interface running on the operating terminal is generated using the QApplication library of PyQt5, and this interface is maintained by the main thread running on the operating terminal. The operating terminal receives video streams sent by the terminal via the RTSP protocol, GPS data sent by the terminal via the TCP protocol, and water transparency data sent by the terminal via the Modbus protocol. The reception method is as follows: after clicking the "Start" button on the UI interface, the operating terminal will start two sub-threads, loop1 and loop2, in the background. Sub-thread loop1 will begin reading each frame of the video stream sent by the terminal frame by frame, while sub-thread loop2 is used to perform real-time identification of the received submerged plant images. When sub-thread loop2 is processing an image, sub-thread loop1 will continuously discard the received frames; when sub-thread loop2 is not processing an image, sub-thread loop1 will pass the latest received frame to sub-thread loop2 for semantic segmentation. After receiving an image frame from child thread loop1, child thread loop2 immediately begins to receive GPS data and water transparency data synchronized with the image using two other protocols. Then, child thread loop2 calls a pre-trained semantic segmentation model to process the image.

[0018] Preferably, the real-time semantic segmentation of the received submerged plant image in the background specifically includes: after receiving all the data from sub-thread loop1, sub-thread loop2 calls a pre-trained semantic segmentation model to process the image. This semantic segmentation model is a neural network trained using deep learning methods. The semantic segmentation neural network structure used in this operation terminal is PSPNet, and the dataset used to train this neural network is manually annotated using the interactive segmentation network SimpleClick.

[0019] Preferably, the real-time display of the terminal's location on the map on the left side of the UI interface running on the operating terminal specifically includes: when the sub-thread loop2 receives GPS data sent by the terminal, the thread calls the map API and sends the GPS data as a parameter along with the GPS data of the center point of the water area to the API. The GPS data generates a marker at the corresponding location on the map to indicate that this location is the terminal's location. Afterwards, the sub-thread loop2 receives the map returned by the API and displays it on the left side of the UI interface using a QWidget component.

[0020] Preferably, the real-time display of the current location GPS data and water transparency data in the text box on the right side of the UI interface running on the operating terminal specifically includes: when the child thread loop2 receives the GPS data and water transparency data, it will parse the two data and then update the two data in the text box on the right side of the UI interface through the setText method of QLineEdit.

[0021] Preferably, the ability to view the latest submerged plant image and its semantic segmentation map captured at the current location of the terminal specifically includes: when the sub-thread is receiving data, clicking the "View Plant Segmentation Map at Current Location" button allows viewing the latest submerged plant image with completed semantic segmentation and its segmentation map captured by sub-thread loop2. When the sub-thread stops receiving data, clicking the "View Plant Segmentation Map at Current Location" button allows viewing the last submerged plant image with completed segmentation captured by sub-thread loop2 and its segmentation map.

[0022] Preferably, the step of statistically analyzing and organizing the semantic segmentation results and saving them in image and text formats respectively includes: after sub-thread loop2 completes the semantic segmentation of an image frame, it counts the number of pixels occupied by each type of plant. Then, sub-thread loop2 locates the folder corresponding to the sub-water area based on the GPS data synchronously received when receiving this image frame, and saves both the image results and the statistical results there.

[0023] Preferably, viewing the plant distribution map based on the segmentation results specifically includes: after the sub-thread finishes receiving data, clicking the "Calculate Plant Distribution Map" button will cause the main thread on the operating terminal to start statistically analyzing the semantic segmentation results of all submerged plant images in the entire water area in the background. After the statistics are completed, the statistical results for the entire water area will be saved in the form of images and text. Then, clicking the "View Plant Distribution Map" button will display the plant distribution map of the entire water area.

[0024] As can be seen from the above technical solution, compared with the prior art, this invention discloses a method for real-time image recognition and distribution display of submerged plants. Through a control UI interface, users can conveniently and quickly control the entire system. The entire method has a simple operation process; users can easily obtain water area maps and, while receiving data, can intuitively see the terminal's real-time location on the map, along with corresponding GPS data and water transparency data. After receiving data, users only need to click the corresponding button to automatically complete the statistical analysis of the distribution of various plants in the entire water area.

[0025] In addition, the semantic segmentation of this application can not only automatically process and analyze a large number of submerged plant images, reducing the workload and time cost of manual processing, but also perform pixel-level segmentation of images, providing more detailed and accurate information on the distribution of submerged plants, and revealing the spatial distribution characteristics of submerged plants. Compared with other image-based tasks, semantic segmentation is completely different and advanced, such as image classification by recognizing the content present in the image; object recognition and detection by recognizing the content and location in the image; and recognition of the content and location present in the image. In this invention, semantic segmentation is applied to submerged plant image data acquired in real time or periodically to achieve continuous monitoring and management of the distribution of submerged plants. By continuously analyzing the distribution of submerged plants, distribution anomalies, changing trends and ecological problems can be detected in a timely manner, and corresponding management measures can be taken. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0028] Figure 2 This describes the process of obtaining a water area map;

[0029] Figure 3 The process of calibrating a map;

[0030] Figure 4 The procedure for viewing the plant segmentation map at the current location;

[0031] Figure 5 View an example image of the plant segmentation diagram at the current location;

[0032] Figure 6 Procedure for viewing plant distribution maps;

[0033] Figure 7 View an example image of the plant distribution map. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] The purpose of this invention is to provide a method for real-time identification and distribution display of submerged plant images. The method includes the following steps: S100: Acquire and calibrate a map of the water area to be measured; S200: Simultaneously receive submerged plant images captured by the terminal camera, GPS data of the terminal's location, and water transparency data measured by the terminal's sensors; S300: Perform real-time semantic segmentation processing on the received submerged plant images; S400: Perform real-time identification on the processed images; S500: Display the identification results on the map in real-time; S600: Display the current location's GPS data and water transparency data in text format; S700: View the latest submerged plant image captured at the terminal's current location and the identification result; S800: Statistically analyze and organize the submerged plant images and identification results, saving them in both image and text formats; S900: Generate a plant distribution map based on the identification results for viewing. This invention enables users to conveniently obtain water area maps and, upon receiving data, can intuitively see the terminal's real-time location on the map, along with corresponding GPS data and water transparency data. After receiving the data, the user only needs to click a button and the system can automatically complete the statistics of the distribution of various plants in the entire water area, realizing efficient real-time image recognition and distribution display of submerged plants.

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] On the one hand, see Figure 1 As shown in the figure, this invention discloses a method for real-time image recognition and distribution display of submerged plants, including the following steps:

[0038] S100: Acquire a map of the water area to be measured and perform calibration;

[0039] S200: Simultaneously receive images of submerged plants captured by the terminal camera, GPS data of the terminal's location, and water transparency data measured by the terminal's sensors;

[0040] S300: Performs real-time semantic segmentation processing on the received submerged plant images;

[0041] S400: Performs real-time recognition on the processed image;

[0042] S500: Displays recognition results on the map in real time;

[0043] S600: Displays current location GPS data and water transparency data in text format;

[0044] S700: View the latest image of a submerged plant captured at the current location of the terminal and the recognition result;

[0045] S800: Statistically analyzes and organizes images and recognition results of submerged plants, and saves them in both image and text formats.

[0046] S900: A plant distribution map is generated based on the identification results and is available for viewing.

[0047] In one specific embodiment, acquiring and calibrating a map of the water area to be measured specifically includes:

[0048] By calling the map API and inputting the name of a body of water, coarse GPS data for that body of water is obtained. Clicking the "Get Current GPS Information" icon then yields fine GPS data for the center point of the body of water. This fine GPS data is then entered into the text box in the UI interface on the operating device. Clicking the "Get Body of Water Map" button again calls the map API to obtain a map image of a certain range centered on this fine GPS data. Next, clicking the "Calibrate Map" button uses a method similar to a "binary search" to obtain the latitude and longitude range of the map. Specifically, the implementation first counts the number of pixels in the icon representing the terminal's location on the map, then continuously fine-tunes the latitude and longitude of the icon until the number of pixels displayed on the map for the icon is exactly half the number of pixels in the complete icon.

[0049] In one specific embodiment, the simultaneous reception of images of submerged plants captured by the terminal camera, GPS data of the terminal's location, and water transparency data measured by the terminal's sensors specifically includes:

[0050] The UI of the operating terminal is generated using the QApplication library in PyQt5 and is maintained by the main thread of the operating terminal. The operating terminal receives video streams from the terminal via the RTSP protocol, GPS data via the TCP protocol, and water transparency data via the Modbus protocol. The receiving method is as follows: after clicking the "Start" button in the UI, the operating terminal starts two background threads, loop1 and loop2. Loop1 begins reading each frame of the video stream sent by the terminal frame by frame, while loop2 performs real-time semantic segmentation on the received submerged plant images. When loop2 is processing an image, loop1 continuously discards received frames; when loop2 is not processing an image, loop1 passes the latest received frame to loop2 for semantic segmentation. After receiving a frame from loop1, loop2 immediately begins receiving GPS data and water transparency data synchronized with the image using the other two protocols. Then, loop2 calls a pre-trained semantic segmentation model to process the image.

[0051] In one specific embodiment, the received submerged plant image is subjected to real-time semantic segmentation in the background, specifically including:

[0052] After receiving all the data from loop1, loop2 calls a pre-trained semantic segmentation model to process the image. This semantic segmentation model is a neural network trained using deep learning. The semantic segmentation neural network architecture used in this method is PSPNet, and the dataset used to train this neural network was manually annotated using the interactive segmentation network SimpleClick, and includes an image database of six common submerged plants.

[0053] In one specific embodiment, the recognition results and the terminal's location are displayed in real time on the map on the left side of the UI interface, specifically including:

[0054] In one specific embodiment, after the loop2 thread receives the GPS data sent by the terminal, it calls the map API and sends the GPS data, along with the GPS data of the center point of the water area, to the API. The GPS data generates a marker at the corresponding location on the map to indicate that this location is the terminal's location. Then, after receiving the map returned by the API, loop2 displays it on the left side of the UI interface using a QWidget component.

[0055] In one specific embodiment, the right side of the UI interface displays real-time GPS data and water transparency data in a text box, specifically including:

[0056] Once the loop2 thread receives the GPS data and water transparency data, it will parse the two data sets and then update them in the text box on the right side of the UI using the QLineEdit's setText method.

[0057] In one specific embodiment, one can view the most recently captured image of a submerged plant and its semantic segmentation map at the current location of the terminal, specifically including:

[0058] When the child thread is receiving data, clicking the "View Current Location Plant Segmentation Image" button will display the latest semantically segmented submerged plant image and its segmentation image from the loop2 thread. When the child thread stops receiving data, clicking the "View Current Location Plant Segmentation Image" button will display the last segmented submerged plant image and its segmentation image from the loop2 thread.

[0059] In one specific embodiment, the semantic segmentation results are statistically analyzed and organized, and saved in both image and text formats, specifically including:

[0060] After loop2 thread completes the semantic segmentation of an image frame, it counts the number of pixels occupied by each type of plant. Then, loop2 thread uses the GPS data synchronously received when receiving this image frame to find the folder of the corresponding sub-water area, and saves both the image results and the statistical results there.

[0061] In one specific embodiment, generating and viewing a plant distribution map based on the segmentation results specifically includes:

[0062] After the sub-thread finishes receiving data, clicking the "Calculate Plant Distribution Map" button will initiate the main thread in the background to perform semantic segmentation of all submerged plant images across the entire water area. Once the statistics are complete, the results for the entire water area will be saved in both image and text formats. Clicking the "View Plant Distribution Map" button will then display the plant distribution map for the entire water area.

[0063] Specifically, the file structure of the operating terminal is as follows:

[0064] File structure:

[0065]

[0066] In one specific embodiment, the "gps" folder contains code for processing GPS data. The "gps_receive.py" file receives GPS data. The "gps_send.py" file sends GPS data; this file is only used for local testing. The "get_gps.py" file obtains GPS data for a location by inputting its Chinese name; this is generally not used because the GPS data obtained this way has a large margin of error. The "gps_conv.py" file is used to convert GPS data between different encoding formats; this file is generally not used because the GPS data received by the client can usually be directly decoded. The "gps_data.txt" file is used to save the received GPS data.

[0067] In one specific embodiment, the "map" folder contains the code for processing the water area map. The "map_get.py" file obtains a map of a certain range centered on the input GPS data. The "map_adjustment.py" file is used to calibrate the map, calculating the latitude and longitude range covered by the obtained map, so as to facilitate subsequent calculation of the sub-water area to which the terminal's current location belongs based on the GPS data sent by the terminal. The "map_patch_index_get.py" file calculates the label of the sub-water area where the terminal is currently located based on the GPS data sent by the terminal.

[0068] In one specific embodiment, the "transparency" folder contains the code for processing transparency data. The "transparency_receive.py" file is used to receive transparency data sent from the terminal. The "transparency_send.py" file is used to send transparency data; this file is only used for local testing. The "transparency_data.txt" file is used to save the received transparency data.

[0069] In one specific embodiment, the "video" folder contains the code for processing the video stream. The "rtsp.py" file is used to test whether the local machine can receive video stream data normally; this file is only used for local testing. The "video_get.py" file is used to receive the video stream sent by the terminal.

[0070] In one specific embodiment, the "plant_statistics" folder contains the code for statistical analysis of the segmentation results. The "plant_statistics.py" file is used to calculate the proportion of each plant species in each sub-water area.

[0071] In one specific embodiment, the "mmsegmentation" folder contains the installed library, version 0.30.0. The "demo / generate_seg_result.py" file is used to test whether the local machine can correctly call the model to segment plant images; this file is only used for local testing. The "demo / seg_plant_img.py" file contains the code for real-time semantic segmentation of submerged plant images in the background of this operation. The "log / best_mIoU_epoch_117.pth" file contains the trained semantic segmentation model for submerged plant images. The "configs / pspnet / plant_7class.py" file is the model's configuration file; the neural network structure needs to be generated based on this file when calling the model.

[0072] In one specific embodiment, the obtained water area map "current_map.png", the water area map with grid segmentation "current_map_gridlines.png", the distribution map of all plants "plant_distribution.png", and the individual distribution map of each plant "H / C / V / M / B / D.png" are stored in the "original_map" folder.

[0073] In one specific embodiment, the "plant_imgs" folder stores images of submerged plants received by the client, wherein each image is stored in the folder of its corresponding sub-water area according to the GPS data at the time of receipt.

[0074] In one specific embodiment, the "plant_imgs_seg" folder stores semantic segmentation maps of submerged plant images received by the client, as well as the semantic segmentation map statistical results stored in text form, and its storage method is the same as that of the "plant_imgs" folder. In addition, this folder also contains two files, "seg_result.txt" and "seg_result_percent.txt", which record the proportion of different plants in all sub-areas of the entire water area, recorded in the form of pixel count and percentage, respectively.

[0075] In one specific embodiment, the "config.yaml" file stores various configurations required for the entire operating system, such as the Baidu Maps API key, the IP address and port for receiving data, etc. The specific configuration needs to be modified according to the file content.

[0076] In one specific embodiment, "ui.py" is the code that generates the UI interface for the entire operating terminal. This file configures the visual interface of the entire operating terminal, as well as various buttons and text boxes, to facilitate operation by staff.

[0077] In one specific embodiment, the "requirements.txt" file records the environment that the entire operating terminal needs to be configured. Before running the code, the environment needs to be configured according to this file, and special attention should be paid to the version and installation path of each environment.

[0078] Step 101: Configure the environment according to the requirement.txt file, paying attention to the versions and installation locations of each library;

[0079] On the other hand, embodiments of the present invention also disclose an operating terminal for real-time image recognition and distribution display of submerged plants, the specific operations of which include:

[0080] Step 1: Configure the environment according to the requirement.txt file, paying attention to the versions and installation locations of each library;

[0081] Step 2: Modify the parameters in the "config.yaml" file to suit the actual use of this project, such as [API][ak], [API][zoom], [gridline][number], [IP], [port], etc.

[0082] Step 3: Open the PROJECT folder using Visual Studio Code (Python 3.10.1164-bit), click the "ui.py" file and run it;

[0083] Step 4: Obtain the water map;

[0084] Step 5: Calibrate the map;

[0085] Step 6: Receive terminal data;

[0086] Step 7: Display the plant segmentation map at the current location;

[0087] Step 8: Calculate and display the plant distribution map.

[0088] Specifically, step 1 involves configuring the environment according to the requirement.txt file, paying attention to the versions and installation locations of each library, including:

[0089] Use pip or conda commands to install each library one by one according to the name and version number of the library in the requirements.txt file. Pay attention to the installation location of each library and switch mirror sources if necessary.

[0090] Specifically, in step 2, modify the parameters in the "config.yaml" file to suit the actual use of this project, such as [API][ak], [API][zoom], [gridline][number], [IP], [port], etc.

[0091] Specifically, in step 3, open the PROJECT folder using Visual Studio Code (Python 3.10.1164-bit), click the "ui.py" file, and run it.

[0092] Specifically, step 4 involves obtaining a water area map, which includes:

[0093] Enter the GPS data of the center point of the water area to be measured in the text box “Please enter GPS center point of water area”. The data format is “longitude, latitude”, where “,” is an English comma, for example: “118.80019307726839,32.074971594853006”. Figure 2 and Figure 3 Next, the user clicks the "Get Water Map" button to obtain and display the map of the water area to be tested. The obtained water map is displayed on the left side of the UI interface. The map has been divided into different sub-water areas according to the [gridline][number] settings in the "config.yaml" file. The yellow icon with the letter "A" on the map is initially positioned at the center point of the water area map; afterwards, this icon represents the terminal's real-time location.

[0094] Specifically, step 5, calibrating the map, includes:

[0095] Clicking the "Calibrate Map" button will display the latitude and longitude range of the currently displayed map, allowing you to determine the sub-water area where the terminal is located at different times based on GPS data. The calibration process will be displayed in the Visual Studio Code terminal window, and the calibration results, along with the GPS data of the water area center point entered in step 104, will be saved in the "config.yaml" file for easy use next time without needing to repeat the calibration.

[0096] Specifically, step 6, receiving terminal data, includes:

[0097] Click the "Start" button, and the client will begin receiving data sent from the terminal, as follows:

[0098]

[0099] On the left side of the UI interface, the terminal displays its current location on a map in real time using a yellow icon; on the right side, the terminal displays the GPS data and water transparency data sent by the terminal in real time in a text box. Simultaneously, the terminal's backend performs real-time semantic segmentation on the received submerged plant images and saves the segmentation results as both images and text, in the folder corresponding to the sub-water area corresponding to the GPS data received simultaneously when the image was received. The RGB color scheme of the semantic segmentation image results is as follows: [Water (W): (1,1,1), Hydrilla verticillata (H): (232,35,244), Ceratophyllum demersum (C): (156,102,102), Vallisneria natans (V): (10,10,190), Myriophyllum spicatum (M): (150,10,50), Potamogeton malaianum (B): (0,220,220), Potamogeton crispus (D): (255,255,0)]. To modify this color scheme, you need to edit the [network][palette] section in the "config.yaml" file. Clicking the "Stop" button will stop the client from receiving data. After stopping data reception, the yellow icon on the map will show the location of the last received GPS data.

[0100] Specifically, step 7 displays the plant segmentation map at the current location, which includes:

[0101] While receiving data, clicking the "View Current Location Plant Segmentation Map" button will display the latest received image of a submerged plant at the current location and its segmentation map. When data reception stops, clicking the "View Current Location Plant Segmentation Map" button will display the last received image of a submerged plant and its segmentation map.

[0102] Figure 4 This describes the steps to view the plant segmentation map at the current location. Figure 5 This is an example image for viewing the plant segmentation diagram at the current location.

[0103] Specifically, in step 8, taking six common submerged plants as examples, including *Potamogeton crispus*, *Hydrilla verticillata*, *Myriophyllum spicatum*, *Potamogeton malaianum* (bamboo-leaved pondweed), *Vallisneria natans*, and *Ceratophyllum demersum*, calculate and display the plant distribution map, specifically including:

[0104] After stopping data reception, click the "Calculate Plant Distribution Map" button. The operating system will then start calculating the distribution of the six plant species in the entire water area in the background and save the statistical results in both text and image formats. Figure 6The workflow for viewing the plant distribution map includes the following: After the sub-thread finishes receiving data, click the "Calculate Plant Distribution Map" button. The main thread of the operating terminal will then begin calculating the semantic segmentation results of all submerged plant images in the entire water area in the background. After the calculation is complete, the statistical results for the entire water area will be saved in both image and text formats. Then, clicking the "View Plant Distribution Map" button will display the plant distribution map for the entire water area. Refer to the description of the "plant_imgs_seg" and "original_map" folders in the file structure section of the operating terminal mentioned above. After clicking the "View Plant Distribution Map" button again, the operating terminal will display the plant distribution of the entire water area. Figure 6 This is the procedure for viewing a plant distribution map. Figure 7 This is an example image for viewing plant distribution. By default, the correspondence between the number of plant categories and RGB colors in a sub-water area is: [0:(0,0,0),1:(255,0,0),2:(255,165,0),3:(255,255,0),4:(0,128,0),5:(0,255,255),6:(128,0,128)]. For example... Figure 7 In the diagram, the black portion on the right has RGB = (0,0,0), indicating that this sub-water area does not contain any of the six plant species. The green portion on the right has RGB = (0,128,0), indicating that this sub-water area contains four of the six plant species. To modify this color scheme, you need to modify the [plant_distribution][color_list] section in the "config.yaml" file. The color scheme for the single plant distribution map is the same as described in step 106.

[0105] This invention targets common submerged plants in aquatic ecosystems (such as six common submerged plants: *Potamogeton crispus*, *Hydrilla verticillata*, *Myriophyllum spicatum*, *Potamogeton malaianum*, *Vallisneria natans*, and *Ceratophyllum demersum*), aiming to achieve real-time identification and distribution statistics of images of these common submerged plants controlled by a user interface. The invention uses a manually annotated submerged plant image dataset generated by the interactive segmentation network SimpleClick. A PSPNet neural network was trained using supervised training in deep learning. Finally, a UI interface was built using PyQt5 for easy operation. In actual operation, staff can click buttons on the UI interface to invoke a pre-trained semantic segmentation model to perform real-time identification of the submerged images of the above six plants captured by the terminal and to statistically analyze the distribution of each plant.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0107] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for real-time recognition and distribution display of submerged plant images, characterized in that, The method comprises the following steps: S100: obtaining a map of a water area to be measured and calibrating the map; S200: synchronously receiving a submerged plant image captured by a terminal camera, GPS data of a location where the terminal is located, and water transparency data measured by a terminal sensor; S300: performing real-time semantic segmentation processing on the received submerged plant image; S400: performing real-time identification on the processed image; S500: displaying the identification result on the map in real time; S600: displaying the GPS data of the current location and the water transparency data in a text form; S700: viewing a latest submerged plant image captured at a current location of the terminal and the identification result; S800: statistically processing the submerged plant image and the identification result, and saving the image and the text in a form of image and text respectively; S900: generating a plant distribution map according to the identification result; The S100 comprises the following steps: S120: obtaining coarse GPS data of the water area to be measured by calling an API of the map and inputting a name of the water area to be measured; S130: selecting a current GPS information icon in the coarse GPS data to obtain fine GPS data of a center point of the water area; S140: calling the API of the map based on the fine GPS data to obtain a map image in a range centered on the fine GPS data; S150: calibrating the map by using a dichotomy method to obtain a longitude and latitude range of the map, and the calibration is implemented in the following manner: S151: counting a number of pixels of an icon representing the location where the terminal is located in the map; S152: finely adjusting the longitude and latitude of the icon until a number of pixels of the icon displayed on the map is half of a complete icon pixel number, and the calibration is completed; The S300 comprises the following steps: The terminal has a sub-thread loop1 and a sub-thread loop2, the sub-thread loop1 is used to read each frame of a video stream sent by the terminal frame by frame, and the sub-thread loop2 is used to perform real-time semantic segmentation on the received submerged plant image; When the sub-thread loop2 is processing an image, the sub-thread loop1 will discard the received frame continuously; When the sub-thread loop2 is not processing an image, the sub-thread loop1 will pass the latest frame to the sub-thread loop2 for semantic segmentation; After the sub-thread loop2 receives the frame of image passed by the sub-thread loop1, the sub-thread loop2 starts to receive GPS data and water transparency data synchronized with the image; After the loop2 receives all the data passed by the loop1, the loop2 calls a pre-trained semantic segmentation model to process the image; The semantic segmentation model is a neural network trained by a deep learning method, a semantic segmentation neural network structure used in the terminal is PSPNet, and a data set used to train the neural network is generated by manual labeling using an interactive segmentation network SimpleClick; The S800 comprises the following steps: S810: When the sub-thread loop2 completes the semantic segmentation of a frame of image, the number of pixels occupied by each plant is counted; S820: The sub-thread loop2 finds the corresponding sub-aquatic area folder according to the GPS data synchronously received when receiving the frame of image; S830: The image result and the statistical result are saved in the folder.

2. The method according to claim 1, wherein, The S200: synchronously receiving the submerged plant image taken by the terminal camera, the GPS data of the location where the terminal is located, and the water transparency data measured by the terminal sensor, comprising: The UI interface is generated by the QApplication library of PyQt5, and the UI interface is maintained by the main thread running by the operation terminal; The operation terminal receives the video stream sent by the terminal camera through the RTSP protocol, receives the GPS data of the terminal positioning through the TCP protocol, and receives the water transparency data measured by the terminal sensor through the Modbus protocol.

3. The method according to claim 1, wherein, The S500: displaying the recognition result on the map in real time, comprising: S510: When the sub-thread loop2 thread receives the GPS data sent by the terminal, the sub-thread loop2 calls the API of the map, and the GPS data is transmitted to the API as a parameter together with the GPS data of the center point of the water area; S520: The GPS data generates a mark at the corresponding position on the map to indicate that the position is the terminal position; S530: The sub-thread loop2 receives the map returned by the API and displays it on the left side of the UI interface through the QWidget component.

4. The method according to claim 3, wherein, The S600: displaying the current position GPS data and the water transparency data in text form, comprising: S610: When the sub-thread loop2 receives the GPS data and the water transparency data, the two data are parsed; S620: The GPS data and the water transparency data are updated in the text box on the right side of the UI interface through the setText method of QLineEdit.

5. The method according to claim 4, wherein the method is characterized by: The S700: viewing the latest submerged plant image taken at the current position of the terminal and the recognition result, comprising: When the sub-thread loop2 receives data, the latest submerged plant image completed semantic segmentation by the sub-thread loop2 and the segmentation image are viewed; When the sub-thread loop2 stops receiving data, the last submerged plant image completed segmentation by the sub-thread loop2 and the segmentation image are viewed.

6. The method according to claim 1, wherein, The S900: generating a plant distribution map according to the recognition result S910: When the sub-thread loop1 and the sub-thread loop2 end receiving data, the semantic segmentation results of all submerged plant images in the water area to be measured are counted; S920: After the counting is completed, the statistical result of the water area to be measured is saved in the form of image and text; S930: The plant distribution map of the water area to be measured is generated and displayed.

Images