Semantic segmentation-based river crab culture area development activity identification method and system

Abstract

The invention discloses a river crab culture area development activity identification method and system based on semantic segmentation, and belongs to the technical field of aquaculture monitoring, the river crab culture area development activity identification method and system based on semantic segmentation comprises the following specific steps: 1, after a front end sends an upload request, the upload request is sent to the front end; the rear end receives and stores the two-time-phase original remote sensing images, generates a corresponding ovr tile file and returns a file address for front-end acceleration display; and 2, under the condition of user-defined filtering, after the front end uploads the two-temporal semantic segmentation tag graph used for filtering the change area and sends a category analysis request, the rear end receives and stores the two-temporal semantic segmentation tag graph, counts tag categories in the two-temporal semantic segmentation tag graph, and returns a category list. According to the method, the change detection model and the semantic segmentation model are trained, and the two-time same satellite remote sensing images uploaded by the user are analyzed, so that the detection efficiency is improved, false alarms are reduced, and more personalized identification requirements are supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of aquaculture monitoring, and in particular relates to a method and system for identifying development activities in river crab breeding areas based on semantic segmentation. Background Art

[0002] With the continuous development of river crab farming industry, the rational development and management of farming areas are becoming more and more important. Traditional farming area monitoring mainly relies on manual inspections, which consumes a lot of manpower and material resources, and it is difficult to timely and accurately detect unreasonable development activities such as illegal reclamation and excessive construction. With the advancement of computer vision technology, semantic segmentation technology has provided a new way for intelligent identification of development activities in farming areas. However, the existing technology still has many shortcomings in identifying development activities in the complex environment of river crab farming. For example, the light and shadow interference caused by the ripples in the aquaculture water body is not accurately identified, and it is difficult to distinguish between the construction of normal aquaculture facilities and the construction of illegal buildings. The current change monitoring methods mainly include the following: (1) Manual visual method: Manually translate the dual-phase remote sensing data within the monitoring range and mark the change range; (2) Direct image comparison method: Directly calculate and transform the pixel values ​​in the two registered phase remote sensing images to find the changed area. Currently, commonly used direct comparison methods for spectral data include image difference method, image ratio method, vegetation index comparison method, principal component analysis method, spectral feature variation method, false color synthesis method, band replacement method, change vector analysis method, band cross-correlation analysis and hybrid detection method; (3) Post-classification comparison method: first classify the remote sensing data of the two phases that have been aligned, and then compare the classification results to obtain change detection information.

[0003] Existing manual visual inspection is inefficient and prone to omissions or subjective judgments. First, manual visual translation requires a lot of time and human resources. When processing large-scale image data, a lot of manual labor is required, which significantly increases the time cost of monitoring projects. Secondly, hiring and training a sufficient number and quality of personnel to perform visual translation is an expensive task. Companies must bear high labor costs, which is not conducive to maintaining competitiveness. In addition, as the scale of monitoring projects increases, traditional manual visual translation methods become impractical. It is difficult to cope with large-scale data sets and vast geographical areas, thus limiting the scalability of monitoring capabilities. Finally, manual visual translation is prone to omissions, errors, or subjective judgments. This may lead to false alarms or omissions of important environmental changes, affecting the accuracy of monitoring.

[0004] Pixel-based classification or clustering methods such as SVM and decision trees are also difficult to meet the needs of large-scale remote sensing information monitoring due to problems such as fragmented results and low universality.

[0005] Therefore, in the context of intelligent environmental protection, using machine recognition methods such as deep learning to fully explore and utilize environmental data such as remote sensing images for ecological environment monitoring is a good way to assist manual interpretation, reduce workload and improve efficiency. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a method and system for identifying river crab breeding area development activities based on semantic segmentation.

[0007] The technical solution adopted to solve the above technical problems is: a method for identifying river crab breeding area development activities based on semantic segmentation, including, characterized in that, including the following specific steps:

[0008] Step 1: After the front-end sends an upload request, the back-end receives and stores the original remote sensing images of the two time phases, generates the corresponding ovr tile file, and returns the file address for the front-end to accelerate display;

[0009] Step 2: In the case of custom filtering, after the front-end uploads the two-phase semantic segmentation label map for filtering the change area and sends a category analysis request, the back-end receives and stores the two-phase semantic segmentation label map and counts the label categories therein, and returns a category list;

[0010] Step 3: In the case of custom filtering, after the front-end sends the analysis request, the back-end receives the blocksize (size of the segmented image), modelvision (model version used for analysis), threshold (area filtering threshold), and filter category parameters selected by the front-end to start analysis and calculation;

[0011] Step 4: After the front end sends the download request, the back end returns the changed area data in shp vector format or tif raster format according to the selected format, so as to monitor the development activities of the river crab breeding area.

[0012] Through the above technical solution, development and construction activities can be detected and identified more accurately, which improves detection efficiency, reduces false alarms, and supports more personalized identification needs.

[0013] Furthermore, the analysis and calculation in step three includes the following specific steps:

[0014] Step 1: Cut the image into overlapping small images and store them temporarily;

[0015] Step 2: Input the change detection model to analyze one by one and output the corresponding change label map (in the case of default filtering, it is necessary to input the semantic segmentation model to analyze one by one and output the corresponding two-phase semantic segmentation label map);

[0016] Step 3: Combine the small images weighted by the confidence matrix to obtain a complete label image;

[0017] Step 4: Calculate the area that is transformed from non-construction land to construction land based on the semantic segmentation label map of the two time phases, and obtain the effective area mask;

[0018] Step 5: Filter out the area within a certain range where plots of different land use properties meet, and obtain a further effective area mask;

[0019] Step 6: The original change labels are filtered through masks to obtain the final change labels and converted into vectorized format for storage;

[0020] Step 7: Return the changed area data in geojson format for front-end display.

[0021] The above technical solution improves the accuracy of analysis and calculation.

[0022] Furthermore, it includes five subsystems, namely data acquisition module, preprocessing module, semantic segmentation model building module, development activity recognition and judgment module, and early warning and report generation module. The data acquisition module is responsible for collecting all-round images of the river crab breeding area through drones equipped with high-definition cameras and multi-spectral camera equipment according to preset routes, and transmits them to the preprocessing module in real time.

[0023] Through the above-mentioned technical solution, the ecological space control zone development and construction activity identification system can more accurately detect and identify development and construction activities in river crab breeding areas, thereby improving detection efficiency and reducing false alarms.

[0024] Furthermore, the preprocessing module is responsible for performing denoising, grayscale correction, and geometric distortion correction on the received image data to eliminate image defects caused by light changes and device jitter, improve image quality, and lay the foundation for subsequent semantic segmentation.

[0025] Furthermore, the semantic segmentation model construction module includes the following specific steps:

[0026] Step 1: Data training and labeling:

[0027] Collect a large number of public datasets and compare the datasets with the resolution that best matches the actual needs, and perform pixel-level annotation for change detection and semantic segmentation on them: assign a binary label of "changed" or "unchanged" to each pixel in a set of images in the training set for change detection, and assign a multi-classification semantic label to each pixel in the images in the training set for semantic segmentation;

[0028] Step 2: Train the deep learning model:

[0029] The architecture of the semantic segmentation model includes an encoder, a decoder, and a pixel-level classification layer. The change detection model uses ResNet18 as the backbone network to extract feature maps corresponding to two times, and the results are convolutionally encoded and segmented to obtain semantic tokens of the number of feature map channels. The context information in the spatial and temporal domains is obtained through encoding and decoding.

[0030] Step 3: Customize the filtering mechanism;

[0031] Embed a custom filtering mechanism to support users to interactively select the filtering threshold, cutting size and the type of change to be identified. Specifically, customize the combination of changed and unchanged categories from the N_classes^2 category combinations, and then calculate the mask of the valid area based on the two-phase semantic segmentation labels, and perform the Hadamard product with the change detection result.

[0032] Through the above technical solution, more detailed monitoring requirements may be proposed in actual monitoring tasks, and the definition of changes and no changes can also be changed, making the overall use more flexible and convenient.

[0033] Furthermore, by establishing an image pyramid, the rendering time is reduced. After the system backend receives the remote sensing image in tif format uploaded by the user, it will generate the corresponding .ovr format pyramid file, divide the original image into multiple resolution levels, and each level is saved as an independent image. The low-resolution image is used to zoom in on a larger map view, and the high-resolution image is used to zoom in on a smaller map view. At the same time, the remote sensing images of the previous and next phases are displayed in a rolling shutter manner and the change detection result layer converted into geojson format.

[0034] Furthermore, the semantic segmentation model building module supports users to upload and update models by themselves and put them into use.

[0035] Through the above technical solution, when the user clicks the corresponding button on the system page, a pop-up upload form will be triggered. In this form, the user needs to provide the name of the model, which helps to identify and label the model; secondly, the user needs to select the category of the model so that it can be correctly classified with other models; finally, the user needs to upload their model weight file. After the user completes and submits the form, the back-end system will receive and process the uploaded file. The model weight file will be saved to a specific folder for use in future analysis. In addition, the back-end will perform some verification and security checks to ensure that the uploaded model weight file is valid and secure. After completion, the model selector on the front-end will be updated immediately, and the newly uploaded model will be added to the list of optional models, enabling users to choose to use them in future analysis processes. No additional settings or configurations are required, just simply select the model they uploaded in the model selector to start using it.

[0036] Furthermore, the development activity identification and determination module is responsible for inputting the pre-processed image into the trained semantic segmentation model to obtain the segmentation result. Through the preset rule library, the segmented different areas are analyzed and compared to determine whether there are any abnormalities; the early warning and report generation module is responsible for once the abnormal development activity is identified, the system immediately triggers the early warning mechanism, sends alarm information to the breeding area managers through SMS and APP push, and generates a detailed abnormal situation report, including the abnormal type, location, and estimated impact range, to assist managers in taking quick response measures.

[0037] The beneficial effects of the present invention are as follows: the present invention trains a change detection model and a semantic segmentation model, analyzes two identical satellite remote sensing images uploaded by users, first identifies generalized construction activities through the change detection model, then inputs the two images into the semantic segmentation model respectively, and then performs custom filtering on the change detection results of the former according to the semantic recognition results of the latter, so that the ecological space control area development and construction activity identification system can more accurately detect and identify the development and construction activities in the river crab breeding area, improves the detection efficiency, reduces false alarms, and supports more personalized identification needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the technical roadmap of the change detection system of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] like Figure 1 As shown, the crab breeding area development activity recognition method and system based on semantic segmentation of this embodiment includes the following specific steps:

[0041] Step 1: After the front-end sends an upload request, the back-end receives and stores the original remote sensing images of the two time phases, generates the corresponding ovr tile file, and returns the file address for the front-end to accelerate display;

[0042] Step 2: In the case of custom filtering, after the front-end uploads the two-phase semantic segmentation label map for filtering the change area and sends a category analysis request, the back-end receives and stores the two-phase semantic segmentation label map and counts the label categories therein, and returns a category list;

[0043] Step 3: In the case of custom filtering, after the front-end sends the analysis request, the back-end receives the blocksize (size of the segmented image), modelvision (model version used for analysis), threshold (area filtering threshold), and filter category parameters selected by the front-end to start analysis and calculation;

[0044] Step 4: After the front end sends the download request, the back end returns the changed area data in shp vector format or tif raster format according to the selected format, so as to monitor the development activities of the river crab breeding area.

[0045] The analysis and calculation in step 3 includes the following specific steps:

[0046] Step 1: Cut the image into overlapping small images and store them temporarily;

[0047] Step 2: Input the change detection model to analyze one by one and output the corresponding change label map (in the case of default filtering, it is necessary to input the semantic segmentation model to analyze one by one and output the corresponding two-phase semantic segmentation label map);

[0048] Step 3: Combine the small images weighted by the confidence matrix to obtain a complete label image;

[0049] Step 4: Calculate the area that is transformed from non-construction land to construction land based on the semantic segmentation label map of the two time phases, and obtain the effective area mask;

[0050] Step 5: Filter out the area within a certain range where plots of different land use properties meet, and obtain a further effective area mask;

[0051] Step 6: The original change labels are filtered through masks to obtain the final change labels and converted into vectorized format for storage;

[0052] Step 7: Return the changed area data in geojson format for front-end display.

[0053] It includes five subsystems: data acquisition module, preprocessing module, semantic segmentation model building module, development activity recognition and judgment module, and early warning and report generation module. The data acquisition module is responsible for using drones equipped with high-definition cameras and multi-spectral camera equipment to collect all-round images of the river crab breeding area according to preset routes, and transmit them to the preprocessing module in real time.

[0054] The preprocessing module is responsible for performing denoising, grayscale correction, and geometric distortion correction on the received image data to eliminate image defects caused by light changes and device jitter, improve image quality, and lay the foundation for subsequent semantic segmentation.

[0055] The semantic segmentation model construction module includes the following specific steps:

[0056] Step 1: Data training and labeling:

[0057] Collect a large number of public datasets and compare the datasets with the resolution that best matches the actual needs, and perform pixel-level annotation for change detection and semantic segmentation on them: assign a binary label of "changed" or "unchanged" to each pixel in a set of images in the training set for change detection, and assign a multi-classification semantic label to each pixel in the images in the training set for semantic segmentation;

[0058] Step 2: Train the deep learning model:

[0059] The architecture of the semantic segmentation model includes an encoder, a decoder, and a pixel-level classification layer. The change detection model uses ResNet18 as the backbone network to extract feature maps corresponding to two times, and the results are convolutionally encoded and segmented to obtain semantic tokens of the number of feature map channels. The context information in the spatial and temporal domains is obtained through encoding and decoding.

[0060] Step 3: Customize the filtering mechanism;

[0061] Embed a custom filtering mechanism to support users to interactively select the filtering threshold, cutting size and the type of change to be identified. Specifically, customize the combination of changed and unchanged categories from the N_classes^2 category combinations, and then calculate the mask of the valid area based on the two-phase semantic segmentation labels, and perform the Hadamard product with the change detection result.

[0062] By establishing an image pyramid, the rendering time is reduced. After the system backend receives the remote sensing image in tif format uploaded by the user, it will generate the corresponding .ovr format pyramid file, divide the original image into multiple resolution levels, and save each level as an independent image. The low-resolution image is used to zoom in on a larger map view, and the high-resolution image is used to zoom in on a smaller map view. At the same time, the remote sensing images of the previous and next phases are displayed in a rolling manner and the change detection result layer converted into geojson format.

[0063] The semantic segmentation model construction module supports users to upload and update models and put them into use. The development activity identification and judgment module is responsible for inputting the pre-processed images into the trained semantic segmentation model to obtain the segmentation results. Through the preset rule library, the segmented different areas are analyzed and compared to determine whether there are any abnormalities; the early warning and report generation module is responsible for once abnormal development activities are identified, the system immediately triggers the early warning mechanism, sends alarm information to the breeding area managers through SMS and APP push, and generates a detailed abnormal situation report, including the abnormal type, location, and estimated impact range, to assist managers in taking quick response measures.

[0064] The front-end is based on the overall architecture and data binding provided by the Vue3 framework, adopts the interface elements and beautification provided by the Element-UI component library, uses the OpenLayers plug-in to display maps and remote sensing images, and uses Axios to efficiently interact with the back-end to form a complete front-end system.

[0065] The backend is based on Python language:

[0066] In the environment configuration of python3.6, pytorch1.6.0, torchvision0.7.0, the UNet semantic segmentation model and the BIT-based Net change detection model are deployed, and the geographic and image data are preprocessed and integrated with the help of gdal, geopandas geographic library and opencv, pillow graphics library to form a complete change recognition system.

[0067] On this basis, the Python-based lightweight Web framework Flask is used for model encapsulation and data transmission, and the Flask-CORS package is used to configure Cors cross-domain to form a complete back-end system.

[0068] The system functions mainly include file transmission and processing, remote sensing information display, and change area calculation.

[0069] For the deep learning model part, for the semantic segmentation and change detection tasks, multiple data sets were selected and noise data enhancement was performed respectively. At the same time, multiple models were selected and trained on multiple data sets in turn and the recall evaluation indicators on the same test set were compared to select the optimal model.

[0070] In the semantic segmentation task, we selected UNet and SegNet, two classic models in the field of semantic segmentation, and trained them on the LoveDA and Guofeng datasets. The UNet model trained on the LoveDA dataset achieved better results.

[0071] In the change detection task, STANet, BIT-CD, and AIE-changedet were selected for training on remote sensing datasets with different focuses on LEVIR, DSIFN, and SenseEarth resolutions and classification labels. Among them, the accuracy of BIT-CD and AIE-changedet on the SenseEarth dataset reached more than 94%.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A method for identifying river crab breeding areas based on semantic segmentation, characterized by , including the following specific steps: Step 1: After the front-end sends an upload request, the back-end receives and stores the original remote sensing images of the two time phases, generates the corresponding ovr tile file, and returns the file address for the front-end to accelerate display; Step 2: In the case of custom filtering, after the front-end uploads the two-phase semantic segmentation label map for filtering the change area and sends a category analysis request, the back-end receives and stores the two-phase semantic segmentation label map and counts the label categories therein, and returns a category list; Step 3: In the case of custom filtering, after the front-end sends the analysis request, the back-end receives the blocksize (size of the segmented image), modelview (model version used for analysis), threshold (area filtering threshold), and filter category parameters selected by the front-end to start analysis and calculation; Step 4: After the front end sends the download request, the back end returns the changed area data in shp vector format or ti f raster format according to the selected format, so as to monitor the development activities of the river crab breeding area.

2. The method for identifying river crab breeding area development activities based on semantic segmentation according to claim 1, characterized in that: The analysis and calculation in step 3 includes the following specific steps: Step 1: Cut the image into overlapping small images and store them temporarily; Step 2: Input the change detection model to analyze one by one and output the corresponding change label map (in the case of default filtering, it is necessary to input the semantic segmentation model to analyze one by one and output the corresponding two-phase semantic segmentation label map); Step 3: Combine the small images weighted by the confidence matrix to obtain a complete label image; Step 4: Calculate the area that is transformed from non-construction land to construction land based on the semantic segmentation label map of the two time phases, and obtain the effective area mask; Step 5: Filter out the area within a certain range where the land parcels with different land use properties meet, and obtain a further effective area mask; Step 6: The original change labels are filtered through masks to obtain the final change labels and converted into vectorized format for storage; Step 7: Return the changed area data in geojson format for front-end display.

3. The crab breeding area development activity identification system based on semantic segmentation according to claim 2 is characterized in that: It includes five subsystems: data acquisition module, preprocessing module, semantic segmentation model building module, development activity recognition and judgment module, and early warning and report generation module. The data acquisition module is responsible for using drones equipped with high-definition cameras and multi-spectral camera equipment to collect all-round images of the river crab breeding area according to preset routes, and transmit them to the preprocessing module in real time.

4. The crab breeding area development activity identification system based on semantic segmentation according to claim 3 is characterized in that: The preprocessing module is responsible for performing denoising, grayscale correction, and geometric distortion correction on the received image data to eliminate image defects caused by light changes and device jitter, improve image quality, and lay the foundation for subsequent semantic segmentation.

5. The crab breeding area development activity identification system based on semantic segmentation according to claim 4 is characterized in that: The semantic segmentation model construction module includes the following specific steps: Step 1: Data training and labeling: Collect a large number of public datasets and compare the datasets with the resolution that best matches the actual needs, and perform pixel-level annotation for change detection and semantic segmentation on them: assign a binary label of "changed" or "unchanged" to each pixel in a set of images in the training set for change detection, and assign a multi-classification semantic label to each pixel in the images in the training set for semantic segmentation; Step 2: Train the deep learning model: The architecture of the semantic segmentation model includes an encoder, a decoder, and a pixel-level classification layer. The change detection model uses ResNet18 as the backbone network to extract feature maps corresponding to two times, and the results are convolutionally encoded and segmented to obtain semantic tokens of the number of feature map channels. The context information in the spatial and temporal domains is obtained through encoding and decoding. Step 3: Customize the filtering mechanism; Embed a custom filtering mechanism to support users to interactively select the filtering threshold, cutting size and the type of change to be identified. Specifically, customize the combination of changed and unchanged categories from the N_classes^2 category combinations, and then calculate the mask of the valid area based on the two-phase semantic segmentation labels, and perform the Hadamard product with the change detection result.

6. The crab breeding area development activity identification system based on semantic segmentation according to claim 5 is characterized in that: By establishing an image pyramid, the rendering time is reduced. After the system backend receives the remote sensing image in tif format uploaded by the user, it will generate the corresponding .ovr format pyramid file, divide the original image into multiple resolution levels, and save each level as an independent image. The low-resolution image is used to zoom in on a larger map view, and the high-resolution image is used to zoom in on a smaller map view. At the same time, the remote sensing images of the previous and next phases are displayed in a rolling manner and the change detection result layer converted into geojson format.

7. The crab breeding area development activity identification system based on semantic segmentation according to claim 6 is characterized in that: The semantic segmentation model building module supports users to upload and update models and put them into use.

8. The crab breeding area development activity identification system based on semantic segmentation according to claim 7 is characterized in that: The development activity recognition and determination module is responsible for inputting the pre-processed images into the trained semantic segmentation model to obtain the segmentation results. Through the preset rule library, it analyzes and compares the different segmented areas to determine whether there are any abnormal situations. The early warning and report generation module is responsible for once abnormal development activities are identified, the system immediately triggers the early warning mechanism, sends alarm information to the breeding area managers through SMS and APP push, and generates a detailed abnormal situation report, including the type of abnormality, location of occurrence, and estimated impact range, to assist managers in taking quick response measures.

Images