Visual identification system of unmanned salvage ship for invasive plants on water surfaces of rivers and lakes

Abstract

The invention provides a visual identification system of an unmanned salvage ship for invasive plants on water surfaces of rivers and lakes, and the visual identification system comprises a visual identification system module which is connected with an invasive plant video image collection module; the invasive plant data set training module is connected with the invasive plant video image acquisition module; and the modeling method module is connected with the invasive plant data set training module, and the modeling method module is connected with the image preprocessing algorithm module. The system is simple in structure, the main core of the system is composed of the water surface invasive plant visual identification system and the salvage equipment, the implementation cost is low, the feasibility is high, the whole system adopts automatic design, the artificial interference factor is small, the stability, accuracy and continuity of the equipment are ensured, the expansibility is high, and the system is suitable for large-scale popularization and application along with continuous training refinement of a model. The application range of the equipment can be continuously expanded, the reusability is high, and energy conservation and environmental protection are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of river and lake ecological protection, and in particular to a visual recognition system for an unmanned salvage ship for invasive plants on the water surface of rivers and lakes. Background Art

[0002] In recent years, the explosive growth of water surface plants such as water hyacinths in rivers and lakes has led to serious water environment, water ecology and social problems, which has caused the imbalance of species distribution in the water ecosystem, the overgrowth of single species, and seriously threatened the growth of indigenous organisms in the water environment, as well as the water conservancy functions of river navigation and flood discharge. The withering of a large number of aquatic plants in autumn and winter every year seriously affects water quality and causes great harm to the water and soil resource environment. The existing treatment processes for water surface floating objects such as water hyacinths mostly involve manual salvage, shore stacking, and selective landfilling.

[0003] Before the outbreak of water hyacinths, there is a process of scattered growth. Traditional manual salvage can only be carried out when water hyacinths break out on a large scale. At this time, the quantity of water hyacinths is large and convenient for centralized interception. The salvaged water hyacinths are all plants with a large biomass, the total salvage volume is large, and the salvage process and later treatment costs are very high. Developing a visual recognition system for an unmanned salvage ship for water surface plants such as water hyacinths in rivers and lakes to automatically identify water hyacinths throughout the growth cycle and continuously salvage them throughout the growth cycle of water hyacinths plays an important role in controlling invasive plants. Summary of the Invention

[0004] The present invention aims to solve the technical problems that traditional manual salvage can only be carried out when water hyacinths break out on a large scale. At this time, the quantity of water hyacinths is large and convenient for centralized interception. The salvaged water hyacinths are all plants with a large biomass, the total salvage volume is large, and the salvage process and later treatment costs are very high. Developing a visual recognition system for an unmanned salvage ship for water surface plants such as water hyacinths in rivers and lakes to automatically identify water hyacinths throughout the growth cycle and continuously salvage them throughout the growth cycle of water hyacinths plays an important role in controlling invasive plants, and provides a visual recognition system for an unmanned salvage ship for invasive plants on the water surface of rivers and lakes.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] The present invention provides a visual recognition system for an unmanned salvage ship for invasive plants on the water surface of rivers and lakes, and the visual recognition system includes:

[0007] A visual recognition system module, which is connected to an invasive plant video image acquisition module;

[0008] An invasive plant data set training module, which is connected to an invasive plant video image acquisition module;

[0009] A modeling method module, which is connected to an invasive plant dataset training module, and the modeling method module is connected to an image preprocessing algorithm module.

[0010] Furthermore, the recognition method of the visual recognition system module includes the following steps:

[0011] S1. Collect a water surface invasive plant dataset and train to obtain training samples;

[0012] S2. The unmanned salvage ship enters the designated area, collects a test set through a camera, and the unmanned salvage ship recognition system judges whether the photographed target is an invasive plant by testing the test set;

[0013] S3. If it is judged to be an invasive plant, the unmanned salvage ship conducts salvage; otherwise, continue to collect the test set.

[0014] Furthermore, the implementation method of the visual recognition system module is to train a training set from the collected water surface invasive plant dataset, and put the test set collected by the camera into the software system for running and recognition.

[0015] Furthermore, the invasive plant dataset includes: videos of different types of invasive plants during the day, in fog, in rain, and at night from the perspective of an unmanned aerial vehicle, and videos of different types of invasive plants during the day, in fog, in rain, and at night from the camera of the unmanned salvage ship. Training sets, validation sets, and test sets are obtained based on the invasive plant dataset;

[0016] The training to obtain a training set based on the invasive plant dataset includes the following steps:

[0017] a. Mark and magnify the features of invasive plants in the video from the perspective of the unmanned salvage ship collected;

[0018] b. Frame the video from the perspective of the unmanned salvage ship with the marked and magnified features;

[0019] c. Conduct invasive plant annotation and inverse annotation on the framed photos to obtain the training set based on the invasive plant dataset.

[0020] Furthermore, the water surface invasive plant visual recognition system module includes a camera, an infrared detection radar, a GPS, a salvage device, and a six-axis attitude sensor, and the camera, infrared detection radar, GPS, salvage device, and six-axis attitude sensor all have waterproof functions.

[0021] Furthermore, the visual recognition system uses a large number of data images for training and recognition, uses interval frame acquisition or frame-by-frame acquisition for recognition, and uses a convolutional neural network for feature vector acquisition to reduce the use of computing resources.

[0022] Further, the modeling method of the modeling method module includes: on the basis of the three-dimensional modeling method of the environmental depth image based on visible light, adding an infrared radar for more detailed modeling to prevent the influence of brightness factors on modeling.

[0023] Further, the steps of the three-dimensional modeling method of the environmental depth image based on visible light are as follows:

[0024] a. Construct a corresponding environmental three-dimensional model according to the current environmental depth image;

[0025] b. Use three-dimensional object recognition technology to identify the object types of each three-dimensional object in the corresponding environmental three-dimensional model;

[0026] c. Obtain the distribution information of invasive plants in the current environmental three-dimensional model according to the identified object types.

[0027] Further, the image preprocessing algorithm of the image preprocessing algorithm module uses the deep learning YOLO V3 model network.

[0028] Further, the YOLO V3 algorithm model includes the following steps:

[0029] a. Adopt a 53-layer convolutional neural network model structure;

[0030] b. Use the K-means algorithm to obtain the prior box scale;

[0031] c. Use the output of logistic to predict the object category.

[0032] On the basis of conforming to the common knowledge in the art, the above preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0033] The positive and progressive effects of the present invention are as follows:

[0034] The proposed visual recognition system for invasive plants on the river and lake surface can identify and salvage more invasive plants on the water surface in the same time, with lower cost and stronger safety compared with traditional manual recognition and salvage. Compared with other recognition software on the market, it reduces the use of computing resources and can save the space occupied by the system hardware on the unmanned salvage ship.

[0035] It has a simple structure, and its main core consists of a visual recognition system for invasive plants on the water surface and a salvage device. The implementation cost is low and the feasibility is strong.

[0036] The whole adopts an automated design, with little interference from human factors, ensuring the stability, accuracy and continuity of the device.

[0037] It has strong scalability. As the model is continuously trained and refined, the applicable range of the device can be continuously expanded. It has strong reusability, is energy-saving and environmentally friendly.

[0038] The recognition is more accurate. It uses infrared radar for three-dimensional modeling to enhance the recognition of invasive plants and increase its recognition efficiency.

[0039] The algorithm has excellent efficiency. It uses the YOLOv3 model network based on deep learning. Compared with the original YOLOv3 training model, the hit rate and recognition time are better. Brief Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application.

[0041] Figure 1 It is a schematic diagram of the system architecture of the visual recognition system of the present invention.

[0042] Figure 2 It is a schematic diagram of the working principle of the visual recognition system of the present invention.

[0043] Figure 3 It is a schematic diagram of the recognition and judgment of invasive plants by the visual recognition system of the present invention.

[0044] Explanation of Reference Numerals in the Drawings

[0045] 1. Visual recognition system module; 2. Invasive plant dataset training module; 3. Invasive plant video image acquisition module; 4. Modeling method module; 5. Image preprocessing algorithm module. Detailed Embodiments

[0046] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0047] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0048] As Figures 1-3 shown, the visual recognition system includes:

[0049] The visual recognition system module 1, and the visual recognition system module 1 is connected to the invasive plant video image acquisition module 3;

[0050] The invasive plant dataset training module 2, and the invasive plant dataset training module 2 is connected to the invasive plant video image acquisition module 3;

[0051] The modeling method module 4, the modeling method module 4 is connected to the invasive plant dataset training module 2, and the modeling method module 4 is connected to the image preprocessing algorithm module 5.

[0052] Furthermore, the recognition method of the visual recognition system module 1 includes the following steps:

[0053] S1. Collect the water surface invasive plant dataset and train to obtain training samples;

[0054] S2. The unmanned salvage ship enters the designated area, collects the test set through the camera, and the unmanned salvage ship recognition system judges whether the shooting target is an invasive plant by testing the test set;

[0055] S3. If it is judged to be an invasive plant, the unmanned salvage ship conducts salvage; otherwise, continue to collect the test set.

[0056] Furthermore, the implementation method of the visual recognition system module 1 is to train the collected water surface invasive plant dataset to obtain a training set, and put the test set collected by the camera into the software system for running and recognition.

[0057] Furthermore, the invasive plant dataset includes: videos of different types of invasive plants during the day, foggy days, rainy days, and nights from the perspective of drones, and videos of different types of invasive plants during the day, foggy days, rainy days, and nights from the cameras of unmanned salvage ships. Based on the invasive plant dataset, training sets, validation sets, and test sets are obtained;

[0058] The steps for obtaining the training set based on the invasive plant dataset include the following:

[0059] a. Mark and magnify the features of invasive plants in the video from the perspective of the unmanned salvage ship collected;

[0060] b. Frame the video from the perspective of the unmanned salvage ship with the marked and magnified features;

[0061] c. Perform invasive plant annotation and reverse annotation on the framed photos to obtain the training set based on the invasive plant dataset.

[0062] The water surface invasive plant visual recognition system module 1 includes a camera, an infrared detection radar, a GPS, a salvage device, and a six-axis attitude sensor, and the camera, infrared detection radar, GPS, salvage device, and six-axis attitude sensor all have waterproof functions.

[0063] Furthermore, the visual recognition system uses a large number of data images for training and recognition, collects images at interval frames or frame by frame for recognition, and uses a convolutional neural network to collect feature vectors, reducing the use of computing resources.

[0064] The modeling method of the modeling method module 4 includes: on the basis of the three-dimensional modeling method of the environmental depth image based on visible light, adding an infrared radar for more detailed modeling to prevent the influence of brightness factors on modeling.

[0065] The steps of the three-dimensional modeling method of the environmental depth image based on visible light are as follows:

[0066] a. Construct a corresponding environmental three-dimensional model according to the current environmental depth image;

[0067] b. Use three-dimensional object recognition technology to identify the object types of each three-dimensional object in the corresponding environmental three-dimensional model;

[0068] c. According to the identified object types, obtain the distribution information of invasive plants in the current environmental three-dimensional model.

[0069] Furthermore, the image preprocessing algorithm of the image preprocessing algorithm module 5 uses the deep learning YOLO V3 model network.

[0070] The YOLO V3 algorithm model includes the following steps:

[0071] a. Adopt a 53-layer convolutional neural network model structure;

[0072] b. Use the K-means algorithm to obtain the prior box scale;

[0073] c. Use the output of logistic to predict the object category.

[0074] In this implementation plan, the device uses a camera to collect a water surface invasive plant data set in the invaded water area and trains to obtain training samples.

[0075] Specifically, in the training process, unzip the collected data set of water surface invasive species to the yolo project folder, divide the data set into a training set, a validation set, and a test set, and continuously train to obtain a model. Test the accuracy of the model for the validation set. If the accuracy is too low, add the data set of water surface invasive plants again for training until the accuracy of the obtained model meets the requirement for the machine to recognize water surface invasive plants in the invaded water area.

[0076] For determining whether the photographed target is an invasive plant, there are mainly the following steps:

[0077] Step 1: The unmanned salvage ship enters the route and the camera takes pictures.

[0078] Step 2: Image preprocessing.

[0079] Step 3: Determine whether it is an invasive species.

[0080] The main steps of image preprocessing include:

[0081] Normalize the captured image, perform network partitioning on it to form S×S grids. Each target has 5 parameters (x, y, w, h, score). The predicted target bounding box width and height are (w, h), the predicted coordinates of the center point of the target bounding box are (x, y), and the predicted confidence score of the target bounding box is (score);

[0082] Suppose A models of alien invasive plants are trained, and determine the model probability P, P(Class_i│object), (i = 1, 2,..., A) to which the target in the captured image belongs;

[0083] The score reflects the accuracy and possibility of the predicted target bounding box containing the target: score = P(object)×IOU, where P(object) is the possibility that there is a target within the predicted target bounding box. P(object) = 1 indicates that the target center is contained, otherwise P(object) = 0. IOU (Intersection Over Union) is the intersection of the predicted and the true target bounding boxes, which can reflect whether the position of the predicted target bounding box is accurate.

[0084] By calculating the product between the target class probability and the predicted target bounding box confidence score, obtain the class confidence score Class_i_score: Class_i_score = score×P(Class_i│object) = P(object)×IOU×P(Class_i│object) = P(Class_i)×IOU, (I = 1, 2,..., A);

[0085] Compare the specified threshold with Class_i_score. If it is greater than the specified threshold, retain the bounding box of the predicted target at this time. If it is less than the specified threshold, discard the bounding box of the predicted target at this time;

[0086] Output the model of the final target;

[0087] The device preprocesses the captured target images in the invaded water area and determines whether it is an invasive plant according to the trained model.

[0088] Specifically, the image preprocessing algorithm normalizes the captured image, divides the network thereof, determines the model probability to which the target in the captured image belongs, and the predicted target bounding box reflects the accuracy and possibility of the target. By calculating the product between the target class probability and the confidence score of the predicted target bounding box, the class confidence score is obtained. The specified threshold is compared with the class confidence score. If it is greater than the specified threshold, the bounding box of the predicted target is retained at this time. If it is less than the specified threshold, the bounding box of the predicted target is discarded and the model of the final target is output.

[0089] In summary, the visual recognition system of the unmanned salvage ship for cleaning invasive plants on the river and lake water surface can accurately identify the invasive plants on the river and lake water surface, and assist the unmanned salvage ship to achieve high energy efficiency, low power consumption and stable operation.

[0090] The circuits, electronic components and modules involved are all prior arts, which can be fully realized by those skilled in the art without further elaboration. The content protected by this application does not involve the improvement of software and methods either.

[0091] The present invention is not limited to the above embodiments. No matter any changes are made in its shape or structure, they all fall within the protection scope of the present invention. The protection scope of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principle and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A visual recognition system for an unmanned salvage vessel for invasive plants on the surface of rivers and lakes, characterized in that: The visual recognition system comprises: A visual recognition system module (1), wherein the visual recognition system module (1) is connected to an invasive plant video image acquisition module (3); An invasive plant data set training module (2), wherein the invasive plant data set training module (2) is connected to an invasive plant video image acquisition module (3); A modeling method module (4), the modeling method module (4) is connected to the invasive plant data set training module (2), and the modeling method module (4) is connected to the image preprocessing algorithm module (5).

2. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The recognition method of the visual recognition system module (1) comprises the following steps: S1. Collect the water surface invasive plant data set and train to obtain training samples; S2, the unmanned salvage ship enters the designated area, collects test sets through the camera, and the unmanned salvage ship recognition system determines whether the photographed target is an invasive plant by testing the test set; S3. If it is judged to be an invasive plant, the unmanned salvage vessel will salvage it; otherwise, continue to collect the test set.

3. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The implementation method of the visual recognition system module (1) is to train the collected water surface invasive plant data set to obtain a training set, and put the test set collected by the camera into the software system for operation and recognition.

4. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 2, characterized in that: The invasive plant dataset includes: different types of invasive plants in the daytime, foggy, rainy and nighttime from the perspective of a drone, and different types of invasive plant videos in the daytime, foggy, rainy and nighttime from the camera of an unmanned salvage ship. A training set, a validation set and a test set are obtained based on the training of the invasive plant dataset.

5. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The water surface invasive plant visual identification system module (1) comprises a camera, an infrared detection radar, a GPS, a salvage device, and a six-axis attitude sensor, and the camera, the infrared detection radar, the GPS, the salvage device, and the six-axis attitude sensor are all waterproof.

6. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The visual recognition system uses a large amount of data images for training and recognition, uses interval frame acquisition or frame-by-frame acquisition for recognition, and uses a convolutional neural network to collect feature vectors, thereby reducing the use of computing resources.

7. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The modeling method of the modeling method module (4) includes: based on the three-dimensional modeling method of the environment depth image based on visible light, more detailed modeling is performed by adding infrared radar to prevent the influence of brightness factors on modeling.

8. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 7, characterized in that: The steps of the three-dimensional modeling method of the environment depth image based on visible light are as follows: a. Construct a corresponding three-dimensional model of the environment based on the current environment depth image; b. Using 3D object recognition technology to identify the object type of each 3D object in the 3D model of the corresponding environment; c. According to the identified object type, obtain the distribution information of the invasive plants in the current three-dimensional model of the environment.

9. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 1, characterized in that: The image preprocessing algorithm of the image preprocessing algorithm module (5) adopts the deep learning YOLO V3 model network.

10. The visual recognition system for an unmanned salvage vessel for invasive plants on river and lake surfaces as claimed in claim 9, characterized in that: The YOLO V3 algorithm model includes the following steps: a. Adopt 53-layer convolutional neural network model structure; b. Use K-means algorithm to obtain the prior frame scale; c. Use the output of logistic to predict the object category.

Images