Real-time monitoring and identification method of three-dimensional enteromorpha based on ABA technology

Abstract

The invention discloses a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology, which belongs to the field of marine environment monitoring, and specifically comprises: obtaining marine environment data, images and video data of enteromorpha areas, and performing preprocessing to generate an image set; extracting feature points by using a pre-trained deep learning model, and obtaining a matching point pair set by using a feature matching algorithm; based on the matching point pair set, generating a preliminary three-dimensional point cloud model by using a camera posture estimation method and a three-dimensional point reconstruction algorithm, and optimizing the optimal three-dimensional point cloud model by using an affine bundle adjustment method; calculating vegetation index NDVI, NR-B and NDWI values ​​based on the optimal three-dimensional point cloud model, and extracting and classifying enteromorpha areas; and improving the accuracy, efficiency and real-time performance of enteromorpha monitoring.

Description

Technical Field

[0001] The invention belongs to the field of marine environment monitoring, and specifically is a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology. Background Art

[0002] Traditional methods of monitoring Ulva mainly rely on manual patrols and satellite remote sensing. Although manual patrols are intuitive, they have problems such as limited monitoring range, low efficiency and high cost. Although satellite remote sensing can cover a large area of ​​sea, it is limited by resolution and weather conditions, making it difficult to accurately identify and dynamically monitor Ulva.

[0003] For example, the patent with the authorization announcement number CN112215857B discloses a method for identifying Enteromorpha based on the growth cycle of Enteromorpha, including: obtaining remote sensing image data, performing cloud recognition processing on low spatial resolution multi-band data, obtaining cloud mask shp data, and using the cloud mask shp data to perform cloud masking on high spatial resolution land mask images; obtaining NDVI images of the Enteromorpha area to be identified based on NDVI data; dividing the Enteromorpha area to be identified into several partitions, obtaining regional multi-band data and regional NDVI images; identifying the Enteromorpha area to be identified based on the regional multi-band data and regional NDVI images; extracting regional Enteromorpha shp data of the Enteromorpha area to be identified based on the regional Enteromorpha pixel decomposition results. This technical solution integrates regional multi-band data and regional NDVI images and obtains regional Enteromorpha pixel decomposition results in combination with the Enteromorpha growth cycle, and uses the characteristics of different growth cycles of Enteromorpha to accurately identify Enteromorpha, which can break through the image resolution and obtain a more accurate Enteromorpha coverage area.

[0004] For example, the patent with publication number CN116523897A discloses a semi-supervised enteromorpha detection method and system based on mutual learning, including: data preprocessing and labeling; building a convolutional neural network model; performing mutual learning of the network model; using two trained networks to detect enteromorpha and weighted fusion of the results. This technical solution aims to solve the problem of small amount of labeled data for enteromorpha detection in remote sensing images in supervised classification. By designing a high-confidence pseudo-label strategy combined with data enhancement, it can efficiently utilize unlabeled data, expand sample data sets, improve the accuracy of training models, and achieve high-quality enteromorpha detection in remote sensing images, providing technical support for disaster monitoring during large-scale enteromorpha outbreaks.

[0005] The above existing technologies all have the following problems: due to the inherent limitations of remote sensing data and the accumulation of errors in the processing process, the accuracy of the final result is still limited, resulting in low monitoring accuracy and efficiency; there is a lack of adaptability and dynamic monitoring capabilities for the changes of Ulva in different sea areas, different seasons or different environmental conditions; there is a lack of three-dimensional spatial information. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology, which obtains marine environmental data, images and video data of enteromorpha areas, and performs preprocessing to generate an image set; uses a pre-trained deep learning model to extract feature points, and obtains a matching point pair set through a feature matching algorithm; based on the matching point pair set, a preliminary three-dimensional point cloud model is generated using a camera posture estimation and a three-dimensional point reconstruction algorithm, and the optimal three-dimensional point cloud model is obtained through the ABA optimization algorithm; the vegetation index NDVI, NR-B and NDWI values ​​are calculated based on the optimal three-dimensional point cloud model, and the enteromorpha areas are extracted and classified; the accuracy, efficiency and real-time performance of enteromorpha monitoring are improved.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The real-time monitoring and identification method of three-dimensional Enteromorpha based on ABA technology includes:

[0009] Step S1: Acquire marine environment data, images and video data of the Enteromorpha area, and perform preprocessing to generate an Enteromorpha area image set;

[0010] Step S2: for different Enteromorpha region images, the pre-trained deep learning model is used to automatically extract feature points of each Enteromorpha region image, and a feature matching algorithm is used to perform feature matching on the feature points to obtain a set of matching point pairs;

[0011] Step S3: Based on the set of matching point pairs, an initial camera pose estimation method and a 3D point reconstruction algorithm are used to generate a preliminary 3D point cloud model, and the preliminary 3D point cloud model is optimized using an ABA optimization algorithm to obtain an optimal 3D point cloud model;

[0012] Step S4: Calculate the vegetation index NDVI, NR-B and NDWI values ​​of each pixel from the Enteromorpha data in the optimal three-dimensional point cloud model, extract the Enteromorpha area according to the NDVI, NR-B and NDWI values, and classify the Enteromorpha in the extracted Enteromorpha area;

[0013] The specific steps of step S3 include:

[0014] S3.1: Obtaining a set of matching point pairs ,from Select at least 8 pairs of points and construct a linear equation system based on the eight-point method principle. , by solving the linear equations , get the initial solution of the basic matrix, where A represents the matrix constructed by matching point pairs, and f represents the vector of the basic matrix;

[0015] S3.2: Generate an initial basic matrix based on the initial solution of the basic matrix, use the SVD algorithm to enforce the rank 2 constraint on the initial basic matrix, and obtain the Enteromorpha area image and The fundamental matrix between

[0016] S3.3: Based on the basic matrix and the known camera intrinsic parameters, the coordinates of the three-dimensional points are recovered from the matching point pairs of the g Enteromorpha area images using the triangulation method, and the coordinates of the three-dimensional points obtained by triangulation are combined into a preliminary three-dimensional point cloud model, where: .

[0017] The specific steps of step S3 also include:

[0018] S3.4: Based on the preliminary 3D point cloud model and the feature points of the Enteromorpha region image, an optimization problem is constructed and the ABA optimization algorithm is used to solve the optimization problem. The formula is:

[0019] ;

[0020] Where E represents the reprojection error, represents the robust loss function, represents the observed image point of the Enteromorpha region, u represents the horizontal coordinate index of the observed Enteromorpha region image, v represents the vertical coordinate index of the observed Enteromorpha region image, t represents the time index, represents the projection function of the prediction model, represents the camera intrinsic parameter matrix, represents the camera posture, represents the coordinates of a three-dimensional point, represents the camera distortion parameters, Indicates the time offset, , represents the regularization parameter, Indicates the change of camera internal parameters, Indicates the change of the position of a 3D point. represents the norm of *;

[0021] S3.5: By comparing the reprojection error before and after optimization and Evaluate the optimization effect and obtain the optimal 3D point cloud model based on the evaluation results.

[0022] The specific steps of S3.3 include:

[0023] S3.31: Get the fundamental matrix in step S3.2, for each , by calculating , convert the two-dimensional image coordinates of the Enteromorpha region image into rays in three-dimensional space, where, Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on represents the fundamental matrix, represents a non-zero constant;

[0024] S3.32: Use the least squares method to solve the intersection of the rays, obtain the coordinates of the three-dimensional point, and store the obtained three-dimensional point in a data structure.

[0025] Specifically, the specific steps of step S2 include:

[0026] S2.1: Acquisition of an image set of the Enteromorpha region , and load the pre-trained deep learning model, where represents the Nth enteromorpha region image in the enteromorpha region image set, where N represents the number of enteromorpha region images;

[0027] S2.2: Use the pre-trained deep learning model to extract feature points from each Enteromorpha region image, and use the feature matching algorithm to match the feature points extracted from different Enteromorpha region images. The formula of the feature matching algorithm is:

[0028] ;

[0029] in, represents the matching result of the nth Enteromorpha region image and the mth Enteromorpha region image based on similarity, represents the set of feature points in the nth Enteromorpha region image, represents the set of feature points in the mth Enteromorpha region image, represents the similarity between the nth Enteromorpha region image and the mth Enteromorpha region image, express The i-th feature point in express Zhongyu Matching feature points, express The feature descriptor of express The feature descriptor of express and The similarity function of Indicated in and Satisfied The feature point pairs, represents conditional probability.

[0030] Specifically, the specific steps of step S2 also include:

[0031] S2.3: Set a dynamic matching threshold based on the characteristics of the acquired Enteromorpha region image set ,

[0032] like , then The corresponding feature points are discarded;

[0033] like , then The corresponding feature points are used as the preliminary matching point pair set ,in, represents the bth preliminary matching point pair, and b represents the number of preliminary matching point pairs;

[0034] S2.4: Apply the random sampling consistency strategy to optimize the preliminary matching point pair set. The formula is:

[0035] ;

[0036] in, It represents the ratio of the nearest neighbor distance to the next nearest neighbor distance of the kth preliminary matching point. Represents the initial matching point pair The distance to its nearest neighbor feature point, Represents the initial matching point pair The distance to the next nearest feature point, represents the weight of the kth preliminary matching point pair, Indicates the offset of the kth preliminary matching point pair;

[0037] S2.5: Setting the ratio threshold ,

[0038] like , then the preliminary matching point pair is discarded;

[0039] like , then the preliminary matching point pair Keep and generate matching point pair sets ,in, represents the cth matching point pair, where c represents the number of matching point pairs.

[0040] Specifically, the specific steps of S4 include:

[0041] S4.1: Extracting Enteromorpha data from the optimal 3D point cloud model;

[0042] S4.2: Read the reflectance values ​​of the near-infrared band and the red light band of each pixel in the Enteromorpha data in the optimal 3D point cloud model and ,based on and , get the NDVI value of each pixel ;

[0043] S4.3: Read the reflectance value of the red edge band of each pixel in the optimal 3D point cloud model ,based on and , get the NR-B value of each pixel ;

[0044] S4.4: Read the reflectance value of the green light band of each pixel in the optimal 3D point cloud model and the reflectance values ​​in the short-wave infrared band ,based on and , get the NDWI value of each pixel .

[0045] Specifically, the specific steps of S4 also include:

[0046] S4.5: Set the NDVI value ;

[0047] like , then it is the preliminary Enteromorpha area;

[0048] like , then it is a water body and non-vegetated area;

[0049] S4.6: Obtain R, G, and B values ​​of the Enteromorpha data in the optimal three-dimensional point cloud model based on the preliminary Enteromorpha region;

[0050] like , then the corresponding pixels in the preliminary Enteromorpha area are blue, red and other artificial features;

[0051] like , then the corresponding pixel in the preliminary Enteromorpha area is Enteromorpha, and the secondary Enteromorpha area is obtained, where Indicates the pixel value of channel B of the Enteromorpha data in the optimal 3D point cloud model;

[0052] S4.7: Set the threshold of NDWI value to , based on the secondary stage Enteromorpha area;

[0053] like , then the corresponding pixel in the Enteromorpha area in the next stage is the water body;

[0054] like , then the corresponding pixel in the next stage Enteromorpha area is Enteromorpha, and the final Enteromorpha area is obtained;

[0055] S4.8: Based on the final Enteromorpha area;

[0056] like , then the corresponding pixel in the final Enteromorpha area is Sargassum;

[0057] like , then the corresponding pixel in the final Enteromorpha area is green algae, where Represents the G channel pixel value of the corresponding pixel in the final Enteromorpha area;

[0058] like or , then the corresponding pixel in the final Enteromorpha area is Porphyra.

[0059] Specifically, the goal of the optimization problem in S3.4 is to minimize the reprojection error, which refers to the difference between the projection of a three-dimensional point to its corresponding image point and the actual observed image point.

[0060] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology when executing the computer program.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention proposes a three-dimensional real-time monitoring and identification method for Enteromorpha based on ABA technology. Through automated feature extraction, feature matching, three-dimensional reconstruction and image recognition methods, not only a fine three-dimensional Enteromorpha model is constructed, but also the growth trend of Enteromorpha is predicted by time series analysis. Once the number of Enteromorpha is detected to exceed a preset threshold, the system can quickly issue an early warning message, providing reaction time and taking timely measures to control the spread of Enteromorpha.

[0063] 2. The present invention proposes a three-dimensional Enteromorpha real-time monitoring and identification method based on ABA technology. By collecting new Enteromorpha area images in real time and repeating three-dimensional reconstruction and Enteromorpha identification, the method can continuously update the distribution of Enteromorpha and monitoring reports, ensuring the real-time and accuracy of monitoring data and improving monitoring accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology of the present invention;

[0065] Figure 2 It is a principle flow chart of the three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology of the present invention;

[0066] Figure 3 A flowchart of obtaining matching point pairs of a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology of the present invention;

[0067] Figure 4 This is a flowchart for realizing the optimal three-dimensional point cloud model of the three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology of the present invention. DETAILED DESCRIPTION

[0068] Example 1

[0069] See also Figure 1 and Figure 2 The present invention provides an embodiment: a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology, wherein ABA refers to Adaptive Bundle Adjustment, i.e., adaptive bundle adjustment method, comprising the following steps:

[0070] Step S1: Acquire marine environment data, images and video data of the Enteromorpha area, and perform preprocessing to generate an Enteromorpha area image set;

[0071] Among them, the types of Enteromorpha include: green algae, Sargassum, and laver.

[0072] Furthermore, the specific steps of step S1 are summarized as follows: 1) using a digital aerial camera dedicated to surveying and mapping, such as the Ultracam Eagle Mark 3 produced by the Austrian company Vexcel, to collect spectra. The aerial camera has four spectral channels of R, G, B, and NIR, and can record information in the visible light and near-infrared bands. It flies at an altitude of 2000 meters, collects data with a resolution of 10 cm, and covers a ground width of 2646 meters; 2) integrating marine environmental data, such as temperature, salinity, and ocean current speed; 3) pre-processing the marine environmental data and the collected images and video data, including correcting for air vapor and light interference, and performing necessary visual and automated interpretation to generate an image set of the Ulva area.

[0073] Step S2: for different Enteromorpha region images, the pre-trained deep learning model is used to automatically extract feature points of each Enteromorpha region image, and a feature matching algorithm is used to perform feature matching on the feature points to obtain a set of matching point pairs;

[0074] Step S3: Based on the set of matching point pairs, an initial camera pose estimation method and a 3D point reconstruction algorithm are used to generate a preliminary 3D point cloud model, and the preliminary 3D point cloud model is optimized using an ABA optimization algorithm to obtain an optimal 3D point cloud model;

[0075] Step S4: Calculate the vegetation index NDVI, NR-B and NDWI values ​​of each pixel from the Enteromorpha data in the optimal three-dimensional point cloud model, extract the Enteromorpha area according to the NDVI, NR-B and NDWI values, and classify the Enteromorpha in the extracted Enteromorpha area.

[0076] Example 2

[0077] See also Figure 3 and Figure 4 In this embodiment, the specific steps of step S2 include:

[0078] S2.1: Acquisition of an image set of the Enteromorpha region , and load the pre-trained deep learning model, where represents the Nth enteromorpha region image in the enteromorpha region image set, and N represents the number of enteromorpha region images. In the present invention, the deep learning model selects a deep learning model based on SURF, and the training process of the deep learning model based on SURF is the prior art content in this field, which is not the inventive solution of the present application and is not described in detail here;

[0079] S2.2: Use the pre-trained deep learning model to extract feature points from each Enteromorpha region image, and use the feature matching algorithm to match the feature points extracted from different Enteromorpha region images. The formula of the feature matching algorithm is:

[0080] ;

[0081] in, represents the matching result of the nth Enteromorpha region image and the mth Enteromorpha region image based on similarity, represents the set of feature points in the nth Enteromorpha region image, represents the set of feature points in the mth Enteromorpha region image, represents the similarity between the nth Enteromorpha region image and the mth Enteromorpha region image, express The i-th feature point in express Zhongyu Matching feature points, express The feature descriptor of express The feature descriptor of express and The similarity function of Indicated in and Satisfied The feature point pairs, represents conditional probability;

[0082] Furthermore, in the present invention, on the one hand, the training process and feature extraction process of the deep learning model are the prior art content in this field, not the inventive solution of the present application, and will not be described in detail here; on the other hand, in the present invention, the similarity measurement is calculated using Euclidean distance, and the calculation process of the Euclidean distance formula is the prior art content in this field, not the inventive solution of the present application, and will not be described in detail here.

[0083] S2.3: Set a dynamic matching threshold based on the characteristics of the acquired Enteromorpha region image set ,

[0084] like , then The corresponding feature points are discarded;

[0085] like , then The corresponding feature points are used as the preliminary matching point pair set ,in, represents the bth preliminary matching point pair, and b represents the number of preliminary matching point pairs;

[0086] S2.4: Apply the random sampling consistency strategy to optimize the preliminary matching point pair set. The formula is:

[0087] ;

[0088] in, It represents the ratio of the nearest neighbor distance to the next nearest neighbor distance of the kth preliminary matching point. Represents the initial matching point pair The distance to its nearest neighbor feature point, Represents the initial matching point pair The distance to the next nearest feature point, represents the weight of the kth preliminary matching point pair, which is used to adjust the importance of the nearest neighbor distance in the ratio calculation, and , Indicates the offset of the kth preliminary matching point pair, which is used to prevent the denominator from being zero and adjust the sensitivity of the ratio test;

[0089] It should be noted that, based on the prior art, the present invention adds , And dynamic adjustability, which can adjust the importance of the distance ratio according to the similarity of feature descriptors or local features around feature points. In addition, the introduced dynamic matching threshold can change according to the overall matching quality of the Ulva region image or the characteristics of a specific region.

[0090] Furthermore, for each pair of matching points in the preliminary matching point pair set, when calculating the distance between the matching point and its nearest neighbor and second nearest neighbor feature point, it should be clarified that the nearest neighbor distance refers to the distance between the matching point and the nearest feature point in the matching image, and the second nearest neighbor distance refers to the distance between the matching point and the second nearest feature point, and the calculation formulas for the nearest neighbor distance and the second nearest neighbor distance are the prior art contents in the field, are not the inventive solutions of the present application, and are not elaborated here.

[0091] S2.5: Setting the ratio threshold , which is used to judge the reliability of the matching point. ;

[0092] like , then the preliminary matching point pair is discarded;

[0093] like , then the preliminary matching point pair Keep and generate matching point pair sets ,in, represents the cth matching point pair, where c represents the number of matching point pairs.

[0094] The specific steps of step S3 include:

[0095] S3.1: Obtaining a set of matching point pairs ,from Select at least 8 pairs of points and construct a linear equation system based on the eight-point method principle. , by solving the linear equations , obtain the initial solution of the basic matrix, where A represents the matrix constructed by the matching point pairs, and f represents the vector of the basic matrix. The specific solution process of the eight-point method is the prior art content in this field, which is not the inventive solution of the present application and will not be described in detail here;

[0096] S3.2: Generate an initial basic matrix based on the initial solution of the basic matrix, use the SVD algorithm to enforce the rank 2 constraint on the initial basic matrix, and obtain the Enteromorpha area image and The fundamental matrix between

[0097] Furthermore, the specific steps of S3.2 include:

[0098] S3.21: Obtain an estimate of the initial fundamental matrix F calculated by the eight-point method in S3.1;

[0099] S3.22: Perform SVD decomposition on the calculated initial basic matrix F , where U and V represent orthogonal matrices, represents a diagonal matrix, the elements on the diagonal are singular values ​​and are arranged from large to small, T represents transpose, it should be noted that the solution process of the SVD algorithm is the prior art content in this field, is not the inventive solution of the present application, and is not described in detail here;

[0100] S3.23: Since the rank of the basic matrix should theoretically be 2, the singular value diagonal matrix obtained after SVD decomposition is Perform a low-rank approximation, retaining only the first two singular values ​​and setting the rest to 0, thus obtaining a new diagonal matrix ;

[0101] S3.24: Modified diagonal matrix Will be used to reconstruct the base matrix .

[0102] S3.3: Based on the basic matrix and the known camera intrinsic parameters, the coordinates of the three-dimensional points are recovered from the matching point pairs of the g Enteromorpha area images using the triangulation method, and the coordinates of the three-dimensional points obtained by triangulation are combined into a preliminary three-dimensional point cloud model, where: , and the triangulation method is the prior art content in this field, and is not the inventive solution of this application, and will not be described in detail here;

[0103] S3.4: Based on the preliminary 3D point cloud model and the feature points of the Enteromorpha region image, an optimization problem is constructed and the ABA optimization algorithm is used to solve the optimization problem. The formula is:

[0104] ;

[0105] Where E represents the reprojection error, represents the robust loss function, represents the observed image point of the Enteromorpha region, u represents the horizontal coordinate index of the observed Enteromorpha region image, v represents the vertical coordinate index of the observed Enteromorpha region image, t represents the time index, represents the projection function of the prediction model, represents the camera intrinsic parameter matrix, represents the camera posture, represents the coordinates of a three-dimensional point, Represents the camera distortion parameters, which are used to model the radial and tangential distortion of the camera lens, thereby improving the accuracy of 3D reconstruction. Indicates the time offset, , represents the regularization parameter, Indicates the change of camera internal parameters, Indicates the change of the position of a 3D point. represents the norm of *;

[0106] Furthermore, the goal of the optimization problem in S3.4 is to minimize the reprojection error, which refers to the difference between the projection of a 3D point to its corresponding image point and the actual observed image point.

[0107] It should be noted that in the present invention, the optimization of multiple key variables is comprehensively considered, including the use of a prediction model The adjusted error, , The smoothness of time changes is improved by introducing a regularization term , It not only ensures the prediction accuracy, but also ensures the stability and interpretability of the model by limiting the change rate of key parameters, avoids the overfitting problem, and makes the optimization results more consistent with physical reality.

[0108] S3.5: By comparing the reprojection error before and after optimization and Evaluate the optimization effect and obtain the optimal 3D point cloud model based on the evaluation results.

[0109] The specific steps of S3.3 include:

[0110] S3.31: Get the fundamental matrix in step S3.2, for each , by calculating , convert the two-dimensional image coordinates of the Enteromorpha region image into rays in three-dimensional space, where, Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on represents the fundamental matrix, represents a non-zero constant;

[0111] Furthermore, the specific steps of converting the two-dimensional image coordinates of the enteromorpha region image into rays in three-dimensional space include:

[0112] (1) Data preparation:

[0113] A pair of matching points is known , located on two Enteromorpha region images, respectively, and are denoted as and , the intrinsic parameter matrices of the two cameras are and , the fundamental matrix F between the views of the two Enteromorpha region images, where Indicates the horizontal coordinate index position of the matching point on the first Enteromorpha region image, Indicates the ordinate index position of the matching point on the first Enteromorpha region image. Indicates the horizontal coordinate index position of the matching point on the first Enteromorpha region image, Indicates the ordinate index position of the matching point on the first Enteromorpha region image;

[0114] (2) Normalized image coordinates:

[0115] The image coordinates and Convert to normalized image coordinates and ,in, , , , , and is the camera intrinsic parameter matrix and The focal length component in Indicated in The horizontal coordinate of a point in the two-dimensional space, Indicated in The horizontal coordinate of a point in the two-dimensional space, Indicated in The ordinate of a point in two-dimensional space, Indicated in The ordinate of a point in the upper two-dimensional space;

[0116] (3) Using the basic matrix:

[0117] according to The nature of In On the ray with direction

[0118] (4) Back-projection through camera internal parameters:

[0119] Use the camera intrinsic parameters to back-project the 2D point onto a ray in the camera coordinate system.

[0120] S3.32: Use the least squares method to solve the intersection of the rays, obtain the coordinates of the three-dimensional points, and store the obtained three-dimensional points in a data structure, wherein the solution process of the least squares method is the prior art content in this field, is not the creative solution of this application, and will not be elaborated here.

[0121] The specific steps of S4 include:

[0122] S4.1: Extracting Enteromorpha data from the optimal three-dimensional point cloud model using a classification algorithm based on a random forest, wherein the extraction process based on the classification algorithm based on a random forest is a prior art content in the art, and is not an inventive solution of the present application, and is not described in detail here;

[0123] S4.2: Read the reflectance values ​​of the near-infrared band and the red light band of each pixel in the Enteromorpha data in the optimal 3D point cloud model and ,based on and , get the NDVI value of each pixel ;

[0124] Among them, the Enteromorpha data extracted from the optimal three-dimensional point cloud model includes the three-dimensional coordinates of each pixel and the corresponding band reflectance value. The pixel value query function of ENVI software can be used to read the band reflectance value.

[0125] S4.3: Read the reflectance value of the red edge band of each pixel in the optimal 3D point cloud model ,based on and , get the NR-B value of each pixel ;

[0126] S4.4: Read the reflectance value of the green light band of each pixel in the optimal 3D point cloud model and the reflectance values ​​in the short-wave infrared band ,based on and , get the NDWI value of each pixel ;

[0127] S4.5: Set the NDVI value ;

[0128] like , then it is the preliminary Enteromorpha area;

[0129] like , then it is water bodies and other non-vegetated areas;

[0130] S4.6: Obtain R, G, and B values ​​of the Enteromorpha data in the optimal three-dimensional point cloud model based on the preliminary Enteromorpha region;

[0131] like , then the corresponding pixels in the preliminary Enteromorpha area are blue, red and other artificial features;

[0132] like , then the corresponding pixel in the preliminary Enteromorpha area is Enteromorpha, and the secondary Enteromorpha area is obtained, where Indicates the pixel value of channel B of the Enteromorpha data in the optimal 3D point cloud model;

[0133] S4.7: Set the threshold of NDWI value to , based on the secondary stage Enteromorpha area;

[0134] like , then the corresponding pixel in the Enteromorpha area in the next stage is the water body;

[0135] like , then the corresponding pixel in the next stage Enteromorpha area is Enteromorpha, and the final Enteromorpha area is obtained;

[0136] S4.8: Based on the final Enteromorpha area;

[0137] like , then the corresponding pixel in the final Enteromorpha area is Sargassum;

[0138] like , then the corresponding pixel in the final Enteromorpha area is green algae, where Represents the G channel pixel value of the corresponding pixel in the final Enteromorpha area;

[0139] like or , then the corresponding pixel in the final Enteromorpha area is Porphyra.

[0140] Example 3

[0141] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology are implemented. For details, please refer to the above method embodiment, which will not be repeated here.

[0142] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

Claims

1. A three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology, characterized in that: include: Step S1: Acquire marine environment data, images and video data of the Enteromorpha area, and perform preprocessing to generate an Enteromorpha area image set; Step S2: for different Enteromorpha region images, the pre-trained deep learning model is used to automatically extract feature points of each Enteromorpha region image, and a feature matching algorithm is used to perform feature matching on the feature points to obtain a set of matching point pairs; Step S3: Based on the set of matching point pairs, an initial camera pose estimation method and a 3D point reconstruction algorithm are used to generate a preliminary 3D point cloud model, and the preliminary 3D point cloud model is optimized using an ABA optimization algorithm to obtain an optimal 3D point cloud model; Step S4: Calculate the vegetation index NDVI, NR-B and NDWI values ​​of each pixel from the Enteromorpha data in the optimal three-dimensional point cloud model, extract the Enteromorpha area according to the NDVI, NR-B and NDWI values, and classify the Enteromorpha in the extracted Enteromorpha area; The specific steps of step S3 include: S3.1: Obtaining a set of matching point pairs ,from Select at least 8 pairs of points and construct a linear equation system based on the eight-point method principle. , by solving the linear equations , get the initial solution of the basic matrix, where A represents the matrix constructed by matching point pairs, and f represents the vector of the basic matrix. Represents the cth matching point pair; S3.2: Generate an initial basic matrix based on the initial solution of the basic matrix, use the SVD algorithm to enforce the rank 2 constraint on the initial basic matrix, and obtain the Enteromorpha area image and The fundamental matrix between S3.3: Based on the basic matrix and the known camera intrinsic parameters, the coordinates of the three-dimensional points are recovered from the matching point pairs of the g Enteromorpha area images using the triangulation method, and the coordinates of the three-dimensional points obtained by triangulation are combined into a preliminary three-dimensional point cloud model, where: , N represents the number of images of Enteromorpha region; The specific steps of step S3 also include: S3.4: Based on the preliminary 3D point cloud model and the feature points of the Enteromorpha region image, an optimization problem is constructed and the ABA optimization algorithm is used to solve the optimization problem. The formula is: ； Where E represents the reprojection error, represents the robust loss function, represents the observed image point of the Enteromorpha region, u represents the horizontal coordinate index of the observed Enteromorpha region image, v represents the vertical coordinate index of the observed Enteromorpha region image, t represents the time index, represents the projection function of the prediction model, represents the camera intrinsic parameter matrix, represents the camera posture, represents the coordinates of a three-dimensional point, represents the camera distortion parameters, Indicates the time offset, , represents the regularization parameter, Indicates the change of camera internal parameters, Indicates the change of the position of a 3D point. represents the norm of *; S3.5: By comparing the reprojection error before and after optimization and Evaluate the optimization effect and obtain the optimal 3D point cloud model based on the evaluation results. Represents the ratio of the nearest neighbor distance to the next nearest neighbor distance of the kth preliminary matching point; The specific steps of S3.3 include: S3.31: Get the fundamental matrix in step S3.2, for each , by calculating , convert the two-dimensional image coordinates of the Enteromorpha region image into rays in three-dimensional space, where, Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on Represents matching point pairs Normalized image in the Enteromorpha region The horizontal index position on represents the fundamental matrix, represents a non-zero constant; S3.32: Use the least squares method to solve the intersection of the rays, obtain the coordinates of the three-dimensional point, and store the obtained three-dimensional point in a data structure.

2. The three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology according to claim 1, characterized in that: The specific steps of step S2 include: S2.1: Acquisition of an image set of the Enteromorpha region , and load the pre-trained deep learning model, where represents the Nth enteromorpha region image in the enteromorpha region image set, where N represents the number of enteromorpha region images; S2.2: Use the pre-trained deep learning model to extract feature points from each Enteromorpha region image, and use the feature matching algorithm to match the feature points extracted from different Enteromorpha region images. The formula of the feature matching algorithm is: ； in, represents the matching result of the nth Enteromorpha region image and the mth Enteromorpha region image based on similarity, represents the set of feature points in the nth Enteromorpha region image, represents the set of feature points in the mth Enteromorpha region image, represents the similarity between the nth Enteromorpha region image and the mth Enteromorpha region image, express The i-th feature point in express Zhongyu Matching feature points, express The feature descriptor of express The feature descriptor of express and The similarity function of Indicated in and Satisfied The feature point pairs, represents conditional probability.

3. The three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology as claimed in claim 2, characterized in that: The specific steps of step S2 also include: S2.3: Set a dynamic matching threshold based on the characteristics of the acquired Enteromorpha region image set , like , then The corresponding feature points are discarded; like , then The corresponding feature points are used as the preliminary matching point pair set ,in, represents the bth preliminary matching point pair, and b represents the number of preliminary matching point pairs; S2.4: Apply the random sampling consistency strategy to optimize the preliminary matching point pair set. The formula is: ； in, It represents the ratio of the nearest neighbor distance to the next nearest neighbor distance of the kth preliminary matching point. Represents the initial matching point pair The distance to its nearest neighbor feature point, Represents the initial matching point pair The distance to the next nearest feature point, represents the weight of the kth preliminary matching point pair, Indicates the offset of the kth preliminary matching point pair; S2.5: Setting the ratio threshold , like , then the preliminary matching point pair is discarded; like , then the preliminary matching point pair Keep and generate matching point pair sets ,in, represents the cth matching point pair, where c represents the number of matching point pairs.

4. The three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology as claimed in claim 3, characterized in that: The specific steps of S4 include: S4.1: Extracting Enteromorpha data from the optimal 3D point cloud model; S4.2: Read the reflectance values ​​of the near-infrared band and the red light band of each pixel in the Enteromorpha data in the optimal 3D point cloud model and ,based on and , get the NDVI value of each pixel ; S4.3: Read the reflectance value of the red edge band of each pixel in the optimal 3D point cloud model ,based on and , get the NR-B value of each pixel ; S4.4: Read the reflectance value of the green light band of each pixel in the optimal 3D point cloud model and the reflectance values ​​in the short-wave infrared band ,based on and , get the NDWI value of each pixel .

5. The three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology as claimed in claim 4, characterized in that: The specific steps of S4 also include: S4.5: Set the NDVI value ; like , then it is the preliminary Enteromorpha area; like , then it is a water body and non-vegetated area; S4.6: Obtain R, G, and B values ​​of the Enteromorpha data in the optimal three-dimensional point cloud model based on the preliminary Enteromorpha region; like , then the corresponding pixels in the preliminary Enteromorpha area are blue and red artificial features; like , then the corresponding pixel in the preliminary Enteromorpha area is Enteromorpha, and the secondary Enteromorpha area is obtained, where Indicates the pixel value of channel B of the Enteromorpha data in the optimal 3D point cloud model; S4.7: Set the threshold of NDWI value to , based on the secondary stage Enteromorpha area; like , then the corresponding pixel in the Enteromorpha area in the next stage is the water body; like , then the corresponding pixel in the next stage Enteromorpha area is Enteromorpha, and the final Enteromorpha area is obtained; S4.8: Based on the final Enteromorpha area; like , then the corresponding pixel in the final Enteromorpha area is Sargassum; like , then the corresponding pixel in the final Enteromorpha area is green algae, where Represents the G channel pixel value of the corresponding pixel in the final Enteromorpha area; like or , then the corresponding pixel in the final Enteromorpha area is Porphyra.

6. The three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology as claimed in claim 5, characterized in that: The goal of the optimization problem in S3.4 is to minimize the reprojection error, which refers to the difference between the projection of a 3D point to its corresponding image point and the actual observed image point.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the three-dimensional enteromorpha real-time monitoring and identification method based on ABA technology described in any one of claims 1 to 6 are implemented.

Images