A method and system for checking and processing forestry area biological survey data

Abstract

The invention relates to the technical field of verification and processing, and specifically discloses a verification and processing method and system for biological survey data in a forestry area, comprising the following steps: step S1: obtaining all biological targets in a forestry area, and dividing the biological targets into several categories; step S2: for any group of biological survey data, dividing the biological survey data in the group into several clusters through cluster analysis, and judging whether outliers are missing; if outliers are missing, they are recorded as abnormal values; step S3: obtaining deviation vectors of outliers and coordinates of various central values, and selecting the coordinates of the central value corresponding to the deviation vector with the shortest length, which are recorded as deviations from the origin; obtaining various deviation vectors with the same deviation from the origin, setting a standard vector, and obtaining the angle between each deviation vector and the standard vector; performing secondary cluster analysis on the angle sequence to obtain several deviation clusters; obtaining outliers corresponding to deviation vectors that do not belong to various deviation clusters, and recording them as zero values.

Description

Technical Field

[0001] The invention relates to the technical field of verification processing, and in particular to a verification processing method and system for biological survey data in forestry areas. Background Art

[0002] Biological survey data generally come from different organisms, experimental conditions, and technical platforms, among which the data of organisms often contain a large number of features and a large number of zero counts; this makes biological survey data diverse, high-dimensional, large in volume, and sparse.

[0003] Verification processing is a process of checking and examining data or records to ensure their accuracy and reliability. Verification processing generally includes data accuracy verification, data integrity verification and design document verification, providing a reliable foundation for subsequent data analysis and engineering design.

[0004] In the prior art, data cleaning is a general verification and processing method, which can detect outliers in the data and delete them. Common data cleaning methods, such as the box plot method, use quartiles and interquartile ranges to find data that deviate from the normal range and record them as outliers. However, there are often a large number of zero counts in biological survey data. This is because biological data has the characteristics of zero inflation and overdispersion, such as variability in biological characteristics or the presence of low-abundance species. As a result, the data cleaning methods in the prior art will misjudge these zero counts as outliers, making the final survey data inaccurate. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for checking and processing forestry area biological survey data to solve the above technical problems.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A method for checking and processing forestry area biological survey data comprises the following steps:

[0008] Step S1: Acquire all biological targets in the forest area and divide the biological targets into several categories; acquire biological survey data in the forest area, record the biological survey data of the same category of biological targets as one group, group the biological survey data, and obtain several groups of biological survey data;

[0009] Step S2: for any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster is recorded as a normal value, and the biological survey data that does not belong to each cluster is recorded as an outlier;

[0010] Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value;

[0011] Step S3: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n}, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m}, where P1 is the coordinate of the first outlier and m is the total number of outliers;

[0012] Get the deviation vector between the i-th outlier and the coordinates of each central value |O e P i |=O e -P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier;

[0013] Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero.

[0014] As a further solution of the present invention: the process of classifying the biological target includes:

[0015] The biological targets are classified once to obtain several biological species, which include animals, fungi, microorganisms and plants; the biological species are then classified twice, and the basis of the secondary classification includes the distribution structure, living environment and sample collection method of the organisms; through the primary classification and secondary classification, the biological targets are divided into several categories.

[0016] As a further solution of the present invention: the process of cluster analysis includes:

[0017] The biological survey data are converted into points in space, and a rectangular coordinate system is established in the space; for any point in the space, a circular neighborhood of the point is obtained with the point as the center and a preset radius R as the radius; the number N of points in the circular neighborhood is obtained, and if N≥Nt, where Nt is a preset number threshold, the center of the circular neighborhood is recorded as the center value; and the center value and all points in its circular neighborhood are recorded as belonging to the same cluster; if N<Nt, the number of points in the circular neighborhood of the next point is determined.

[0018] As a further solution of the present invention: the process of dividing the biological survey data in the group into several clusters includes:

[0019] After obtaining the first cluster in the biological survey data, all points in the first cluster are obtained to obtain the point set J1, then the point set is excluded in the space, and the second cluster is obtained from all the remaining points; and so on, the biological survey data in the group is divided into several clusters.

[0020] As a further solution of the present invention: the method for determining whether the outlier value is missing includes filtering out the biological survey data with missing values ​​in the outlier value by combining the isnull() function and the Boolean index.

[0021] As a further solution of the present invention: the process of setting the standard vector S includes:

[0022] Get the deviation vector of the same deviation from the origin {|O1P i |,|O2P i |, ..., |O n P i |}, where |O n P i |=(x n-i ,y n-i , z n-i ); then the coordinates of the standard vector S are:

[0023] .

[0024] As a further solution of the present invention: each angle is converted into a point in space through cluster analysis, recorded as an angle point, and a new rectangular coordinate system is established in the space; each angle point is selected in turn, with the angle point as the center of the circle and a preset second radius R´ as the radius, to obtain a new circular neighborhood of the angle point; the number N´ of points in the new circular neighborhood is obtained, and if N´≥Nt´, where Nt´ is the preset second numerical threshold, the center of the new circular neighborhood is recorded as the center value of the angle; and the center value of the angle and all the angle points in its circular neighborhood are recorded as belonging to the same deviation cluster.

[0025] As a further solution of the present invention: a verification and processing system for forestry area biological survey data, comprising:

[0026] Data processing module: obtain all biological targets in the forest area and divide the biological targets into several categories; obtain biological survey data in the forest area, record the biological survey data of the same category of biological targets as a group, group the biological survey data, and obtain several groups of biological survey data;

[0027] Missing value filtering module: For any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster are recorded as normal values, and the biological survey data that do not belong to each cluster are recorded as outliers;

[0028] Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value;

[0029] Zero-value extraction module: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n}, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m}, where P1 is the coordinate of the first outlier and m is the total number of outliers;

[0030] Get the deviation vector between the i-th outlier and the coordinates of each central value |O e P i |=O e -P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier;

[0031] Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero.

[0032] Beneficial effects of the present invention:

[0033] In the prior art, data cleaning is a general verification and processing method, which can detect outliers in the data and delete them. Common data cleaning methods, such as the box plot method, use quartiles and interquartile ranges to find data that deviate from the normal range and record them as outliers. However, there are often a large number of zero counts in biological survey data. This is because biological data has the characteristics of zero inflation and overdispersion, such as variability in biological characteristics or the presence of low-abundance species. As a result, the data cleaning methods in the prior art will misjudge these zero counts as outliers, making the final survey data inaccurate.

[0034] Compared with the prior art, the present invention distinguishes zero counts from abnormal values ​​through secondary cluster analysis according to the difference between zero counts and abnormal values ​​in biological survey data; firstly, the biological survey data is subjected to the first cluster analysis to preliminarily divide the outliers in the biological survey data that are different from most of the biological survey data, and since the zero counts are normal values ​​different from most of the data, the zero counts must be complete values, so the missing values ​​are filtered out from the outliers; the missing values ​​must be abnormal values; then the remaining outliers are analyzed, and the deviation trend of each outlier compared with the normal value is reflected by the deviation vector; since the abnormal values ​​are often caused by external factors such as operating errors or system failures, the abnormal values ​​of the biological survey data in the group generally have similar deviation trends; therefore, the deviation vector is subjected to secondary cluster analysis to obtain several deviation clusters, and the deviation trend shared by most of the deviation values ​​is screened out; if there is an outlier whose deviation trend does not belong to any deviation cluster, the outlier is recorded as zero count; in the data cleaning of the biological survey data, the situation that the zero count is mistakenly judged as an abnormal value is avoided, and the accuracy of the biological survey data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings.

[0036] Figure 1 The present invention is a flowchart of a method and system for checking and processing forestry area biological survey data. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 As shown, the present invention is a method for checking and processing forestry area biological survey data, comprising the following steps:

[0039] Step S1: Acquire all biological targets in the forest area and divide the biological targets into several categories; acquire biological survey data in the forest area, record the biological survey data of the same category of biological targets as one group, group the biological survey data, and obtain several groups of biological survey data;

[0040] As a preferred embodiment of the present invention, the process of classifying the biological target includes:

[0041] The biological target is classified once to obtain several biological species, wherein the biological species include animals, fungi, microorganisms and plants; the biological species are then classified twice, wherein the basis of the secondary classification includes the distribution structure, living environment and sample collection method of the organisms; and the biological target is divided into several categories through the primary classification and the secondary classification;

[0042] It should be noted that when collecting biological survey data, biological targets can be classified through systematic taxonomy, ecology, genetics and other methods; systematic taxonomy can classify and differentiate species by comparing the morphological, anatomical, genetic and other characteristics of different species; ecology classifies organisms based on their adaptability to the environment, the ecological niches they occupy, and their interactions with other species; genetic classification uses differences in gene sequences to identify and classify species, and this method is particularly suitable for identifying species that are similar in appearance but have significant genetic differences; each method has its own unique application scenarios and advantages, so in actual operations, the most appropriate classification strategy should be selected according to specific needs;

[0043] Step S2: for any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster is recorded as a normal value, and the biological survey data that does not belong to each cluster is recorded as an outlier;

[0044] Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value;

[0045] It should be noted that the zero count is the data that is different from most normal values ​​when the biological survey data is collected due to the characteristics of zero inflation and overdispersion in the biological survey data; although the zero count is different from the normal value, it is also valid data and not an outlier; cluster analysis is performed on the biological survey data to divide the data into several clusters, most of which belong to the main cluster, and some data points that do not conform to the main trend are regarded as outliers; since the zero count is a normal value different from most of the data, the zero count must be a complete value, so the missing value is filtered out from the outlier value, and the missing value must be an outlier;

[0046] As a preferred embodiment of the present invention, the cluster analysis process includes:

[0047] The biological survey data are converted into points in space, and a rectangular coordinate system is established in the space; for any point in the space, a circular neighborhood of the point is obtained with the point as the center and a preset radius R as the radius; the number of points N in the circular neighborhood is obtained, and if N≥Nt, where Nt is a preset number threshold, the center of the circular neighborhood is recorded as the center value; and the center value and all points in its circular neighborhood are recorded as belonging to the same cluster; if N<Nt, the number of points in the circular neighborhood of the next point is determined;

[0048] The process of dividing the biological survey data within a group into several clusters includes:

[0049] After obtaining the first cluster in the biological survey data, all points in the first cluster are obtained to obtain a point set J1, then the point set is excluded in the space, and the second cluster is obtained from all the remaining points; and so on, the biological survey data in the group is divided into several clusters;

[0050] It should be noted that the present invention adopts DBSCAN density clustering to obtain outliers in the biological survey data. In DBSCAN density clustering, the method for determining the preset radius mainly includes using a k-distance graph or calculating the average distance between data points, and selecting the optimal radius value through experiments and observation of clustering effects; the determination of the preset number is usually based on several key factors, including data dimension, data distribution characteristics and expected cluster granularity; for lower dimensional data, such as 2D or 3D, Nt=4 or Nt=5 is usually set; this is because a lower Nt may lead to overly sensitive cluster division, while a higher value may miss some clusters that should be identified; the preset number can also be determined by a k-distance graph;

[0051] As a preferred embodiment of the present invention, the process of dividing the biological survey data in a group into a plurality of clusters includes:

[0052] After obtaining the first cluster in the biological survey data, all points in the first cluster are obtained to obtain a point set J1, then the point set is excluded in the space, and the second cluster is obtained from all the remaining points; and so on, the biological survey data in the group is divided into several clusters;

[0053] Step S3: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n}, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m}, where P1 is the coordinate of the first outlier and m is the total number of outliers;

[0054] Get the deviation vector between the i-th outlier and the coordinates of each central value |Oe P i |=O e -P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier;

[0055] Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero value;

[0056] It should be noted that by calculating the deviation vector of each outlier from the normal value (the center value of the cluster), their deviation trend relative to normal data can be analyzed; the direction and size of the deviation vector can reflect the degree and direction of the deviation of the outlier; because outliers are usually caused by similar external factors (such as operating errors, system failures, etc.), the manifestations of these outliers in the data space often have similar deviation trends; therefore, the deviation vectors of these outliers show consistency in direction; in order to identify the deviation trends shared by most outliers, all deviation vectors are subjected to secondary clustering analysis. This process can group outliers with similar deviation trends into the same deviation cluster; the deviation clusters obtained through secondary clustering analysis can help screen out the main deviation trends shared by most outliers; these main deviation trends have the same abnormal reasons, and for outliers whose deviation trends do not belong to any deviation cluster, they may be counted as zero and not directly recorded as outliers for processing;

[0057] As a preferred embodiment of the present invention, the process of setting the standard vector S includes:

[0058] Get the deviation vector of the same deviation from the origin {|O1P i |,|O2P i |, ..., |O n P i |}, where |O n P i |=(x n-i ,y n-i , z n-i ); then the coordinates of the standard vector S are:

[0059] ;

[0060] It can be understood that the deviation vector reflects the deviation trend of the outlier relative to the cluster center. When performing spatial analysis, the average vector is used as the standard vector to provide a unified comparison benchmark for all outliers; this helps to standardize the degree of deviation of different outliers, which is convenient for comparison and analysis;

[0061] As a preferred embodiment of the present invention, each angle is converted into a point in space through cluster analysis, recorded as an angle point, and a new rectangular coordinate system is established in the space; each angle point is selected in turn, with the angle point as the center of the circle and a preset second radius R' as the radius, to obtain a new circular neighborhood of the angle point; the number of points N' in the new circular neighborhood is obtained, and if N'≥Nt', where Nt' is the preset second number threshold, the center of the new circular neighborhood is recorded as the center value of the angle; and the center value of the angle and all the angle points in its circular neighborhood are recorded as belonging to the same deviation cluster;

[0062] It can be understood that the process of determining the preset second radius and the preset second number threshold is the same as the preset radius and the preset number threshold in step S2.

[0063] A verification and processing system for forestry area biological survey data, comprising:

[0064] Data processing module: obtain all biological targets in the forest area and divide the biological targets into several categories; obtain biological survey data in the forest area, record the biological survey data of the same category of biological targets as a group, group the biological survey data, and obtain several groups of biological survey data;

[0065] Missing value filtering module: For any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster are recorded as normal values, and the biological survey data that do not belong to each cluster are recorded as outliers;

[0066] Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value;

[0067] Zero-value extraction module: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n}, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m}, where P1 is the coordinate of the first outlier and m is the total number of outliers;

[0068] Get the deviation vector between the i-th outlier and the coordinates of each central value |O e P i |=O e-P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier;

[0069] Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero.

[0070] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for checking and processing forestry area biological survey data, characterized in that: The following steps are involved: Step S1: Acquire all biological targets in the forest area and divide the biological targets into several categories; acquire biological survey data in the forest area, record the biological survey data of the same category of biological targets as one group, group the biological survey data, and obtain several groups of biological survey data; Step S2: for any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster is recorded as a normal value, and the biological survey data that does not belong to each cluster is recorded as an outlier; Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value; Step S3: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n }, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m }, where P1 is the coordinate of the first outlier and m is the total number of outliers; Get the deviation vector between the i-th outlier and the coordinates of each central value |O e P i |=O e -P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier; Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero.

2. The method for checking and processing forestry area biological survey data according to claim 1 is characterized in that: In step S1, the process of classifying the biological target includes: The biological targets are classified once to obtain several biological species, which include animals, fungi, microorganisms and plants; the biological species are then classified twice, and the basis of the secondary classification includes the distribution structure, living environment and sample collection method of the organisms; through the primary classification and secondary classification, the biological targets are divided into several categories.

3. The method for checking and processing forestry area biological survey data according to claim 1 is characterized in that: In step S2, the cluster analysis process includes: The biological survey data are converted into points in space, and a rectangular coordinate system is established in the space; for any point in the space, a circular neighborhood of the point is obtained with the point as the center and a preset radius R as the radius; the number N of points in the circular neighborhood is obtained, and if N≥Nt, where Nt is a preset number threshold, the center of the circular neighborhood is recorded as the center value; and the center value and all points in its circular neighborhood are recorded as belonging to the same cluster; if N<Nt, the number of points in the circular neighborhood of the next point is determined.

4. The method for checking and processing forest area biological survey data according to claim 3 is characterized in that: In step S2, the process of dividing the biological survey data in the group into several clusters includes: After obtaining the first cluster in the biological survey data, all points in the first cluster are obtained to obtain the point set J1, then the point set is excluded in the space, and the second cluster is obtained from all the remaining points; and so on, the biological survey data in the group is divided into several clusters.

5. The method for checking and processing forestry area biological survey data according to claim 1 is characterized in that: In step S2, the method for determining whether the outlier is missing includes filtering out missing biological survey data in the outlier by combining an isnull() function and a Boolean index.

6. The method for checking and processing forestry area biological survey data according to claim 1, characterized in that: In step S3, the process of setting the standard vector S includes: Get the deviation vector of the same deviation from the origin {|O1P i |,|O2P i |, ..., |O n P i |}, where |O n P i |=(x n-i ,y n-i , z n-i ); then the coordinates of the standard vector S are: 。 7. The method for checking and processing forestry area biological survey data according to claim 1 is characterized in that: In step S3, each angle is converted into a point in space through cluster analysis, recorded as an angle point, and a new rectangular coordinate system is established in the space; each angle point is selected in turn, with the angle point as the center of the circle and the preset second radius R' as the radius, to obtain a new circular neighborhood of the angle point; the number of points N' in the new circular neighborhood is obtained, and if N'≥Nt', where Nt' is the preset second number threshold, the center of the new circular neighborhood is recorded as the center value of the angle; and the center value of the angle and all the angle points in its circular neighborhood are recorded as belonging to the same deviation cluster.

8. A verification and processing system for biological survey data in forestry areas, characterized in that: include: Data processing module: obtain all biological targets in the forest area and divide the biological targets into several categories; obtain biological survey data in the forest area, record the biological survey data of the same category of biological targets as a group, group the biological survey data, and obtain several groups of biological survey data; Missing value filtering module: For any group of biological survey data, the biological survey data in the group are divided into several clusters through cluster analysis, the biological survey data in each cluster are recorded as normal values, and the biological survey data that do not belong to each cluster are recorded as outliers; Determine whether the outlier value is missing, if so, directly record the outlier value as an abnormal value, and delete the abnormal value; Zero-value extraction module: Establish a rectangular coordinate system in the cluster analysis space, obtain the center value in each cluster, and obtain several center value coordinates {O1, O2, ..., O n }, where O1 is the coordinate of the center value of the first cluster, and n is the total number of clusters; get the coordinates of each outlier {P1, P2, ..., P m }, where P1 is the coordinate of the first outlier and m is the total number of outliers; Get the deviation vector between the i-th outlier and the coordinates of each central value |O e P i |=O e -P i , where e∈[1,n], e is a positive integer; and the center value coordinate corresponding to the deviation vector with the shortest length is selected and recorded as the deviation origin of the i-th outlier; Obtain each deviation vector that deviates from the origin in the same way, set a standard vector S, obtain the angle between each deviation vector and the standard vector, obtain an angle sequence, and perform secondary clustering analysis on the angle sequence to obtain several deviation clusters; obtain the outlier value corresponding to the deviation vector that does not belong to each deviation cluster and record it as zero.

Images