A real-time three-dimensional lightning positioning method
The real-time three-dimensional lightning positioning method addresses environmental noise and data processing challenges by using support vector machines and Apache Spark for accurate and efficient lightning positioning.
Patent Information
- Application Number
- CN202310828460.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-07
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-07-07
AI Technical Summary
Existing technologies face challenges in accurately determining the real-time three-dimensional positioning of lightning, including the accuracy of existing technologies in addressing the technical problem of environmental noise interference, initial calculation accuracy, and data processing efficiency in lightning positioning systems.
A real-time three-dimensional lightning positioning method utilizing data preprocessing, pulse identification with support vector machines, initial positioning through triangulation and optimization with differential evolution, and real-time processing with Apache Spark and edge computing to enhance accuracy and efficiency.
The method improves pulse identification accuracy, optimizes positioning results, and ensures efficient real-time processing by reducing noise interference and enhancing data processing capabilities.
Smart Images

Figure CN116992215B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of real-time lightning positioning, and in particular to a real-time three-dimensional lightning positioning method. Background Art
[0002] Lightning is one of the major serious natural disasters, which can cause forest and oil depot fires, power supply and communication information system failures or damage, and pose a major threat to aerospace, mining and some important and sensitive high-tech equipment. After the 1980s, the hazards caused by lightning increased significantly, especially in areas closely related to high-tech, such as aerospace, national defense, communications, electricity, computers, and the electronics industry. The widespread use of large-scale and ultra-large-scale integrated circuits that are extremely sensitive to lightning electromagnetic interference has greatly increased the probability of being struck by lightning.
[0003] The development of real-time three-dimensional lightning positioning services can provide timely warnings for lightning activities, thereby reducing the harm of lightning to people and property. The quality of lightning positioning algorithms directly affects the accuracy and efficiency of lightning positioning results, so the development of new lightning positioning algorithms and technologies is a very urgent task at present.
[0004] At present, Lightning real-time 3D technology faces three problems:
[0005] First, the accuracy of lightning three-dimensional positioning calculation is affected by the environmental noise of the observation station. Most of the environmental noise has a wide frequency domain and is difficult to filter out using conventional frequency domain filters, which seriously affects pulse matching and thus affects the positioning accuracy. Therefore, it is urgent to improve the quality of the original lightning signal. In the paper "A low frequency 3D lightning mapping network innorth China" ("Atmospheric Research" (2021)), the regular environmental noise pulses in each observation station are fitted, and the sliding cross-correlation recognition method is used to zero the lightning signal pulses that have a high match with the fitted noise pulses. This method can reduce the interference of noise pulses to a certain extent, but it requires manual identification of noise pulses, and the sliding matching also increases the calculation amount of filtering processing, and the filtering efficiency is low. At the same time, if the correlation coefficient is set improperly, it is easy to cause invalid filtering or excessive filtering. The paper "A New Method of Three-Dimensional Location for Low-frequency Electric Field Detection Array" (Journal of Geophysical Research (2018)) uses EMD to process the original lightning signal, which overcomes the influence of environmental noise to a certain extent. However, EMD may have modal aliasing when decomposing IMF, and there is no clear standard for how many iterations are required to decompose IMF.
[0006] Secondly, many scientific research teams often use the Levenberg-Marquardt (LM) algorithm to fit and optimize the initial three-dimensional lightning location initial calculation results. "Beijing Lightning Comprehensive Detection Network (BLNET): Network Composition and Preliminary Location Results" (Atmospheric Science (2015)), but LM is a "trust region" algorithm and has a high dependence on the accuracy of the initial results. When the initial solution accuracy of the lightning radiation source location is low, the fitting efficiency of the LM algorithm will be greatly reduced. If a grid search location algorithm is used, "An Improved Grid Search Lightning Location Algorithm" (Computer Engineering and Applications (2016)), it is easy to obtain high-precision results, but the calculation amount is very large and the calculation time is long.
[0007] Finally, for real-time lightning location technology, it is particularly important that data collection matches the backend data processing speed. When there is a strong thunderstorm, data collection is frequent, which can easily lead to data congestion on the acquisition card, causing instability in the lightning location system. Therefore, in the face of large-scale data real-time processing, it is necessary to introduce a reliable computing engine to ensure the stability of data collection, data reliability, and low latency of data analysis and processing in the real-time lightning location system.
[0008] How to solve the above problems is the subject faced by the present invention. Summary of the invention
[0009] The purpose of the present invention is to provide a real-time three-dimensional lightning positioning method.
[0010] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically as follows:
[0011] A lightning real-time three-dimensional positioning method, the method comprising the following steps:
[0012] S1: Data preprocessing, preprocessing the collected electric field signals; including denoising and normalization operations to reduce the impact of noise on recognition.
[0013] S2: Pulse identification, using support vector machine for binary classification to classify the signal into pulse signal and non-pulse signal;
[0014] Before performing positioning calculations, pulse matching is required to determine that the pulse signals detected by each measuring station are emitted by the same radiation source; this is a key pre-processing step to ensure that the correct data is used when calculating the lightning source location.
[0015] The pulses are identified by machine learning methods, which include support vector machine model, decision tree model and neural network model.
[0016] S3: Initial positioning calculation, using the triangulation method, based on the principle of multi-station time difference measurement (TDOA) to measure the time difference between the signal arriving at multiple receiving stations, and using the CHAN positioning algorithm to calculate the initial position of the lightning source;
[0017] S4: Optimization algorithm solution, using differential evolution algorithm to optimize the initial positioning result obtained in step S3 to obtain the accurate lightning source location;
[0018] S5: Real-time positioning, using the Apache Spark distributed computing framework to process and analyze data in real time, while using edge computing technology to reduce transmission delays and improve real-time lightning positioning performance.
[0019] Step S2 includes:
[0020] S21: Feature extraction: extracting features from the electric field signal preprocessed in step S1, selecting time domain, frequency domain or time-frequency domain features (including peak value, mean value, energy, and spectrum features), and performing feature selection methods (including stepwise regression and principal component analysis (PCA)) to obtain the final features as input of a support vector machine for training a classifier;
[0021] S22: Data labeling: label the electric field signals through actual observation data; divide the signals into lightning pulse signals: positive class, and non-lightning pulse signals: negative class;
[0022] S23: Training support vector machine classifier: Use the electric field signal annotated in step S22 to train the support vector machine classifier to obtain an optimal hyperplane to separate positive signals from negative signals. The objective function of the support vector machine can be expressed as:
[0023] minimize:
[0024] subject to:y i ·(w T x i +b)≥1-ξ i ,i=1,...,N
[0025] ξ i ≥0
[0026] Among them, w is the weight vector, b is the bias term, and x i is the input feature vector, y i is the corresponding label +1 or -1, ξ i is the slack variable, C is the penalty parameter; T is the transposed matrix, N is the total sample size;
[0027] In machine learning, a label is the expected model output and a method for identifying sample categories. In a binary classification task, i.e., the classification problem of "positive signals and negative signals" in this invention, the labels of the two types of samples are +1 and -1 respectively.
[0028] S24: Use kernel function: In order to solve nonlinear problems, kernel function is introduced to map the original feature space to a high-dimensional space (a higher dimension than the original feature space). The kernel functions used include linear kernel, polynomial kernel, and radial basis function kernel (RBF).
[0029] The form of the RBF kernel function is:
[0030] K(x i ,x j) = exp(-γ||x i -x j || 2 )
[0031] Among them, γ is the parameter of the RBF kernel function; Xi and Xj are input samples, and ||Xi-Xj|| represents the Euclidean distance (i.e., the distance between samples).
[0032] The benefits of using RBF kernel function compared to linear kernel and polynomial kernel are as follows:
[0033] (1) Nonlinear modeling capability: The RBF kernel function can handle nonlinear relationships. Compared with linear kernels and polynomial kernels, the RBF kernel function has stronger nonlinear modeling capabilities and can better capture complex patterns and structures in data.
[0034] (2) Parameter adjustment: Compared with the polynomial kernel, the RBF kernel function only has one parameter γ that needs to be adjusted. This makes the parameter adjustment of the model relatively simple and it is easier to find the appropriate parameter value through methods such as cross-validation.
[0035] S25: Predicting pulse signals: using the trained support vector machine classifier to classify the new electric field signal; for the electric field signal x marked in step S22, the prediction result can be calculated by the following formula:
[0036] f(x)=w T ·φ(x)+b
[0037] Among them, φ(x) is the feature vector after the kernel function maps the input signal to a high-dimensional space; if f(x)>0, then the signal x is classified as a pulse signal; if f(x)<0, it is classified as a non-pulse signal;
[0038] S26: Set appropriate time windows and thresholds to match the same radiation source pulses measured at different stations:
[0039] S261: Set time window: In order to correctly match the same radiation source pulses measured at different measuring stations, set a suitable time window; the time window should be determined according to the signal propagation speed and the distance between the measuring stations, that is, the time difference from the radiation source to the two measuring stations is less than the time it takes for the speed of light to propagate between the two measuring stations.
[0040] S262: Peak-finding threshold: Calculate the cumulative probability distribution of the electric field signal pulse and set a peak-finding threshold; when the cumulative probability reaches 99%, the corresponding amplitude is used as the peak-finding threshold;
[0041] S263: Positioning pulse selection: in the time window, find the electric field signal with the largest amplitude as the positioning pulse; for pulses with an interval exceeding 15 μs, select an electric field signal with a suitable percentage threshold as the positioning pulse according to the distribution of the amplitude;
[0042] S264: Pulse matching by cross-correlation method: pulse matching is performed by using the cross-correlation method; the time window is set to 350 μs, the sliding step is 50 μs, and the cross-correlation function of the electric field signals between each measuring station in the time window is calculated; the cross-correlation function is expressed as:
[0043] R xy (τ) = ∑ (x(t) · y(t + τ))
[0044] Where x(t) and y(t) represent the electric field signals of the two measuring stations respectively, and τ represents the time delay.
[0045] S27: Find the time delay corresponding to the peak of the cross-correlation function; the peak of the cross-correlation function indicates the maximum similarity between the two signals; by finding the time delay corresponding to the peak, match the same radiation source pulses measured at different stations; set the threshold to 0.8 to determine whether the two signals come from the same radiation source; if the peak of the cross-correlation function is greater than the threshold, it is considered that the two signals come from the same radiation source; match the positioning pulses of all stations in pairs to obtain a pulse combination.
[0046] Step S3 specifically includes:
[0047] S31, set the positioning system to have H measuring stations, use S i (x i ,y i ,z i ), where i = 1, 2, ..., H; let S1 be the primary station and the others be auxiliary stations; the spatial position of the lightning radiation source is P (x, y, z).
[0048] S32. Calculate the time difference between the primary station and the secondary station:
[0049] Δt i =t i -t1
[0050] Where i = 2, 3, ..., H; t i is the time when the lightning radiation source arrives at the i-th measuring station.
[0051] S33, calculating the distance difference according to the time difference:
[0052] Δd i =c·Δt i
[0053] Where i = 2, 3, ..., H; c is the speed of light;
[0054] S34, construct the super time difference equations:
[0055] (xx i ) 2 +(yy i ) 2 +(zz i ) 2 -(x-x1) 2 -(y-y1) 2 -(z-z1) 2 =Δd i 2
[0056] Where i = 2, 3, ..., H;
[0057] S35. Simplify the system of equations, eliminate the square terms, and obtain the linear system of equations:
[0058] A·X=B
[0059] in,
[0060] A=[2(x2-x1),2(y2-y1),2(z2-z1);
[0061] 2(x3-x1),2(y3-y1),2(z3-z1); ....;
[0063] 2(x N -x1),2(y N -y1),2(z N -z1)]
[0064] X=[x,y,z] T
[0065] B=[Δd2 2 +x1 2 +y1 2 +z1 2 -x2 2 -y2 2 -z2 2 ; Δd3 2 +x1 2 +y1 2 +z1 2 -x3 2 -y3 2 -z3 2 ; ...; Δd N 2 +x1 2 +y1 2+z1 2 -x N 2 -y N 2 -z N 2 ]
[0066] S36. Solve the linear equations using the iterative least squares method (ILS) to obtain the initial position estimate of the lightning radiation source:
[0067] P(x,y,z)=(A T A) -1 ·A T ·B
[0068] Innovations to the CHAN positioning algorithm are made to improve the accuracy of the initial location calculation of the lightning source:
[0069] (1) Introducing weighted least squares (WLS)
[0070] When there are large differences in the distances and signal propagation conditions between stations, the weighted least squares (WLS) method can be used to weight the measurement errors of different stations. This can improve the accuracy of the positioning results by assigning weights to each equation based on the signal quality or distance of the station.
[0071] (2) Using high-order polynomial fitting
[0072] In the case of nonlinearity, a high-order polynomial fitting method can be considered to replace the linear equations to solve the lightning source location. This method can better handle the super-time difference equations in the nonlinear case, thereby improving the accuracy of the positioning results.
[0073] In step S4, the differential evolution algorithm DE is used to optimize the initial positioning result to obtain a more accurate lightning source location.
[0074] S42 initialization: randomly generate an initial population containing V solution vectors, each solution vector has D dimensions; set algorithm parameters, mutation factor F, crossover rate CR and maximum number of iterations T;
[0075] S42 mutation operation: For each solution vector xi, three different solution vectors r1, r2 and r3 are randomly selected from the population to calculate the difference vector: vi = r1 + F*(r2-r3), where r1, r2 and r3 are different from each other and have different magnitudes and directions from xi;
[0076] S43 Crossover operation: For each solution vector xi, generate a test vector ui according to the crossover rate CR and the difference vector vi; in D dimensions, perform the following operations on each dimension j:
[0077] Generate a random number randj, the value range is [0,1];
[0078] If randj≤CR or j is equal to a random integer (the range of the random integer is [1, D]), then set ui[j]=vi[j];
[0079] Otherwise, set ui[j] = xi[j];
[0080] Selection operation: For each solution vector xi, calculate its fitness value f(xi) and the fitness value f(ui) of the test vector ui. If f(ui)≤f(xi), replace xi with ui; otherwise, keep the solution vector xi
[0081] Convergence check: Check whether the maximum number of iterations T is reached; if reached, output the optimal solution; otherwise, return to step S42 to continue iteration.
[0082] Step S5 is specifically as follows:
[0083] Spark applications are launched on a distributed cluster and consist of a Driver program. The Driver program creates a SparkContext, which is the main connection between the application and the Spark cluster.
[0084] S51. Data partitioning and task allocation: Let the collected lightning data set be D, and divide D into k subsets: D1, D2, ..., D k ; Each subset is assigned to a computing node to achieve distributed data processing; the set of computing nodes is designed to be
[0085] in represents the i-th computing node;
[0086] The collected data set D is received by the Spark application from each station. On these computing nodes, the Spark application applies operations (Transformations and Actions) to the distributed data, including data cleaning, denoising, and feature extraction. Spark transforms these operations into a directed acyclic graph (DAG), which is then divided into a series of stages (Stages) by the DAG scheduler, and each stage consists of a series of tasks (Tasks).
[0087] The task scheduler assigns these tasks to various Executors. These Executors run on worker nodes, and each Executor has a certain number of cores and memory.
[0088] Data is partitioned across the nodes in the cluster and transferred over the network when needed.
[0089] S52, Real-time data processing and analysis: Each computing node According to the assigned sub-dataset D i Real-time data processing and analysis; To achieve real-time processing, the distributed computing framework will dynamically adjust the processing speed according to system resources; let the processing speed be P, and the data processing function be f, then within time t, the amount of data processed by the i-th computing node can be expressed as: P i (t)=|f(D i )|;
[0090] S53, Edge computing technology: Reduce the complexity of subsequent data processing. By deploying edge computing devices at the source of data generation (such as measurement stations), transmission delay can be reduced; let the set of edge computing devices be E, where E i represents the ith edge computing device; the edge computing device can perform preliminary processing (including filtering and denoising operations) when the data is generated to reduce the complexity of subsequent data processing; let the edge computing processing function be g, then the data set processed by the edge computing device is expressed as: D'=g(D); the edge computing processing function is a data preprocessing operation performed on the edge computing device, which is mainly used to perform preliminary processing of data near the data source (such as a measuring station) to reduce the burden of subsequent data processing and reduce network transmission delay. Specific edge computing processing functions may include the following operations:
[0091] S531 Data cleaning: including denoising and removing outliers from raw data to improve data quality;
[0092] S532 Data Compression: Reduce the size of data through lossy or lossless compression algorithms to reduce transmission delays and storage costs;
[0093] S533 Feature Extraction: Extract useful features from raw data to reduce data dimension and computational complexity;
[0094] After the Executor executes the task, it sends the result back to the Driver program. After the Driver program receives the results of all tasks, it aggregates the data for global analysis and calculates the real-time location results of the lightning source.
[0095] S54: Data aggregation and result calculation: The data processed by the computing nodes needs to be aggregated for global analysis; let the data aggregation function be h, then the aggregated data is expressed as: D agg =h(P1(t),P2(t),...P k(t)); h is the operation of finding the average value of each data, that is:
[0096]
[0097] P1, P2...Pk are the positions of k lightning bolts obtained on k subsets,
[0098] Based on the calculated D agg , calculate the real-time location result of the lightning source. Assume that the lightning location function is L, then the real-time location result is expressed as: L(t) = L(D agg );
[0099] The process of implementing real-time positioning based on Apache Spark is as follows:
[0100] Spark applications (consisting of Driver programs) are launched on a distributed cluster.
[0101] Spark creates a SparkContext object, which is the main connection to the cluster.
[0102] After creating a SparkContext, the application applies operations (Transformations and Actions) to distributed data, typically loaded from HDFS or other data sources.
[0103] Spark converts operations into a series of stages, each of which consists of a series of tasks. These tasks are scheduled by TaskScheduler and run on Executor.
[0104] These Executors run on worker nodes, and each Executor has a certain number of cores and memory.
[0105] Data is partitioned across the nodes in the cluster and transferred over the network.
[0106] When all tasks are completed, the application ends.
[0107] Compared with the prior art, the present invention has the following beneficial effects:
[0108] (1) Enhanced pulse identification accuracy: Pulse identification using machine learning methods such as support vector machines (SVM) has stronger generalization capabilities and can improve the accuracy of pulse identification compared to traditional methods. This helps to use the correct data in the positioning process, thereby improving the accuracy of positioning results.
[0109] (2) Optimized positioning results: The initial positioning results are optimized by differential evolution algorithm (DE) to obtain a more accurate lightning source location. The introduction of genetic algorithm (GA) and differential evolution algorithm (DE) enables the solution to have global search capabilities and can find the optimal solution in the solution space, thereby improving positioning accuracy. This makes the positioning results more accurate and helps to improve the accuracy of lightning monitoring and warning.
[0110] (3) Efficient real-time processing capability: Use distributed computing frameworks (such as Apache Spark or Flink) and edge computing technology to process and analyze data in real time, reduce transmission delays, and improve real-time lightning location performance. This parallel data processing and near-source preprocessing method can quickly process large amounts of data, ensuring that the lightning monitoring system can still operate efficiently when faced with large amounts of data, thereby improving system efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0111] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0112] Figure 1 It is a schematic diagram of the process of the present invention;
[0113] Figure 2 This is a schematic diagram of the real-time lightning location operation of the Apache Spark distributed computing framework;
[0114] Figure 3 Schematic diagram of real-time three-dimensional positioning of ground lightning in the present invention; (a) is a schematic diagram of the change of electric field waveform over time, (b) is a schematic diagram of the change of radiation source height over time, (c) is a schematic diagram of the vertical section of the lightning discharge process from west to east, (d) is a planar schematic diagram of the lightning discharge process, (e) is a schematic diagram of the distribution of the number of radiation sources with height, (f) is a schematic diagram of the vertical section of the lightning discharge process from south to north, (g) is a schematic diagram of the lightning discharge process Figure 3 Dimensional schematic diagram;
[0115] Figure 4 Schematic diagram of real-time three-dimensional positioning of cloud flash in the present invention; (a) is a schematic diagram of the change of electric field waveform over time, (b) is a schematic diagram of the change of radiation source height over time, (c) is a schematic diagram of the vertical section of the lightning discharge process from west to east, (d) is a planar schematic diagram of the lightning discharge process, (e) is a schematic diagram of the distribution of the number of radiation sources with height, (f) is a schematic diagram of the vertical section of the lightning discharge process from south to north, (g) is a schematic diagram of the lightning discharge process Figure 3 Vittu.
[0116] In the figure: ※ indicates the name of the observation station. DETAILED DESCRIPTION
[0117] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0118] Embodiment 1:
[0119] Reference Figure 1 A real-time three-dimensional lightning positioning method is provided, the method comprising the following steps:
[0120] S1: Data preprocessing, preprocessing the collected electric field signals; including denoising and normalization operations to reduce the impact of noise on recognition.
[0121] S2: Pulse identification, using support vector machine for binary classification to classify the signal into pulse signal and non-pulse signal;
[0122] Before performing positioning calculations, pulse matching is required to determine that the pulse signals detected by each measuring station are emitted by the same radiation source; this is a key pre-processing step to ensure that the correct data is used when calculating the lightning source location.
[0123] The pulses are identified by using a machine learning method. The machine learning method uses a support vector machine model, a decision tree model, and a neural network model. In this embodiment, the support vector machine model is taken as an example.
[0124] S3: Initial positioning calculation, using the triangulation method, based on the multi-station time difference measurement TDOA principle to measure the time difference between the signal arriving at multiple receiving stations, and using the CHAN positioning algorithm to calculate the initial position of the lightning source;
[0125] S4: Optimization algorithm solution, using differential evolution algorithm to optimize the initial positioning result obtained in step S3 to obtain the accurate lightning source location;
[0126] S5: Real-time positioning, using the Apache Spark distributed computing framework to process and analyze data in real time, while using edge computing technology to reduce transmission delays and improve real-time lightning positioning performance.
[0127] Step S2 includes:
[0128] S21: Feature extraction: extracting features from the electric field signal preprocessed in step S1, selecting time domain, frequency domain or time-frequency domain features (including peak value, mean value, energy, and spectrum features), and performing feature selection methods (including stepwise regression and principal component analysis (PCA)) to obtain the final features as input of a support vector machine for training a classifier;
[0129] S22: Data labeling: label the electric field signals through actual observation data; divide the signals into lightning pulse signals: positive class, and non-lightning pulse signals: negative class;
[0130] S23: Training support vector machine classifier: Use the electric field signal annotated in step S22 to train the support vector machine classifier to obtain an optimal hyperplane to separate positive signals from negative signals. The objective function of the support vector machine can be expressed as:
[0131] minimize:
[0132] subject to:y i ·(w T x i +b)≥1-ξ i ,i=1,...,N
[0133] ξ i ≥0
[0134] Among them, w is the weight vector, b is the bias term, and x i is the input feature vector, y i is the corresponding label +1 or -1, ξ i is the slack variable, C is the penalty parameter; T is the transposed matrix, N is the total sample size;
[0135] In machine learning, a label is the expected model output and a method for identifying sample categories. In a binary classification task, i.e., the classification problem of "positive signals and negative signals" in this invention, the labels of the two types of samples are +1 and -1 respectively.
[0136] S24: Use kernel function: In order to solve nonlinear problems, kernel function is introduced to map the original feature space to a high-dimensional space (a higher dimension than the original feature space). The kernel functions used include linear kernel, polynomial kernel, and radial basis function kernel (RBF).
[0137] The form of the RBF kernel function is:
[0138] K(x i ,x j ) = exp(-γ||x i -x j || 2 )
[0139] Among them, γ is the parameter of the RBF kernel function; Xi and Xj are input samples, and ||Xi-Xj|| represents the Euclidean distance (i.e., the distance between samples).
[0140] The benefits of using RBF kernel function compared to linear kernel and polynomial kernel are as follows:
[0141] (1) Nonlinear modeling capability: The RBF kernel function can handle nonlinear relationships. Compared with linear kernels and polynomial kernels, the RBF kernel function has stronger nonlinear modeling capabilities and can better capture complex patterns and structures in data.
[0142] (2) Parameter adjustment: Compared with the polynomial kernel, the RBF kernel function only has one parameter γ that needs to be adjusted. This makes the parameter adjustment of the model relatively simple and it is easier to find the appropriate parameter value through methods such as cross-validation.
[0143] S25: Predicting pulse signals: using the trained support vector machine classifier to classify the new electric field signal; for the electric field signal x marked in step S22, the prediction result can be calculated by the following formula:
[0144] f(x)=w T ·φ(x)+b
[0145] Among them, φ(x) is the feature vector after the kernel function maps the input signal to a high-dimensional space; if f(x)>0, then the signal x is classified as a pulse signal; if f(x)<0, it is classified as a non-pulse signal;
[0146] S26: Set appropriate time windows and thresholds to match the same radiation source pulses measured at different stations:
[0147] S261: Set time window: In order to correctly match the same radiation source pulses measured at different measuring stations, set a suitable time window; the time window should be determined according to the signal propagation speed and the distance between the measuring stations; that is, the time difference from the radiation source to the two measuring stations is less than the time it takes for the speed of light to propagate between the two measuring stations;
[0148] S262: Peak-finding threshold: Calculate the cumulative probability distribution of the electric field signal pulse and set a peak-finding threshold; when the cumulative probability reaches 99%, the corresponding amplitude is used as the peak-finding threshold;
[0149] S263: Positioning pulse selection: In the time window, find the electric field signal with the largest amplitude as the positioning pulse; for pulses with an interval of more than 15μs, select an electric field signal with a suitable percentage threshold as the positioning pulse according to the distribution of the amplitude; in this embodiment, 10% of the signal amplitude distribution is selected as the threshold. In this way, the signal with a higher amplitude can be automatically determined according to the overall distribution of the signal amplitude.
[0150] S264: Pulse matching by cross-correlation method: pulse matching is performed by using the cross-correlation method; for each pair of measuring stations, a time window of 350 μs and a sliding step of 50 μs are set, and the cross-correlation function of the electric field signals between the measuring stations in the time window is calculated; the cross-correlation function can be expressed as:
[0151] R xy (τ) = ∑ (x(t) · y(t + τ))
[0152] Where x(t) and y(t) represent the electric field signals of the two measuring stations respectively, and τ represents the time delay.
[0153] S27: Find the time delay corresponding to the peak of the cross-correlation function; the peak of the cross-correlation function indicates the maximum similarity between the two signals; by finding the time delay corresponding to the peak, match the same radiation source pulses measured at different stations; set the threshold to 0.8 to determine whether the two signals come from the same radiation source; if the peak of the cross-correlation function is greater than the threshold, it is considered that the two signals come from the same radiation source; match the positioning pulses of all stations in pairs to obtain a pulse combination.
[0154] Step S3 specifically includes:
[0155] S31, set the positioning system to have H measuring stations (in this embodiment, the measuring stations are set to at least 4), respectively using S i (x i ,y i ,z i ), where i = 1, 2, ..., H; let S1 be the primary station and the others be auxiliary stations; the spatial position of the lightning radiation source is P (x, y, z).
[0156] S32. Calculate the time difference between the primary station and the secondary station:
[0157] Δt i =t i -t1
[0158] Where i = 2, 3, ..., H; t i is the time when the lightning radiation source arrives at the i-th measuring station.
[0159] S33, calculating the distance difference according to the time difference:
[0160] Δd i =c·Δt i
[0161] Where i = 2, 3, ..., H; c is the speed of light;
[0162] S34, construct the super time difference equations:
[0163] (xxi ) 2 +(yy i ) 2 +(zz i ) 2 -(x-x1) 2 -(y-y1) 2 -(z-z1) 2 =Δd i 2
[0164] Where i = 2, 3, ..., H;
[0165] S35. Simplify the system of equations, eliminate the square terms, and obtain the linear system of equations:
[0166] A·X=B
[0167] in,
[0168] A=[2(x2-x1),2(y2-y1),2(z2-z1);2(x3-x1),2(y3-y1),2(z3-z1);....;2(x N -x1),2(y N -y1),2(z N -z1)]
[0169] X=[x,y,z] T
[0170] B=[Δd2 2 +x1 2 +y1 2 +z1 2 -x2 2 -y2 2 -z2 2 ; Δd3 2 +x1 2 +y1 2 +z1 2 -x3 2 -y3 2 -z3 2 ; ...; Δd N 2 +x1 2 +y1 2 +z1 2 -x N 2 -y N 2 -z N 2 ]
[0171] S36. Solve the linear equations using the iterative least squares method (ILS) to obtain the initial position estimate of the lightning radiation source:
[0172] P(x,y,z)=(A T A) -1 ·A T ·B
[0173] Innovations to the CHAN positioning algorithm are made to improve the accuracy of the initial location calculation of the lightning source:
[0174] (3) Introducing weighted least squares (WLS)
[0175] When there are large differences in the distances and signal propagation conditions between stations, the weighted least squares (WLS) method can be used to weight the measurement errors of different stations. This can improve the accuracy of the positioning results by assigning weights to each equation based on the signal quality or distance of the station.
[0176] (4) Using high-order polynomial fitting
[0177] In the case of nonlinearity, a high-order polynomial fitting method can be considered to replace the linear equations to solve the lightning source location. This method can better handle the super-time difference equations in the nonlinear case, thereby improving the accuracy of the positioning results.
[0178] In step S4, the differential evolution algorithm DE is used to optimize the initial positioning result to obtain a more accurate lightning source location.
[0179] S42 initialization: randomly generate an initial population containing V solution vectors, each solution vector has D dimensions; set algorithm parameters, mutation factor F, crossover rate CR and maximum number of iterations T;
[0180] S42 mutation operation: For each solution vector xi, three different solution vectors r1, r2 and r3 are randomly selected from the population to calculate the difference vector: vi = r1 + F*(r2-r3), where r1, r2 and r3 are different from each other and have different magnitudes and directions from xi;
[0181] S43 Crossover operation: For each solution vector xi, generate a test vector ui according to the crossover rate CR and the difference vector vi; in D dimensions, perform the following operations on each dimension j:
[0182] Generate a random number randj, the value range is [0,1];
[0183] If randj≤CR or j is equal to a random integer (the range of the random integer is [1, D]), then set ui[j]=vi[j];
[0184] Otherwise, set ui[j] = xi[j];
[0185] Selection operation: For each solution vector xi, calculate its fitness value f(xi) and the fitness value f(ui) of the test vector ui. If f(ui)≤f(xi), replace xi with ui; otherwise, keep the solution vector xi
[0186] Convergence check: Check whether the maximum number of iterations T is reached; if reached, output the optimal solution; otherwise, return to step S42 to continue iteration.
[0187] refer to Figure 2 , step S5 is specifically as follows:
[0188] Spark applications are launched on a distributed cluster and consist of a Driver program. The Driver program creates a SparkContext, which is the main connection between the application and the Spark cluster.
[0189] S51. Data partitioning and task allocation: Let the collected lightning data set be D, and divide D into k subsets: D1, D2, ..., D k ; Each subset is assigned to a computing node to achieve distributed data processing; the set of computing nodes is designed to be
[0190] in represents the i-th computing node;
[0191] The collected data set D is received by the Spark application from each station. On these computing nodes, the Spark application applies operations (Transformations and Actions) to the distributed data, including data cleaning, denoising, and feature extraction. Spark transforms these operations into a directed acyclic graph (DAG), which is then divided into a series of stages (Stages) by the DAG scheduler, and each stage consists of a series of tasks (Tasks).
[0192] The task scheduler assigns these tasks to various Executors. These Executors run on worker nodes, and each Executor has a certain number of cores and memory.
[0193] Data is partitioned across the nodes in the cluster and transferred over the network when needed.
[0194] S52, Real-time data processing and analysis: Each computing node According to the assigned sub-dataset Di Real-time data processing and analysis; To achieve real-time processing, the distributed computing framework will dynamically adjust the processing speed according to system resources; let the processing speed be P, and the data processing function be f, then within time t, the amount of data processed by the i-th computing node can be expressed as: P i (t)=|f(D i )|;
[0195] S53, Edge computing technology: Reduce the complexity of subsequent data processing. By deploying edge computing devices at the source of data generation (such as measurement stations), transmission delay can be reduced; let the set of edge computing devices be E, where E i represents the ith edge computing device; the edge computing device can perform preliminary processing (including filtering and denoising operations) when the data is generated to reduce the complexity of subsequent data processing; let the edge computing processing function be g, then the data set processed by the edge computing device is expressed as: D'=g(D); the edge computing processing function is a data preprocessing operation performed on the edge computing device, which is mainly used to perform preliminary processing of data near the data source (such as a measuring station) to reduce the burden of subsequent data processing and reduce network transmission delay. Specific edge computing processing functions may include the following operations:
[0196] S531 Data cleaning: including denoising and removing outliers from the original data to improve data quality; in this embodiment, a sliding average filter, a median filter, and other methods are used to remove noise.
[0197] S532 Data compression: reducing the volume of data by lossy or lossless compression algorithm to reduce transmission delay and storage cost; in this embodiment, discrete cosine transform or wavelet transform is used to compress data;
[0198] S533 Feature extraction: extract useful features from the original data to reduce the data dimension and reduce the computational complexity. In this embodiment, the amplitude, duration, frequency and other features of the pulse signal are extracted from the lightning data.
[0199] After the Executor executes the task, it sends the result back to the Driver program. After the Driver program receives the results of all tasks, it aggregates the data for global analysis and calculates the real-time location results of the lightning source.
[0200] S54: Data aggregation and result calculation: The data processed by the computing nodes needs to be aggregated for global analysis; let the data aggregation function be h, then the aggregated data is expressed as: D agg =h(P1(t),P2(t),...P k (t)); h is the operation of finding the average value of each data, that is:
[0201]
[0202] P1, P2...Pk are the positions of k lightning bolts obtained on k subsets,
[0203] Based on the calculated D agg , calculate the real-time location result of the lightning source. Assume that the lightning location function is L, then the real-time location result is expressed as: L(t) = L(D agg );
[0204] The process of implementing real-time positioning based on Apache Spark is as follows:
[0205] Spark applications (consisting of Driver programs) are launched on a distributed cluster.
[0206] Spark creates a SparkContext object, which is the main connection to the cluster.
[0207] After creating a SparkContext, the application applies operations (Transformations and Actions) to distributed data, typically loaded from HDFS or other data sources.
[0208] Spark converts operations into a series of stages, each of which consists of a series of tasks. These tasks are scheduled by TaskScheduler and run on Executor.
[0209] These Executors run on worker nodes, and each Executor has a certain number of cores and memory.
[0210] Data is partitioned across the nodes in the cluster and transferred over the network.
[0211] When all tasks are completed, the application ends.
[0212] Example 2
[0213] On the basis of Example 1:
[0214] Figure 3 , Figure 4The three-dimensional real-time positioning diagram of ground-to-ground lightning and cloud-to-cloud lightning is shown. The experimental data comes from the three-dimensional positioning network of Shandong lightning low-frequency radiation sources. The positioning network includes 8 stations, namely Jiushan Station (JIUS), Liyazhuang Station (LIYA), Fengjia Station (FENG), Xiyang Station (XIYA), Dongshanhou Station (DOSH), Beishao Station (BEIS), Binhai Station (BIHN), among which JIUS Station is the central station of the positioning network. The baseline length of the station ranges from 6.3km to 30km, and the average baseline length is 14.9km. All stations are located in plain areas, with an altitude of 0-15m, and there is almost no difference in the height of the stations.
[0215] Each measuring station collects lightning signals through fast antenna and slow antenna. The effective frequency band ranges are 1.5kHz-2MHz and 10Hz-1MHz, and the time constants are 100μs and 220ms respectively.
[0216] The acquisition module is a dual-time base acquisition mode of ABA, refer to CN216350956U, lightning signal acquisition system. The dual-time base acquisition mode includes continuous acquisition of time base A and triggered acquisition of time base B. The sampling rates of time base A and time base B are different. The data signal output end of the antenna module is connected to the acquisition channel of the acquisition module. The acquisition module of this embodiment uses the German spectrum acquisition card to collect the change signal of the lightning electric field. The vertical resolution is 16 bits, and the maximum sampling rate of the four channels is 60MS / s. The dual-time base acquisition mode ABA with both trigger acquisition (B time base) and continuous acquisition (A time base) functions is adopted. The sampling rates of A and B modes can be adjusted appropriately according to needs. The number of sampling points triggered at one time in mode B can be set to 2 n (n=1, 2, 3...), using a higher sampling rate of 10MS / s, the sampling length of 223 sampling points is about 840ms, which is equivalent to the duration of a lightning strike. In mode A, the sampling rate is set to 400kS / s. The synchronous electric field signal for lightning location is mainly based on the data collected by mode B. The continuous acquisition mode A can detect weak radiation signals (signals below the trigger threshold may not be collected by mode B).
[0217] The collected electric field signals are denoised and normalized, and machine learning methods such as support vector machines (SVM) are used to identify pulses. The cross-correlation method is used to match the same radiation source pulses measured at different stations. The CHAN positioning algorithm is used to calculate the initial position of the lightning source, and then the genetic algorithm (GA) or differential evolution algorithm (DE) is used to optimize the initial positioning results. Finally, the distributed computing framework (Apache Spark) and edge computing technology are used to process and analyze the data in real time to achieve high-precision and efficient real-time three-dimensional lightning positioning.
[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A real-time three-dimensional lightning positioning method, characterized in that: The method comprises the following steps: S1: Data preprocessing, preprocessing the collected electric field signals; Including denoising and normalization operations to reduce the impact of noise on recognition; S2: Pulse identification, using support vector machine for binary classification to classify the signal into pulse signal and non-pulse signal; S3: Initial positioning calculation, using the triangulation method, based on the multi-station time difference measurement TDOA principle to measure the time difference between the signal arriving at multiple receiving stations, and using the CHAN positioning algorithm to calculate the initial position of the lightning source; S4: Optimization algorithm solution, using differential evolution algorithm to optimize the initial positioning result obtained in step S3 to obtain the accurate lightning source location; S5: Real-time positioning, using the Apache Spark distributed computing framework to process and analyze data in real time, while using edge computing technology to reduce transmission delays and improve real-time lightning positioning performance; Step S2 includes: S21: Feature extraction: extract features from the electric field signal preprocessed in step S1, select time domain, frequency domain or time-frequency domain features, perform feature selection method processing, and obtain the final features as the input of the support vector machine for training the classifier; S22: Data labeling: label the electric field signals through actual observation data; divide the signals into lightning pulse signals: positive class, and non-lightning pulse signals: negative class; S23: Training support vector machine classifier: Use the electric field signal annotated in step S22 to train the support vector machine classifier to obtain an optimal hyperplane to separate positive signals from negative signals. The objective function of the support vector machine can be expressed as: minimize: subjectto:y i ·(w T x i +b)≥1-ξ i ,i=1,...,N x i ≥0 Among them, w is the weight vector, b is the bias term, and x i is the input feature vector, y i is the corresponding label +1 or -1, ξ i is the slack variable, C is the penalty parameter; T is the transposed matrix, N is the total sample size; S24: Use kernel function: Introduce kernel function to map the original feature space to high-dimensional space The form of the RBF kernel function is: K(x i ,x j )=exp(-γx i -x j 2 ) Among them, γ is the parameter of the RBF kernel function; Xi and Xj are input samples, and ||Xi-Xj|| represents the Euclidean distance; S25: Predicting pulse signals: using the trained support vector machine classifier to classify the new electric field signal; for the electric field signal x marked in step S22, the prediction result can be calculated by the following formula: f(x)=w T ·φ(x)+b Among them, φ(x) is the feature vector after the kernel function maps the input signal to a high-dimensional space; if f(x)>0, then the signal x is classified as a pulse signal; if f(x)<0, it is classified as a non-pulse signal; S26: Set appropriate time windows and thresholds to match the same radiation source pulses measured at different stations: S261: Set time window: In order to correctly match the same radiation source pulses measured at different measuring stations, set a suitable time window; the time window should be determined according to the signal propagation speed and the distance between the measuring stations, that is, the time difference between the lightning signal to the two measuring stations is less than the time it takes for the speed of light to propagate between the two measuring stations; S262: Peak-finding threshold: Calculate the cumulative probability distribution of the electric field signal pulse and set a peak-finding threshold; when the cumulative probability reaches 99%, the corresponding amplitude is used as the peak-finding threshold; S263: Positioning pulse selection: in the time window, find the electric field signal with the largest amplitude as the positioning pulse; for pulses with an interval exceeding 15 μs, select an electric field signal with a suitable percentage threshold as the positioning pulse according to the distribution of the amplitude; S264: Pulse matching by cross-correlation method: pulse matching is performed by using the cross-correlation method; the time window is set to 350 μs, the sliding step is 50 μs, and the cross-correlation function of the electric field signals between each measuring station in the time window is calculated; the cross-correlation function is expressed as: R xy (τ)=Σ(x(t)·y(t+τ)) Where x(t) and y(t) represent the electric field signals of the two measuring stations, respectively, and τ represents the time delay; S27: Find the time delay corresponding to the peak of the cross-correlation function; the peak of the cross-correlation function represents the maximum similarity between the two signals; by finding the time delay corresponding to the peak, the same radiation source pulses measured at different measuring stations are matched.
2. A lightning real-time three-dimensional positioning method according to claim 1, characterized in that: Step S3 specifically includes: S31, set the positioning system to have H measuring stations, use S i (x i ,y i ,z i ), where i = 1, 2, ..., H; let S1 be the primary station and the others be the secondary stations; the spatial position of the lightning radiation source is P (x, y, z); S32. Calculate the time difference between the primary station and the secondary station: Δt i =t i -t1 Where i = 2, 3, ..., H; t i is the time when the lightning radiation source arrives at the i-th station; S33, calculating the distance difference according to the time difference: Δd i =c·Δt i Where i = 2, 3, ..., H; c is the speed of light; S34, construct the super time difference equations: (xx i ) 2 +(yy i ) 2 +(zz i ) 2 -(x-x1) 2 -(y-y1) 2 -(z-z1) 2 =Δd i 2 Where i = 2, 3, ..., H; S35. Simplify the system of equations, eliminate the square terms, and obtain the linear system of equations: A·X=B in, A=[2(x2-x1),2(y2-y1),2(z2-z1); 2(x3-x1),2(y3-y1),2(z3-z1); ....; 2(x N −x1),2(y N −y1),2(z N −z1)] X=[x,y,z] T B=[Δd2 2 +x1 2 +y1 2 +z1 2 -x2 2 -y2 2 -z2 2 ; <h2 style=";text-align:left;direction:ltr">Δd3<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +x1<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +y1<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +z1<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> -x3<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> -y3<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> -z3<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> ; ...; Δd N 2 +x1 2 +y1 2 +z1 2 -x N 2 -y N 2 -z N 2 ] S36. Use iterative least squares method: ILS to solve the linear equations and obtain the initial position estimate of the lightning radiation source: P(x,y,z)=(A T ·A) -1 ·A T ·B。 3. A lightning real-time three-dimensional positioning method according to claim 2, characterized in that: In step S4, the differential evolution algorithm DE is used to optimize the initial positioning result to obtain a more accurate lightning source location. S42 initialization: randomly generate an initial population containing V solution vectors, each solution vector has D dimensions; set algorithm parameters, mutation factor F, crossover rate CR and maximum number of iterations T; S42 mutation operation: For each solution vector xi, three different solution vectors r1, r2 and r3 are randomly selected from the population to calculate the difference vector: vi = r1 + F*(r2-r3), where r1, r2 and r3 are different from each other and have different magnitudes and directions from xi; S43 Crossover operation: For each solution vector xi, generate a test vector ui according to the crossover rate CR and the difference vector vi; in D dimensions, perform the following operations on each dimension j: Generate a random number randj, the value range is [0,1]; If randj≤CR or j is equal to a random integer, and the range of the random integer is [1,D], then set ui[j]=vi[j]; Otherwise, set ui[j] = xi[j]; Selection operation: For each solution vector xi, calculate its fitness value f(xi) and the fitness value f(ui) of the test vector ui; if f(ui)≤f(xi), replace xi with ui, otherwise keep the solution vector xi; Convergence check: Check whether the maximum number of iterations T is reached; if reached, output the optimal solution; otherwise, return to step S42 to continue iteration.
4. A lightning real-time three-dimensional positioning method according to claim 3, characterized in that: Step S5 is specifically as follows: S51. Data partitioning and task allocation: Let the collected lightning data set be D, and divide D into k subsets: D1, D2, ..., D k ; Each subset is assigned to a computing node to achieve distributed data processing; the set of computing nodes is designed to be in represents the i-th computing node; S52, Real-time data processing and analysis: Each computing node According to the assigned sub-dataset D i Perform real-time data processing and analysis; let the processing speed be P, and the data processing function be f, then the amount of data processed by the i-th computing node in time t can be expressed as: P i (t)=|f(D i )|; S53, edge computing technology: reduce the complexity of subsequent data processing; let the edge computing device set be E, where E i represents the i-th edge computing device; the edge computing device can perform preliminary processing when the data is generated, reducing the complexity of subsequent data processing; Assuming the edge computing processing function is g, the data set processed by the edge computing device is expressed as: D'=g(D); the specific edge computing processing function may include the following operations: S531 Data cleaning: including denoising and removing outliers from raw data to improve data quality; S532 Data Compression: Reduce the size of data through lossy or lossless compression algorithms to reduce transmission delays and storage costs; S533 Feature Extraction: Extract useful features from raw data to reduce data dimension and computational complexity; S54: Data aggregation and result calculation: The data processed by the computing nodes needs to be aggregated for global analysis; let the data aggregation function be h, then the aggregated data is expressed as: D agg =h(P1(t),P2(t),...P k (t)); h is the operation of finding the average value of each data, that is: P1, P2...Pk are the positions of k lightning bolts obtained on k subsets, Based on the calculated D agg , calculate the real-time location result of the lightning source; let the lightning location function be L, then the real-time location result is expressed as: L(t)=L(D agg ).
