A method for processing full upper atmosphere lidar data based on WASM technology

Abstract

A method for processing full-high-atmosphere lidar data based on WASM technology includes the following steps: first, determining the structure and semantics of the query, interpreting the SQL query, obtaining the interpretation result, and then generating a QEP according to the interpretation result, determining the operations that need to be materialized for each lidar data in the QEP, and disassembling the QEP into multiple pipelines, using the operations for materializing each lidar data as the splitting points of the pipeline; converting each pipeline in the QEP into WebAssembly code, and then merging the WebAssembly codes of each pipeline to generate a specific WebAssembly code, storing it in the memory of the WebAssembly module to obtain a result set; writing the result set into the memory, and then reading the result set from the module through the host. The present invention has a faster data reading and processing speed when the data volume is very large.

Description

Technical Field

[0001] The present invention relates to an improvement of a laser radar data processing technology, belongs to the field of atmosphere detection, and in particular to a full high-altitude atmosphere laser radar data processing method based on WASM technology. Background Art

[0002] The Earth's near-space ranges from a dozen kilometers to 100 kilometers above sea level, including multiple atmospheric layers such as the stratosphere, mesosphere and lower thermosphere. It is the only way for space launch and return, and is also the activity area for long-range missiles, hypersonic vehicles and resident airships. The detection of parameters such as atmospheric density, temperature and wind field in this space is a major demand for the recognition, development and utilization of the atmospheric characteristics and laws of near-space and the guarantee of space activities. Due to the different composition and activity laws of different atmospheric layers, the detection of atmospheric parameters requires different detection mechanisms to obtain more accurate atmospheric parameters.

[0003] The use of lidar to detect near-space atmospheric parameters is currently the method with the widest altitude coverage and the highest temporal and spatial resolution. It usually uses Raman scattering, Rayleigh scattering and atomic fluorescence scattering mechanisms generated by the interaction between laser and atmosphere. Below 30km, because of the atmospheric aerosols, pure rotational Raman scattering is usually used to obtain the scattered echo light signals of high and low quantum numbers of nitrogen and oxygen molecules that vary with altitude. There are 2 sets of data in total, which can be used to invert the change of atmospheric temperature with altitude. The altitude range of 30-80km is mainly composed of nitrogen and oxygen molecules. Rayleigh scattering is usually used to obtain Rayleigh scattering echo signals of nitrogen and oxygen molecules, 1 set of data in the vertical direction, and the change of atmospheric density and temperature with altitude is inverted. 4 sets of data are obtained in the two directions of longitude and latitude to invert the change of atmospheric horizontal wind field with altitude. There are many kinds of atoms in the altitude range of 80-100km, and the resonance fluorescence scattering of atoms is used to invert the change of atmospheric horizontal wind field with altitude. Combined with the laser triple-frequency switching, a total of 9 groups of data are obtained in the vertical, longitude and latitude directions, and the changes in atomic density, temperature and horizontal wind field with height are inverted, which is superior to the atmospheric detection lidar. The detection mechanism of different atmospheric layers is different. Usually, different data are accessed and read, quality screened, denoised, etc., and then inverted to obtain physical parameters such as atmospheric density, temperature and wind field, and then stored, displayed, and transmitted. With the development and promotion of atmospheric detection lidar's application, faced with the processing of massive heterogeneous data from multi-site observation points, traditional data processing methods have complex data access, low data reading and writing storage efficiency, and are restricted by platform software, which limits the speed and efficiency of data processing.

[0004] The amount of data collected by lidar is very large, containing rich time, space and scattering information. In traditional data processing methods, the efficiency of accessing and reading this data is low, especially when processing multi-site observation data. Traditional systems usually need to perform multiple reading and conversion operations on the data, which not only increases processing time, but also may cause data access bottlenecks and affect overall performance.

[0005] A Chinese patent application with application number CN202210718580.2 and application date June 23, 2022 discloses a laser radar data processing method and device, including: collecting laser laser radar signals, wherein the laser laser radar signals include a local oscillator signal emitted by a laser seed source and an echo signal reflected by a target; performing digital down-conversion processing on the laser laser radar signal to obtain a first signal, which is a complex signal vector; in the frequency domain, performing pulse compression processing on the first signal, wherein the pulse compression processing is implemented based on the first signal and a template signal to obtain a second signal, which is a real number vector; based on the second signal, determining the moment of the effective signal, so as to determine the distance information of the target based on the moment of the effective signal. This scheme greatly improves the data processing efficiency by utilizing the parallel computing of the GPU, but the above technology does not solve the problem of slow data reading and processing speed when the data volume is very large.

[0006] The information disclosed in this background technology section is only intended to increase the understanding of the overall background of this patent application, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention

[0007] The purpose of the present invention is to overcome the problem in the prior art that the data reading and processing speed is slow when the data volume is very large, and to provide a full high-altitude atmosphere lidar data processing method based on WASM technology with a faster data reading and processing speed when the data volume is very large.

[0008] To achieve the above objectives, the technical solution of the present invention is: a full upper atmosphere lidar data processing method based on WASM technology, the full upper atmosphere lidar data processing method based on WASM technology comprises the following steps:

[0009] Step 1: First, determine the structure and semantics of the query, interpret the SQL query, obtain the interpretation result, and then generate a QEP based on the interpretation result. In the QEP, determine the operations that need to be materialized for each lidar data. At the same time, disassemble the QEP into multiple pipelines, and use the operation of materializing each lidar data as the split point of the pipeline;

[0010] The second step is to convert each pipeline in QEP into WebAssembly code, then merge the WebAssembly codes of each pipeline, and finally generate a complete WebAssembly module containing all pipeline operations;

[0011] Step 3: Select a WebAssembly engine that supports fast compilation and optimization features from the WebAssembly module, set the memory of the WebAssembly module, and then execute the WebAssembly code in the WebAssembly engine that supports fast compilation and optimization features to generate WebAssembly code for specific algorithms and data structures, and store the generated code in the memory of the WebAssembly module to obtain a result set;

[0012] Step 4: Write the result set to the memory of the WebAssembly module, then read the result set from the WebAssembly module through the host and convert the result set into a form used by the host.

[0013] In the first step, the QEP is simultaneously disassembled into multiple pipelines, which means that after the QEP is generated, it is disassembled into multiple independent pipelines, and each pipeline operation is allocated to an independent memory space.

[0014] During the execution of each pipeline, dedicated WebAssembly code is dynamically generated according to its operation type.

[0015] The specific operations of disassembling QEP into multiple pipelines are as follows:

[0016] A. Scan data from table R and perform selection operations;

[0017] B. Scan data from table S and perform hash join;

[0018] C. Group and aggregate the join results.

[0019] After the QEP is disassembled into a pipeline, a table scan operation is performed to traverse the data rows in the database table in order, and then a selection operation is performed to select the records that meet the conditions. The data is then connected using a hash connection. The data integrated by the hash connection is then used to quickly match the related records of the two data sets, and the data integrated by the hash connection is used as the input for the aggregation operation.

[0020] The second step also includes custom memory mapping, specifically: in the WebAssembly module, the lidar data is directly mapped to the memory address space of the WebAssembly module through the custom memory mapping function SetModuleMemory().

[0021] During the query execution process, the generated result set will be written directly to the pre-allocated WebAssembly module memory area. By defining an efficient result writing function write_result() in the WebAssembly code, it is ensured that the results can be accurately stored in memory in order.

[0022] In the third step, a WebAssembly engine that supports fast compilation and optimization features is first selected from the WebAssembly module, specifically: the basic algorithms and data structures required are determined from the QEP, and according to the above results, corresponding WebAssembly codes are generated for these commonly used algorithms and data structures, and the compiled WebAssembly codes are delegated to the WebAssembly engine for execution. The WebAssembly engine will be responsible for compiling the WebAssembly code into machine code and optimizing the execution.

[0023] The WebAssembly engine adopts the V8 engine.

[0024] In the fourth step, the result set is written into the memory of the WebAssembly module, and then the result set is read from the WebAssembly module through the host, and the result set is converted into a form used by the host. Specifically, during the execution process, the query results are written into a pre-allocated memory area, and the result set is transmitted back to the host system through a memory mapping or callback mechanism.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. In a method for processing full-high-atmosphere lidar data based on WASM technology, the query QEP is compiled into WebAssembly code and executed in the V8 engine, and lidar data, such as detection height, photon count, etc., are directly mapped to the memory address space of the WebAssembly module, avoiding the common multiple data copying and moving in the traditional processing method, thereby reducing the memory overhead and delay during processing, improving the reading speed of large amounts of data, and realizing efficient and low-latency data processing. Through the efficient processing capability of WebAssembly, lidar data at different heights and directions can be quickly analyzed and processed to generate key parameters such as atmospheric density, temperature, and wind field. The high efficiency of WebAssembly can ensure that when the lidar data is inverted, signal screened, denoised, etc., real-time processing capabilities are provided to meet the high-speed acquisition and analysis requirements of lidar data. Therefore, the present invention has a faster data reading and processing speed when the data volume is very large.

[0027] 2. In a method for processing full-high-atmosphere lidar data based on WASM technology, during query execution, the result set is written into the WebAssembly module memory and read through memory mapping or callback mechanism, which can avoid unnecessary data copying and improve the efficiency of result set processing. WASM has cross-platform characteristics, so that data processing no longer depends on a specific operating system or hardware. LiDAR data can be seamlessly transmitted and processed between different platforms without additional adaptation or conversion steps. This advantage not only improves the flexibility of processing, but also greatly reduces the development and deployment costs between different environments. Therefore, the present invention improves the flexibility of processing and reduces costs.

[0028] 3. In the method for processing full-high-atmosphere lidar data based on WASM technology, WASM allows complex lidar data processing tasks to be decomposed into multiple parallel execution processes, which can fully utilize the multi-core and parallel processing capabilities of modern computing platforms to speed up data processing. At the same time, the WASM engine can optimize the code in real time during execution to ensure that it always runs at optimal performance during the processing process, which is particularly important for real-time analysis of lidar data and can meet the needs of concurrent processing of multiple stations. Therefore, the present invention can process multiple stations concurrently and speed up data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a trade-off diagram of the engine compilation time and execution time of the present invention.

[0030] Figure 2 It is the execution flow chart of QEP of the present invention.

[0031] Figure 3 It is an architecture diagram of the WebAssembly platform of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] See also Figures 1 to 3 A method for processing full upper atmosphere lidar data based on WASM technology, the method comprising the following steps:

[0034] Step 1: First, determine the structure and semantics of the query, interpret the SQL query, obtain the interpretation result, and then generate a QEP based on the interpretation result. In the QEP, determine the operations that need to be materialized for each lidar data. At the same time, disassemble the QEP into multiple pipelines, and use the operation of materializing each lidar data as the split point of the pipeline;

[0035] The second step is to convert each pipeline in the QEP into WebAssembly code, and then merge the WebAssembly codes of each pipeline to finally generate a complete WebAssembly module containing all pipeline operations, which can execute the entire query operation;

[0036] Step 3: Select a WebAssembly engine that supports fast compilation and optimization features from the WebAssembly module, set the memory of the WebAssembly module, and then execute the WebAssembly code in the WebAssembly engine that supports fast compilation and optimization features to generate WebAssembly code for specific algorithms and data structures, and store the generated code in the memory of the WebAssembly module to obtain a result set;

[0037] Step 4: Write the result set to the memory of the WebAssembly module, then read the result set from the WebAssembly module through the host and convert the result set into a form used by the host.

[0038] In the first step, the QEP is simultaneously disassembled into multiple pipelines, which means that after the QEP is generated, it is disassembled into multiple independent pipelines, and each pipeline operation is allocated to an independent memory space.

[0039] During the execution of each pipeline, dedicated WebAssembly code is dynamically generated according to its operation type.

[0040] The specific operations of disassembling QEP into multiple pipelines are as follows:

[0041] A. Scan data from table R and perform selection operations;

[0042] B. Scan data from table S and perform hash join;

[0043] C. Group and aggregate the join results.

[0044] After the QEP is disassembled into a pipeline, a table scan operation is performed to traverse the data rows in the database table in order, and then a selection operation is performed to select the records that meet the conditions. The data is then connected using a hash connection. The data integrated by the hash connection is then used to quickly match the related records of the two data sets, and the data integrated by the hash connection is used as the input for the aggregation operation.

[0045] The second step also includes custom memory mapping, specifically: in the WebAssembly module, the lidar data is directly mapped to the memory address space of the WebAssembly module through the custom memory mapping function SetModuleMemory().

[0046] During the query execution process, the generated result set will be written directly to the pre-allocated WebAssembly module memory area. By defining an efficient result writing function write_result() in the WebAssembly code, it is ensured that the results can be accurately stored in memory in order.

[0047] In the third step, a WebAssembly engine that supports fast compilation and optimization features is first selected from the WebAssembly module, specifically: the basic algorithms and data structures required are determined from the QEP, and according to the above results, corresponding WebAssembly codes are generated for these commonly used algorithms and data structures, and the compiled WebAssembly codes are delegated to the WebAssembly engine for execution. The WebAssembly engine will be responsible for compiling the WebAssembly code into machine code and optimizing the execution.

[0048] The WebAssembly engine adopts the V8 engine.

[0049] In the fourth step, the result set is written into the memory of the WebAssembly module, and then the result set is read from the WebAssembly module through the host, and the result set is converted into a form used by the host. Specifically, during the execution process, the query results are written into a pre-allocated memory area, and the result set is transmitted back to the host system through a memory mapping or callback mechanism.

[0050] The supplementary description of the present invention is as follows:

[0051] WASM (WebAssembly) technology can get rid of the limitations of data processing platforms. Its cross-platform characteristics allow high-performance applications to be executed directly in web browsers, providing a platform-independent way for efficient processing of lidar data. It can not only overcome the limitations of traditional data processing methods, including extended incremental encoding technology, but also provide higher data processing efficiency and accuracy.

[0052] Embodiment 1:

[0053] A method for processing full upper atmosphere lidar data based on WASM technology, the method comprising the following steps:

[0054] Step 1: First, determine the structure and semantics of the query, interpret the SQL query, obtain the interpretation result, and then generate a QEP based on the interpretation result. In the QEP, determine the operations that need to be materialized for each lidar data. At the same time, disassemble the QEP into multiple pipelines, and use the operation of materializing each lidar data as the split point of the pipeline;

[0055] The second step is to convert each pipeline in the QEP into WebAssembly code, and then merge the WebAssembly codes of each pipeline to finally generate a complete WebAssembly module containing all pipeline operations, which can execute the entire query operation;

[0056] Step 3: Select a WebAssembly engine that supports fast compilation and optimization features from the WebAssembly module, set the memory of the WebAssembly module, and then execute the WebAssembly code in the WebAssembly engine that supports fast compilation and optimization features to generate WebAssembly code for specific algorithms and data structures, and store the generated code in the memory of the WebAssembly module to obtain a result set;

[0057] Step 4: Write the result set to the memory of the WebAssembly module, then read the result set from the WebAssembly module through the host and convert the result set into a form used by the host.

[0058] When applied: compile the query execution plan (QEP) into WebAssembly. Once the WebAssembly code is generated, it can be quickly compiled into machine code in the WebAssembly engine and start executing. When the WebAssembly engine executes the code, the code will first be executed in the form of fast compilation. As the data processing proceeds, the WebAssembly engine will gradually optimize the code to achieve a seamless transition from unoptimized code to optimized code. Through the gradually optimized code, the processing efficiency of lidar data has been improved. This efficient processing can not only reflect data changes in real time, but also ensure that the processing results are accurate and reliable, ultimately providing accurate data support for atmospheric detection.

[0059] Embodiment 2:

[0060] Embodiment 2 is substantially the same as Embodiment 1, except that:

[0061] In the first step, the QEP is disassembled into multiple pipelines at the same time, which means that after the QEP is generated, it is disassembled into multiple independent pipelines, and each pipeline operation is allocated to an independent memory space. This memory allocation method can effectively reduce the memory access conflict between pipeline operations, so that each pipeline can be executed in parallel, which improves the overall data processing speed. During the execution of each pipeline, a dedicated WebAssembly code is dynamically generated according to its operation type; after the QEP is disassembled into pipelines, When performing table scan operations, dedicated WebAssembly code can be generated according to the specific needs of each pipeline, avoiding the redundancy of general code and improving execution efficiency. The data rows in the database table are traversed in order, and then selection operations are performed to select qualified records. The data is then connected using a hash connection. The data integrated by the hash connection is then used to quickly match the relevant records of the two data sets. The data integrated by the hash connection is used as the input for the aggregation operation. This customized code generation method can avoid the function call overhead brought by traditional precompiled libraries, thereby improving the overall execution efficiency of the query.

[0062] When applied: QEP is a structured representation that defines how to execute each step of the query. After the QEP is generated, it needs to be further decomposed into multiple pipelines that can be executed independently. Each pipeline corresponds to one or more specific operations, such as table scan, selection, connection and aggregation. This decomposition process is crucial because it determines how to parallelize the execution of the query to improve processing efficiency. Once the QEP is decomposed into pipelines, the next step is to execute the table scan operation. Table scan is the first step to read data from the database. It usually traverses the data rows in the database table in order. In this process, all eligible data rows will be retrieved as input for subsequent steps. After the table scan is completed, the next step is the selection operation. The selection operation is to filter data according to specific conditions. , for example, select records that meet a certain condition. This step ensures that only relevant data will be passed to subsequent processing steps, thereby reducing unnecessary data processing and improving overall efficiency. After data selection, the next step is data connection, usually using hash connection. Hash connection is an efficient data connection method, especially suitable for large-scale data sets. It quickly matches related records of two data sets through hash tables. This step integrates related data in different tables together to provide complete information input for the final result generation. The data integrated by hash connection will be used as the input of the aggregation operation. At this time, the data has been streamlined and organized, and only further grouping and calculation are required. Finally, the data after hash connection will enter the aggregation operation stage. In this stage, the data is grouped according to certain specific attributes, such as date, category, etc., and then the data in each group is calculated, such as sum, average, etc., to obtain the final query result, covering all the information required by the SQL query.

[0063] Embodiment 3:

[0064] Embodiment 3 is substantially the same as Embodiment 1, except that:

[0065] Interpret the SQL query and generate the corresponding QEP, which is a tree structure representing the sequence of query operations.

[0066] For example, parsing the following SQL query:

[0067] SELECT Rx, MIN(Sx)

[0068] FROM R, S

[0069] WHERE Rx<42 AND R.id = S.rid

[0070] GROUP BY Rx;

[0071] The generated QEP is similar to a tree, which includes operations such as table scan, selection, join, and grouping.

[0072] Identify operations that need to be materialized, such as grouping, joining, and sorting, and use them as split points in the pipeline;

[0073] Disassemble the above QEP into the following pipeline;

[0074] A. Scan data from table R and perform selection operations;

[0075] B. Scan data from table S and perform hash join;

[0076] C. Group and aggregate the connection results;

[0077] Compile each pipeline into WebAssembly code and into a loop structure for reading table data, as follows:

[0078] (loop

[0079] (if (i32.lt_s (local.get $i) (global.get $table_length))

[0080] (then

[0081] ;; Operations to read table data

[0082] (local.set $data (i32.load (i32.add (global.get $table_base)(local.get$i))))

[0083] (local.set $i (i32.add (local.get $i) (i32.const 4))) ) ) )

[0087] Select: Compile as conditional branch;

[0088] (if (f64.lt (local.get $R_x)(f64.const 42))

[0089] (then

[0090] ;; Operations when conditions are met ) )

[0093] Join: Compiles to a hash join operation;

[0094] (loop

[0095] (if (i32.eq (i32.load (local.get $R_id)) (i32.load (local.get$S_rid)))

[0096] (then

[0097] ;; Perform connection operation ) ) )

[0101] Grouping and aggregation: compiled into hash grouping and aggregation operations;

[0102] (loop

[0103] (if (i32.eq (local.get $group_key) (i32.load (local.get$hash_table_key)))

[0104] (then

[0105] ;; Update aggregate value )

[0107] (else

[0108] ;; Insert a new group ) ) )

[0112] Use a custom SetModuleMemory() function to map the data in the database system to the memory of the WebAssembly module to avoid unnecessary data copying overhead;

[0113] For example, map the data addresses of table R and table S into the memory of the WebAssembly module:

[0114] void SetModuleMemory(uint32_t module_memory_base,void* data_address,size_t size);

[0115] SetModuleMemory(module_memory_base, table_R_data, table_R_size);

[0116] SetModuleMemory(module_memory_base + table_R_size, table_S_data,table_S_size);

[0117] Delegate the compiled WebAssembly code to the WebAssembly engine for execution. The V8 engine will be responsible for compiling the WebAssembly code into machine code and optimizing the execution.

[0118] v8::Isolate* isolate = v8::Isolate::New(create_params);

[0119] v8::HandleScope handle_scope(isolate);

[0120] v8::Local <v8::context>context = v8::Context::New(isolate);

[0121] v8::Context::Scope context_scope(context);

[0122] v8::Local <v8::object>global = context->Global();

[0123] v8::Local <v8::function>func =v8::Local <v8::function>::Cast(global->Get(context, v8::String::NewFromUtf8(isolate, "execute_query")).ToLocalChecked());

[0124] func->Call(context, global, 0, nullptr);

[0125] Generate WebAssembly code for specific algorithms and data structures;

[0126] By analyzing QEP, determine the required algorithms and data structures, such as hash tables, sorting algorithms, etc.;

[0127] For example, the WebAssembly code that generates a hash table:

[0128] (func $insert_into_hash_table (param $key i32) (param $value i32)

[0129] ;; Calculate the hash value

[0130] (local $hash i32)

[0131] (local.set $hash (i32.rem_u (local.get $key) (i32.const 1024)))

[0132] ;; Insert into hash table

[0133] (i32.store (i32.add (global.get $hash_table_base) (local.get$hash))(local.get $value)) )

[0135] Generate and compile highly specialized WebAssembly code on the fly;

[0136] Write the result set into the WebAssembly module’s memory;

[0137] During execution, query results are written to a pre-allocated memory area;

[0138] For example, writing query results to WebAssembly module memory:

[0139] (func $write_result (param $result i32)

[0140] (i32.store (i32.add (global.get $result_base) (global.get $result_offset)) (local.get$result))

[0141] (global.set $result_offset (i32.add (global.get $result_offset)(i32.const 4))) )

[0143] Read the result set from the WebAssembly module;

[0144] The result set is transferred back to the host system via memory mapping or callback mechanism.

[0145] For example, to read a result set from a WebAssembly module:

[0146] uint32_t* result = reinterpret_cast<uint32_t*> (module_memory_base +result_offset);

[0147] for (size_t i = 0; i <result_count; ++i) {

[0148] std::cout<<"Result: "< <result[i]<<std::endl;

[0149] }

[0150] Embodiment 4:

[0151] Embodiment 4 is substantially the same as Embodiment 1, except that:

[0152] The V8 engine first quickly compiles the WebAssembly code into machine code so that it can be executed immediately to meet the low latency requirements. During the execution process, the V8 engine also optimizes the code in real time, especially the hot code paths, so that it can run with higher efficiency in subsequent executions. For example, in the initial execution stage, the code may access memory through direct address calculation, and in the subsequent stages, through the optimization of the V8 engine, memory access operations may be further simplified or pre-fetched, thereby improving performance.

[0153] Embodiment 5:

[0154] Embodiment 5 is substantially the same as Embodiment 1, except that:

[0155] like Figure 1 As shown, Figure 1 The trade-offs between compilation time and execution time of different query execution engines are shown. The generation and disassembly of QEP are key steps to achieve efficient query execution. Complex queries are decomposed into multiple executable pipelines. Each pipeline contains operations such as table scan, selection, join, and aggregation. Table scan operations are performed by traversing table data. Selection operations filter data according to conditions. Join operations are performed through hash tables. Aggregation operations group and calculate data.

[0156] like Figure 2 As shown, Figure 2 The execution process of QEP is shown. The WebAssembly code of each pipeline is compiled independently and data is transferred through shared memory during execution. The WebAssembly module is executed in the V8 engine, and its fast compilation and optimization features are used to significantly improve query execution performance.

[0157] like Figure 3 As shown, Figure 3 The architecture of the WebAssembly platform in the browser engine is demonstrated. For specific query requirements, highly specialized WebAssembly code is dynamically generated and compiled, avoiding the limitations of traditional pre-compiled libraries. The WebAssembly code directly interacts with the data in the database system, and low-latency data processing is achieved through efficient memory mapping technology. The method of the present invention is particularly suitable for processing large amounts of high-speed data, such as a large amount of scattered echo light signals generated by lidar.

[0158] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A method for processing full upper atmosphere lidar data based on WASM technology, characterized in that: The full upper atmosphere lidar data processing method based on WASM technology comprises the following steps: Step 1: First, determine the structure and semantics of the query, interpret the SQL query, obtain the interpretation result, and then generate a query execution plan (QEP) based on the interpretation result. In the QEP, determine the operations that need to be materialized for each lidar data, and disassemble the QEP into multiple pipelines, using the operation of materializing each lidar data as the split point of the pipeline; After the QEP is generated, it is disassembled into multiple independent pipelines, and each pipeline operation is allocated to an independent memory space; During the execution of each pipeline, dedicated WebAssembly code is dynamically generated according to its operation type; The specific operations of disassembling QEP into multiple pipelines are as follows: A. Scan data from table R and perform selection operations; B. Scan data from table S and perform hash join; C. Group and aggregate the connection results; The second step is to convert each pipeline in QEP into WebAssembly code, then merge the WebAssembly codes of each pipeline, and finally generate a complete WebAssembly module containing all pipeline operations; Step 3: Select a WebAssembly engine that supports fast compilation and optimization features from the WebAssembly module, set the memory of the WebAssembly module, and then execute the WebAssembly code in the WebAssembly engine that supports fast compilation and optimization features to generate WebAssembly code for specific algorithms and data structures. Store the generated code in the memory of the WebAssembly module to obtain a result set. Step 4: Write the result set to the memory of the WebAssembly module, then read the result set from the WebAssembly module through the host and convert the result set into a form used by the host.

2. According to claim 1, a method for processing full upper atmosphere lidar data based on WASM technology is characterized in that: After the QEP is disassembled into a pipeline, a table scan operation is performed to traverse the data rows in the database table in order, and then a selection operation is performed to select the records that meet the conditions. The data is then connected using a hash connection. The data integrated by the hash connection is then used to quickly match the related records of the two data sets, and the data integrated by the hash connection is used as the input for the aggregation operation.

3. The method for processing full upper atmosphere lidar data based on WASM technology according to claim 1 is characterized in that: The second step also includes custom memory mapping, specifically: in the WebAssembly module, the lidar data is directly mapped to the memory address space of the WebAssembly module through the custom memory mapping function SetModuleMemory().

4. The method for processing full upper atmosphere lidar data based on WASM technology according to claim 3 is characterized in that: During the query execution process, the generated result set will be written directly to the pre-allocated WebAssembly module memory area. By defining an efficient result writing function write_result() in the WebAssembly code, it is ensured that the results can be accurately stored in memory in order.

5. The method for processing full upper atmosphere lidar data based on WASM technology according to claim 1 is characterized in that: In the third step, a WebAssembly engine that supports fast compilation and optimization features is first selected from the WebAssembly module, specifically: the basic algorithms and data structures required are determined from the QEP, and according to the above results, corresponding WebAssembly codes are generated for these commonly used algorithms and data structures, and the compiled WebAssembly codes are delegated to the WebAssembly engine for execution. The WebAssembly engine will be responsible for compiling the WebAssembly codes into machine codes and optimizing the execution.

6. The method for processing full upper atmosphere lidar data based on WASM technology according to claim 5 is characterized in that: The WebAssembly engine adopts the V8 engine.

7. The method for processing full upper atmosphere lidar data based on WASM technology according to claim 1 is characterized in that: In the fourth step, the result set is written into the memory of the WebAssembly module, and then the result set is read from the WebAssembly module through the host, and the result set is converted into a form used by the host. Specifically, during the execution process, the query results are written into a pre-allocated memory area, and the result set is transmitted back to the host system through a memory mapping or callback mechanism.

Images