Method and device for testing and evaluating performance of urban low-altitude airspace multi-modal perception network

Abstract

The application discloses a kind of urban low-altitude airspace multi-modal sensing network performance test and evaluation method and equipment, it is related to urban low-altitude airspace multi-modal sensing network performance test technical field, method includes: the test environment including multiple urban complex scenes is constructed;Based on test environment, simulation test is carried out to urban low-altitude airspace multi-modal sensing network, and using the analog signal and analog data provided by test environment, the data input end, data processing end and index calculation end of urban low-altitude airspace multi-modal sensing network are successively verified;Based on real test data and true value data, the score of multiple performance indexes of urban low-altitude airspace multi-modal sensing network is calculated.Further determine the performance level of urban low-altitude airspace multi-modal sensing network.The application can provide a set of objective, quantitative, graded comprehensive performance evaluation system for the construction and application of low-altitude sensing network.

Description

Technical Field

[0001] This invention relates to the field of performance testing technology for multimodal sensing networks in urban low-altitude airspace, and particularly to a method and equipment for performance testing and evaluation of multimodal sensing networks in urban low-altitude airspace. Background Technology

[0002] With the increasing application of low-altitude airspace and the growing variety of aircraft, the low-altitude industry is expanding further. While the low-altitude economy is developing rapidly, safety hazards are also increasing. In particular, the effective management of non-cooperative drones, due to their large scale and difficulty in detection, is a key focus and challenge in low-altitude control. A crucial means to improve control capabilities is to enhance low-altitude perception capabilities. Currently, low-altitude airspace generally suffers from insufficient perception capabilities, severe information silos, and prominent operational safety hazards, lacking a unified, continuous, and real-time airspace perception infrastructure. Furthermore, the urban low-altitude airspace environment is complex, and single or limited technical means are insufficient to effectively perceive non-cooperative target drones. Therefore, it is necessary to design an urban low-altitude drone perception network that combines multiple perception technologies based on the characteristics of urban scenarios to solve the above problems and achieve effective perception of non-cooperative target drones.

[0003] To assess and improve low-altitude sensing capabilities, existing technologies typically involve performance testing of sensing devices or networks. These testing methods are often conducted in controlled laboratory environments or simple outdoor settings, measuring fundamental parameters such as detection range and positioning accuracy by injecting standard signals or launching cooperative target drones. The testing focuses on verifying the tracking capabilities of a single type of sensor or a known cooperative target, aiming to obtain basic performance data of the device under ideal or simplified conditions.

[0004] However, existing testing methods have significant drawbacks. Current tests primarily focus on laboratory environments, making it difficult to simulate complex urban scenarios, leading to a disconnect between test results and the actual performance of sensing networks in real-world urban environments. Due to the lack of effective verification of multi-sensor fusion performance in complex scenarios, the assessment of sensing capabilities against non-cooperative targets is weak. Consequently, the constructed sensing networks often exhibit high false alarm probabilities, poor detection continuity, and a sharp decline in performance under harsh environments or complex electromagnetic conditions in actual deployments. Furthermore, existing tests lack standardized verification processes across the entire data input, data processing, and index calculation chain, as well as a comprehensive performance rating system based on real-world scenarios, resulting in insufficient comprehensiveness and authority of the test conclusions.

[0005] Therefore, it is necessary to design a full-process testing and evaluation method that can closely integrate with the real characteristics of the urban environment and conduct full-process testing and evaluation of the urban low-altitude airspace multimodal sensing network, from simulation to actual measurement, from local verification to comprehensive evaluation, so as to accurately measure its actual ability to cope with complex scenarios and non-cooperative targets, thereby guiding the standardized construction and performance improvement of the low-altitude sensing network. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of existing technologies. Specifically, it provides a method and equipment for performance testing and evaluation of multimodal sensing networks in urban low-altitude airspace, as detailed below:

[0007] 1) In a first aspect, the present invention provides a method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace, the specific technical solution of which is as follows:

[0008] Construct a test environment that includes various complex urban scenarios;

[0009] Based on the test environment, a simulation test was conducted on the multimodal sensing network of urban low-altitude airspace. Using the simulated signals and data provided by the test environment, the data input end, data processing end, and index calculation end of the multimodal sensing network of urban low-altitude airspace were verified in sequence.

[0010] In a real physical environment, the validated urban low-altitude airspace multimodal sensing network is tested. The ground truth data provided by the corresponding ground truth system and the real test data output by the urban low-altitude airspace multimodal sensing network are collected simultaneously. Based on the real test data and ground truth data, the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network are calculated.

[0011] The performance level of the urban low-altitude airspace multimodal sensing network is determined based on the scores of multiple performance indicators.

[0012] The beneficial effects of the performance testing and evaluation method for a multimodal sensing network in urban low-altitude airspace according to the present invention are as follows:

[0013] By constructing a test environment encompassing various complex urban scenarios, the limitations of existing testing methods in simulating real-world urban environments are effectively overcome, laying the foundation for evaluating the performance of the sensing network under near-real-world conditions. Simulation tests were conducted on the urban low-altitude airspace multimodal sensing network based on this test environment. Using simulated signals and data provided by the test environment, the data input end, data processing end, and index calculation end were verified sequentially, achieving standardized verification of the entire information processing chain of the sensing network. This enables systematic problem localization and improves the overall perception capability against non-cooperative targets. The verified urban low-altitude airspace multimodal sensing network was then tested in a real physical environment. Simultaneously, ground truth data provided by the corresponding ground truth system and real test data output by the sensing network were collected, ensuring that the performance evaluation results are derived from actual physical conditions and enhancing the reliability and authority of the test conclusions. Performance levels were determined based on multiple performance index scores calculated from real test data and ground truth data, ultimately providing an objective, quantitative, and graded comprehensive performance evaluation system for the construction and application of low-altitude sensing networks.

[0014] Based on the above scheme, the performance testing and evaluation method of a multimodal sensing network in urban low-altitude airspace of the present invention can be further improved as follows.

[0015] Furthermore, based on the test environment, simulation tests were conducted on the urban low-altitude airspace multimodal sensing network, including:

[0016] Run the urban low-altitude airspace multimodal sensing network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expectations.

[0017] The beneficial effects of adopting the above-mentioned further approach are: by running the urban low-altitude airspace multimodal perception network in the test environment and generating preliminary performance evaluation results, it is possible to quickly identify the basic performance of the system in simulated complex scenarios before it is put into subsequent detailed verification and real-world testing. This step plays an early screening role, avoiding the deployment of systems with obvious defects or that do not meet basic requirements to more time-consuming subsequent stages, thereby improving the efficiency and resource utilization of the overall testing process.

[0018] Furthermore, using simulated signals and data provided by the test environment, the data input end, data processing end, and index calculation end of the urban low-altitude airspace multimodal sensing network were verified sequentially, including:

[0019] When the initial performance evaluation results meet the expectations, the simulated signal provided by the test environment will be injected into the data input terminal to verify the data reception function;

[0020] Based on the requirements of the simulated urban test scenario in the test environment, the simulated data corresponding to the requirements of the urban test scenario is input into the data processing terminal for logic verification and debugging.

[0021] Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal.

[0022] The beneficial effects of adopting the above-mentioned further solution are as follows: by conducting targeted verification at the data input end, data processing end, and indicator calculation end in sequence, step-by-step control over the entire chain of information processing for the urban low-altitude airspace multimodal sensing network is achieved. This method can systematically isolate and locate problematic links, ensuring the reliability of data reception, the correctness of fusion logic, and the accuracy of indicator calculation, thus laying a solid foundation for the stable and accurate operation of the entire system in a real environment.

[0023] Furthermore, the performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability.

[0024] Several performance metrics of the urban low-altitude airspace multimodal sensing network were calculated based on real test data and ground truth data, including:

[0025] Align the collected ground truth data with the actual test data in terms of time and space;

[0026] In the ground truth data and real test data after time and space alignment, according to the set evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is statistically analyzed, and this ratio is used as the detection probability.

[0027] In the real data and actual test data after time and space alignment, the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network to the same test target is updated per unit time, and this number of updates is used as the detection information update frequency.

[0028] In the real data and the actual test data after time and space alignment, the horizontal error and vertical error between the actual test data and the real data are calculated respectively, which are used as the detection accuracy.

[0029] When the test target triggers a preset abnormal event, the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal perception network are recorded, and the detection alarm delay is calculated based on the time difference between the first timestamp and the second timestamp.

[0030] In the real data and actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability.

[0031] The beneficial effects of adopting the above-mentioned further approach are: by clarifying the definitions of multiple performance indicators and the calculation methods based on aligned data, an objective and quantitative evaluation system has been established. Indicators such as detection probability, detection information update frequency, detection accuracy, detection alarm latency, and false alarm probability cover key dimensions from target discovery, continuous tracking, accurate positioning to anomaly response, and can comprehensively and accurately measure the actual perception capability of urban low-altitude airspace multimodal perception networks in dealing with complex scenarios and non-cooperative targets.

[0032] Furthermore, it also includes generating a test report based on the data obtained when testing the multimodal sensing network in urban low-altitude airspace.

[0033] The beneficial effects of adopting the above-mentioned further solutions are: generating structured test reports based on test data, and comprehensively and systematically recording and archiving the entire testing process, intermediate results, and final conclusions. This ensures the traceability of testing activities and the authority of evaluation conclusions, providing detailed and objective documentation for performance analysis, technical rectification, system acceptance, and standard setting.

[0034] 2) Secondly, the present invention also provides a performance testing and evaluation system for a multimodal sensing network in urban low-altitude airspace, the specific technical solution of which is as follows:

[0035] It includes a test environment construction module, a first test module, a second test module, and a determination module;

[0036] The test environment building module is used to: build test environments that include various complex urban scenarios;

[0037] The first test module is used to: conduct simulation tests on the urban low-altitude airspace multimodal sensing network based on the test environment, and use the simulated signals and simulated data provided by the test environment to verify the data input end, data processing end and index calculation end of the urban low-altitude airspace multimodal sensing network in sequence;

[0038] The second testing module is used to: test the verified urban low-altitude airspace multimodal sensing network in a real physical environment, simultaneously collect the ground truth data provided by the corresponding ground truth system and the real test data output by the urban low-altitude airspace multimodal sensing network, and calculate the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network based on the real test data and ground truth data.

[0039] The determination module is used to determine the performance level of the urban low-altitude airspace multimodal sensing network based on the scores of multiple performance indicators.

[0040] Based on the above scheme, the performance testing and evaluation system for urban low-altitude airspace multimodal sensing network of the present invention can be further improved as follows.

[0041] Furthermore, the first test module is specifically used to: run the urban low-altitude airspace multimodal sensing network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expected results.

[0042] Furthermore, the first test module is also specifically used for:

[0043] When the initial performance evaluation results meet the expectations, the simulated signal provided by the test environment will be injected into the data input terminal to verify the data reception function;

[0044] Based on the requirements of the simulated urban test scenario in the test environment, the simulated data corresponding to the requirements of the urban test scenario is input into the data processing terminal for logic verification and debugging.

[0045] Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal.

[0046] Furthermore, the performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability.

[0047] The second testing module is also used for:

[0048] Align the collected ground truth data with the actual test data in terms of time and space;

[0049] In the ground truth data and real test data after time and space alignment, according to the set evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is statistically analyzed, and this ratio is used as the detection probability.

[0050] In the real data and actual test data after time and space alignment, the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network to the same test target is updated per unit time, and this number of updates is used as the detection information update frequency.

[0051] In the real data and the actual test data after time and space alignment, the horizontal error and vertical error between the actual test data and the real data are calculated respectively, which are used as the detection accuracy.

[0052] When the test target triggers a preset abnormal event, the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal perception network are recorded, and the detection alarm delay is calculated based on the time difference between the first timestamp and the second timestamp.

[0053] In the real data and actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability.

[0054] Furthermore, it also includes a generation module, which is used to generate a test report based on the data obtained when testing the multimodal sensing network in the urban low-altitude airspace.

[0055] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so as to enable the electronic device to implement any of the above-mentioned methods for performance testing and evaluation of urban low-altitude airspace multimodal sensing networks.

[0056] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned methods for performance testing and evaluation of urban low-altitude airspace multimodal sensing networks.

[0057] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below:

[0059] Figure 1 This is a flowchart illustrating a method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace according to an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the structure of a multimodal sensing network performance testing and evaluation system for urban low-altitude airspace according to an embodiment of the present invention. Detailed Implementation

[0061] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0062] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0063] like Figure 1 As shown in the figure, a method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace according to an embodiment of the present invention includes the following steps:

[0064] S1. Construct a test environment containing various complex urban scenarios. The specific implementation process is as follows:

[0065] S10. Conduct in-depth research and analysis on various typical and extreme conditions that may be encountered in urban low-altitude airspace operations. These conditions are transformed into a series of quantifiable and simulable test elements. For example, it is necessary to clarify that rainy weather mainly affects the propagation of radio signals, leading to signal attenuation; foggy weather mainly affects the penetration capability of optical sensors, leading to image degradation; and low-line-of-sight obstruction scenarios may simultaneously cause multiple effects such as signal reflection, obstruction, and communication delay. All these analysis results are systematically recorded to form a detailed scenario feature database. This database not only lists various types of complex urban scenarios but also records in detail the specific types and degrees of impact of each scenario on different sensors and technologies in the urban low-altitude airspace multimodal sensing network, such as the evolution of 5G mobile communication technology, radar, and photoelectric tracking equipment. This database is the logical foundation and data source for all subsequent simulation and reproduction work.

[0066] S11. Based on the specific scenario requirements defined in S10, a dedicated outdoor physical testing site needs to be selected and constructed. This site is typically a planned open area that can be divided into multiple grid areas, each representing a typical urban area type. The site must be equipped with a high-precision environmental monitoring and control system capable of simulating or accurately recording specific environmental physical parameters, such as ambient temperature, relative humidity, atmospheric pressure, and illuminance, within a local grid area according to test instructions. To achieve the physical reproduction of various complex urban scenarios, the site needs to deploy corresponding controllable facilities. For example, a large adjustable spray system can simulate foggy scenarios of different concentrations and durations within a specific grid; by constructing a movable and reconfigurable array of building models, low-viewpoint occlusion scenarios with different building densities, heights, and spatial layouts can be constructed. The core function of the physical testing site is to provide a controlled and repeatable "stage" for subsequent tests in a real physical environment, while also serving as a real carrier for the physical effects of some scenarios (such as foggy weather).

[0067] S12 involves configuring a high-fidelity software simulation platform. This step is the core technology for building the test environment, aiming to generate simulated signals and data that strictly match the physical scene through software. The software simulation platform comprises three key components. The first part is the scene signal simulator, which dynamically generates and injects corresponding simulated noise and interference signals into the interface of the urban low-altitude airspace multimodal sensing network data acquisition terminal based on various complex urban scenes selected from the scene feature database. For example, when testing a rainy scene, the simulator injects signal attenuation and background noise conforming to the precipitation attenuation model in the ITU Recommendations into the sensing link data stream that relies on 5G mobile communication technology evolution. When testing a low-line-of-sight obstruction scene, it simulates multipath effects and rapid fading of the communication signal. The intensity of the simulation can be continuously adjusted through parameters. The second part is the data processing logic injection module, which allows testers to preset or trigger specific processing rules at the data processing terminal according to the characteristics of specific scenes. For example, when simulating key location scenarios, this module can generate control commands to forcibly block or eliminate simulated data from the radar data input terminal to test the performance of the multi-source fusion system when a key data source is missing. The third part is the indicator calculation benchmark generator, which provides the expected benchmark data and performance qualification thresholds required for indicator calculation in the verification stage. For example, a qualification threshold of greater than or equal to 80% is set for the detection probability indicator, and a qualification threshold of less than or equal to 30 meters is set for the horizontal detection accuracy indicator.

[0068] S13. Deeply integrate and calibrate the software simulation platform with the physical test site to form a unified and coordinated test environment. This includes accurately mapping various complex urban scenarios defined in the software to specific grid areas of the physical test site. For example, associating the "Grid A - Heavy Rain" scenario configuration in the software with the grid A area in the physical site where a strong spray device is installed. Next, rigorous calibration work is required, such as calibrating the simulated signal strength injected by the software simulator to make it equivalent to the interference level measured in the corresponding physical environment (such as under high humidity spray). It is also necessary to verify the consistency of linkage to ensure that when the test drone flies in the physical grid of the simulated rain scene, the software platform can synchronously and accurately apply the corresponding signal attenuation to the relevant data stream. Through this series of integration and debugging, a "virtual-real combined" test environment is finally formed. This environment supports both pure simulation mode, which uses simulated signals and data to conduct preliminary verification and debugging of the urban low-altitude airspace multimodal perception network, and "semi-physical" or full-physical mode, which means launching drones as test targets under real physical environment conditions, while the software platform assists with signal simulation and logic control for specific scenarios to obtain test data that is closer to actual combat, thus fully supporting the entire process of testing and evaluation from simulation to actual testing.

[0069] Various complex urban scenarios, including rainy scenes, foggy scenes, key location scenes, high-traffic areas, low-view-distance obstruction scenes, and other targeted scenarios, specifically:

[0070] 1) The rainy day scenario is a complex urban scenario simulating precipitation conditions. This scenario primarily evaluates the impact of rainwater on the performance of sensing and communication technologies that rely on radio wave propagation, such as its effect on 5G mobile communication technology signals and radar waves. Raindrops absorb and scatter radio waves, leading to signal attenuation and a decrease in signal-to-noise ratio (SNR), which may reduce detection range, increase bit error rate, or affect positioning accuracy. In the test environment construction, this effect is simulated by establishing a mathematical model in a software simulation platform. For example, the correction for SNR can be modeled as follows: In this formula, This indicates the signal-to-noise ratio in a rainy scene. Represents the original signal power; symbol It is a rain attenuation factor greater than one, used to simulate the attenuation of signal power; Indicates the base noise power; It is the rain-induced noise enhancement factor, used to simulate the additional noise introduced by rainfall.

[0071] 2) The foggy scenario simulates a complex urban scene with low visibility weather conditions. This scenario primarily evaluates the impact of suspended water droplets or particulate matter on the performance of optical and infrared sensing methods, such as photoelectric tracking devices, visible light cameras, and thermal imagers. Fog scatters and absorbs visible light and infrared radiation, severely reducing the effective range and image quality of optical devices, leading to difficulties in target recognition and tracking loss. In the test environment construction, a combination of physical and software methods is used for simulation: different concentrations of fog are actually generated using a fogging device at the physical test site; simultaneously, the image data streams transmitted from the photoelectric devices are digitally blurred and have their contrast reduced in the software simulation platform. The degree of image degradation can be described by a mathematical model, with the relationship approximating as follows: In this formula, Indicates the degree of image degradation in foggy scenes; Indicates the initial scene contrast; symbol It is the attenuation coefficient; symbol It is the distance between the target and the sensor; It refers to meteorological visibility distance.

[0072] 3) The key area scenario simulates a complex urban environment with special security or confidentiality requirements. In this scenario, due to security or regulatory restrictions, certain active detection methods, such as radar, may be prohibited or severely limited. The test in this scenario aims to evaluate the capability and reliability of an urban low-altitude airspace multimodal sensing network to perform fusion sensing relying solely on the remaining sensors when one or more key detection data sources are missing. When constructing this scenario, rules are primarily set in the data processing logic injection module of the software simulation platform to actively eliminate or ignore simulated radar data input, thereby forcibly simulating the condition of missing radar data during testing.

[0073] 4) High-traffic areas simulate complex urban scenarios in densely populated areas, such as commercial centers, transportation hubs, and venues for large events. In this scenario, a large number of concurrent mobile communication terminals can place a significant load on the cellular network, potentially leading to communication link congestion, increased data transmission latency, and even temporary service degradation. This poses a challenge to the ability to identify drones relying on wide-area cellular network sensing. When constructing this scenario, a statistically significant random latency is proactively added to the sensing data transmitted back via the network in the data processing logic injection module of the software simulation platform, and a small amount of data packet loss may be simulated. The network latency can be modeled as follows: In this formula, This represents the total transmission latency in high-traffic areas. Indicates the latency of the underlying network; It is a random variable representing the additional delay caused by crowd congestion, and its distribution can be fitted by historical network congestion data.

[0074] 5) The low-line-of-sight obstruction scenario simulates a complex urban environment with numerous high-rise buildings and narrow streets. These structures severely obstruct the direct path of wireless signals, causing problems such as non-line-of-sight propagation, multipath effects, and rapid signal fading, affecting the stability of 5G mobile communication technology evolution and radio detection technologies. Simultaneously, buildings also obstruct the line of sight of photoelectric sensors and radar, creating detection blind spots. Constructing this scenario requires a combination of physical and software methods: setting up building models on a physical test site to create realistic physical obstruction; and adding a multipath interference model to the wireless signal in the scenario signal simulator on the software simulation platform to simulate the superposition effect of signals after reflection and diffraction along different paths. The received signal strength in the obstructed environment can be expressed as: In this formula, This represents the total power of the received signal in scenarios with low line-of-sight obstruction. Indicates the transmitted signal power; Indicates the number of multipaths; Indicates the first Gain factor for each path; Indicates the first The occlusion loss factor of the path; Indicates the first The propagation distance along each path; It is the path loss index.

[0075] 6) Other targeted scenarios refer to special complex urban scenarios customized according to specific test tasks or application requirements, in addition to the typical scenarios mentioned above. For example, fire rescue scenarios require simulating the complex effects of high temperatures and dense smoke on various sensor signals; complex electromagnetic environment scenarios require simulating the anti-interference capabilities of radio detection and communication systems under conditions of strong co-channel or adjacent-channel interference. The construction of these scenarios is more customized, requiring the test organizer and the tested party to negotiate, clarify the key influencing factors of the scenario, and design corresponding signal processing and data modification rules in the software simulation platform. For example, in fire rescue scenarios, specific interference patterns are applied to infrared signals, and fluctuations caused by thermal turbulence are simulated for communication signals.

[0076] S2. Based on the test environment, a simulation test is conducted on the urban low-altitude airspace multimodal sensing network. Using the simulated signals and simulated data provided by the test environment, the data input end, data processing end, and index calculation end of the urban low-altitude airspace multimodal sensing network are verified in sequence.

[0077] Among these, simulation tests were conducted on the multimodal sensing network for urban low-altitude airspace based on the test environment, including:

[0078] S20. Run the urban low-altitude airspace multimodal sensing network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expected results. The specific implementation process is as follows:

[0079] S200. Deploy the physical system of the urban low-altitude airspace multimodal sensing network into the constructed test environment. Deployment involves two levels. The first level is hardware deployment, which involves installing the various sensor nodes of the urban low-altitude airspace multimodal sensing network, such as automatic correlation surveillance receivers (ACRs), wide-area cellular network surveillance base stations, radar equipment, and photoelectric tracking equipment, at designated grid locations on the physical test site according to the test plan, and completing power supply and physical connections. The second level is software integration, which ensures that the software components of the urban low-altitude airspace multimodal sensing network—the data acquisition end, data processing end, and index calculation end—are integrated and interfaced with the software simulation platform of the test environment. All interfaces of the data acquisition end must be able to receive simulated signals and simulated data streams injected from the software simulation platform. The operating status, internal data streams, and output results of the data processing end and index calculation end must be within the observable and recordable range of the test environment monitoring system. This step ensures that the urban low-altitude airspace multimodal sensing network can operate as a complete system in a controlled test environment, with all its inputs precisely controlled and all its outputs fully captured.

[0080] S201. Based on the objectives of this simulation test, select one or more combinations of complex urban scenarios from the defined scenario library, such as simultaneously selecting a rainy day scenario and a low-line-of-sight obstructed location scenario for composite testing. Load the configuration files for these scenarios into the control software of the test environment. The software simulation platform then dynamically generates two aspects based on the scenario definition. One aspect is generating simulated low-altitude aircraft flight trajectory data, which represents cooperative or non-cooperative targets flying in the selected scenario, such as drones, and includes high-precision spatiotemporal information as a baseline truth. The other aspect is generating simulated noise and interference signals corresponding to the scenario. For example, for the selected rainy day scenario, the platform will inject signal attenuation and background noise conforming to the ITU Recommendation model into all radio frequency-dependent sensor data links, such as 5G mobile communication technology evolution data streams and radar echo signals. The simulated flight trajectory data and scenario-specific interference signals are synthesized within the software platform to generate the final simulated signal stream with scenario characteristics, ready for injection into the system.

[0081] S202. The urban low-altitude airspace multimodal sensing network was powered on and started in the test environment. The software simulation platform began injecting the synthesized simulated signal stream into the data acquisition terminal of the urban low-altitude airspace multimodal sensing network. The urban low-altitude airspace multimodal sensing network began to run its complete sensing, transmission, processing, and calculation process, attempting to detect, identify, and track targets in the simulated signals. The test ran continuously for a preset duration, for example, thirty minutes. During the entire operation, two sets of data were collected and recorded simultaneously. The first set of data consisted of the output data from the urban low-altitude airspace multimodal sensing network itself, including the original target reports reported by the data acquisition terminal, the fused tracks output by the data processing terminal, and any intermediate results that may be generated by the index calculation terminal. This data was fully logged. The second set of data was the baseline data provided by the test environment, namely the ground truth data of the simulated target tracks initially generated by the software simulation platform, and the detailed parameter logs of the injected simulated signals. These two sets of data were synchronized with timestamps using a unified high-precision time source, laying the foundation for subsequent performance comparisons.

[0082] S203. After the test run, the collected data is analyzed and processed to calculate the specific values ​​of several core performance indicators. The calculation process uses the ground truth data of the simulated flight path as an objective benchmark, comparing each item with the output results of the urban low-altitude airspace multimodal sensing network. For example, when calculating the probability of non-cooperative target detection, the number of valid track points output by the urban low-altitude airspace multimodal sensing network that successfully match ground truth track points within a preset spatiotemporal tolerance throughout the entire simulated flight path is counted, and then divided by the total number of ground truth track points. The calculation formula is: In this formula, the symbol Indicates the calculated probability of detecting a non-cooperative target; symbol Indicates the number of target reports that were successfully matched; symbol This represents the total number of target points in the baseline ground truth. When calculating the detection accuracy of non-cooperative targets, each successfully matched reported point is compared to its corresponding ground truth point, and the horizontal error is calculated. and vertical error Finally, the average level accuracy is calculated using the following formula: In this formula, the symbol Indicates the calculated average level of detection accuracy; symbol Indicates the total number of matching points; symbol Indicates the first The horizontal error value of each matching point. Similarly, according to the definition, indicators such as false alarm probability, update frequency, and alarm latency are calculated. Finally, each indicator yields a specific numerical result, such as a detection probability of 85% and a horizontal detection accuracy of 20 meters. This set of values ​​constitutes the raw data for the preliminary performance evaluation results.

[0083] S204. Convert the raw values ​​of each indicator calculated in S203 into one or more evaluable summary scores. A weighted comprehensive scoring method is typically used. Operationally, initial weights are assigned to each performance indicator. These weights are pre-set according to general principles before the test, and the sum of all weights is one. Then, a scoring function is used. The actual value of each indicator This is mapped to a percentage score. The formula for calculating the overall score is: In this formula, This represents the preliminary overall performance evaluation score calculated from the data. It is the summation index, from 1 to... ,represent The performance indicators used in the evaluation; Indicates the first The weight of each indicator; Indicates the first The actual calculated value of each indicator; It is the actual value A function that maps to a percentage score. This yields the overall score. Then, compare it with the expected results that were explicitly defined before the test. Compare. Expected results. It is a pre-set score threshold, such as 80 points. The judgment logic is: if the overall score... Furthermore, if all key individual indicators, such as detection probability and false alarm probability, reach their respective preset qualified thresholds, the preliminary performance evaluation result is considered to have met expectations; otherwise, it is considered to have failed to meet expectations. This judgment will be automatically output to the test personnel and serve as a process control signal to determine whether to initiate detailed verification work on the data input end, data processing end, and indicator calculation end.

[0084] The urban low-altitude airspace multimodal sensing network is a networked system integrating multiple heterogeneous sensing technologies for continuous, real-time, and unified monitoring and sensing of aircraft, especially drones, in urban low-altitude airspace. This network consists of various sensor nodes deployed in different geographical locations, including but not limited to automatic dependent surveillance receivers (ADS-B), wide-area cellular network surveillance base stations, multi-source fusion surveillance systems, radar, electro-optical tracking equipment, and radio detection equipment. These sensors operate in different frequency bands of the electromagnetic spectrum and employ different physical principles, such as radio signal reception, optical imaging, and radar detection, to independently acquire characteristic information of aerial targets from multiple dimensions. The core function of the urban low-altitude airspace multimodal sensing network is to correlate, calibrate, estimate, and synthesize information from these different modal sensors through data fusion technology to form a comprehensive, accurate, and reliable air situation map. Its focus is on the effective detection, tracking, and identification of non-cooperative targets, i.e., drones that do not actively broadcast their identity and location information, providing data support for urban low-altitude safety management.

[0085] The preliminary performance evaluation results are a summary of a series of quantitative analysis conclusions generated during the simulation testing phase, after the urban low-altitude airspace multimodal sensing network was run in the constructed test environment. These results are derived from the simulated signals and data provided by the test environment, calculated through the calculation of several predefined core performance indicators. These performance indicators typically include the probability of detecting non-cooperative targets, the probability of false alarms from non-cooperative targets, the update frequency of non-cooperative target detection information, the accuracy of non-cooperative target detection, and the alarm latency for non-cooperative targets. The preliminary performance evaluation results are presented in specific numerical forms, such as a detection probability of 90%, a horizontal accuracy of 25 meters, and an alarm latency of 3 seconds, objectively reflecting the basic performance of the urban low-altitude airspace multimodal sensing network in a controlled, simulated complex urban scenario. The purpose of generating these results is to quickly verify the overall architecture, algorithm logic, and parameter configuration of the urban low-altitude airspace multimodal sensing network before investing in more resource-intensive real physical environment testing and detailed module verification, identifying any obvious performance defects or design flaws, thereby playing a role in early risk control and test process optimization.

[0086] The expected outcome is a pre-defined standard or threshold used to judge the pass / fail status of the preliminary performance evaluation results, based on the testing objectives, the technical specifications of the system under test, or industry application requirements, before the simulation test begins. The expected outcome typically comprises two levels. The first level is the pass / fail threshold for individual performance indicators, such as requiring a detection probability of no less than 80%, a false alarm probability of no more than 1%, and a horizontal detection accuracy better than 30 meters. The second level is the overall comprehensive performance scoring threshold for the system, i.e., a minimum acceptable comprehensive score, such as 80 points. The expected outcome serves as the objective scale and decision-making basis for judging whether the preliminary performance evaluation "meets" the requirements. It is clearly defined by the testing organization during the test plan design phase and solidified in the test procedures or control software before test execution to ensure the objectivity and consistency of the evaluation judgment and avoid disputes arising after testing due to ambiguity in the standards.

[0087] The process involved using simulated signals and data provided by the test environment to sequentially verify the data input, data processing, and index calculation ends of the urban low-altitude airspace multimodal sensing network, including:

[0088] S21. When the preliminary performance evaluation results meet the expectations, the analog signal provided by the test environment is injected into the data input terminal to verify the data receiving function. The specific implementation process is as follows:

[0089] S210. Verification of condition satisfaction and preparation of dedicated test vectors. This step requires that the simulation of the urban low-altitude airspace multimodal sensing network in the test environment has been completed, and the preliminary performance evaluation results have been generated with a comprehensive score. Greater than or equal to the preset expected result Simultaneously, key individual indicators also reach thresholds. Upon receiving this positive judgment signal, the test system automatically or manually initiates the data input verification process. Preparation includes generating or calling a batch of test vectors specifically for verifying data reception functionality from the software simulation platform of the test environment. These test vectors differ from the comprehensive signal streams used to evaluate overall performance in simulation tests; they are more targeted, standardized, and traceable. The test vectors need to cover the technical characteristics and data protocols of each sensor type accessed by the urban low-altitude airspace multimodal sensing network. For example, to verify the data input of the corresponding broadcast automatic dependent surveillance receiver, a sequence of simulated signal frames strictly conforming to the 1090ES or UAT data link format needs to be generated, with each frame containing fields such as aircraft identification code, latitude and longitude, altitude, and speed conforming to the protocol specifications. To verify the data input of the corresponding wide-area cellular network surveillance, a simulated signal stream conforming to the signaling and data packet format of the fifth-generation mobile communication technology evolution network needs to be generated, simulating the remote identification and location information of unmanned aerial vehicles reported by drones through the cellular network. The contents and timing of all test vectors are accurately recorded, forming an "injection signal truth log".

[0090] S211. Based on the actual physical interface type and communication protocol of the data input terminal of the urban low-altitude airspace multimodal sensing network, configure the signal injection path and parameters for the test environment. The data input terminal of the urban low-altitude airspace multimodal sensing network typically consists of multiple independent physical or logical interfaces. For example, radar equipment may provide data via RF cables or dedicated network interfaces, photoelectric tracking equipment may provide video streams via video cables or Ethernet, and radio detection equipment may receive signals via antenna interfaces. Operators need to connect the signal output terminal of the test environment software simulation platform to these corresponding data input terminals through appropriate signal converters, protocol simulators, or direct network connections. The configuration process must ensure that all parameters of the injected signal, such as level, frequency, modulation method, data packet format, baud rate, network IP address, and port number, are completely consistent with the signals generated by the actual sensors. For RF signal injection, an RF signal generator, up-converter, and coupler are required; for network data injection, it is done directly through a network switch and protocol simulation software. This step ensures that the simulated signal provided by the test environment can enter the data input terminal in a "realistic" manner.

[0091] S212. On the test environment control software, start the signal injection program. The injection process adopts a structured and sequential approach, rather than injecting all signals at once. The structured approach is reflected in two dimensions. The first dimension is injection by channel according to sensor type. For example, first, inject the prepared analog signal test vector into the broadcast automatic correlation monitoring data input terminal separately for a predetermined period, such as five minutes; then, inject the corresponding test vector into the wide area cellular network monitoring data input terminal separately; then, proceed sequentially to the radar data input terminal, photoelectric data input terminal, etc. The second dimension is injection in stages according to signal quality or scene type within each channel. For example, in the wide area cellular network monitoring channel, first inject an ideal analog signal with good signal strength, no packet loss, and no delay; then inject a network data stream simulating a high-traffic area scene with random delay and controllable packet loss rate; finally, inject a analog signal simulating a low-line-of-sight obstruction scene with rapidly fading signal strength. Each injection sub-process has a clear start timestamp, end timestamp, and content index of the injected test vector. The test environment monitoring system synchronously records the precise content and transmission time of every data packet or signal fragment it sends to every data input terminal of the urban low-altitude airspace multimodal sensing network, and updates the "injected signal truth log" in real time.

[0092] S213. On one side of the urban low-altitude airspace multimodal sensing network, after receiving the injected analog signal, the data input end initiates its internal signal conditioning, analog-to-digital conversion, and protocol parsing processes. To verify the data reception function, data monitoring and acquisition are required at the first logical data output point after the data input end. This point is usually the output buffer of the data acquisition end, or the data stream interface after preliminary parsing and standardization. By calling the debugging interface at the software level of the urban low-altitude airspace multimodal sensing network, or by deploying data probes on the hardware link, the data content reported after being received, parsed, and preprocessed by the data input end is captured and recorded in real time. Each captured data needs to be stamped with a high-precision timestamp synchronized with the injected signal. For example, record each frame of target information reported by the data acquisition end, including the target identification code parsed from the analog signal, the calculated latitude and longitude coordinates, altitude, received signal strength indication value, and the local timestamp added by the data acquisition end. This step generates a "received data response log".

[0093] S24. The collected "Received Data Response Log" is time-aligned and content-compared with the "Injected Signal Truth Log" recorded in the test environment, and quantitative analysis is performed from three dimensions. The first dimension is data integrity verification, calculating the data reception rate using the formula: In this formula, Indicates the data reception rate; This indicates the number of data units (such as data frames and target reports) that were correctly parsed and reported in the "Received Data Response Log"; This represents the total number of corresponding data units recorded in the "Injected Signal Truth Log" and sent to the corresponding data input terminal. The data reception rate should be close to 100%. The second dimension is data accuracy verification. For each successfully received data unit, its key fields are compared with the truth value of the injected signal. For example, comparing whether the parsed target latitude and longitude are consistent with the latitude and longitude encoded in the injected signal within the allowable error range; comparing whether the parsed identity identifier is correct. The data accuracy rate is calculated using the formula: In this formula, Indicates data accuracy; This represents the number of data units whose content is completely correct or whose errors are within acceptable limits. The third dimension is timing characteristic verification, which calculates the average processing latency using the following formula: In this formula, This represents the average processing delay from signal injection to data acquisition terminal output; Indicates the first The timestamp of each data unit appearing in the "Received Data Response Log"; Indicates the corresponding number The transmission timestamp of each data unit in the "Injected Signal Truth Log". This delay should be within the range allowed by the sensor's technical specifications. Based on the evaluation results of the above three dimensions, if the data reception rate... Data accuracy All values ​​exceeded the preset verification threshold (e.g., all were greater than 99.5%), and the average processing latency was [missing information]. If the data reception function of the urban low-altitude airspace multimodal sensing network's data input end is verified as successful, the system remains stable within the expected range. Otherwise, the problem needs to be identified based on the logs, and adjustments and optimizations should be made before re-verification.

[0094] The simulated signals provided by the test environment are electronic signals or data streams dynamically generated by the software simulation platform within the constructed test environment according to test requirements, or retrieved from the database. These signals simulate the outputs of various sensors and the effects of complex scenarios in the real world. These signals strictly adhere to international standards, industry protocols, and physical layer specifications for various aviation surveillance sensors and communication systems. The types of simulated signals fully cover the sensor types integrated into the urban low-altitude airspace multimodal sensing network, such as simulating radio frequency signals and data packets of Automatic Dependent Surveillance-Broadcast (ADS-B), simulating radio frequency pulse sequences and echoes transmitted and received by radar, simulating video streams and attitude data of photoelectric tracking equipment, simulating network signaling and service data packets between 5G base stations and UAV terminals, and simulating specific frequency band spectrum signals captured by radio detection equipment. More importantly, these simulated signals can be embedded with predefined noise, interference, attenuation, and time delay to accurately reproduce the impact of various complex urban scenarios such as rainy days, foggy days, high-traffic areas, and low-line-of-sight obstructed areas on the sensing system, providing a high-fidelity, repeatable, and controllable test input source for system verification.

[0095] The data input terminal of the urban low-altitude airspace multimodal sensing network is the collective term for the hardware interfaces and software protocol stacks responsible for receiving raw information from various external sensing sensors or data sources within this networked system. It is the sole entry point for all external information flowing into the urban low-altitude airspace multimodal sensing network for subsequent processing. Physically, the data input terminal manifests as a series of interfaces adapted to different signal types, such as RF antenna interfaces, fiber optic interfaces, Ethernet ports, and serial communication ports. Logically, each physical interface is backed by a dedicated signal conditioning module, analog-to-digital converter, protocol parser, or driver. These components work together to convert raw analog RF signals, digital video streams, network data packets, and other signals into a standardized data format that can be recognized and processed by the system's unified processing framework. An urban low-altitude airspace multimodal sensing network typically has multiple independent data input terminals, corresponding to different modalities of sensors, such as automatic correlation surveillance receivers (ACCSIRs), wide-area cellular network surveillance base stations, radar, photoelectric tracking equipment, and radio detection equipment. The performance and reliability of the data input terminal directly determine the quality of the acquired raw information and are the foundation of the entire system's sensing capability.

[0096] S22. Based on the requirements of the simulated urban test scenario in the test environment, the simulated data corresponding to the urban test scenario requirements is input into the data processing terminal for logic verification and debugging. The specific implementation process is as follows:

[0097] S220 involves test requirement analysis and scenario model parameterization. This step requires in-depth analysis of the urban test scenario requirements simulated in the test environment, transforming them from abstract textual descriptions into concrete mathematical models and parameter sets that can be directly executed by the software simulation platform. Operationally, testers select one or more combinations of requirements to be covered in this verification from a predefined "Urban Test Scenario Requirements List," such as simultaneously verifying the impact of "rainy day scenario" and "low line-of-sight occlusion scenario" on data fusion. For each selected requirement, the corresponding pre-set mathematical model in the test environment software simulation platform is invoked, and specific test intensity parameters are configured for it. For example, for the "rainy day scenario" requirement, the specific value of the rainfall intensity simulated in this test (e.g., 20 mm / hour) needs to be determined. This value will be substituted into the rainfall attenuation model recommended by the International Telecommunication Union to calculate the specific signal attenuation coefficient. and noise enhancement factor For scenarios requiring high foot traffic, it is necessary to determine the statistical distribution parameters of the simulated network latency, such as the mean. and variance All these parameters constitute a structured "parameter set for this verification scenario," which is the direct basis for generating simulation data with accurate scenario characteristics.

[0098] S221. The software simulation platform of the test environment loads the "parameter set of this verification scenario". Based on these parameters, the platform dynamically generates test vectors in two aspects. First, it generates the ground truth track of the target with scenario characteristics, i.e., the perfect spatiotemporal trajectory data that the simulated target (such as a drone) should produce under assumed ideal conditions. Second, and more crucially, it "contaminates" or "modifies" the ground truth track according to the scenario parameters to generate simulated data corresponding to the requirements of the urban test scenario. This simulated data simulates the data that various sensors can actually report under the corresponding scenario conditions, with typical limitations. For example, for radar sensors, the simulated data will add additional attenuation to the simulated target echo signal according to the rainfall attenuation model, and may generate false alarm points that conform to the statistical characteristics of rain clutter; for wide-area cellular network surveillance data, the simulated data will add random transmission delay to the target location report according to the network latency model. It may conform to the formula ,in These simulated data are packaged in a standardized format according to the data acquisition terminal of the urban low-altitude airspace multimodal sensing network, and are accompanied by data quality labels (such as confidence level and error estimation) to form a sequence of "scenario-based test vectors" that can be directly input into the data processing terminal.

[0099] S222. The sequence of "scenario-based test vectors" generated in S221 is injected into the data processing terminal of the urban low-altitude airspace multimodal perception network through the verified data input channel or directly through the test interface. The injection process is controlled and usually proceeds in order of increasing scenario complexity. For example, test vectors containing only the influence of a single scenario (such as pure fog) are injected first, followed by test vectors containing the influence of composite scenarios (such as rain with low line-of-sight occlusion). Simultaneously, comprehensive monitoring of key nodes and the final output within the data processing terminal is initiated. The monitoring content includes: the output of the data preprocessing module, the correlation matching results between data from different sources, the state of the fusion filter (such as the covariance matrix of the Kalman filter), and the fused trajectory and target attribute information of the final output. The monitoring data is strictly time-synchronized with the input test vectors to ensure that each output result can be traced back to its corresponding input conditions and processing time.

[0100] S223. After processing the injected "scenario-based test vector," the data processing end outputs information such as the fused target trajectory. Verification involves comparing the output with predefined expected behavior. This comparison falls into two categories. The first is numerical accuracy comparison, which compares the fused trajectory position output by the data processing end with the original target ground truth trajectory encapsulated in the "scenario-based test vector" to calculate the fusion accuracy under specific scene interference. For example, calculating the horizontal root mean square error of the fused trajectory under a test vector in a low-line-of-sight occlusion scene. The formula is: In this formula, This represents the root mean square error of the water level. This indicates the total number of comparison points; Indicates the first output from the data processing end The horizontal coordinates of each point; The first category represents the corresponding truth coordinates. The second category is logical rule verification, which verifies certain scenario requirements that directly affect the processing logic. For example, for the "key location scenario" requirement, the corresponding rule is to block radar data. During verification, after injecting the test vector for blocking radar data, the output log of the data processing end is checked to confirm whether it has triggered the "radar data missing" alarm flag, and to check whether the output fused track does not actually use radar source information, or whether it has switched to the preset radar-free fusion mode. This can be determined by checking the data source contribution field of the output track.

[0101] S224. Based on the comparison results in S223, quantitatively evaluate the performance of the data processing end. The evaluation criteria have been set before testing, for example, requiring a certain level of fusion accuracy when processing test vectors for a specific scenario. Must be below the threshold Furthermore, all logical rule verifications must pass. If the evaluation results meet all criteria, the logical verification of the data processing end for the city's test scenario requirements is deemed successful. If the criteria are not met, such as excessive fusion error or incorrect execution of logical rules, the debugging phase begins. Debugging is handled by developers who use monitored internal process data to analyze the root cause of the problem. Problems may stem from improper threshold settings for data association gates, process noise parameters of the fusion filter not adapting to dynamic changes in the scenario, or unreasonable fusion weight allocation in cases of missing data. After adjusting the relevant algorithm parameters or logical judgment conditions on the data processing end, developers need to re-inject the same test vector for verification, forming an iterative cycle of "test-evaluation-debugging" until the data processing end's performance for the current city's test scenario requirements reaches the predetermined target. After verifying one combination of requirements, the test vectors for other combinations of requirements are changed, and the above steps are repeated until all planned city test scenario requirements are covered.

[0102] The urban test scenario requirements simulated in the test environment refer to a set of test requirements closely related to specific urban environmental characteristics, airspace usage features, and safety management objectives, aimed at evaluating the performance, functionality, and reliability of the urban low-altitude airspace multimodal sensing network under urban low-altitude conditions. These requirements stem from the actual challenges of urban low-altitude operations, such as complex electromagnetic environments, dense building obstruction, variable weather conditions, high-density population areas, and protection of key areas. In the test environment, these requirements are concretized into a series of simulable and loadable configuration items, such as requirements for rainy weather scenarios, foggy weather scenarios, key location scenarios, high-traffic areas scenarios, and low-line-of-sight obstruction scenarios. Each requirement clearly defines the environmental conditions to be simulated, the types of interference to be injected, the specific system operating modes to be verified, and the expected performance thresholds, serving as the fundamental basis for designing targeted test cases and generating specific simulation data.

[0103] The simulated data corresponding to urban test scenario requirements refers to a standardized dataset generated by the test environment software simulation platform based on the specific definition of the urban test scenario requirements. This dataset contains the specific scenario characteristics and effects described by the requirements. These data simulate the raw or pre-processed information that various sensing sensors can actually collect and report under real-world urban conditions. For example, for the "rainy day scenario" requirement, the simulated radar spot data will include additional false alarm points conforming to the precipitation scattering model, and the signal strength of the target echo will be attenuated. For the "high-traffic area scenario" requirement, the simulated wide-area cellular network surveillance and positioning data will have statistically consistent random transmission delays added. These data strictly adhere to the physical model and statistical laws defined in the requirements and encapsulate timestamps, sensor types, observation values, and data quality tags derived from the scenario model. They serve as a bridge connecting abstract test requirements with specific system logic verification.

[0104] The data processing unit of the urban low-altitude airspace multimodal sensing network is the core functional module responsible for integrating, correlating, calibrating, estimating, and fusing multi-source, heterogeneous sensing information from the data input end within this networked system. It receives data streams from various sensors after preliminary preprocessing and standardization by the data acquisition end. These data streams may include target reports from Automatic Dependent Surveillance-Broadcast (ADBS), remote identification information of unmanned aerial vehicles (UAVs) from wide-area cellular network surveillance, radar tracks and trajectories, image recognition results from electro-optical tracking devices, and spectral characteristics from radio detection devices. The data processing unit uses a series of complex algorithms, such as time synchronization, spatial registration, data correlation (e.g., nearest neighbor or joint probability data correlation), state estimation (e.g., Kalman filtering), and multi-source information fusion (e.g., weighted fusion or feature-level fusion), to synthesize these independent, potentially contradictory, redundant, or missing pieces of information into a unified, continuous, and reliable global low-altitude situation map. Its final output is a comprehensive set of target trajectories, attributes, and alarm information with higher accuracy, completeness, and real-time performance, which is directly provided to the performance evaluation end.

[0105] S23. Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal. The specific implementation process is as follows:

[0106] S230. Verification Data Input and Expected Benchmark Preparation: Two sets of key input data are required. The first set is simulated data processed by the data processing unit. This data originates from the previous data processing verification phase and is a structured result output after the data processing unit fuses scenario-based test vectors. It typically exists in the form of a comprehensive target track list, with each track containing information such as target identifier, timestamp, fused 3D position, velocity, heading, and data source composition. The second set of input data is expected benchmark data independent of the urban low-altitude airspace multimodal perception network. This data is directly provided by the software simulation platform of the test environment and consists of two parts: one part is the original "ground truth" used to calculate indicators, i.e., the precise target trajectory and event time used in the simulated test; the other part is the "benchmark indicator value" pre-calculated based on this "ground truth" using an independently verified and authoritative reference algorithm. For example, the reference algorithm uses a high-precision mathematical library to calculate the theoretical detection probability benchmark value based on the ground truth track and known test conditions. Positioning accuracy benchmark value These two sets of data—the processed simulation data and the baseline data—must be aligned using a unified time base to ensure consistency of data segments and events.

[0107] S231. The simulated data prepared in S230 and processed by the data processing terminal is used as standard input and completely imported into the indicator calculation terminal of the urban low-altitude airspace multimodal perception network. The indicator calculation terminal automatically and sequentially executes the calculation process based on its pre-programmed and integrated indicator calculation modules. The calculation is entirely based on the input data. For example, for the detection probability indicator, the algorithm inside the indicator calculation terminal retrieves all fused track reports from the input data and performs spatiotemporal correlation matching with the known ground truth values ​​of the test targets. The matching algorithm sets a spatiotemporal correlation threshold; reports with successful correlation are considered valid detections. Finally, the formula for calculating the detection probability is executed: In this formula, This represents the detection probability value calculated by the indicator calculation terminal; This represents the number of target reports that successfully match the true value in the input data; symbol This represents the total number of true targets (or the total number of sampling points) within the testing period. For the detection accuracy index, the index calculation unit selects all successfully matched point pairs and calculates the horizontal error for each pair. Then calculate the average level precision: In this formula, This indicates the average level of detection accuracy calculated by the indicator calculation terminal; This represents the total number of matching point pairs; This indicates the first data input to the indicator calculation terminal. The horizontal coordinates of the matching points; This represents the corresponding truth coordinates. Following this method, the indicator calculation module sequentially calculates all predetermined indicators such as false alarm probability, update frequency, and alarm latency, ultimately outputting a dataset containing the calculation results of each indicator. .

[0108] S232. Set the calculation results output from the indicator calculation terminal in S231. , with the prepared set of expected benchmark values A point-by-point comparison is required. The purpose of the comparison is to verify the numerical accuracy of the calculation results from the indicator calculation end, not to evaluate the performance of the sensing network. Therefore, it is necessary to perform a point-by-point comparison for each indicator. Preset an allowable error range This error range is determined based on the nature and magnitude of the indicator; for example, for probability indicators (… (Probability value), allowable absolute error range It can be set to 0.001 (0.1%); for accuracy-related indicators ( (distance value), allowable relative error range It can be set to 0.01 (1%). The comparison calculation is as follows: for each indicator... Calculate the deviation Then judge Whether it holds true. Simultaneously, check if any runtime errors or logical exceptions are thrown during the calculation process. The entire comparison process can be completed through an automated script, generating a detailed deviation analysis table listing the baseline value for each indicator. Calculated values ,deviation Allowable error And the preliminary judgment status of "pass / fail".

[0109] S233. Based on the item-by-item comparison results generated in S232, a comprehensive judgment is made on the overall calculation accuracy of the indicator calculation end. The judgment logic is based on two levels. The first level is the pass rate of individual indicators. Assuming a total of [number missing] indicators have been verified... Performance indicators. The number of indicators that passed the comparison. Calculate the pass rate of the indicators: In this formula, The first level represents the pass rate of the metrics calculated on the metrics side. The second level is the veto power of key metrics. For certain metrics, such as system availability and data consistency, the correctness of their calculation logic is crucial, and they may be defined as key metrics, requiring strict adherence to deviations. It must be zero or infinitely close to zero (e.g., less than a very small threshold). The comprehensive judgment rule is: if the pass rate of the indicator... If the accuracy of the calculations at the indicator calculation end of the urban low-altitude airspace multimodal sensing network is verified as passed if the preset qualification threshold (e.g., 99%) is reached or exceeded, and all key indicators show a "pass" result, then the verification is deemed successful. Otherwise, the verification is deemed unsuccessful.

[0110] S234. If S233 determines that the verification passed, then the specific "expected benchmark" dataset used in this verification will be... The software version and configuration parameters of the indicator calculation terminal are archived together as a standard verification benchmark in subsequent testing. If the verification fails, the problem localization and debugging phase must be initiated. Developers need to identify which indicators have exceeded the tolerance based on the deviation analysis table. Then, they need to analyze the reasons for the deviation by combining the design documents and logs of the indicator calculation terminal. The reasons may include: incorrect algorithm formula coding, misunderstanding of input data format, insufficient numerical precision in the intermediate calculation process, or fundamental differences from the mathematical methods used in the benchmark calculation (such as interpolation methods or statistical methods). After modifying the algorithm code or configuration of the indicator calculation terminal, the complete verification process must be re-executed until it passes. Regardless of whether the verification passes or fails, a detailed "Indicator Calculation Terminal Accuracy Verification Report" must be generated, recording the verification configuration, input data summary, calculation process, comparison results, judgment conclusions, and any problems found and corrective measures, to ensure the integrity and traceability of the verification activities.

[0111] The simulated data processed by the data processing end refers to the structured data set output by the data processing end of the urban low-altitude airspace multimodal perception network after verification at the data input end and logic verification and debugging at the data processing end. This data is no longer the original or single sensor signal, but the final product after a series of complex algorithm processing such as time alignment, spatial registration, data association, trajectory filtering and fusion estimation. Its standard form is a unified list of comprehensive target tracks. Each track contains a unique identifier of a target continuously tracked by the system, a timestamp, fused three-dimensional position coordinates, velocity vector, heading angle, target type classification and its confidence level, and the contribution information of each original sensor data constituting the track. This data represents the urban low-altitude airspace multimodal perception network's final "understanding" and "reconstruction" of the air situation in the test scenario, and is the direct and only data source input to the index calculation end for calculating the scores of various performance indicators.

[0112] The index calculation module of the urban low-altitude airspace multimodal sensing network is the core functional module in the system specifically responsible for performance quantification and evaluation. Located after the data processing module, it receives the integrated target trajectory data, processed by fusion, as its sole input. Internally, the index calculation module encapsulates the calculation algorithms, mathematical formulas, and processing logic for all preset performance indicators. These indicators include, but are not limited to, non-cooperative target detection probability, non-cooperative target false alarm probability, non-cooperative target detection information update frequency, non-cooperative target detection accuracy (horizontal and vertical), non-cooperative target alarm latency, system availability, and data consistency. The core task of the index calculation module is to automatically analyze the input integrated trajectory data according to strict mathematical definitions and calculation procedures, combining necessary benchmark truth information (provided by the test environment during the verification phase and by an independent measurement system during the real testing phase) to calculate the specific value of each performance indicator. Its output is a structured set of performance indicator data, which will be directly used to generate performance test reports, perform weighted comprehensive scoring, and ultimately determine the performance level of the urban low-altitude airspace multimodal sensing network. This is a crucial link in the transformation from system operating status to the final performance evaluation conclusion.

[0113] S3. In a real physical environment, test the verified urban low-altitude airspace multimodal sensing network, simultaneously collect the ground truth data provided by the corresponding ground truth system and the real test data output by the urban low-altitude airspace multimodal sensing network, and calculate the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network based on the real test data and ground truth data.

[0114] In a real physical environment, the validated urban low-altitude airspace multimodal sensing network was tested. Simultaneously, ground truth data provided by the corresponding ground truth system and real test data output by the urban low-altitude airspace multimodal sensing network were collected. The specific implementation process is as follows:

[0115] 1) Based on the technical solution's requirements for the real physical environment, the testing organizer will select a representative outdoor site in a city or suburb as the test site. This site should be able to support the reproduction or natural inclusion of various complex urban scene features, such as the presence of nearby real building complexes to create low-line-of-sight obstruction conditions, and include different terrains such as open areas and populated areas. Furthermore, the site's airspace and surrounding environment should meet the safety regulations for UAV flight. Before the official start of the test, professional environmental monitoring equipment must be used to measure and record the site's physical parameters, including ambient temperature, relative humidity, atmospheric pressure, illuminance, background noise level, and wind speed and direction. These parameters are recorded in the test log as background conditions for the test. Simultaneously, the test site will be physically isolated according to safety regulations, a safety warning zone will be set up, and necessary electromagnetic shielding and network isolation equipment will be deployed to protect the test system and prevent interference with the outside world.

[0116] 2) Deploy the verified urban low-altitude airspace multimodal sensing network, which has completed all the aforementioned simulation and verification stages, into the selected real physical environment. Deployment includes installing and securing all sensor nodes, such as automatic dependent surveillance receivers (ADS-B), wide-area cellular network surveillance base stations, radar, photoelectric tracking equipment, and radio detection equipment, ensuring stable power supply and uninterrupted communication links. After deployment, conduct on-site power-on self-tests and basic function verifications. A crucial preliminary step is high-precision time and space synchronization. All sensor nodes of the urban low-altitude airspace multimodal sensing network, the real UAV test targets ready for flight, and independently deployed ground truth system measurement stations must all be connected to the same high-precision time source, such as a BeiDou or GPS timing module, ensuring that all system-generated data carries a unified reference microsecond-level precision timestamp. Simultaneously, using high-precision surveying equipment such as total stations or real-time dynamic differential positioning technology, accurately measure the geographic coordinates of the antenna phase centers of all sensor nodes, the ground truth system reference station, and the test take-off and landing points. Record this coordinate information into the relevant systems as a unified geographic reference framework for all spatial data calculations.

[0117] 3) Establish and calibrate a truth system to provide reference data. This truth system is independent of the multimodal sensing network in the low-altitude airspace of the city being tested and is typically composed of higher-precision measurement equipment. For example, using carrier phase real-time dynamic differential positioning technology, a high-precision real-time dynamic differential positioning receiver and antenna are installed on the test UAV, and a real-time dynamic differential positioning reference station is established on the ground. This system can provide the test UAV with real-time position, velocity, and time information with centimeter-level accuracy. Another supplementary method is to use a laser tracker or total station to optically track the UAV and obtain high-precision angle and distance data. Before the test begins, all real UAVs used as test targets must undergo safety checks, battery charging, and the high-precision real-time dynamic differential positioning receiver and other truth measurement equipment must be securely installed on the UAVs to ensure they are functioning properly and do not affect the UAV's flight. The accuracy of the truth system itself must be verified through static and dynamic calibration before the test.

[0118] 4) According to a pre-established detailed test plan, operators control one or more real UAVs equipped with a truth-based measurement system to fly in a real physical environment along various preset routes. The route design covers different altitudes, speeds, and different typical areas to fully stimulate the performance of the urban low-altitude airspace multimodal sensing network. At the same time as the flight test command is issued, three data acquisition systems are simultaneously activated and continuously operate. The first system is the verified urban low-altitude airspace multimodal sensing network itself. It operates continuously, its data acquisition end receives signals from real sensors, and its data processing end performs real-time fusion processing. Its internal output data at all levels (including raw reports, fused tracks, and system status) are completely recorded. These records are collectively referred to as the real test data output by the urban low-altitude airspace multimodal sensing network. The second system is an independent truth-based system. It continuously receives and records measurement data transmitted from devices such as high-precision real-time dynamic differential positioning receivers on the UAVs, forming a truth-based data stream provided by the truth-based system. This data stream contains the most accurate spatiotemporal trajectory of the UAV. The third system is the test control system, which synchronously records the global timeline of the entire flight test, UAV remote control commands, system status heartbeats, and any operational events or abnormal alarms.

[0119] 5) After the flight test, raw recorded data was exported from three independent data sources. From the data recording unit of the urban low-altitude airspace multimodal sensing network, all logs and output data files generated by its data acquisition, processing, and index calculation ends throughout the test were exported. From the ground station of the truth system, high-precision trajectory data files, typically including timestamps, were exported. ,longitude ,latitude Altitude Eastward speed Northbound speed Horizontal speed Fields such as [field name missing]. Export task timeline logs from the test control system. The key to preprocessing is aligning data streams from different systems along the time dimension. Due to varying internal processing latency across systems, although the timestamp baseline is unified, the actual physical time corresponding to the data may have a fixed offset. It is necessary to identify common event markers (such as the time of the drone's takeoff radio command, the time of crossing specific spatial marker points) and calculate and compensate for the system time offset between each data stream. After time alignment, the target locations reported by the urban low-altitude airspace multimodal sensing network are uniformly transformed to the same coordinate system as the ground truth data. Ultimately, this results in two sets of core data that are strictly synchronized in time and share a unified spatial coordinate system: one set is the ground truth data provided by the ground truth system. The other set consists of real test data output from the urban low-altitude airspace multimodal sensing network. This prepares the ground for subsequent performance metric calculations.

[0120] The real physical environment refers to a specific outdoor location existing in the real world, possessing actual geographical features, terrain undulations, real buildings, natural vegetation, real meteorological conditions, and a complex electromagnetic background, rather than a virtual space simulated by computer software. In this environment, conditions such as air, temperature, humidity, light, and wind and rain are naturally occurring or present, and electromagnetic noise originates from real civilian communication, broadcasting, television, and other electronic devices within the environment. It serves as an irreplaceable ultimate testing ground for evaluating the final performance of urban low-altitude airspace multimodal sensing networks in actual deployment. All the uncertainties and complexities it encompasses ensure that the test data obtained here has the highest credibility and practical reference value.

[0121] The verified urban low-altitude airspace multimodal sensing network refers to an entity system that has successfully completed all the aforementioned simulation and module verification steps in the technical solution. Specifically, it has passed simulation tests in a test environment containing various complex urban scenarios, and its preliminary performance evaluation results have met expectations. Its data input terminal's data receiving function has been verified through simulated signal injection and is functioning normally. Its data processing terminal has completed logic verification and debugging based on simulated data input according to the requirements of the urban test scenario. The accuracy of its index calculation terminal has also been confirmed by comparison with the expected benchmark. A system in this state, with its hardware connections, software algorithms, parameter configurations, and internal processing logic having undergone rigorous testing in a controlled environment, is considered to possess the basic capabilities for correct operation and can be deployed to a real physical environment for final capability verification and performance evaluation.

[0122] The ground truth system is a suite of instruments and equipment deployed independently of the multimodal sensing network in the urban low-altitude airspace under test, possessing higher measurement accuracy. Its sole task is to continuously and accurately measure the spatial position, velocity, attitude, and temporal information of the actual UAV or other aircraft being tested, and to generate records during the test. The ground truth system typically uses high-precision real-time dynamic differential positioning technology as its core, and may be supplemented by equipment such as laser trackers and optical theodolites. The ground truth system itself must undergo rigorous calibration before testing, and its measurement accuracy and uncertainty must be known and significantly higher than the theoretical accuracy of the sensing network being evaluated, thus serving as an objective and authoritative reference benchmark for evaluating the performance of the sensing network.

[0123] The truth data provided by the truth system refers to a set of high-precision measurement data about the spatiotemporal state of the test target, collected and recorded by an independently deployed truth system during the real physical environment testing phase. This data is typically stored in time-series format, with each data point containing a high-precision timestamp and the target state information corresponding to that moment, such as 3D geographic coordinates, 3D velocity vectors, and attitude angles. Because the truth system employs precision measurement technologies such as real-time dynamic differential positioning with carrier phase, the accuracy of the data it provides can reach the centimeter level, far exceeding the nominal accuracy of the tested urban low-altitude airspace multimodal sensing network. This data serves as the "standard answer" or "ground truth" throughout the performance evaluation process, and is an indispensable benchmark for calculating the scores of various performance indicators of the urban low-altitude airspace multimodal sensing network.

[0124] The real test data output by the urban low-altitude airspace multimodal sensing network refers to the total data generated and output by all internal processing stages of the verified urban low-altitude airspace multimodal sensing network during continuous operation in the real physical environment testing phase. It encompasses everything from the raw target reports received and initially processed by each sensor at the data acquisition end, to the comprehensive target trajectory and situational information generated after multi-source information fusion at the data processing end, as well as the system's operational status logs and internal messages. Unlike the simulated data used in simulation tests, this data is the direct output of the sensing network after responding to real targets, real signals, and real interference in the real physical world. It includes all the characteristics, advantages, and errors exhibited by the system in the actual environment. This data is the direct object for evaluating the real performance of the sensing network and needs to be compared and analyzed with the true data provided by the ground truth system to determine the actual level of its various performance indicators.

[0125] The performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability. These performance indicators are calculated based on real test data and ground truth data, and include:

[0126] 1) Align the collected ground truth data with the actual test data in terms of time and space. The specific implementation process is as follows:

[0127] ① After the test is completed, the raw data files recorded by the truth system and the raw data files recorded by the urban low-altitude airspace multimodal sensing network are loaded into dedicated data analysis software or processing platforms, respectively. The processors first need to manually or automatically identify several clear and identifiable common physical events in these two independent time-series data sets. These events should have unique and identifiable characteristic markers in both data streams. Typical common events include: the precise time when the UAV remote control takeoff command is issued (which can be provided by the test master control log); the time when the UAV crosses a pre-set geographic marker with significant spatial characteristics (such as the center point at a specific altitude above the site); the start or end time of the UAV performing a specific maneuver (such as hovering or uniform circular motion); or the transmission time of a special beacon signal triggered manually during the test. Each common event corresponds to a timestamp in the truth data stream. In the real test data stream of the urban low-altitude airspace multimodal sensing network, there is a corresponding timestamp. .

[0128] ② Based on the multiple common event pairs identified in the previous step, calculate the system time offset between the ground truth system and the urban low-altitude airspace multimodal sensing network. Although both were connected to a unified high-precision time source before testing, due to the different data acquisition, processing, encapsulation, and storage links within their respective systems, a fixed or slowly drifting transmission and processing delay difference will occur. Assuming that... The first common event, for the first Calculate the time difference for each event. Due to the presence of random errors, statistical methods are needed to estimate the optimal system time offset. The least squares method or direct average calculation is usually used. In this formula, This represents the calculated estimated value of the system time offset; Indicates the number of common events; Indicates the first The timestamp difference of a common event in the two sets of data. Subsequently, global compensation is performed on all timestamps in the real test data stream of the urban low-altitude airspace multimodal perception network to generate new timestamps. After this step, the two sets of data are aligned on the time base, meaning that data points describing the same physical moment have the same or very similar time labels.

[0129] ③True value data typically uses a high-precision global geodetic coordinate system, such as the WGS-84 coordinate system, whose location is in latitude and longitude. as high as the earth This indicates that, for computational convenience, the urban low-altitude airspace multimodal sensing network may use a local Cartesian coordinate system, such as a spatial rectangular coordinate system established with a point in the site as the origin, whose output position is... Spatial alignment requires transforming the coordinates output by the perceptron to the coordinate system of the ground truth data. This necessitates solving for a set of coordinate transformation parameters, including the translation vector. Rotation matrix and scale factor Based on time alignment, a data segment of stable drone flight over a certain period is selected. Within this data segment, the ground truth system provides a series of high-precision position points. The urban low-altitude airspace multimodal sensing network provides corresponding (time-aligned) location points. Using these matching point pairs, the optimal transformation parameters are solved using the least squares adjustment method through the Bursa seven-parameter model or a simplified model. The transformation relationship can be expressed as: In this formula, This represents the coordinates of the true data in the geodetic rectangular coordinate system; It is a translation vector; It is a scaling factor; It is a rotation matrix; These are the local coordinates output by the urban low-altitude airspace multimodal sensing network. Solving for them yields... , , Then, the calibration of the spatial transformation model was completed.

[0130] ④ After time compensation and coordinate transformation model calibration, the two sets of data have a basis for comparison in time and space. However, since the original sampling frequencies of the two sets of data may be different, the corresponding timestamps are not strictly one-to-one, and data resampling is required. A unified, high-frequency time series is selected as the reference time axis, for example, the high-frequency timestamps of the true data. Use this as a reference. For each moment on the reference time axis... The corresponding state already exists in the truth data. For real test data of the urban low-altitude airspace multimodal sensing network, the timestamps of its original output points are: The status is Interpolation algorithms are used to calculate the multimodal sensing network in the urban low-altitude airspace. The state value at any given time. For location data, linear interpolation is typically used: In this formula, Indicates the interpolated result The target location (local coordinates) of the multimodal sensing network in the urban low-altitude airspace at any time; and It is the adjacent area in the raw data of the urban low-altitude airspace multimodal sensing network. Two timestamps; symbols and This corresponds to the original position. Then, using the transformation parameters obtained from the solution, the interpolated local coordinates are... The data was uniformly transformed to the same WGS-84 geodetic rectangular coordinate system as the true data, resulting in... It can be further converted into latitude and longitude height format. .

[0131] ⑤ After completing the above steps, two sets of data sequences with completely synchronized time and unified spatial coordinate system are generated: the true value data sequence. Real test data sequence aligned with urban low-altitude airspace multimodal sensing network Alignment quality verification is required. Verification methods include: checking if the timestamp difference between the two datasets is close to zero at known common event points; selecting a segment of flight data not involved in the transformation parameter calculation, plotting the trajectory overlap, and visually checking the overlap degree; calculating statistics on the position difference between the two datasets during a period of stable flight, such as the mean and standard deviation, to ensure they meet expectations. Finally, according to the test task requirements, extract the effective data time period for subsequent index calculations, for example, removing segments where the UAV's takeoff, landing, and turning maneuvers cause tracking instability, retaining data from stable flight segments such as straight lines and constant speeds. At this point, the temporal and spatial alignment of the collected ground truth data with the actual test data is complete, outputting two sets of aligned data that can be directly used for performance index calculations.

[0132] 2) Based on the time- and space-aligned ground truth data and actual test data, and according to the established evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is calculated, and this ratio is used as the detection probability. The specific implementation process is as follows:

[0133] ① Call upon the time- and space-aligned ground truth data sequence and the real test data sequence output by the urban low-altitude airspace multimodal sensing network. Based on the test objectives and technical specifications, select the applicable rule from the established evaluation rules. If the test focuses on evaluating the system's instantaneous response and positioning accuracy at each ground truth sampling moment, then rule one: proportional statistics based on the number of data points is adopted. If the test focuses on evaluating the system's ability to continuously provide effective sensing information at a specified time granularity, especially when the system update cycle differs from the ground truth system, then rule two: coverage statistics based on fixed time intervals is adopted.

[0134] ② If rule one is used, then the following operation is performed: traverse each aligned truth data point. For the first... From 10 data points, obtain the longitude from the ground value data. ,latitude ,high And the longitude in the corresponding real test data output by the urban low-altitude airspace multimodal sensing network. ,latitude ,high Calculate the horizontal error. and vertical error The horizontal error is calculated using the semi-versus formula:

[0135]

[0136] In this formula, express Horizontal error at any given time; The average radius of the Earth; Indicates latitude difference; This represents the difference in longitude. The formula for calculating the vertical error is: In this formula, express Vertical error at time. Determine if the data point is a valid sensing point: If simultaneously satisfying... and If it is, then it is considered valid, where The horizontal error threshold (e.g., 10 meters). Set a vertical error threshold (e.g., 15 meters). Count the total number of all valid sensing points. Expected amount of perceived data The total number of ground truth points participating in the comparison. Detection probability. The calculation formula is: .

[0137] If rule two is adopted, the following operations are performed: The entire test timeline... Divided into consecutive and non-overlapping fixed time intervals, each interval having a length of [length missing]. (e.g., 1 second), the total number of intervals is For the first time interval Check if there is at least one valid sensing point within this window that meets the accuracy threshold criterion of Rule 1. If so, determine that time interval as a valid coverage interval. Count the number of all valid coverage intervals. The expected amount of perceived data is equal to the total number of time intervals. Detection probability The calculation formula is: .

[0138] ③ Output the calculated detection probability value. The above calculation can be performed independently with a single three-dimensional test grid cell as the spatial unit. All data points or time intervals falling into each grid cell are counted separately to calculate the local detection probability of each grid cell, thereby achieving a refined spatial distribution evaluation of performance.

[0139] 3) In the ground truth data and actual test data after time and space alignment, count the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network for the same test target is updated per unit time, and use this number of updates as the detection information update frequency. The specific implementation process is as follows:

[0140] From the temporally and spatially aligned sequence of real test data, extract all real test data points belonging to the same specific test objective based on the target identifier. Then, categorize these data points according to their timestamps. The data is sorted from smallest to largest to form an ordered, continuous stream of real-world test data for that target. To avoid the influence of unstable states, such as when the target is just acquired or about to be lost, on the statistical results, a segment of data where the target is stably tracked is typically selected for calculation; for example, data from the beginning and end of the track is removed. A unit of statistical time is selected. For example, 1 second. Within the selected stable tracking segment time range, multiple segments of length are divided using a sliding window or consecutive non-overlapping windows. The time window. For a time window The system iterates through the real test data stream belonging to the target and counts the number of valid sensing information entries whose timestamps fall within this window. Valid sensing information refers to data records output by the urban low-altitude airspace multimodal sensing network that contain valid timestamps and complete target status (such as location). The counted number of entries represents the update count within this time window. For each time window, calculate its detection information update frequency. For example, if If two data points are counted within the window within a second, then... Hertz. To characterize the update frequency performance of a target across the entire flight segment or a specific area, the Hertz frequency can be calculated over all time windows. Calculate the average value to obtain the average update frequency. Simultaneously, the distribution of update frequencies can be analyzed, such as maximum, minimum, and standard deviation. This calculation can also be performed using three-dimensional test grid cells to analyze the update frequency characteristics of different spatial regions.

[0141] 4) Calculate the horizontal and vertical errors between the ground truth data and the actual test data after time and space alignment, respectively, as the detection accuracy. The specific implementation process is as follows:

[0142] Iterate through each data pair after time and space alignment. For the first... For each data pair, calculate its horizontal error using the method for calculating the detection probability rule. and vertical error The calculation formulas and symbols are completely consistent with those described above. This applies to all selected evaluation segments or specific three-dimensional test grid cells. Statistical analysis is performed on the error values ​​of each data pair. Commonly used statistical features include: average mean error. Root mean square error (RMS): The 95th percentile error refers to: Sort the level error values ​​from smallest to largest, and take the nth value. The value of each position is used as Vertical statistics , , The calculation method is similar. These statistics together constitute a complete description of the detection accuracy of the multimodal sensing network in urban low-altitude airspace.

[0143] 5) When the test target triggers a preset abnormal event, record the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal sensing network, and calculate the detection alarm delay based on the time difference between the first and second timestamps. The specific implementation process is as follows:

[0144] Before testing begins, the logical rules and thresholds for triggering alarms are precisely pre-defined in the rule engine of the integrated management service platform, such as electronic fence intrusion, safety interval conflicts, and excessive flight path deviations. This ensures that the truth system, the urban low-altitude airspace multimodal sensing network, the integrated management service platform, and the independent data recording system are all synchronized to a unified high-precision time source, with synchronization errors controlled within milliseconds. In laboratory simulations or field tests, the test target is manipulated to perform specific actions according to a predetermined plan, causing its state to violate preset rules, thereby triggering a preset abnormal event. For example, controlling a drone to fly into a no-fly zone. The truth system or a dedicated event monitoring module continuously monitors the target's state. When the target state is detected to meet the abnormal conditions, the precise moment of this event is immediately recorded as the first timestamp. This moment is determined through state interpolation or a high-frequency decision strategy. An independent end-to-end data recording system continuously monitors the alarm information output interface of the integrated management service platform. When it captures an alarm information network packet issued by the platform corresponding to the triggered event, it immediately assigns it a receiving timestamp based on the same time base, serving as a second timestamp. Calculate the original time difference. To accurately measure the platform's internal processing latency, the network transmission time of alarm information from the platform's exit point to the recording system must be deducted. . The time-to-round-trip time of the network path is estimated beforehand. Ultimately, the detection alarm latency is determined. The calculation formula is: In this formula, This indicates the calculated detection and alarm delay; Indicates the time when the recording system received the alarm; Indicates the precise time when the abnormal event occurred; This indicates the estimated network transmission latency. Verify the consistency between the alarm information and the triggering event, and confirm the timestamp reference is correct. Include the event type, spatial location (associated with grid cells), and... , , , and whether the requirements are met (e.g.) Complete records, including seconds, are archived.

[0145] In this context, the effective sensing data volume refers to the number of data units that meet quality requirements, selected according to established evaluation rules, from the real test data output by the urban low-altitude airspace multimodal sensing network when calculating detection probability or similar performance indicators. Under evaluation rules based on the number of data points, it specifically refers to the total number of individual sensing data points whose horizontal and vertical errors, when compared with ground truth data, simultaneously do not exceed preset thresholds. Under evaluation rules based on fixed time intervals, it refers to the total number of fixed-length time intervals containing at least one effective sensing point. The effective sensing data volume is a core molecule for quantifying the usability and accuracy of the system's output data, directly determining the final performance indicator score.

[0146] The expected sensing data volume refers to the total amount of data units used as an ideal reference standard when evaluating the performance of a multimodal sensing network in urban low-altitude airspace. It represents the total scale of sensing data that the system should provide under perfect conditions. Its specific meaning varies depending on the selected evaluation rule: when using a rule based on the number of data points, it equals the total number of all data sampling points provided by the true system for comparison; when using a rule based on time intervals, it equals the total number of time intervals obtained by dividing the entire evaluation period into fixed windows. The expected sensing data volume serves as the denominator in performance indicator calculations, providing an objective and absolute benchmark for measuring the system's data completeness or time coverage capability.

[0147] In this context, valid sensing information output for the same test target refers to all compliant data records belonging to the same tracked test target within the data stream of the urban low-altitude airspace multimodal sensing network. This information is extracted from a time- and space-aligned sequence of real test data, arranged chronologically based on a unique target identifier. Each piece of valid sensing information is a complete data unit, requiring a valid timestamp and target state parameters at the current moment, such as the fused 3D position coordinates. This series of continuous data records constitutes a continuous trajectory description of the test target, providing the direct data basis for calculating the detection information update frequency and analyzing time-related performance indicators such as trajectory continuity.

[0148] The test target triggering a pre-defined abnormal event refers to, during performance testing, the controlled operation causing the flight state or behavior of the drone or other aircraft used as the test target to violate one or more safety monitoring rules pre-set in the integrated management service platform. These rules clearly define the judgment conditions and thresholds for abnormal situations, such as intrusion into an electronic fence, deviation from the planned flight path exceeding the limit, or distance from other targets being less than the safe interval. The triggering behavior is part of the test plan, aiming to proactively create a measurable risk scenario to evaluate the ability of the urban low-altitude airspace multimodal perception network and its upper-level platform to perceive, judge, and respond to abnormal situations. The moment of occurrence of this event is the starting reference point for subsequent calculation of detection and alarm latency.

[0149] 6) In the real data and actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability. The specific implementation process is as follows:

[0150] 1) After completing temporal and spatial alignment, two sets of data are obtained: a truth data sequence provided by the truth system, containing the precise trajectories of all real test targets; and a real test data sequence output by the urban low-altitude airspace multimodal sensing network, containing all target track reports generated by the system. First, all independent tracks reported by the system are extracted from the real test data sequence. Each independent track consists of a series of data points with the same target identifier that are temporally continuous or nearly continuous. Next, these system-reported tracks need to be correlated and matched with the truth target tracks. The matching process typically employs data association algorithms, such as the nearest neighbor method, setting a reasonable spatiotemporal association threshold. For each track reported by the system, it is checked whether a truth target track exists within its lifecycle, allowing for successful temporal and spatial correlation. Successful correlation means that the system track correctly tracks a real-world target.

[0151] 2) After association matching, system-reported tracks that failed to be successfully associated with any ground truth target track are initially identified as "unassociated tracks". However, not all unassociated tracks should be counted as false alarms. Judgment rules need to be introduced to exclude momentary false tracks caused by brief interference or data jitter. An unassociated track must meet the following conditions to be ultimately classified as a "false alarm track": First, the duration of the track must exceed a preset minimum threshold. For example, 5 seconds. This ensures that the statistics are based on meaningful false tracks that the system continuously misidentifies as real targets, rather than transient noise. Second, the track must contain a certain number of data points, and the track must exhibit a certain continuity or motion logic in space, rather than being completely random scattered points. By setting these conditions, most invalid alarms caused by random noise or single misjudgments can be effectively filtered out, focusing on the persistent false alarm problem at the system level.

[0152] 3) After completing the false alarm track determination, count the total number of tracks ultimately determined to be false alarms during the entire evaluation period, and record it as follows: Calculating the false alarm probability requires determining a suitable baseline for the "total detection opportunities." According to industry practice, one of two baselines is typically used. The first baseline is the "total number of association attempts," which is approximately equal to the total number of times the system tracks and identifies all real targets plus the number of false alarms. Operationally, this can be expressed as the total number of real targets. Total system runtime Joint estimation. The second type of base, especially in the context of a system scanning at fixed periods, divides the total evaluation time into time units and counts the total number of times the system outputs tracks. For clarity, a common simplified form of the first base is used: false alarm probability. Defined as the average number of false alarm tracks occurring per unit time, or more classically, as the ratio of the number of false alarm tracks to the total number of actual target occurrences. A commonly used formula is: In this formula, This represents the calculated false alarm probability; This represents the total number of false alarm tracks obtained from statistics; Indicates the selected total base. This represents the total number of independent tracks (including correctly correlated and false alarms) output by the urban low-altitude airspace multimodal sensing network throughout the entire testing process. Thus, the formula... The percentage of false alarm tracks to all output tracks of the system is given directly. This represents the number of correctly associated tracks. This value directly reflects the purity of the system's output information.

[0153] 4) To more precisely evaluate system performance, the calculation of false alarm probability can be spatially decomposed into three-dimensional test grid cells. Specifically, the entire test airspace is divided into grids. For each grid cell, the number of false alarm tracks within its spatial range and the number of real target tracks passing through that grid (or the total number of output tracks within that grid) are counted individually. Then, the local false alarm probability of that grid cell is calculated. This method generates a spatial distribution map of the false alarm probability, helping to identify which areas (such as building obstruction edges or near electromagnetic interference sources) are more prone to false alarms, providing directional guidance for system optimization. Finally, the overall false alarm probability value is output. The evaluation of this indicator is completed using the optional spatial distribution analysis results.

[0154] The process of obtaining scores for multiple performance indicators of the urban low-altitude airspace multimodal sensing network includes:

[0155] 1) Calculate the score for the detection probability. Specifically, based on the selected evaluation rule (Rule 1 or Rule 2), calculate the raw value of the detection probability, for example... or The raw value needs to be mapped to a percentage score according to a preset scoring criterion. The scoring criterion is typically a piecewise function or a lookup table. For example, setting a score based on the detection probability. The mapping rule is: if the detection probability ,but points; if ,but ;like ,but In this rule, This indicates the score of the detection probability index; This represents the calculated raw detection probability value. The calculated... or Substitution By calculating according to the above mapping rules, the final score of the detection probability can be obtained. If the detection probability is calculated separately for each three-dimensional grid cell, then each grid cell is mapped to a score independently, or the original probability values ​​of all grid cells are averaged and then mapped to a single score.

[0156] 2) Calculate the score for the detection information update frequency, specifically based on the calculated average update frequency. For example, 2.5 Hz, which is mapped to a percentage score. The scoring criteria need to define a desired update frequency. and the lowest acceptable frequency For example, for a specific type of low-altitude target, the desired update frequency is... Hertz, minimum requirement Hertz. Score. The mapping rule can be designed as follows: If ,but points; if ,but ;like ,but In this rule, The score represents the frequency of detection information updates; This represents the calculated average update frequency; and These are the desired frequency and the minimum frequency threshold, respectively. The score can be calculated by substituting the values. .

[0157] 3) Calculate the detection accuracy score. Specifically, detection accuracy includes two dimensions: horizontal and vertical. The scores are usually calculated separately and then combined. For horizontal detection accuracy, a representative statistic is selected, such as the root mean square error (RMSE). Set a threshold standard for accuracy scores, for example, require... The passing score is 60 points (meters). A meter is considered excellent (corresponding to 100 points). Horizontal accuracy score. The mapping formula can be designed as follows:

[0158]

[0159] In this formula, This indicates the score for horizontal detection accuracy. This represents the calculated root mean square error of the water level; and These are the acceptable and excellent thresholds, respectively. Vertical accuracy score. The calculation method is similar, using the vertical root mean square error. And its corresponding threshold. Finally, the total detection accuracy score. It can be the average of the horizontal and vertical scores: .

[0160] 4) Calculate the score for detection and alarm latency, specifically, the average detection and alarm latency calculated based on multiple test events. For example, 3.2 seconds, for score mapping. Set a latency requirement, such as a target value. Seconds (corresponding to 100 points), maximum allowed value Seconds (corresponding to 60 points). Detection alarm latency score. The mapping rule can be designed as follows: If ,but points; if ,but ;like ,but Points. In this rule... This represents the score for the detection alarm latency indicator; This represents the calculated average alarm latency; and These are the target latency and the maximum allowable latency, respectively.

[0161] 5) Calculate the score for the false alarm probability. Specifically, based on the calculated raw value of the false alarm probability... For example, 0.5%, which is mapped to a percentage score. A tolerance upper limit for the false alarm probability is set, for example... The passing grade is 60 points. The cutoff score is 100 (excellent). False alarm probability score. The mapping rule can be designed as follows: If ,but points; if ,but ;like ,but (Minimum score is 0). Under this rule, This represents the score of the false alarm probability index; This represents the calculated false alarm probability; and These are the maximum allowed value and the excellent value, respectively.

[0162] Through the above five steps, the detection probability score was obtained respectively. Detection information update frequency score Detection accuracy score Detection alarm delay score And false alarm probability score These five scores constitute a standardized score set for evaluating the performance of urban low-altitude airspace multimodal sensing networks, and can be directly input into the subsequent performance level determination module for comprehensive processing. All scoring mapping rules and thresholds are clearly defined before testing to ensure the objectivity and consistency of the evaluation.

[0163] S4. Based on the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network, determine the performance level of the urban low-altitude airspace multimodal sensing network. The specific implementation process is as follows:

[0164] S40. Extract the percentage scores for various performance indicators of the urban low-altitude airspace multimodal sensing network. These scores include at least the detection probability score. Detection information update frequency score Detection accuracy score Detection alarm delay score And false alarm probability score These five scores are organized into an ordered set of values, forming a performance index score vector, denoted as... , This represents a vector composed of scores for all performance metrics. After extraction, data verification must be performed immediately to ensure the accuracy of each score. ( The values ​​of the representative indicator numbers are all within a reasonable percentage range, that is... Furthermore, all scoring calculation processes and original data are verifiable and can only proceed to the next stage after verification.

[0165] S41. The overall score is not a simple average of the scores across all indicators. Instead, it requires assigning different importance weights to different indicators based on the specific application needs of the urban scenario. Determining the weights is a formalized process. Operationally, the test organizers convene a review panel composed of multiple industry experts. These experts conduct multiple rounds of independent, anonymous scoring using the Delphi method, based on the specific urban scenario targeted in this test—for example, a densely populated city center, a key location with electromagnetic shielding requirements, or a typical suburban scenario. Each expert... For each indicator Assign a weight value All weight values ​​satisfy the normalization condition After multiple rounds of feedback and statistical analysis, a convergent weight vector representing the group's consensus was finally obtained. . Represents a defined vector of indicator weights; , , , , These represent the weights of detection probability, update frequency, detection accuracy, alarm latency, and false alarm probability, respectively, and satisfy the following conditions: For example, in key location security scenarios, the detection probability... and false alarm probability The weight may be assigned a higher value.

[0166] S42. Obtain the performance index score vector. With a defined weight vector Combined, a weighted composite score is calculated. The calculation formula is: In this formula, This represents the calculated comprehensive performance score of the urban low-altitude airspace multimodal sensing network, a value between 0 and 100. The calculation process involves multiplying the score of each indicator by its corresponding weight, and then summing the weighted scores of all five indicators. This calculation is typically automated using specialized evaluation software or scripts to ensure the accuracy and traceability of the calculation process. For example, if the score vector is... The weight vector is ,but .

[0167] S43, Obtain overall performance score Next, this numerical score needs to be mapped to predefined, discrete performance levels. The mapping rules are strict and well-defined, explicitly stated before testing. For example, a standard four-level mapping rule is: if the overall score... If so, the performance level is determined to be Grade A. If so, the performance level is determined to be B. If so, the performance level is determined to be C. If the performance rating is not met, the performance level is determined to be D. This determination process is automatic and requires no manual intervention. For example, the calculated performance rating is D. According to the mapping rules, it falls within the range of 80 to 90, therefore the performance level of the city's low-altitude airspace multimodal sensing network is determined to be Level B.

[0168] S44. Organize the process and results of the first four steps to form a formal rating conclusion. The conclusion document should clearly list the score for each performance indicator. Weights determined by expert review The calculated comprehensive score The final performance level will be determined and integrated into the final test report. Before the report is released, the technical lead of the testing organization or a designated third-party auditing expert must conduct a final review of the entire evaluation process. The review will focus on the compliance of the weight determination procedure, the correctness of the comprehensive score calculation, and the accuracy of the level mapping. Once the review is passed, the final determination of the performance level of the city's low-altitude airspace multimodal sensing network will take effect. This level conclusion, along with the complete calculation process record, will be archived to provide an authoritative decision-making basis for the capability assessment, operational permitting, or subsequent technical improvements of the low-altitude sensing network.

[0169] The above technical solution also includes:

[0170] S5. Based on the data obtained during the testing of the multimodal sensing network in the urban low-altitude airspace, a test report is generated. The specific implementation process is as follows:

[0171] S50. After completing all testing phases of the urban low-altitude airspace multimodal sensing network, initiate the report generation process. The primary task is to establish a centralized test database, collecting test data scattered across various stages and subsystems by category. This data is mainly divided into several categories: The first category is environmental and configuration data, including test time, location, physical environment parameter records, a list of urban low-altitude airspace multimodal sensing network equipment and deployment location coordinates, and the selection and description of test scenarios; the second category is process log data, including simulation test platform operation logs, signal injection records, data processing terminal debugging records, and flight path plans and flight control logs for real flight tests; the third category is core input and output data, including simulated signals and data used in the simulation phase, benchmark true data collected by the high-precision benchmark measurement system in the real test phase, raw outputs from the urban low-altitude airspace multimodal sensing network data acquisition terminal, fused trajectory data output from the data processing terminal, and calculation results of various indicators generated by the indicator calculation terminal; the fourth category is evaluation result data, including scores for various performance indicators, records of the Delphi method weight determination process, comprehensive score calculation process and results, and performance level determination results.

[0172] S51. After data aggregation, it cannot be directly used to generate reports; it must undergo data quality verification. Verification includes checking the temporal continuity, logical rationality, and correlation between different data sources. For example, it is necessary to verify whether the target track timestamps output by the urban low-altitude airspace multimodal sensing network are correctly aligned with the timestamps of the baseline ground truth data under a unified time reference. The system will automatically check the integrity of data packets and mark potentially contradictory abnormal data points. For abnormal data, it is necessary to review the test logs to determine whether it is caused by a momentary system failure, environmental interference, or test operation error, and decide whether to remove the data or add annotations based on established data validity rules. For data that needs to be calculated, such as performance indicators, ensure that the original input data is complete and error-free.

[0173] S52. The test report should include a test overview, equipment list, measured data, calculation results, overall score and grade determination, and must be accompanied by an evaluation log and data summary table. Therefore, report generation is a structured filling process. Test personnel or the report generation system, following this architecture, organize the data verified in step two into standard descriptions, tables, charts, and attachments. The test overview section should describe in detail the test purpose, basis, organizer, tested system, test environment, and scenario; the equipment list should list the models, serial numbers, and key parameters of all sensors, drones, and measuring equipment involved in the test in tabular form; the measured data section should display representative raw data samples, such as a comparison chart of the baseline and fused tracks of a typical flight; the calculation results section should detail the calculation process, intermediate values, and final scores for each core performance indicator; the overall score and grade determination section should display the weight vector, the calculation process of the overall scoring formula, and the final grade mapping results.

[0174] S53. The test log is a chronological record of the testing process, detailing every key operation, system state change, anomaly, and handling measures from test preparation to completion. The data summary table is a highly condensed version of the massive test data, typically presenting statistical information on key results in tabular form. For example, a single data summary table might summarize the average detection probability, average positioning accuracy, and average alarm latency at different altitudes. Attachments may also include important raw data graphs, spectrum analysis charts, fusion situational awareness screenshots, and raw statistical tables of expert weighted scores. These attachments are crucial support for the conclusions in the main report, ensuring the testing process is traceable and reviewable.

[0175] S54. Integrate all written chapters and attachments into a unified document framework to form a draft test report. The test technical lead and project lead will conduct multiple rounds of review of the report, focusing on: data accuracy, objectivity of conclusions, depth of analysis, compliance with formatting requirements, and full adherence to testing specifications. During the review process, areas requiring supplementary analysis or clarification may be identified, and the report writers must revise accordingly. After finalization, the test report will be converted to a non-editable portable document format or other prescribed standard format for output and archiving. The final test report, as the official deliverable of the urban low-altitude airspace multimodal sensing network performance testing and evaluation work, will be submitted to the client or relevant management departments, providing authoritative technical basis for their decisions on low-altitude sensing network construction, equipment selection, or system acceptance.

[0176] The data obtained during the testing of the urban low-altitude airspace multimodal sensing network refers to the sum of all original and derived information generated and recorded throughout the entire testing lifecycle, from simulation verification to real-world testing. It is not merely the final performance metrics, but a complete data set encompassing input, process, output, and environment. Specifically, it includes: configuration parameters and real-time monitoring data of the testing environment; characteristics of simulated signals and true values ​​of simulated flight paths used in simulation testing; signal sequences injected during data input verification and received response logs; fusion results of scenario-based test vectors input and output during data processing verification; high-precision true value data collected by an independent benchmark measurement system in the real physical environment; data streams from the data acquisition end, data processing end, and performance indicator calculation end of the urban low-altitude airspace multimodal sensing network itself during real-world testing; and system status logs, event logs, and operation logs recorded by the monitoring system throughout the testing process. This data forms the sole factual basis for objectively recording testing activities, reproducing the testing process, analyzing system performance, and generating final evaluation conclusions.

[0177] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation. The scheme after adjusting the order is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0178] like Figure 2 As shown, an embodiment of the present invention provides a performance testing and evaluation system 200 for a multimodal sensing network in urban low-altitude airspace, comprising a test environment construction module 201, a first test module 202, a second test module 203, and a determination module 204.

[0179] The test environment construction module 201 is used to: build a test environment containing various complex urban scenarios;

[0180] The first test module 202 is used to: conduct simulation tests on the urban low-altitude airspace multimodal sensing network based on the test environment, and use the simulated signals and simulated data provided by the test environment to verify the data input end, data processing end and index calculation end of the urban low-altitude airspace multimodal sensing network in sequence;

[0181] The second test module 203 is used to: test the verified urban low-altitude airspace multimodal sensing network in a real physical environment, simultaneously collect the truth data provided by the corresponding truth system and the real test data output by the urban low-altitude airspace multimodal sensing network, and calculate the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network based on the real test data and the truth data.

[0182] The determination module 204 is used to determine the performance level of the urban low-altitude airspace multimodal sensing network based on the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network.

[0183] Optionally, in the above technical solution, the first test module 202 is specifically used to: run the urban low-altitude airspace multimodal perception network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expected results.

[0184] Optionally, in the above technical solution, the first test module 202 is further specifically used for:

[0185] When the initial performance evaluation results meet the expectations, the simulated signal provided by the test environment will be injected into the data input terminal to verify the data reception function;

[0186] Based on the requirements of the simulated urban test scenario in the test environment, the simulated data corresponding to the requirements of the urban test scenario is input into the data processing terminal for logic verification and debugging.

[0187] Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal.

[0188] Optionally, in the above technical solution, the performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability.

[0189] The second test module 203 is also used for:

[0190] Align the collected ground truth data with the actual test data in terms of time and space;

[0191] In the ground truth data and real test data after time and space alignment, according to the set evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is statistically analyzed, and this ratio is used as the detection probability.

[0192] In the real data and actual test data after time and space alignment, the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network to the same test target is updated per unit time, and this number of updates is used as the detection information update frequency.

[0193] In the real data and the actual test data after time and space alignment, the horizontal error and vertical error between the actual test data and the real data are calculated respectively, which are used as the detection accuracy.

[0194] When the test target triggers a preset abnormal event, the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal perception network are recorded, and the detection alarm delay is calculated based on the time difference between the first timestamp and the second timestamp.

[0195] In the real data and actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability.

[0196] Optionally, the above technical solution also includes a generation module, which is used to generate a test report based on the data obtained when testing the multimodal sensing network in urban low-altitude airspace.

[0197] It should be noted that the beneficial effects of the urban low-altitude airspace multimodal sensing network performance testing and evaluation system 200 provided in the above embodiments are the same as the beneficial effects of the urban low-altitude airspace multimodal sensing network performance testing and evaluation method described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0198] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned methods for performance testing and evaluation of urban low-altitude airspace multimodal sensing networks.

[0199] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for performance testing and evaluation of urban low-altitude airspace multimodal sensing networks.

[0200] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0201] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace, characterized in that, include: Construct a test environment that includes various complex urban scenarios, including rainy weather scenarios, foggy weather scenarios, key location scenarios, high-traffic areas scenarios, low-view-distance occlusion scenarios, and targeted scenarios; Based on the test environment, a simulation test was conducted on the urban low-altitude airspace multimodal sensing network. Using the simulated signals and data provided by the test environment, the data input end, data processing end, and index calculation end of the urban low-altitude airspace multimodal sensing network were verified in sequence. In a real physical environment, the verified urban low-altitude airspace multimodal sensing network is tested. The ground truth data provided by the corresponding ground truth system and the real test data output by the urban low-altitude airspace multimodal sensing network are collected simultaneously. Based on the real test data and ground truth data, the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network are calculated. The performance level of the urban low-altitude airspace multimodal sensing network is determined based on the scores of multiple performance indicators. Based on the test environment, simulation tests were conducted on the multimodal sensing network in the urban low-altitude airspace, including: Run the urban low-altitude airspace multimodal sensing network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expected results; The performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability. Based on the real test data and ground truth data, several performance indicators of the urban low-altitude airspace multimodal sensing network were calculated, including: The collected truth data is aligned with the actual test data in both time and space. In the real data and the actual test data after time and space alignment, according to the set evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is calculated, and this ratio is used as the detection probability. In the real data and the actual test data after time and space alignment, the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network to the same test target is updated per unit time, and this number of updates is used as the detection information update frequency. In the real data and the real test data after time and space alignment, the horizontal error and vertical error between the real test data and the real data are calculated respectively, which are used as the detection accuracy. When the test target triggers a preset abnormal event, the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal sensing network are recorded, and the detection alarm delay is calculated based on the time difference between the first timestamp and the second timestamp. In the real data and the actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability.

2. The method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace according to claim 1, characterized in that, Using the simulated signals and data provided by the test environment, the data input end, data processing end, and index calculation end of the urban low-altitude airspace multimodal sensing network are verified sequentially, including: When the preliminary performance evaluation results meet the expectations, the simulated signal provided by the test environment is injected into the data input terminal to verify the data receiving function; Based on the urban test scenario requirements simulated by the test environment, the simulated data corresponding to the urban test scenario requirements are input into the data processing terminal for logic verification and debugging. Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal.

3. A method for performance testing and evaluation of a multimodal sensing network in urban low-altitude airspace according to any one of claims 1 to 2, characterized in that, Also includes: A test report is generated based on the data obtained during the testing of the multimodal sensing network in the urban low-altitude airspace.

4. A performance testing and evaluation system for a multimodal sensing network in urban low-altitude airspace, characterized in that, It includes a test environment construction module, a first test module, a second test module, and a determination module; The test environment construction module is used to: construct a test environment containing various complex urban scenarios, including rainy day scenarios, foggy day scenarios, key location scenarios, high-traffic area scenarios, low-view-distance occlusion area scenarios, and targeted scenarios; The first test module is used to: conduct simulation tests on the urban low-altitude airspace multimodal sensing network based on the test environment, and use the simulated signals and simulated data provided by the test environment to verify the data input end, data processing end and index calculation end of the urban low-altitude airspace multimodal sensing network in sequence; The second testing module is used to: test the verified urban low-altitude airspace multimodal sensing network in a real physical environment, simultaneously collect the truth data provided by the corresponding truth system and the real test data output by the urban low-altitude airspace multimodal sensing network, and calculate the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network based on the real test data and the truth data. The determining module is used to: determine the performance level of the urban low-altitude airspace multimodal sensing network based on the scores of multiple performance indicators of the urban low-altitude airspace multimodal sensing network; The first testing module is specifically used to: run the urban low-altitude airspace multimodal sensing network in the test environment, generate preliminary performance evaluation results, and determine whether the preliminary performance evaluation results meet the expected results; The performance indicators of the urban low-altitude airspace multimodal sensing network include: detection probability, detection information update frequency, detection alarm latency, detection accuracy, and false alarm probability. The second test module is also used for: The collected truth data is aligned with the actual test data in both time and space. In the real data and the actual test data after time and space alignment, according to the set evaluation rules, the ratio of the effective sensing data output by the urban low-altitude airspace multimodal sensing network to the expected sensing data is calculated, and this ratio is used as the detection probability. In the real data and the actual test data after time and space alignment, the number of times the effective sensing information output by the urban low-altitude airspace multimodal sensing network to the same test target is updated per unit time, and this number of updates is used as the detection information update frequency. In the real data and the real test data after time and space alignment, the horizontal error and vertical error between the real test data and the real data are calculated respectively, which are used as the detection accuracy. When the test target triggers a preset abnormal event, the first timestamp of the event and the second timestamp of the alarm information output by the urban low-altitude airspace multimodal sensing network are recorded, and the detection alarm delay is calculated based on the time difference between the first timestamp and the second timestamp. In the real data and the actual test data after time and space alignment, the probability of erroneously detecting the test target is statistically analyzed and used as the false alarm probability.

5. The urban low-altitude airspace multimodal sensing network performance testing and evaluation system according to claim 4, characterized in that, The first test module is also specifically used for: When the preliminary performance evaluation results meet the expectations, the simulated signal provided by the test environment is injected into the data input terminal to verify the data receiving function; Based on the urban test scenario requirements simulated by the test environment, the simulated data corresponding to the urban test scenario requirements are input into the data processing terminal for logic verification and debugging. Based on the simulated data processed by the data processing terminal, the indicator calculation terminal performs indicator calculations and compares the calculation results with the expected benchmark to verify the accuracy of the indicator calculation terminal.

6. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the performance testing and evaluation method for a multimodal sensing network in urban low-altitude airspace as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the performance testing and evaluation method for a multimodal sensing network in urban low-altitude airspace as described in any one of claims 1 to 3.

Images