Unmanned aerial vehicle positioning and tracking method and system based on millimeter wave and optoelectronic space-time data fusion

Abstract

The application provides a method and system for positioning and tracking unmanned aerial vehicles based on millimeter wave and photoelectric space-time data fusion. Firstly, the millimeter wave spatial information of the reflection point cloud is obtained by using a millimeter wave detection source. Secondly, multi-angle photoelectric space-time flow data is obtained to determine the potential area of the unmanned aerial vehicle. Then, the multi-angle photoelectric space-time flow data is quickly reconstructed into a three-dimensional trajectory. Next, the reflection point position and the three-dimensional attitude of the unmanned aerial vehicle are spatially interwoven and fused. Finally, the real-time spatial coordinate cluster and continuous flight vector are calculated based on the continuous anti-shielding space trajectory formed to complete the positioning and tracking of the unmanned aerial vehicle. The technical solution provided by the application not only improves the perception robustness and reliability of the system in a dynamic shielding scene, but also realizes comprehensive, continuous and stable positioning and tracking of the unmanned aerial vehicle, thereby providing reliable state perception capability for safe flight and control of the unmanned aerial vehicle in a complex environment.

Description

Technical Field

[0001] This application relates to the field of radio detection and optical image processing fusion technology, and in particular to a UAV positioning and tracking method and system based on the fusion of millimeter wave and photoelectric spatiotemporal data. Background Technology

[0002] With the widespread application of drones in urban logistics scenarios, the demand for high-precision and high-reliability positioning and tracking technologies in dynamic environments with flying obstacles such as buildings and vegetation is becoming increasingly urgent. These scenarios require technical solutions that can not only penetrate some obstructions to perceive targets, but also accurately depict the target's motion details using visual gaps, thereby achieving continuous and stable tracking of the drone.

[0003] Current technical solutions mainly rely on single-type sensor data, such as using LiDAR point clouds for 3D scene reconstruction and target tracking. This approach constructs a high-precision 3D model of the surrounding environment by actively emitting laser beams and receiving reflected signals, and then identifies and tracks UAV targets through point cloud registration and segmentation techniques.

[0004] However, the existing solution has significant drawbacks. LiDAR has limited penetration capability against non-rigid obstructions (such as dense foliage, rain, and fog). Its laser beam is easily absorbed or scattered, resulting in missing target point clouds or a sharp drop in signal-to-noise ratio behind the obstruction. It cannot stably provide continuous position information of the obstructed target, thus causing tracking loss or trajectory interruption in complex obstruction scenarios. Summary of the Invention

[0005] This application provides a UAV positioning and tracking method and system based on the fusion of millimeter wave and photoelectric spatiotemporal data, in order to solve the problem in the prior art that it is difficult to penetrate obstacles and obtain high-precision visual details in complex occlusion environments when relying on a single sensor source, which leads to discontinuous target tracking trajectory and easy interruption.

[0006] Firstly, this application provides a UAV positioning and tracking method based on the fusion of millimeter-wave and optoelectronic spatiotemporal data, including:

[0007] In dynamic flight scenarios with flying obstacles, millimeter-wave spatial information containing the reflection point cloud of flying obstacles is obtained using a millimeter-wave detection source;

[0008] Simultaneously acquire multi-angle photoelectric spatiotemporal flow data of the UAV, which includes dynamic image sequences of the UAV captured between the flight obstacles;

[0009] based on the potential area of the unmanned aerial vehicle determined in the millimeter wave spatial information, the multi-angle photoelectric space-time flow data is subjected to rapid three-dimensional trajectory reconstruction of the flight obstacle gap area enhancement, and a three-dimensional trajectory reconstruction result is obtained;

[0010] The reflection point position in the millimeter wave spatial information is spatially inter-embedded and fused with the unmanned aerial vehicle three-dimensional attitude in the three-dimensional trajectory reconstruction result to form a continuous anti-shielding space trajectory.

[0011] According to the continuous anti-shielding space trajectory, the real-time space coordinate cluster and the continuous flight vector of the unmanned aerial vehicle under the dynamic influence of the flight obstacle are calculated to complete the positioning and tracking of the unmanned aerial vehicle.

[0012] Optionally, in the dynamic flight scene with flight obstacles, millimeter wave spatial information containing flight obstacle reflection point clouds is obtained by using a millimeter wave detection source, comprising:

[0013] A millimeter wave detection signal with penetration characteristics is emitted to the dynamic flight scene with flight obstacles;

[0014] Based on the millimeter wave detection signal, the first echo signal reflected from the surface of the flight obstacle and the second echo signal reflected from the surface of the unmanned aerial vehicle are captured by a millimeter wave receiving antenna array;

[0015] The first echo signal and the second echo signal are subjected to synchronous receiving processing to obtain a mixed millimeter wave echo signal containing time domain and frequency domain characteristics;

[0016] The reflection energy distribution characteristics matched with the motion characteristics of the unmanned aerial vehicle are separated from the mixed millimeter wave echo signal;

[0017] Based on the reflection energy distribution characteristics, the dynamic reflection point position formed behind the flight obstacle by the unmanned aerial vehicle is marked in a three-dimensional space coordinate system;

[0018] The dynamic reflection point position is spatially superimposed with the static reflection point position of the flight obstacle to generate millimeter wave spatial information containing the spatial relationship between the unmanned aerial vehicle and the flight obstacle.

[0019] Optionally, multi-angle photoelectric space-time flow data of the unmanned aerial vehicle is synchronously obtained, and the multi-angle photoelectric space-time flow data contains a dynamic image sequence of the unmanned aerial vehicle captured in the flight obstacle gap, comprising:

[0020] The photoelectric sensing device group is deployed in a spatial distribution arrangement manner according to a preset position in the dynamic flight scene;

[0021] control each photoelectric sensing device in the photoelectric sensing device group to collect multi-view optical images of the UAV in the flight obstacle gap in a time-synchronous triggering manner;

[0022] establish a spatial view angle correspondence relationship for the multi-view optical images collected by each photoelectric sensing device to form a multi-angle observation network;

[0023] extract an optical feature change pattern of the UAV in the continuous movement process in the flight obstacle gap from the multi-angle observation network;

[0024] combine the optical feature change pattern with the corresponding timestamp information to generate multi-angle photoelectric space-time flow data.

[0025] Optionally, based on the potential area of the UAV determined in the millimeter wave spatial information, the multi-angle photoelectric space-time flow data is subjected to rapid three-dimensional trajectory reconstruction with flight obstacle gap area enhancement, and a three-dimensional trajectory reconstruction result is obtained, including:

[0026] extract spatial boundary information of the potential area of the UAV from the millimeter wave spatial information;

[0027] map the spatial boundary information of the potential area to each optical observation plane corresponding to the multi-angle photoelectric space-time flow data to form a key attention area under multiple view angles;

[0028] In the dynamic image sequence of the multi-angle photoelectric space-time flow data, optical signal enhancement processing is performed on the key attention area under each view angle to obtain enhanced optical signals of each view angle;

[0029] Based on the spatial boundary information of the potential area, a spatial constraint relationship between different view angles in the multi-angle photoelectric space-time flow data is established;

[0030] According to the spatial constraint relationship, three-dimensional motion features of the UAV in the movement in the flight obstacle gap are extracted from the enhanced optical signals of each view angle;

[0031] The three-dimensional motion features are used to reconstruct the spatial motion trajectory of the UAV over a continuous time to generate a three-dimensional trajectory reconstruction result.

[0032] Optionally, in the dynamic image sequence of the multi-angle photoelectric space-time flow data, optical signal enhancement processing is performed on the key attention area under each view angle to obtain enhanced optical signals of each view angle, including:

[0033] extract a regional optical feature distribution for the key attention area under each view angle;

[0034] determine an optical parameter adjustment strategy according to the optical feature distribution of the region, to generate an optical enhancement parameter set;

[0035] perform parameter adjustment processing on the optical signals in the focus region based on the optical enhancement parameter set, to obtain preliminary enhanced optical signals;

[0036] establish an optical contrast relationship between the UAV movement region and the background region in the focus region, to generate a region contrast enhancement model;

[0037] perform contrast optimization processing on the preliminary enhanced optical signals using the region contrast enhancement model, to obtain optimized enhanced optical signals;

[0038] integrate the optimized enhanced optical signals under each viewing angle to generate enhanced optical signals under each viewing angle.

[0039] Optionally, the reflection point positions in the millimeter wave spatial information and the UAV three-dimensional attitude in the three-dimensional trajectory reconstruction result are spatially embedded and fused to form a continuous anti-shielding spatial trajectory, including:

[0040] establish a spatial correspondence relationship between the reflection point positions in the millimeter wave spatial information and the three-dimensional attitude of the UAV, to obtain a position-attitude mapping relationship;

[0041] based on the position-attitude mapping relationship, embed the reflection point positions in the millimeter wave spatial information into the spatial structure corresponding to the three-dimensional attitude of the UAV, to obtain preliminary fused spatial data;

[0042] extract millimeter wave feature information in the reflection point positions in the millimeter wave spatial information and optical feature information in the three-dimensional attitude of the UAV, to generate a multi-source feature set;

[0043] construct a spatial complementary processing rule according to the multi-source feature set, to form a feature complementary model;

[0044] perform spatial feature complementary processing on the preliminary fused spatial data using the feature complementary model, to obtain optimized fused spatial data;

[0045] perform continuous processing on the optimized fused spatial data in a preset time dimension, to generate a continuous anti-shielding spatial trajectory

[0046] Optionally, according to the continuous anti-shielding spatial trajectory, calculate real-time spatial coordinate clusters and continuous flight vectors of the UAV under the dynamic influence of the flight obstacle, to complete positioning and tracking of the UAV, including:

[0047] extracting spatial position parameters from the continuous anti-occlusion spatial trajectory to obtain a spatial position sequence of the UAV;

[0048] analyzing spatial distribution characteristics in the spatial position sequence of the UAV to determine a coordinate point gathering area;

[0049] generating a real-time spatial coordinate cluster of the UAV according to a spatial density distribution of the coordinate point gathering area;

[0050] extracting spatial displacement changes in a time sequence from the continuous anti-occlusion spatial trajectory to obtain UAV motion displacement data;

[0051] establishing a flight direction change pattern based on the UAV motion displacement data to generate a flight vector calculation model;

[0052] processing the real-time spatial coordinate cluster by using the flight vector calculation model to generate a continuous flight vector of the UAV;

[0053] real-time positioning and tracking the UAV according to the continuous flight vector of the UAV.

[0054] In a second aspect, the application provides an unmanned aerial vehicle positioning and tracking system based on millimeter wave and photoelectric space-time data fusion, comprising:

[0055] A first acquisition module is configured to acquire millimeter wave spatial information containing flight obstacle reflection point clouds by using a millimeter wave detection source in a dynamic flight scene with flight obstacles;

[0056] A second acquisition module is configured to synchronously acquire multi-angle photoelectric space-time flow data of the UAV, which contains a dynamic image sequence of the UAV captured in the flight obstacle gap;

[0057] A reconstruction module is configured to perform fast three-dimensional trajectory reconstruction of the multi-angle photoelectric space-time flow data in the flight obstacle gap area based on a potential area of the UAV determined from the millimeter wave spatial information, to obtain a three-dimensional trajectory reconstruction result;

[0058] A fusion module is configured to perform spatial mutual embedding fusion of reflection point positions in the millimeter wave spatial information and UAV three-dimensional poses in the three-dimensional trajectory reconstruction result, to form a continuous anti-occlusion spatial trajectory;

[0059] A calculation module is configured to calculate real-time spatial coordinate clusters and continuous flight vectors of the UAV under the dynamic influence of the flight obstacles according to the continuous anti-occlusion spatial trajectory, to complete positioning and tracking of the UAV.

[0060] In a third aspect, the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component, and implement the unmanned aerial vehicle positioning and tracking method based on millimeter wave and photoelectric space-time data fusion as described in the first aspect above.

[0061] In a fourth aspect, the present application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the unmanned aerial vehicle positioning and tracking method based on millimeter wave and photoelectric space-time data fusion as described in the first aspect.

[0062] The unmanned aerial vehicle positioning and tracking method provided by the present application effectively overcomes the limitations of a single sensing source in a complex occlusion environment by fusing millimeter wave and photoelectric space-time data. The method uses the penetration characteristics of millimeter waves to obtain spatial information of the target behind the obstacle, simultaneously captures the fine motion details of the target in the gap through a multi-angle photoelectric system, and then generates a continuous anti-occlusion trajectory through a space mutual embedding fusion technology, finally realizes stable and continuous accurate positioning and tracking of the unmanned aerial vehicle, and significantly improves the perception robustness and reliability of the system in a dynamic occlusion scene.

[0063] Further, by extracting spatial position sequences and displacement change data from the fused trajectory, a coordinate cluster representing the accurate position distribution of the unmanned aerial vehicle and a flight vector representing the motion trend thereof can be intelligently generated. This solving method not only ensures the space-time continuity of the positioning result, but also provides a complete description of the motion state of the unmanned aerial vehicle, thereby realizing real-time and smooth tracking of the target under dynamic interference of the flight obstacle, and effectively avoiding tracking interruption and trajectory jump problems.

[0064] These aspects or other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0066] Figure 1 A flowchart of the unmanned aerial vehicle positioning and tracking method based on millimeter wave and photoelectric space-time data fusion provided by the present application is shown;

[0067] Figure 2 A structural schematic diagram of the unmanned aerial vehicle positioning and tracking system based on millimeter wave and photoelectric space-time data fusion provided by the present application is shown;

[0068] Figure 3 A structural schematic diagram of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0069] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the present application will be clearly and completely described below in combination with the drawings in the present application.

[0070] In some processes described in the specification and claims of the present application and the above description, a plurality of operations are included which occur in a specific order, but it should be clearly understood that these operations can be executed or in parallel without the order in which they appear in the text, and the serial numbers of the operations such as 101, 102, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in the text are used to distinguish different messages, devices, modules, etc. and do not represent the order of precedence. Also, "first" and "second" are not of different types.

[0071] The technical scheme in the present application will be clearly and completely described below in combination with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0072] Figure 1 A flowchart of a method for unmanned aerial vehicle positioning and tracking based on millimeter wave and photoelectric space-time data fusion is provided in the present application, as shown in Figure 1 The method comprises the following steps.

[0073] Step 101: In a dynamic flight scene with flight obstacles, millimeter wave spatial information containing flight obstacle reflection point clouds is obtained by using a millimeter wave detection source.

[0074] Optionally, step 101 can specifically include the following steps:

[0075] Step 1011: A millimeter wave detection signal with penetration characteristics is emitted to a dynamic flight scene with flight obstacles;

[0076] Step 1012: Based on the millimeter wave detection signal, a first echo signal reflected from the surface of the flight obstacle and a second echo signal reflected from the surface of the unmanned aerial vehicle are captured by a millimeter wave receiving antenna array;

[0077] Step 1013, synchronously receiving and processing the first echo signal and the second echo signal to obtain a mixed millimeter wave echo signal containing time domain and frequency domain features;

[0078] Step 1014, separating a reflected energy distribution feature matching the motion feature of the unmanned aerial vehicle from the mixed millimeter wave echo signal;

[0079] Step 1015, marking a dynamic reflection point position of the unmanned aerial vehicle formed behind the flight obstacle in a three-dimensional space coordinate system based on the reflected energy distribution feature;

[0080] Step 1016, spatially superimposing the dynamic reflection point position and a static reflection point position of the flight obstacle to generate millimeter wave space information containing the spatial relationship between the unmanned aerial vehicle and the flight obstacle.

[0081] In the above scheme, the millimeter wave detection source refers to a radar device capable of emitting high-frequency millimeter waves and receiving echoes, used to detect the existence and position of a long-distance object; the flight obstacle refers to a static object such as a building or a tree appearing on the flight path of the unmanned aerial vehicle; the flight obstacle reflection point cloud is a three-dimensional space point set formed by the reflection of millimeter wave signals from the surface of these obstacles; the millimeter wave space information is the comprehensive data finally generated, integrating the positional relationship between the obstacle and the unmanned aerial vehicle; the millimeter wave detection signal is a specific frequency radio wave actively emitted by the detection source; the millimeter wave receiving antenna array is a group of antennas arranged according to a specific rule, used to synchronously receive multi-directional echo signals; the first echo signal is from obstacle reflection, and the second echo signal is from unmanned aerial vehicle reflection; the time domain and frequency domain features respectively describe the characteristics of the echo signal changing with time and its frequency composition; the mixed millimeter wave echo signal is the original signal containing all reflection information without processing; the reflected energy distribution feature is a unique signal pattern extracted from the mixed signal, which can identify moving objects (such as unmanned aerial vehicles); the three-dimensional space coordinate system is an X, Y, Z axis reference system used to accurately mark the position of a point; the dynamic reflection point position represents the position point of the moving unmanned aerial vehicle, and the static reflection point position represents the position point of the stationary obstacle.

[0082] In this embodiment, first, a millimeter wave detection source actively emits a millimeter wave detection signal with good penetration ability to the scene where there is a flight obstacle through step 1011. Second, when the signal encounters the flight obstacle and the unmanned aerial vehicle, it will produce reflections respectively, and the millimeter wave receiving antenna array simultaneously captures the first echo signal reflected from the surface of the flight obstacle and the second echo signal reflected from the surface of the unmanned aerial vehicle through step 1012. Then, the receiving system synchronously receives and preliminarily processes the two echo signals to obtain a mixed millimeter wave echo signal containing the reflection information of the obstacle and the unmanned aerial vehicle and retaining the original time and frequency characteristics through step 1013. Then, the signal processing unit separates the reflection energy distribution characteristics matched with the motion characteristics of the unmanned aerial vehicle from the mixed signal according to the principle that the echo signal of the unmanned aerial vehicle will exhibit unique frequency changes (such as Doppler effect) when it is in motion through step 1014. After that, the system calculates and locates in the three-dimensional coordinate system according to the extracted reflection energy distribution characteristics to accurately mark the dynamic reflection point position formed by the unmanned aerial vehicle currently flying behind the flight obstacle through step 1015. Finally, the system superimposes and integrates the dynamic reflection point position and all the static reflection point positions of the flight obstacle identified before in the same three-dimensional space to finally generate a millimeter wave space information completely describing the relative spatial relationship between the unmanned aerial vehicle and the surrounding flight obstacles.

[0083] For example, in a city area with a high-rise building group (A obstacle) and a communication tower (B obstacle), a delivery unmanned aerial vehicle is performing a delivery task. The millimeter wave radar emits a detection signal to the area, and the signal penetrates some materials on the surface of the building. The antenna array of the radar receives strong echoes (first echo signal) reflected from the solid wall surface of the A obstacle and relatively weak echoes (second echo signal) reflected from the unmanned aerial vehicle body flying between the two buildings. These echoes are synchronously processed into a mixed signal, and the system successfully separates the unique signal characteristics representing the motion of the unmanned aerial vehicle by analyzing the subtle changes in the frequency of the mixed signal. Based on this characteristic, the system accurately calculates and marks the real-time position point of the unmanned aerial vehicle relative to the A and B obstacles in the three-dimensional coordinate system of the digital map, and superimposes the dynamic point and the static contour point cloud of the A and B obstacles to generate a clear spatial relationship diagram of the surrounding environment marked with the position of the unmanned aerial vehicle.

[0084] The scheme can effectively penetrate some barriers, detect the unmanned aerial vehicle behind the barriers, and successfully distinguish the signal characteristics of the moving target and the static obstacle from the complex mixed echoes by actively emitting and receiving and processing millimeter wave signals, and finally accurately generate a three-dimensional spatial relationship diagram containing both dynamic targets and static environment, thereby providing reliable and basic environment and target position information for subsequent tracking procedures.

[0085] Step 102, synchronously acquiring multi-angle photoelectric space-time flow data of the UAV, the multi-angle photoelectric space-time flow data containing a dynamic image sequence of the UAV captured in the flight obstacle gap.

[0086] Optionally, step 102 can specifically include the following steps:

[0087] Step 1021, deploying photoelectric sensing devices in a spatially distributed manner according to a preset position in the dynamic flight scene;

[0088] Step 1022, controlling each photoelectric sensing device in the photoelectric sensing device group to collect multi-view optical images of the UAV appearing in the flight obstacle gap in a time-synchronous triggering manner;

[0089] Step 1023, establishing a spatial view angle correspondence relationship for the multi-view optical images collected by each photoelectric sensing device to form a multi-angle observation network;

[0090] Step 1024, extracting an optical feature change pattern of the UAV in a continuous movement process in the flight obstacle gap from the multi-angle observation network;

[0091] Step 1025, combining the optical feature change pattern with corresponding timestamp information to generate multi-angle photoelectric space-time flow data.

[0092] In the above scheme, the multi-angle photoelectric space-time flow data is a data set that synchronously records the appearance form, movement process and accurate time information of the UAV when viewed from different directions; the dynamic image sequence refers to a group of images arranged in chronological order obtained by continuous shooting of a camera, recording the continuous movement process of the target; the dynamic flight scene refers to an actual operating environment in which the UAV and the obstacle can be in a movement state; the spatially distributed setting manner refers to a strategy of installing multiple optical sensors in different physical positions according to a specific geometric layout (such as surrounding or interleaving); the photoelectric sensing device group is a collection of multiple optical cameras or infrared sensors, etc., used to capture light signals from different angles; each photoelectric sensing device refers to each independent camera or sensor unit in the sensor group; the multi-view optical image refers to multiple pictures about the same scene taken from different spatial position points at the same time; the spatial view angle correspondence relationship is a mathematical relationship that establishes how the same physical point in the images taken by different cameras maps to each other; the multi-angle observation network is a virtual and unified collaborative observation system constructed based on the spatial positions and view angle relationships of all cameras; the optical feature change pattern refers to the change rule of the appearance, shape or texture that can represent the movement law of the target identified from the image sequence; and the corresponding timestamp information refers to the shooting time data accurately recorded for each frame of image.

[0093] In this embodiment, first, the operator deploys a group of photoelectric sensors consisting of multiple cameras in a spatially distributed manner at multiple key locations around the dynamic flight scene to be monitored according to a pre-planned layout scheme in step 1021, ensuring that these cameras can cover the entire area from different directions. Second, a central controller sends a synchronous triggering instruction to all cameras in the group of photoelectric sensors in step 1022, ensuring that each photoelectric sensor takes a photo at exactly the same moment, thereby collecting a complete set of multi-view optical images of the UAV passing through the gap between the obstacles. Third, the data processing system establishes the spatial perspective correspondence between the cameras by calculating their known precise installation positions and orientations in step 1023, which means that the system can know how to correspond the same object point seen from different cameras, thereby connecting these independent cameras into a unified multi-angle observation network. Fourth, the system uses this multi-angle observation network to analyze the collected continuous images in step 1024, specifically extracting unique optical feature change patterns that can describe the motion pattern of the UAV moving in the gap between the obstacles, such as the continuous change of its shape or the moving track of a specific color. Finally, the system binds the optical feature change pattern representing the motion pattern with the precise timestamp information of each image in step 1025, and finally generates a complete multi-angle photoelectric space-time flow data containing spatial three-dimensional information and time information.

[0094] In the embodiment of the previous step, in a city area with A and B obstacles, to make up for the lack of millimeter waves in visual details, the operation team deploys a high-definition camera on the roof of each of the four corners of the area (C1, C2, C3, C4 positions), which together form a group of photoelectric sensors. When the millimeter wave detects that the UAV is flying into the gap between the A and B buildings, the central controller immediately sends a synchronous shooting instruction to all cameras from C1 to C4, and the four cameras capture four different angle photos (multi-view optical images) of the UAV passing through the gap at the same moment. The system quickly calculates and establishes the spatial perspective correspondence according to the precise GPS coordinates and lens angles of the four cameras, forming a virtual collaborative observation network. By analyzing this set of synchronous image sequences, the system successfully extracts the moving track of the UAV body logo in the continuous frames (optical feature change pattern), and binds this motion pattern with the millisecond-level precise shooting time of each picture, finally generating a multi-angle photoelectric space-time flow data that records in detail the flight attitude and trajectory of the UAV in the gap.

[0095] The scheme successfully captures the dynamic moment of the target in the gap between obstacles from multiple optimal angles through the deployment of a sensor group arranged in a spatially distributed manner and high-precision time synchronization acquisition; by establishing an accurate spatial perspective relationship network, multiple independent perspectives are integrated into a unified observation system, thereby enabling the effective extraction of the essential characteristics of target motion from complex image sequences; ultimately, by deeply integrating visual features and time information, observation data with high spatial resolution and time continuity are generated, providing a rich and reliable visual basis for subsequent accurate three-dimensional reconstruction.

[0096] In step 103, based on the potential area of the unmanned aerial vehicle determined in the millimeter wave spatial information, the multi-angle photoelectric space-time flow data is subjected to rapid three-dimensional trajectory reconstruction with flight obstacle gap region enhancement, and a three-dimensional trajectory reconstruction result is obtained.

[0097] Optionally, step 103 can specifically include the following steps:

[0098] In step 1031, spatial boundary information of the potential area of the unmanned aerial vehicle is extracted from the millimeter wave spatial information.

[0099] In step 1032, the spatial boundary information of the potential area is mapped to each optical observation plane corresponding to the multi-angle photoelectric space-time flow data, forming a key attention area under multiple perspectives.

[0100] In step 1033, in the dynamic image sequence of the multi-angle photoelectric space-time flow data, optical signal enhancement processing is performed on the key attention area under each perspective to obtain an enhanced optical signal under each perspective.

[0101] In step 1033, the following steps can be specifically included:

[0102] For the key attention area under each perspective, the distribution of optical features in the area is extracted, an optical parameter adjustment strategy is determined according to the distribution of optical features in the area, an optical enhancement parameter set is generated, the optical signal in the key attention area is subjected to parameter adjustment processing based on the optical enhancement parameter set, a preliminary enhanced optical signal is obtained, an optical contrast relationship between the unmanned aerial vehicle movement area and the background area in the key attention area is established to generate a regional contrast enhancement model, the preliminary enhanced optical signal is subjected to contrast optimization processing using the regional contrast enhancement model to obtain an optimized enhanced optical signal, the optimized enhanced optical signals under each perspective are integrated for multi-perspective consistency to generate an enhanced optical signal under each perspective.

[0103] In step 1034, based on the spatial boundary information of the potential area, a spatial constraint relationship between different perspectives in the multi-angle photoelectric space-time flow data is established.

[0104] Step 1035, extracting the three-dimensional motion features of the UAV when moving in the flight obstacle gap from the enhanced optical signals of each view according to the spatial constraint relationship;

[0105] Step 1036, reconstructing the spatial motion trajectory of the UAV in continuous time using the three-dimensional motion features to generate a three-dimensional trajectory reconstruction result.

[0106] In the above scheme, the flight obstacle gap region refers to a physical space surrounded by obstacles but not completely blocked, which can be used for the UAV to pass through; the fast three-dimensional trajectory reconstruction refers to an efficient calculation process aimed at recovering the motion path of the target in three-dimensional space from a two-dimensional image sequence; the three-dimensional trajectory reconstruction result is the output of the process, i.e., the set of position points of the UAV moving continuously in three-dimensional space; the spatial boundary information of the potential region is derived from the millimeter wave data, which is the spatial range and its limits where the UAV may exist; the corresponding optical observation planes refer to the imaging planes of the two-dimensional images captured by each camera; the key attention region is the specific area that needs special processing after mapping on each camera image; the optical signal enhancement processing is a technique for improving image quality by adjusting image parameters; the enhanced optical signal is the processed image data with better quality; the optical feature distribution describes the statistical regularity of the brightness, color, and other attributes of different pixel points in the image; the optical parameter adjustment strategy is a plan for how to adjust parameters (such as brightness, contrast) according to image features; the optical enhancement parameter set is a set containing specific adjustment values (such as gain value); the preliminary enhanced optical signal is the image after preliminary parameter adjustment; the optical contrast relationship refers to the difference degree of the target region and the background region in optical characteristics in the image; the region contrast enhancement model is a calculation model for further expanding the difference between the target and the background; the optimized enhanced optical signal is the final high-quality image after contrast optimization; the multi-view consistency integration is a process to ensure that the images processed from different angles are consistent in content; the spatial boundary information defines the range of attention in three-dimensional space; the spatial constraint relationship is the geometric correspondence rule between different camera views; the three-dimensional motion features are characteristics extracted from multiple view images that can reflect the motion of the target in three-dimensional space; the spatial motion trajectory in continuous time refers to a continuous path formed by the target in three-dimensional space over time.

[0107] In this embodiment, first, through step 1031, the processing system extracts the specific spatial boundary information of the region where the drone is most likely to exist from the approximate range provided by the millimeter wave spatial information, which defines the range of the region in the form of three-dimensional coordinates. Secondly, through step 1032, the system projects these three-dimensional spatial boundary information onto the two-dimensional optical observation plane corresponding to each camera according to the imaging geometry model of each camera, thereby delineating a region of interest in each camera's picture. Then, through step 1033, the system performs optical signal enhancement processing on the dynamic image sequence within the delineated region of interest: first, analyze the optical feature distribution of the image in the region, and accordingly develop an optical parameter adjustment strategy and generate a specific set of optical enhancement parameters (such as brightness gain value) to preliminarily adjust the image and obtain a preliminary enhanced optical signal; then, analyze the optical contrast relationship between the drone movement region and the background region, construct a region contrast enhancement model, and use this model to optimize the preliminary enhanced optical signal to highlight the differences between the target and the background, obtaining an optimized enhanced optical signal; finally, integrate all the optimized images from different angles to ensure their mutual matching, and finally generate the enhanced optical signals of each view. Then, through step 1034, the system again uses the spatial boundary information of the potential region obtained from the millimeter wave data as a geometric constraint to establish precise spatial constraint relationships between different camera perspectives in the multi-angle photoelectric spatiotemporal flow data. After that, through step 1035, the system, according to these spatial constraint relationships, cross-analyzes and extracts three-dimensional motion features that can reflect the essence of the drone's movement in the gap between obstacles from the enhanced optical signals of each perspective, just like solving a geometry problem. Finally, through step 1036, the system uses the extracted three-dimensional motion features to reverse deduce and reconstruct the precise position of each point in three-dimensional space of the drone in the past continuous time in the computer, and connects these points to generate the final complete three-dimensional trajectory reconstruction result.

[0108] In the embodiment of the previous step, in a city area with A and B obstacles, the system calculates the potential area of the UAV in the gap between A and B buildings from the millimeter wave spatial information, and extracts its three-dimensional spatial boundary (a long strip-shaped space). The system projects this three-dimensional boundary onto the two-dimensional pictures of C1, C2, C3, and C4 cameras respectively, and frames the approximate rectangular area (the area of interest) where the gap is located in each picture. For these areas, the system finds that the UAV image is dark due to backlight (optical feature distribution), and then decides to increase the brightness (optical parameter adjustment strategy) and set the gain parameter (optical enhancement parameter set) for processing, obtaining a brighter image (preliminary enhanced optical signal). However, the contrast between the UAV and the dark background is still not enough, so the system establishes a contrast enhancement model (regional contrast enhancement model) to further process and obtain an optimized image with clear black and white (optimized enhanced optical signal), and ensures that the image contents after optimization of the four perspectives are consistent in time and space (multi-perspective consistency integration). Subsequently, the system calculates the corresponding position of a point from the C1 camera to the C2 camera picture using the three-dimensional spatial boundary provided by the millimeter wave (spatial constraint relationship). Based on these constraints, the system accurately extracts the swing features of the UAV tail in the three-dimensional space (three-dimensional motion features) from the four enhanced video streams. Finally, the system successfully reconstructs the complete and smooth three-dimensional flight path of the UAV from flying into the gap to flying out of the gap (three-dimensional trajectory reconstruction result) according to these feature points.

[0109] The core value of this scheme is to cleverly use the coarse-grained spatial prior information provided by the millimeter wave to guide and optimize the fine photoelectric data processing process. By mapping the potential area determined by the millimeter wave to the image, the image range that needs complex processing is greatly reduced, achieving rapid focusing; on this basis, targeted optical signal enhancement is carried out, effectively overcoming the imaging quality degradation problem caused by the occlusion, significantly improving the image clarity and contrast of the target area; by establishing accurate spatial constraints, the accuracy and consistency of the motion features extracted from the multi-perspective data in the three-dimensional space are ensured.

[0110] Step 104, spatially embedding the reflection point position in the millimeter wave spatial information and the three-dimensional attitude of the UAV in the three-dimensional trajectory reconstruction result to form a continuous anti-shielding spatial trajectory.

[0111] Optionally, step 104 can specifically include the following steps:

[0112] Step 1041, establishing a spatial correspondence between the reflection point position in the millimeter wave spatial information and the three-dimensional attitude of the UAV to obtain a position-attitude mapping relationship;

[0113] Step 1042: Based on the position and attitude mapping relationship, the reflection point position in the millimeter-wave spatial information is embedded into the spatial structure corresponding to the three-dimensional attitude of the UAV to obtain preliminary fused spatial data.

[0114] Step 1043: Extract millimeter-wave feature information from the reflection point location in the millimeter-wave spatial information and optical feature information from the three-dimensional attitude of the UAV to generate a multi-source feature set;

[0115] Step 1044: Construct spatial complementary processing rules based on the multi-source feature set to form a feature complementary model;

[0116] Step 1045: Use the feature complementarity model to perform spatial feature complementarity processing on the preliminary fused spatial data to obtain optimized fused spatial data;

[0117] Step 1046: The optimized and fused spatial data is processed to be continuous in a preset time dimension to generate a continuous anti-occlusion spatial trajectory.

[0118] In the above scheme, the reflection point location refers to the three-dimensional spatial coordinate point calculated after the millimeter-wave signal is reflected back from the surface of the UAV; spatial inter-embedded fusion is a special fusion method that refers to combining data from different sensors not simply by superimposing them, but like stitching together... Figure 1 The data are embedded into each other's data structures to form an organic whole; the continuous anti-occlusion spatial trajectory is a continuous and smooth three-dimensional motion path generated after fusion, which can resist interference from occlusion objects; the three-dimensional attitude of the UAV describes the position of the UAV in three-dimensional space and its pitch, yaw and roll rotation states; the position and attitude mapping relationship is the mathematical rule that establishes the correspondence between millimeter-wave reflection points and specific parts on the UAV's three-dimensional attitude model; the preliminary fused spatial data is the fused data obtained after the reflection points are initially embedded into the attitude model, which has not yet been optimized; millimeter-wave feature information is the information extracted from the position of the reflection point that characterizes its physical properties (such as reflection intensity); optical feature information is the information extracted from the three-dimensional attitude that characterizes its visual properties (such as edge contours); the multi-source feature set is the summary of feature information from the above different sources; the construction of spatial complementary processing rules is to formulate rules on how different features complement each other based on their advantages and disadvantages; the feature complementary model is a computational model built according to these rules to optimize the fused data; the optimized fused spatial data is the higher-quality fused data after being processed by the feature complementary model; the preset time dimension refers to a predefined time axis or time series.

[0119] In this embodiment, firstly through step 1041, the system establishes the accurate spatial correspondence between each reflection point position in the millimeter wave spatial information and a specific part on the UAV three-dimensional attitude model described by the three-dimensional trajectory reconstruction result through geometric calculation and model matching, thereby obtaining the position and attitude mapping relationship, which is like determining which component of the UAV model the millimeter wave point cloud should correspond to. Secondly, through step 1042, the system inserts the sparse but strong penetration reflection point positions provided by the millimeter wave into the spatial structure corresponding to the detailed UAV three-dimensional attitude reconstructed by the photoelectric data like a mortise, thereby obtaining a preliminary fusion spatial data that preliminarily combines the two kinds of data together. Then through step 1043, the system extracts the inherent millimeter wave feature information (such as the reflection intensity of the point) from the reflection point positions of the millimeter wave and extracts the rich optical feature information (such as the texture gradient of the surface) from the three-dimensional attitude of the UAV, and collects these information from different sources together to generate a multi-source feature set. Then through step 1044, the system analyzes the multi-source feature set, studies how the millimeter wave features make up for the lack of optical features (such as millimeter wave points when optical is missing), and vice versa, thereby constructing a set of spatial complementary processing rules and forming a computing model that can intelligently complement features, i.e., a feature complementary model. Then through step 1045, the system uses the feature complementary model to process the preliminary fusion spatial data, uses the millimeter wave features to repair or enhance the part that is blurred and missing due to occlusion in the optical attitude, and uses the optical features to refine the accuracy and semantics of the millimeter wave points, thereby obtaining an optimized fusion spatial data that is more complete and accurate in space. Finally, through step 1046, the system performs smoothing connection and interpolation processing on the optimized fusion spatial data along a preset time dimension (i.e., a continuous time axis), ensures that the data at each time is seamlessly connected, and finally generates a continuous anti-occlusion spatial trajectory that is stable, continuous and can effectively resist the influence of occlusion.

[0120] In the embodiment of the previous step, in the urban area with A and B obstacles, the system now has several sparse reflection point positions of the millimeter wave detected in the gap, and the fine three-dimensional attitude of the UAV reconstructed by the four cameras (including the body angle and the rotor state). First, calculate and determine that a certain reflection point of the millimeter wave corresponds to the left front rotor tip on the three-dimensional attitude model of the UAV (establish a position and attitude mapping relationship). Then, embed this millimeter wave point into the three-dimensional attitude model reconstructed by the optical-electricity, so that the model has a very accurate coordinate point at that part (obtain preliminary fused spatial data). Then, extract the strong penetration feature of the millimeter wave point (millimeter wave feature information) and the delicate surface texture feature of the optical model (optical feature information) (generate a multi-source feature set). The system finds that when the UAV is temporarily blocked by the smoke floating out of the A building, the optical model becomes blurred, but the millimeter wave point remains stable. Therefore, the system formulates a rule: when the optical feature confidence decreases, preferentially trust the millimeter wave feature (construct a spatial complementary processing rule and form a feature complementary model). Using this model, the system stabilizes the pose of the optical model with the millimeter wave point within a few frames of smoke blocking, and obtains better data (obtain optimized fused spatial data). Finally, the system smoothly connects the data of all time frames after processing on the timeline, forming a flight trajectory that is still stable and continuous even through smoke blocking (generate a continuous anti-shielding spatial trajectory).

[0121] By establishing an accurate mapping relationship, the present scheme realizes the deep mutual embedding of the millimeter wave sparse point cloud and the optical-electricity delicate three-dimensional attitude in the spatial structure, rather than simply superimposing them; by extracting and fusing multi-source heterogeneous features, the respective advantages of the strong penetration of the millimeter wave and the high resolution of the optical data are fully utilized; by constructing a feature complementary model, the performance degradation of a single sensor in a specific situation (such as shielding and blurring) is intelligently solved, and the advantages are complementary; finally, through continuous processing in the time dimension, a spatial motion trajectory is generated that can still maintain high precision, high stability and strong robustness under complex shielding environment, significantly improving the performance of the overall positioning and tracking system under adverse conditions.

[0122] Step 105, according to the continuous anti-shielding spatial trajectory, calculating the real-time spatial coordinate cluster and continuous flight vector of the UAV under the dynamic influence of the flight obstacle, to complete the positioning and tracking of the UAV.

[0123] Optionally, step 105 can specifically include the following steps:

[0124] Step 1051, extracting a spatial position parameter from the continuous anti-shielding spatial trajectory to obtain a spatial position sequence of the UAV;

[0125] Step 1052, analyze the spatial distribution characteristics in the spatial position sequence of the UAV to determine a coordinate point aggregation area;

[0126] Step 1053, generate a real-time spatial coordinate cluster of the UAV according to the spatial density distribution of the coordinate point aggregation area;

[0127] Step 1054, extract spatial displacement changes in the time sequence from the continuous anti-occlusion spatial trajectory to obtain UAV motion displacement data;

[0128] Step 1055, establish a flight direction change pattern based on the UAV motion displacement data to generate a flight vector calculation model;

[0129] Step 1056, process the real-time spatial coordinate cluster using the flight vector calculation model to generate a continuous flight vector of the UAV;

[0130] Step 1057, according to the continuous flight vector of the UAV, real-time positioning and tracking of the UAV is performed.

[0131] In the above scheme, the real-time spatial coordinate cluster refers to a set of multiple spatial coordinate points representing the possible position of the UAV at a specific time, reflecting the positioning accuracy and uncertainty; the continuous flight vector of the UAV is a vector sequence describing the direction and speed of the UAV motion at each time point; the positioning and tracking of the UAV refers to the process of continuously determining and predicting the spatial position and motion state of the UAV; the spatial position parameter is the basic coordinate information extracted from the trajectory data; the spatial position sequence of the UAV is a set of UAV position coordinates arranged in chronological order; the spatial distribution characteristics are the statistical rules and geometric characteristics of the position points in space; the coordinate point aggregation area is the spatial range where the position points are concentrated; the spatial density distribution describes the number of position points in a unit of space volume; the spatial displacement change is the difference in position coordinates between adjacent time points; the UAV motion displacement data is a set of information recording the position change value and direction of the UAV; the flight direction change pattern is the regular characteristics of the change of the UAV motion direction with time; the flight vector calculation model is a mathematical relationship framework for generating flight vectors based on motion data.

[0132] In this embodiment, first, through step 1051, the processing system extracts the most basic spatial position parameters from the continuous anti-occlusion spatial trajectory data, which are the original three-dimensional coordinate points that constitute the trajectory. By arranging these points in chronological order, a spatial position sequence that can reflect the complete moving path of the UAV is obtained. Second, through step 1052, the system performs statistical analysis on the spatial position sequence to identify the rules and characteristics of the distribution of these coordinate points in three-dimensional space, i.e., the spatial distribution characteristics, and determines the coordinate point aggregation area where the UAV most frequently appears or is most likely to exist according to these characteristics. Next, through step 1053, the system further analyzes the density of points in these coordinate point aggregation areas, i.e., the spatial density distribution, and generates a real-time spatial coordinate cluster representing the most accurate possible position of the UAV through density clustering and other methods. Then, through step 1054, the system extracts the change in position and direction of the UAV between adjacent time points, i.e., the spatial displacement change, from the same continuous anti-occlusion spatial trajectory, and obtains UAV motion displacement data by calculating these changes to describe the motion state of the UAV. After that, through step 1055, the system analyzes the rules and trends of changes in the flight direction of the UAV based on these motion displacement data, establishes a flight direction change pattern, and constructs a flight vector calculation model that can accurately calculate the flight vector. Finally, through steps 1056 and 1057, the system processes the real-time spatial coordinate cluster obtained earlier using this flight vector calculation model to calculate the motion direction and speed of the UAV at each time point, i.e., the continuous flight vector, and realizes continuous positioning and motion state tracking of the UAV based on this vector information.

[0133] In the embodiment of the previous step, in a city area with A and B obstacles, the system obtains a continuous anti-occlusion spatial trajectory of the UAV passing through the gap between A and B buildings. The system first extracts the three-dimensional coordinate points recorded every millisecond (spatial position parameters) from this trajectory and arranges them in chronological order to form a spatial position sequence. Through analysis, it is found that these coordinate points are mainly concentrated in a belt-shaped area with a width of about 2 meters when passing through the gap (spatial distribution characteristics), which is the coordinate point aggregation area. It is found later that the point density is highest in the center of this area and decreases towards the edges (spatial density distribution), so the point group with the highest density in the center is output as the real-time spatial coordinate cluster of the UAV. At the same time, the moving distance and direction between adjacent millisecond points on the trajectory are calculated (spatial displacement change) to obtain displacement data. Analysis shows that the UAV basically maintains straight flight in the gap (flight direction change pattern), and accordingly a flight vector calculation model is established. Finally, the system processes the coordinate cluster using this model to calculate that the UAV is flying at a speed of 5 meters per second in the direction of 270 degrees (continuous flight vector), and accordingly stably tracks its movement.

[0134] The scheme can intelligently solve the fused anti-occlusion trajectory, provide accurate spatial position information of the unmanned aerial vehicle, and reveal the motion state characteristics; the spatial distribution characteristics and density information are extracted from the trajectory data to generate coordinate cluster output reflecting the position reliability; the vector calculation model accurately describing the motion law is established by analyzing the displacement change and direction mode; and finally, the comprehensive, continuous and stable positioning and tracking of the unmanned aerial vehicle are realized by comprehensively considering the position and vector information, thereby providing reliable state sensing capability for safe flight and control of the unmanned aerial vehicle in complex environment.

[0135] Figure 2 A structure diagram of an unmanned aerial vehicle positioning and tracking system based on millimeter wave and photoelectric space-time data fusion is provided for the present application, as shown in Figure 2 The system comprises:

[0136] A first acquisition module 21 is configured to acquire millimeter wave spatial information containing reflection point clouds of flight obstacles by using a millimeter wave detection source in a dynamic flight scene with flight obstacles.

[0137] A second acquisition module 22 is configured to synchronously acquire multi-angle photoelectric space-time flow data of the unmanned aerial vehicle, which contains dynamic image sequences of the unmanned aerial vehicle captured in the gap between the flight obstacles.

[0138] A reconstruction module 23 is configured to perform rapid three-dimensional trajectory reconstruction of the multi-angle photoelectric space-time flow data in the gap region between the flight obstacles based on the potential region of the unmanned aerial vehicle determined from the millimeter wave spatial information, to obtain a three-dimensional trajectory reconstruction result.

[0139] A fusion module 24 is configured to perform spatial mutual embedding fusion of the reflection point positions in the millimeter wave spatial information and the three-dimensional attitude of the unmanned aerial vehicle in the three-dimensional trajectory reconstruction result, to form a continuous anti-occlusion spatial trajectory.

[0140] A solving module 25 is configured to solve real-time spatial coordinate clusters and continuous flight vectors of the unmanned aerial vehicle under the dynamic influence of the flight obstacles according to the continuous anti-occlusion spatial trajectory, to complete the positioning and tracking of the unmanned aerial vehicle.

[0141] Figure 2 The unmanned aerial vehicle positioning and tracking system based on millimeter wave and photoelectric space-time data fusion can perform Figure 1 The implementation principle and technical effects of the unmanned aerial vehicle positioning and tracking method based on millimeter wave and photoelectric space-time data fusion are not described again. The specific operation modes of each module and unit of the unmanned aerial vehicle positioning and tracking system based on millimeter wave and photoelectric space-time data fusion in the above embodiments have been described in detail in the embodiments related to the method, and will not be described in detail here.

[0142] In one possible design, Figure 2 The unmanned aerial vehicle positioning and tracking system based on millimeter wave and optoelectronic space-time data fusion of the illustrated embodiment can be implemented as a computing device, such as a computer. Figure 3 As shown, the computing device can include a storage component 31 and a processing component 32.

[0143] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32.

[0144] The processing component 32 is configured to perform the above Figure 1 The embodiment of the method for positioning and tracking an unmanned aerial vehicle based on millimeter wave and optoelectronic space-time data fusion.

[0145] The processing component 32 can include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, for executing the above method.

[0146] The storage component 31 is configured to store various types of data to support the operation of the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0147] Of course, the computing device can also include other components, such as an input / output interface, a display component, a communication component, etc.

[0148] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0149] The communication component is configured to facilitate wired or wireless communication between the computing device and other devices, etc.

[0150] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform, and the computing device can be a cloud server, and the processing component, the storage component, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0151] The embodiment of the present application further provides a computer storage medium storing a computer program, and the computer program can realize the above method when being executed by a computer. Figure 1 The embodiment of the present application further provides a computer storage medium storing a computer program, and the computer program can realize the above method when being executed by a computer.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0153] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0154] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0155] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for positioning and tracking unmanned aerial vehicles based on millimeter wave and optoelectronic space-time data fusion, characterized in that, The application relates to a method for tracking a UAV in a dynamic flight scene with flight obstacles. In a dynamic flight scene with flight obstacles, millimeter wave spatial information containing reflection point clouds of the flight obstacles is obtained by using a millimeter wave detection source; Synchronously acquiring multi-angle photoelectric space-time flow data of the UAV, which contains a dynamic image sequence of the UAV captured in the gap between the flight obstacles; Based on the potential area of the UAV determined in the millimeter wave spatial information, the multi-angle photoelectric space-time flow data is subjected to rapid three-dimensional trajectory reconstruction with flight obstacle gap area enhancement, and a three-dimensional trajectory reconstruction result is obtained; The reflection point positions in the millimeter wave spatial information are spatially interembedded with the UAV three-dimensional posture in the three-dimensional trajectory reconstruction result to form a continuous anti-shielding space trajectory; According to the continuous anti-shielding space trajectory, real-time space coordinate clusters and continuous flight vectors of the UAV under the dynamic influence of the flight obstacles are calculated to complete the positioning and tracking of the UAV.

2. The method of claim 1, wherein, In a dynamic flight scene with flight obstacles, millimeter wave spatial information containing reflection point clouds of the flight obstacles is obtained by using a millimeter wave detection source, comprising: A millimeter wave detection signal with penetration characteristics is emitted to a dynamic flight scene with flight obstacles; Based on the millimeter wave detection signal, a first echo signal reflected from the surface of the flight obstacle and a second echo signal reflected from the surface of the UAV are captured by a millimeter wave receiving antenna array; The first echo signal and the second echo signal are subjected to synchronous receiving processing to obtain a hybrid millimeter wave echo signal containing time domain and frequency domain characteristics; From the hybrid millimeter wave echo signal, a reflection energy distribution feature matching the motion characteristics of the UAV is separated out; Based on the reflection energy distribution feature, a dynamic reflection point position formed behind the flight obstacle by the UAV is marked in a three-dimensional space coordinate system; The dynamic reflection point position is spatially superimposed with the static reflection point position of the flight obstacle to generate millimeter wave spatial information containing the spatial relationship between the UAV and the flight obstacle.

3. The method of claim 1, wherein, Synchronously acquiring multi-angle photoelectric space-time flow data of the UAV, which contains a dynamic image sequence of the UAV captured in the gap between the flight obstacles, comprising: Deploying a photoelectric sensing device group in a spatial distribution setting mode according to a preset position in the dynamic flight scene; Controlling each photoelectric sensing device in the photoelectric sensing device group to collect multi-view optical images of the UAV appearing in the gap between the flight obstacles in a time-synchronous triggering mode; Establishing a spatial view angle correspondence relationship of the multi-view optical images collected by each photoelectric sensing device to form a multi-angle observation network; Extracting an optical feature change mode of the UAV in a continuous movement process in the gap between the flight obstacles from the multi-angle observation network; Combining the optical feature change mode with corresponding timestamp information to generate multi-angle photoelectric space-time flow data.

4. The method of claim 1, wherein, based on the potential area of the unmanned aerial vehicle determined in the millimeter wave spatial information, performing rapid three-dimensional trajectory reconstruction of the flight obstacle gap area enhancement on the multi-angle photoelectric space-time flow data to obtain a three-dimensional trajectory reconstruction result, including: extracting spatial boundary information of the potential area of the unmanned aerial vehicle from the millimeter wave spatial information; mapping the spatial boundary information of the potential area to each optical observation plane corresponding to the multi-angle photoelectric space-time flow data to form a key attention area under multiple perspectives; in the dynamic image sequence of the multi-angle photoelectric space-time flow data, performing optical signal enhancement processing on the key attention area under each perspective to obtain enhanced optical signals of each perspective; based on the spatial boundary information of the potential area, establishing a spatial constraint relationship between different perspectives in the multi-angle photoelectric space-time flow data; according to the spatial constraint relationship, extracting three-dimensional motion features of the unmanned aerial vehicle when moving in the flight obstacle gap from the enhanced optical signals of each perspective; reconstructing the spatial motion trajectory of the unmanned aerial vehicle over continuous time using the three-dimensional motion features to generate a three-dimensional trajectory reconstruction result.

5. The method of claim 4, wherein, in the dynamic image sequence of the multi-angle photoelectric space-time flow data, performing optical signal enhancement processing on the key attention area under each perspective to obtain enhanced optical signals of each perspective, including: extracting a regional optical feature distribution from the key attention area under each perspective; determining an optical parameter adjustment strategy according to the regional optical feature distribution to generate a set of optical enhancement parameters; based on the set of optical enhancement parameters, performing parameter adjustment processing on the optical signals in the key attention area to obtain preliminary enhanced optical signals; establishing an optical contrast relationship between the motion area and the background area of the unmanned aerial vehicle in the key attention area to generate a regional contrast enhancement model; using the regional contrast enhancement model to perform contrast optimization processing on the preliminary enhanced optical signals to obtain optimized enhanced optical signals; integrating the optimized enhanced optical signals under each perspective for multi-perspective consistency to generate enhanced optical signals of each perspective.

6. The method of claim 1, wherein, spatially inter-embedding the reflection point position in the millimeter wave spatial information and the three-dimensional attitude of the unmanned aerial vehicle in the three-dimensional trajectory reconstruction result to form a continuous anti-shielding spatial trajectory, including: establishing a spatial correspondence relationship between the reflection point position in the millimeter wave spatial information and the three-dimensional attitude of the unmanned aerial vehicle to obtain a position-attitude mapping relationship; based on the position-attitude mapping relationship, embedding the reflection point position in the millimeter wave spatial information into the spatial structure corresponding to the three-dimensional attitude of the unmanned aerial vehicle to obtain preliminary fusion spatial data; extracting millimeter wave feature information in the reflection point position in the millimeter wave spatial information and optical feature information in the three-dimensional attitude of the unmanned aerial vehicle to generate a multi-source feature set; constructing a spatial complementary processing rule according to the multi-source feature set to form a feature complementary model; using the feature complementary model to perform spatial feature complementary processing on the preliminary fusion spatial data to obtain optimized fusion spatial data; The optimized fusion space data is continuously processed in a preset time dimension to generate a continuous anti-occlusion space trajectory.

7. The method of claim 1, wherein, According to the continuous anti-occlusion space trajectory, real-time space coordinate clusters and continuous flight vectors of the UAV under the dynamic influence of the flight obstacles are calculated to complete the positioning tracking of the UAV, including: Space position parameters are extracted from the continuous anti-occlusion space trajectory to obtain a space position sequence of the UAV; The spatial distribution characteristics in the space position sequence of the UAV are analyzed to determine a coordinate point aggregation area; According to the spatial density distribution of the coordinate point aggregation area, real-time space coordinate clusters of the UAV are generated; Space displacement changes in the time sequence are extracted from the continuous anti-occlusion space trajectory to obtain UAV motion displacement data; Based on the UAV motion displacement data, a flight direction change pattern is established to generate a flight vector calculation model; The real-time space coordinate clusters are processed by using the flight vector calculation model to generate continuous flight vectors of the UAV; According to the continuous flight vectors of the UAV, real-time positioning tracking of the UAV is performed.

8. An unmanned aerial vehicle positioning and tracking system based on millimeter wave and optoelectronic space-time data fusion, characterized in that, Comprise: The first acquisition module is used for acquiring millimeter wave space information containing flight obstacle reflection point cloud by using a millimeter wave detection source in a dynamic flight scene with flight obstacles; The second acquisition module is used for synchronously acquiring multi-angle photoelectric space-time flow data of the UAV, which contains dynamic image sequences of the UAV captured in the flight obstacle gaps; The reconstruction module is used for performing fast three-dimensional trajectory reconstruction of the multi-angle photoelectric space-time flow data in the flight obstacle gap area based on the potential area of the UAV determined from the millimeter wave space information to obtain a three-dimensional trajectory reconstruction result; The fusion module is used for spatially inter-embedding fusion of reflection point positions in the millimeter wave space information and UAV three-dimensional attitude in the three-dimensional trajectory reconstruction result to form a continuous anti-occlusion space trajectory; The calculation module is used for calculating real-time space coordinate clusters and continuous flight vectors of the UAV under the dynamic influence of the flight obstacles according to the continuous anti-occlusion space trajectory to complete the positioning tracking of the UAV.

9. A computing device, comprising: Comprise a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to realize the unmanned aerial vehicle positioning tracking method based on millimeter wave and photoelectric space-time data fusion according to any one of claims 1-7.

10. A computer storage medium, characterized in that, A computer program is stored, and when the computer program is executed by a computer, the unmanned aerial vehicle positioning tracking method based on millimeter wave and photoelectric space-time data fusion according to any one of claims 1-7 is realized.

Images