Unmanned aerial vehicle visual positioning service system based on event camera

Abstract

The invention provides an event camera-based unmanned aerial vehicle visual positioning service system, and belongs to the technical field of Internet of Things. Event camera data and inertial measurement unit data of an unmanned aerial vehicle are acquired; performing time surface separation processing on the event camera data to obtain visual plane time sequence features corresponding to the event camera data; registering the visual plane time sequence characteristics with three-dimensional point cloud map data of a flight environment where the unmanned aerial vehicle is located to obtain a registration result; and performing cross-modal fusion on the registration result and the inertial measurement unit data to obtain pose positioning data of the unmanned aerial vehicle. According to the technical scheme, on the premise that the perception real-time performance of the unmanned aerial vehicle is guaranteed, based on the assistance of the three-dimensional point cloud map, high-precision positioning of the unmanned aerial vehicle is completed only through input of the inertial measurement unit and the event camera.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and in particular to an event camera-based unmanned aerial vehicle visual positioning service system. Background Art

[0002] As the application scenarios of drones gradually expand to urban environments, the safe operation of drones faces greater challenges. Drone positioning technology plays a vital role in enabling drones to achieve safe and reliable low-altitude operations, and is also one of the core areas of current drone technology research and development.

[0003] Although relevant technologies have been used to conduct real-time positioning of UAVs during flight, these research works still face many problems, such as low positioning accuracy of UAVs, limitation of fixed sensor update frequency and noise, and reliance on external signals. Therefore, how to achieve high-precision positioning of UAVs while ensuring the real-time perception of UAVs is still a technical problem that needs to be solved urgently in the relevant technical field. Summary of the invention

[0004] The main purpose of the embodiment of the present application is to propose a UAV visual positioning service system based on an event camera, which aims to complete high-precision positioning of the UAV based on the assistance of a three-dimensional point cloud map and only with the input of an inertial measurement unit and an event camera, while ensuring the real-time perception of the UAV.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a drone visual positioning service system based on an event camera, the system comprising:

[0006] The acquisition module is used to acquire the drone’s event camera data and inertial measurement unit data;

[0007] An event processing module, used for performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data;

[0008] A registration module, used to register the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a registration result;

[0009] A fusion module is used to perform cross-modal fusion on the registration result and the inertial measurement unit data to obtain the posture positioning data of the UAV.

[0010] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a drone visual positioning service method based on an event camera, the method comprising:

[0011] Get the drone’s event camera data and inertial measurement unit data;

[0012] Performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data;

[0013] Registering the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a registration result;

[0014] The registration result is cross-modally fused with the inertial measurement unit data to obtain the posture positioning data of the UAV.

[0015] In some embodiments, the visual plane temporal feature is a temporal distance field;

[0016] The performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data includes:

[0017] Separating and accumulating the event camera data into a positive event time surface and a negative event time surface based on event polarity;

[0018] A reverse time stamp is constructed based on the positive event time surface and the negative event time surface to obtain a temporal distance field.

[0019] In some embodiments, performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data includes:

[0020] filtering the event camera data based on the inertial measurement unit data to obtain filtered event camera data;

[0021] The filtered event camera data is subjected to a time surface separation process to obtain visual plane temporal features corresponding to the event camera data.

[0022] In some embodiments, cross-modal fusion of the registration result and the inertial measurement unit data to obtain the posture positioning data of the drone includes:

[0023] Optimizing the registration result by using the inertial measurement unit data as an observation value to obtain an optimized registration result;

[0024] The optimized registration result is used as the posture positioning data of the UAV.

[0025] In some embodiments, the method further comprises:

[0026] constructing a factor graph based on the registration result and the inertial measurement unit data;

[0027] The posture positioning data of the UAV in a preset sliding window is optimized based on the factor graph to obtain the optimized posture positioning data of the UAV.

[0028] In some embodiments, the jointly optimizing the posture positioning data of the UAV within a preset sliding window based on the factor graph includes:

[0029] Based on maximum a posteriori estimation, the prior factors, the temporal registration factors and the motion factors in the factor graph are jointly solved to optimize the posture positioning data of the UAV within a preset sliding window;

[0030] Among them, the prior factor is the prior pose and motion model estimation of the drone, the temporal registration factor is the registration residual extracted from the registration result, and the motion factor is the motion residual calculated based on the inertial measurement unit data.

[0031] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the second aspect when executing the computer program.

[0032] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the second aspect is implemented.

[0033] To achieve the above-mentioned purpose, the fifth aspect of the embodiments of the present application proposes a computer program product, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the second aspect.

[0034] The event camera-based UAV visual positioning service system, method, device, storage medium and product proposed in the present application first obtain the event camera data and inertial measurement unit data of the UAV; then perform time surface separation processing on the event camera data to obtain the visual plane timing features corresponding to the event camera data; and further align the visual plane timing features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain the alignment result; finally, cross-modally fuse the alignment result with the inertial measurement unit data to obtain the UAV's posture positioning data.

[0035] In this way, the embodiment of the present application makes full use of the new sensor of the event camera, combines the endogenous motion data (inertial measurement unit data) provided by the inertial measurement unit and the prior three-dimensional point cloud map data, performs high-frequency cross-modal positioning of the drone, and constrains and optimizes the estimated drone's posture positioning data (such as six-degree-of-freedom posture), thereby obtaining accurate and high-frequency real-time posture. In other words, the embodiment of the present application can ensure the real-time perception of the drone, so that the drone can complete high-precision positioning with the assistance of the three-dimensional point cloud map, using only the inertial measurement unit and the event camera as input sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the steps of the event camera-based drone visual positioning service method in some embodiments provided in the embodiments of the present application;

[0037] Figure 2 A schematic diagram of a process of separating polar time surfaces from event camera data and extracting event temporal distance fields in some embodiments of the method for visual positioning service of a drone based on an event camera provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the operation flow of the visual positioning system involved in some embodiments of the drone visual positioning service method based on an event camera provided in an embodiment of the present application;

[0039] Figure 4 for Figure 1 Schematic diagram of detailed process flow of step S102;

[0040] Figure 5 for Figure 1 Schematic diagram of detailed process flow of step S103;

[0041] Figure 6 A schematic diagram of the steps of the drone visual positioning service method based on the event camera provided in the embodiment of the present application in other embodiments;

[0042] Figure 7 A schematic diagram of a factor graph optimization process involved in some embodiments of the event camera-based drone visual positioning service method provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of the overall structure of a visual positioning system involved in some embodiments of the event camera-based drone visual positioning service method provided in the embodiments of the present application;

[0044] Fig. 9 A schematic diagram of the structure of the UAV visual positioning service system based on the event camera provided in the embodiment of the present application;

[0045] Fig.10 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that, although the functional modules are divided in the system / device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system / device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] First, the overall concept of the embodiments of the present application is described.

[0050] Drones are an important core project in the rapidly developing field of robotics. With their flexibility and efficiency, they have become a hot topic of concern in many industries. At present, drones have been widely used in the low-altitude economic field, undertaking diverse tasks such as logistics distribution, rapid rescue, and infrastructure inspection. However, as the application scenarios of drones gradually expand to urban environments, their safe operation also faces greater challenges. This is mainly because the dense building structures and busy human activities in urban environments have put forward higher requirements for the navigation and positioning accuracy of drones.

[0051] In order to ensure the safety of urban buildings and pedestrians, the position of drones must be precisely controlled within the route to avoid the risk of deviation and collision. At the same time, in the case of collaborative operation of multiple drones, accurate positioning is also an important prerequisite for ensuring task coordination and flight safety. Therefore, drone positioning technology plays a vital role in achieving safe and reliable low-altitude operations. It plays a vital role in ensuring the safety and reliability of low-altitude operations. Therefore, drone positioning technology is also one of the core areas of current drone technology research and development.

[0052] In the related technologies, the positioning technology of unmanned aerial vehicles mainly relies on the Global Navigation Satellite System (GNSS) to realize the absolute positioning of unmanned aerial vehicles. For example, unmanned aerial vehicles that are put into operation are usually equipped with signal receivers on board, and the absolute positioning of the current unmanned aerial vehicle is obtained by receiving satellite signals and processing corresponding data. In addition, unmanned aerial vehicles can also be equipped with sensors such as cameras, inertial measurement units, and barometers. During the flight, the onboard computing equipment processes the input data of these sensors and estimates its own position and posture using methods such as Simultaneous Localization and Mapping (SLAM). By running unmanned aerial vehicles or robots in this way, positioning and mapping can be achieved in unknown environments, thereby improving the autonomy of unmanned aerial vehicles or robots, and thus getting rid of dependence on satellite signals. Furthermore, unmanned aerial vehicle positioning can also rely on pre-built high-definition three-dimensional maps, and positioning can be achieved by matching lidar or visual data with maps.

[0053] However, the various drone positioning technologies mentioned above, especially those that rely on the global satellite navigation system, are easily affected by satellite signal obstruction in environments with tall buildings such as urban canyons, which can lead to unstable drone positioning. In addition, when a drone enters a residential area with dense signals, it may encounter signal congestion due to excessive signal access from the service provider, which further affects the accuracy of positioning. In other words, these satellite navigation-based positioning solutions can usually only provide meter-level positioning accuracy, which is difficult to meet the needs of high-precision positioning. At the same time, although drone real-time positioning and mapping technologies based on airborne sensors can provide more accurate positioning information, they are difficult to resist the error caused by sensor noise. In large-scale outdoor scenarios, these technologies have high requirements for computing resources, which not only leads to slow processing speed and high resource consumption, but also easily generates cumulative errors and error drift problems during long-term operation. Although some positioning methods in related technologies attempt to improve positioning accuracy by using urban high-precision maps or point cloud maps and combining them with airborne sensor input for registration, due to the limitation of the update frequency of ordinary radar or visual sensors, when performing real-time positioning of high-speed moving drones, it is easy to have positioning tracking loss problems.

[0054] In summary, although there are academic studies on the positioning technology of drones during flight, these studies still face problems such as low accuracy, limitation of fixed sensor update frequency and noise, and reliance on external signals. Therefore, it is still a challenge to use urban point cloud maps to achieve high-precision positioning of drones while ensuring the real-time perception of drones.

[0055] Based on this, the embodiments of the present application propose a drone visual positioning service system, method, device, storage medium and product based on an event camera, which aims to perform high-frequency cross-modal positioning of the drone, that is, under the premise of ensuring the real-time perception of the drone, the drone can complete high-precision positioning with the assistance of the city point cloud map, using only the inertial measurement unit and the event camera as input sensors.

[0056] Based on the overall concept of the above-mentioned embodiments of the present application, specific embodiments of the drone visual positioning service system, method, device, storage medium and product based on event camera provided in the embodiments of the present application are proposed. First, each specific embodiment of the drone visual positioning service method based on event camera in the embodiments of the present application is described in detail.

[0057] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0058] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0060] In addition, the drone visual positioning service method based on event cameras provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a drone, a ground base station of a drone, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the drone visual positioning service method based on event cameras, etc., but is not limited to the above forms.

[0061] Alternatively, the drone visual positioning service method based on event cameras provided in the embodiments of the present application can also be used in many general or special computer system environments or configurations. For example: drone positioning system, personal computer, server computer, handheld device or portable device, tablet device, multiprocessor system, microprocessor-based system, set-top box, programmable consumer computer device, network PC, minicomputer, mainframe computer, distributed computing environment including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] For ease of understanding and explanation, the following text will take the terminal device applying the drone visual positioning service method based on the event camera provided by the embodiment of the present application as an example for detailed explanation. The implementation of the drone visual positioning service method based on the event camera provided by the embodiment of the present application in any of the above-mentioned forms of the subject application can refer to the process of the terminal device applying the drone visual positioning service method based on the event camera described below.

[0063] Please refer to Figure 1 , Figure 1 A schematic diagram of the steps of the event camera-based drone visual positioning service method in some embodiments of the present application. It should be understood that although Figure 1The execution order of some method steps is shown in the figure, but based on different design requirements of actual applications, the drone visual positioning service method based on event camera provided in the embodiment of the present application can certainly adopt an execution order different from the method steps shown in the figure. That is, Figure 1 The order of the steps of the method shown does not constitute a limitation on the execution logic order of the event camera-based drone visual positioning service method provided in the embodiment of the present application. Figure 1 Reasonable changes in the order of the method steps shown should be included in the protection scope of the event camera-based drone visual positioning service method provided in the embodiments of the present application.

[0064] like Figure 1 As shown, in some embodiments, the event camera-based drone visual positioning service method provided in the embodiments of the present application may include but is not limited to steps S101 to S104.

[0065] Step S101: Acquire event camera data and inertial measurement unit data of the drone.

[0066] During the flight of the drone, the terminal device continuously collects corresponding sensor data through various sensors configured on the drone, and then processes the collected sensor data to realize real-time positioning of the drone to obtain corresponding posture positioning data. Among them, the terminal device first collects the drone's event camera data through at least the drone's event camera, and collects the drone's endogenous motion information through the drone's inertial measurement unit to obtain the drone's inertial measurement unit data.

[0067] Step S102: performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data.

[0068] After obtaining the drone's event camera data and inertial measurement unit data, the terminal device first estimates the drone's position based on the event camera data based on spatiotemporal features. In this process, the terminal device first uses a method of separating polar time surfaces to perform time surface separation processing on the drone's event camera data, thereby obtaining the visual plane temporal features of the drone corresponding to the event camera data.

[0069] In some embodiments, the visual plane temporal feature may be a temporal distance field. In this case, the above step S102 may include the following steps:

[0070] Separating and accumulating the event camera data into a positive event time surface and a negative event time surface based on event polarity;

[0071] A reverse time stamp is constructed based on the positive event time surface and the negative event time surface to obtain a temporal distance field.

[0072] When processing the event camera data of the drone, the terminal device adopts the method of separating polarity time surfaces, dividing the single events in the event camera data into positive events or negative events based on the event polarity, thereby separating and accumulating the entire event camera data into positive event time surfaces and negative event time surfaces. Afterwards, the terminal device further inverts the positive event time surface and the negative event time surface to construct a reverse timestamp, thereby extracting the temporal distance field through a series of calculations. In addition, the terminal device uses the extracted temporal distance field as the feature of the event visual plane (visual plane temporal feature).

[0073] For example, Figure 2 As shown, when the terminal device separates the polarity time surface and extracts the event temporal distance field from the event camera data of the drone, the terminal device accumulates the events in the event camera data into two time surfaces (positive event time surface and negative event time surface) according to the polarity. In the time surface, each pixel value represents the most recent event timestamp at that location, thereby emphasizing the most recent event.

[0074] Based on this, the end device defines the time surface as: Among them, i last (x) is the most recent timestamp at each pixel x, and δ is the set attenuation factor. Then, the terminal device inverts T To construct a reverse timestamp, this will make the pixel value of the edge smaller, that is, the direction of increasing value corresponds to the distance from the edge from far to near, forming a temporal distance field.

[0075] Step S103: aligning the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a alignment result.

[0076] After the terminal device performs time surface separation processing on the event camera data to obtain the visual plane temporal features, it further performs event plane three-dimensional point cloud registration processing, that is, the terminal device registers the visual plane temporal features with the pre-built three-dimensional point cloud map data of the flight environment in which the drone is located, and obtains the registration result between the visual plane temporal features and the three-dimensional point cloud map data.

[0077] Exemplarily, the terminal device may Figure 2 As shown, the temporal distance field generated by separating the polar time surface of the event camera data and extracting the event temporal distance field is used as a temporal feature (visual plane temporal feature) to be aligned with the point cloud (three-dimensional point cloud map data).

[0078] It should be noted that the terminal device separates the polar time surface and extracts the event temporal distance field from the event camera data, and then uses the temporal distance field as a temporal feature to align with the point cloud, thereby converting the registration of events and point clouds into calculations on the temporal distance field, thereby expressing the registration problem of events and point clouds as a minimization problem. For example, a point cloud is projected into the visual plane of the current camera pose, and the value of the coordinate in the temporal distance field is used as the residual of the point cloud, and the coordinate gradient is the residual optimization direction.

[0079] Step S104: cross-modal fusion is performed on the registration result and the inertial measurement unit data to obtain the posture positioning data of the UAV.

[0080] After obtaining the registration results between the visual plane temporal features and the three-dimensional point cloud map data, the terminal device optimizes the drone's posture estimation by performing cross-modal fusion processing on the registration results and the inertial measurement unit data to obtain the drone's precise posture positioning data.

[0081] For example, Figure 3 As shown in the figure, when the terminal device locates and tracks the UAV during flight through the visual positioning system, the event camera and the inertial measurement unit are used as input sensors to provide real-time data observation during the flight of the UAV. And through the fusion of the registration results of the three-dimensional point cloud map data and the event camera data and the measurement results of the inertial measurement unit (inertial measurement unit data), the relative posture update of the UAV at the current moment (posture positioning data) is obtained.

[0082] In the embodiment of the present application, the terminal device first acquires the event camera data of the drone through the event camera of the drone, and acquires the endogenous motion information of the drone through the inertial measurement unit of the drone to obtain the inertial measurement unit data of the drone. Then, a method of separating polar time surfaces is used to perform time surface separation processing on the event camera data of the drone, thereby obtaining the visual plane time series features of the drone corresponding to the event camera data, and further aligning the visual plane time series features with the pre-built three-dimensional point cloud map data of the flight environment of the drone, and obtaining the alignment result between the visual plane time series features and the three-dimensional point cloud map data. Finally, the terminal device performs cross-modal fusion processing on the alignment results together with the inertial measurement unit data, thereby optimizing the posture estimation of the drone to obtain accurate posture positioning data of the drone.

[0083] In this way, the embodiment of the present application makes full use of the new sensor of the event camera, combines the endogenous motion data (inertial measurement unit data) provided by the inertial measurement unit and the prior three-dimensional point cloud map data, performs high-frequency cross-modal positioning of the drone, and constrains and optimizes the estimated drone's posture positioning data (such as six-degree-of-freedom posture), thereby obtaining accurate and high-frequency real-time posture. In other words, the embodiment of the present application can ensure the real-time perception of the drone, so that the drone can complete high-precision positioning with the assistance of the three-dimensional point cloud map, using only the inertial measurement unit and the event camera as input sensors.

[0084] In addition, the embodiment of the present application uses the temporal distance field as a temporal feature for registration with the point cloud through the terminal device, thereby converting the registration of events and point clouds into calculations on the temporal distance field, thereby expressing the registration problem of events and point clouds as a minimization problem, thereby realizing a novel event point cloud registration method, abandoning the traditional idea based on similarity comparison between image frames and point clouds or feature point registration, and enabling more efficient cross-modal data fusion and pose estimation.

[0085] Please refer to Figure 4 , Figure 4 for Figure 1 Schematic diagram of the detailed step flow of step S102.

[0086] In some embodiments, Figure 4 As shown, the above step S102: performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data may also include steps S401 and S402 as shown below.

[0087] Step S401: filtering the event camera data based on the inertial measurement unit data to obtain filtered event camera data.

[0088] When the terminal device estimates the position and posture of the drone based on the spatiotemporal characteristics based on the drone's event camera data, it can also perform hierarchical fusion processing based on inertial measurement based on the inertial measurement unit data. That is, the terminal device integrates the input data of the inertial measurement unit (inertial measurement unit data) to fuse the motion information of the drone with the event camera data in a hierarchical manner in the front and back stages. Among them, in the early stage of processing the event camera data, the terminal device filters and cleans the event camera data through the motion information of the drone represented by the inertial measurement unit data, thereby obtaining the filtered event camera data.

[0089] In some embodiments, the terminal device can determine the polarity of the event by comparing the motion information and the reprojected point cloud gradient attributes, thereby achieving accurate noise filtering of the event camera data to reduce the number of events to be used in the subsequent matching process.

[0090] Step S402: performing time surface separation processing on the filtered event camera data to obtain visual plane temporal features corresponding to the event camera data.

[0091] After filtering the event camera data based on the inertial measurement unit data, the terminal device further performs the time surface separation processing operation on the filtered event camera data as described in the above step S102 and its detailed steps, thereby obtaining the visual plane temporal features corresponding to the event camera data.

[0092] Please refer to Figure 5 , Figure 5 for Figure 1 Schematic diagram of the detailed process flow of step S103.

[0093] In some embodiments, Figure 5 As shown, the above step S103: cross-modal fusion of the registration result and the inertial measurement unit data to obtain the posture positioning data of the drone may include the following steps S501 and S502.

[0094] Step S501: optimizing the registration result by taking the inertial measurement unit data as observation values ​​to obtain an optimized registration result.

[0095] In the process of integrating the input data of the inertial measurement unit, the terminal device integrates the motion information of the drone with the event camera data in a hierarchical manner in the front and back two stages. In the early stage of processing the event camera data, the event camera data is filtered and cleaned by the motion information of the drone represented by the inertial measurement unit data to obtain the filtered event camera data. In the later stage, when the registration result between the filtered event camera data and the three-dimensional point cloud map data is cross-modally fused with the inertial measurement unit data, the terminal device introduces the inertial measurement unit data as an observation value, thereby optimizing the pose estimation result of the drone represented by the registration result to obtain the optimized registration result.

[0096] Step S502: Using the optimized registration result as the posture positioning data of the UAV.

[0097] After obtaining the optimized registration result, the terminal device can directly use the optimized registration result as the posture positioning data of the UAV.

[0098] In this embodiment, the terminal device makes full use of the high-frequency result output (event camera data and inertial measurement unit data) of the event camera and the inertial measurement unit to locate the drone, which can greatly improve the estimation frequency of the drone's own positioning, thereby effectively coping with the rapidly changing environment or high flight speed of the drone. In addition, by fusing the two real-time sensor data through the terminal device, it not only meets the premise of the real-time requirement for drone positioning, but also enables the drone in flight to maintain high-precision positioning. In addition, by using the event camera and the inertial measurement unit as input sensors for the system for positioning the drone, the flight capability of the system in high dynamic range or low light scenes is significantly enhanced, thereby effectively avoiding the image blur or distortion problems of traditional image sensors in such environments, and can also expand the application scope of drones, improve the adaptability and autonomy of drones in complex environments, and avoid the accumulation of errors in positioning the drone due to environmental changes, thereby improving the flight safety and mission execution efficiency of the drone.

[0099] In some embodiments, the terminal device can adopt a method based on factor graph optimization to fuse the event registration estimate (registration result) with the estimate of the inertial measurement unit (inertial measurement unit data), thereby significantly improving the accuracy of positioning the drone while ensuring high-frequency output. The use of this fusion method by the terminal device can not only enhance the robustness of the drone positioning system in a dynamic environment, but also improve the accuracy of the drone pose estimation.

[0100] Please refer to Figure 6 , Figure 6 A schematic diagram of the step flow of the event camera-based drone visual positioning service method provided in the embodiment of the present application in other embodiments.

[0101] In some embodiments, Figure 6 As shown, the event camera-based drone visual positioning service method provided in the embodiment of the present application may also include step S601 and step S602 as shown below.

[0102] Step S601: constructing a factor graph based on the registration result and the inertial measurement unit data.

[0103] When the terminal device performs cross-modal fusion of the registration results and the inertial measurement unit data to determine the position and positioning data of the drone, a factor graph optimization method is adopted to first construct a factor graph consisting of variable nodes and factor nodes based on the registration results and the inertial measurement unit data.

[0104] It should be noted that in the factor graph constructed by the terminal device, the variable nodes represent the posture positioning data of the drone (such as six-degree-of-freedom posture), while the factor nodes express the probability distribution of the drone's posture based on the results of data collection by the event camera and the inertial measurement unit (event camera data and inertial measurement unit data).

[0105] Step S602: Optimizing the posture positioning data of the UAV within a preset sliding window based on the factor graph to obtain optimized posture positioning data of the UAV.

[0106] After constructing the factor graph, the terminal device further optimizes the posture positioning data of the UAV within a pre-set sliding window based on the factor graph, thereby obtaining the precise posture positioning data of the UAV after optimization.

[0107] It should be noted that when the terminal device sets a sliding window to perform joint optimization of the drone's posture positioning data, the optimization result of the drone's current posture is used to constrain the optimization results of the previous times.

[0108] In some embodiments, the above step S602: jointly optimizing the posture positioning data of the UAV in the preset sliding window based on the factor graph may include the following steps:

[0109] The prior factors, the temporal registration factors and the motion factors in the factor graph are jointly solved based on the maximum a posteriori estimation to optimize the posture positioning data of the UAV within a preset sliding window.

[0110] It should be noted that when the terminal device constructs the factor graph based on the registration results and the inertial measurement unit data, it also uses the UAV's prior pose and the pose estimation of the UAV by a simple motion model as a priori factors, and, through the temporal registration of the event plane (visual plane temporal features) with the three-dimensional point cloud map data (registration results), extracts the registration residual and introduces it into the factor graph as a temporal registration factor to capture the alignment error between the event mode and the geometric map, and, also uses the observation data of the inertial measurement unit (inertial measurement unit data) to calculate the motion residual and construct the motion factor, thereby further strengthening the constraints on the pose estimation.

[0111] When the terminal device optimizes the posture positioning data of the UAV in the sliding window based on the factor graph, the prior factors, temporal registration factors and motion factors in the factor graph are jointly solved based on the maximum a posteriori estimation, so as to optimize the posture positioning data of the UAV in the sliding window based on the synergistic effect of the three types of factors, namely the prior factors, the temporal registration factors and the motion factors, so as to achieve high-precision estimation of the posture positioning data of the UAV (such as six-degree-of-freedom posture), and at the same time ensure the adaptability and robustness of the positioning of the UAV to complex dynamic scenes.

[0112] For example, Figure 7 As shown in the figure, when the terminal device performs cross-modal fusion of the registration results and the inertial measurement unit data based on factor graph optimization to determine the pose positioning data of the drone, it first constructs a factor graph consisting of variable nodes and factor nodes, where the variable nodes represent the six-degree-of-freedom pose (pose positioning data) of the drone, and the factor nodes express the probability distribution of the pose based on the event camera and inertial measurement unit results (event camera data and inertial measurement unit data), and use the prior pose and simple motion model estimation as prior factors. Subsequently, the terminal device extracts the registration residual through the temporal registration of the event plane and the three-dimensional point cloud map and introduces it into the factor graph as a temporal registration factor to capture the alignment error between the event mode and the geometric map. In addition, the terminal device also uses the observation data of the inertial measurement unit to calculate the motion residual and construct the motion factor to further strengthen the constraints on the pose estimation. The synergy of these three types of factors is jointly solved in the factor graph optimization process, thereby achieving high-precision estimation of the six-degree-of-freedom pose of the drone, while ensuring the adaptability and robustness of the system to complex dynamic scenes. The terminal device performs joint optimization by setting a sliding window, and the optimization result of the drone's current posture will constrain the optimization results of the previous times.

[0113] In the factor graph optimization framework, for the set X of drone pose positioning data in the sliding window, the terminal sets the maximum a posteriori estimation to achieve joint optimization. The goal is to find the most likely pose X under the measurement input (event camera data and inertial measurement unit data). This process can be expressed as:

[0114]

[0115] Since P(T,M) is a constant independent of X, the optimization objective can be further simplified to minimize the following objective function:

[0116]

[0117] Right now:

[0118]

[0119] Among them, Φ i (X) represents the negative log-probability contribution corresponding to each factor.

[0120] Next, a complete embodiment of the drone visual positioning service method based on an event camera proposed in the embodiment of the present application is presented.

[0121] like Figure 8 As shown, the unmanned aerial vehicle visual positioning service method based on event camera proposed in the embodiment of the present application includes two modules: pose estimation based on spatiotemporal features and hierarchical fusion based on inertial measurement. The input of the visual positioning system includes event camera data, inertial measurement unit (IMU) data, and pre-built three-dimensional point cloud map data, and the output is the pose positioning data of the unmanned aerial vehicle (such as six-degree-of-freedom pose). Since both the event camera and the inertial measurement unit have a high update frequency, the unmanned aerial vehicle visual positioning service method based on event camera proposed in the embodiment of the present application makes full use of this advantage. Through the efficient coordinated operation of the above two modules, the system for positioning the unmanned aerial vehicle can generate high-precision positioning results at a high frequency, thereby meeting the dual requirements of real-time and accuracy of the unmanned aerial vehicle in a complex dynamic environment.

[0122] The pose estimation module based on spatiotemporal features in the above modules is a basic module to ensure the completion of the visual positioning system, including methods for event input processing and event plane three-dimensional point cloud registration. By processing event camera data and three-dimensional point cloud map data of point cloud modality, the basic operation of the visual positioning system and the output of preliminary pose estimation results are ensured. On this basis, the hierarchical fusion module based on inertial measurement of the visual positioning system further performs modal data fusion and result optimization. In terms of event data processing, the pose estimation module based on spatiotemporal features adopts a method of separating polar time surfaces, accumulating single events into positive and negative event time surfaces, and extracting the temporal distance field through a series of calculations as the features of the event visual plane. These features are then aligned with the point cloud map to output accurate pose transformation results. It is worth noting that the embodiment of the present application realizes a novel event point cloud registration, abandons the traditional idea of ​​comparing the similarity between image frames and point clouds or registering feature points, and realizes more efficient cross-modal data fusion and pose estimation.

[0123] The above-mentioned hierarchical fusion module based on inertial measurement is a further optimization of the basic module. It integrates the input data of the inertial measurement unit and fuses the motion information in a hierarchical manner in the front and back stages. In the early event processing stage, the motion information is used for event data filtering and cleaning. By comparing the motion information and the gradient properties of the reprojected point cloud, the polarity of the event can be determined and the noise can be accurately filtered to reduce the number of events used in the matching process. The later inertial measurement values ​​are introduced as observation values ​​to optimize the pose estimation results. The hierarchical fusion module based on inertial measurement adopts a method based on factor graph optimization. By fusing the event registration estimate with the estimate of the inertial measurement unit, it significantly improves the positioning accuracy while ensuring high-frequency output. This fusion method not only enhances the robustness of the system in dynamic environments, but also improves the accuracy of pose estimation.

[0124] See also Fig. 9 The embodiment of the present application also provides a drone visual positioning service system based on an event camera, which can implement the above-mentioned drone visual positioning service method based on an event camera, and the system includes:

[0125] The acquisition module is used to acquire the drone’s event camera data and inertial measurement unit data;

[0126] An event processing module, used for performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data;

[0127] A registration module, used to register the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a registration result;

[0128] A fusion module is used to perform cross-modal fusion on the registration result and the inertial measurement unit data to obtain the posture positioning data of the UAV.

[0129] In some embodiments, the visual plane temporal feature is a temporal distance field;

[0130] The event processing module is further used to separate and accumulate the event camera data into a positive event time surface and a negative event time surface based on event polarity; and to construct a reverse time stamp based on the positive event time surface and the negative event time surface to obtain a temporal distance field.

[0131] In some embodiments, the event processing module is also used to filter the event camera data based on the inertial measurement unit data to obtain filtered event camera data; and perform time surface separation processing on the filtered event camera data to obtain visual plane timing features corresponding to the event camera data.

[0132] In some embodiments, the fusion module is further used to optimize the registration result by using the inertial measurement unit data as an observation value to obtain an optimized registration result; and to use the optimized registration result as the posture positioning data of the UAV.

[0133] In some embodiments, the fusion module is further used to construct a factor graph based on the alignment result and the inertial measurement unit data; and, based on the factor graph, optimize the posture positioning data of the drone within a preset sliding window to obtain the optimized posture positioning data of the drone.

[0134] In some embodiments, the fusion module is further used to jointly solve the prior factors, the temporal registration factors and the motion factors in the factor graph based on the maximum a posteriori estimation, so as to optimize the posture positioning data of the UAV within the preset sliding window;

[0135] Among them, the prior factor is the prior pose and motion model estimation of the drone, the temporal registration factor is the registration residual extracted from the registration result, and the motion factor is the motion residual calculated based on the inertial measurement unit data.

[0136] It should be noted that the specific implementation of the event camera-based drone visual positioning service system provided in the embodiment of the present application is basically the same as the specific implementation of the event camera-based drone visual positioning service method mentioned above, and will not be repeated here.

[0137] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned drone visual positioning service method based on event camera when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0138] See also Fig.10 , Fig.10 The hardware structure of a computer device according to an embodiment is shown, and the computer device includes:

[0139] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0140] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the drone visual positioning service method based on the event camera in the embodiment of this application;

[0141] Input / output interface 1003, used to implement information input and output;

[0142] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0143] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0144] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0145] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned event camera-based drone visual positioning service method is implemented.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] An embodiment of the present application also provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the above-mentioned event camera-based drone visual positioning service method is implemented.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A UAV visual positioning service system based on event camera, characterized in that: The system comprises: The acquisition module is used to acquire the drone’s event camera data and inertial measurement unit data; An event processing module, used for performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data; A registration module, used for registering the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a registration result; A fusion module is used to perform cross-modal fusion on the registration result and the inertial measurement unit data to obtain the posture positioning data of the UAV.

2. A drone visual positioning service method based on event camera, characterized in that: The method comprises: Get the drone’s event camera data and inertial measurement unit data; Performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data; Registering the visual plane temporal features with the three-dimensional point cloud map data of the flight environment of the UAV to obtain a registration result; The registration result is cross-modally fused with the inertial measurement unit data to obtain the posture positioning data of the UAV.

3. The method according to claim 2, characterized in that The visual plane temporal feature is a temporal distance field; The performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data includes: Separating and accumulating the event camera data into a positive event time surface and a negative event time surface based on event polarity; A reverse time stamp is constructed based on the positive event time surface and the negative event time surface to obtain a temporal distance field.

4. The method according to claim 2, characterized in that: The performing time surface separation processing on the event camera data to obtain visual plane temporal features corresponding to the event camera data includes: filtering the event camera data based on the inertial measurement unit data to obtain filtered event camera data; The filtered event camera data is subjected to a time surface separation process to obtain visual plane temporal features corresponding to the event camera data.

5. The method according to claim 4, characterized in that The cross-modal fusion of the registration result and the inertial measurement unit data to obtain the posture positioning data of the UAV includes: Optimizing the registration result by using the inertial measurement unit data as an observation value to obtain an optimized registration result; The optimized registration result is used as the posture positioning data of the UAV.

6. The method according to any one of claims 2 to 5, characterized in that The method further comprises: constructing a factor graph based on the registration result and the inertial measurement unit data; The posture positioning data of the UAV in a preset sliding window is optimized based on the factor graph to obtain the optimized posture positioning data of the UAV.

7. The method according to claim 6, characterized in that The jointly optimizing the posture positioning data of the UAV in a preset sliding window based on the factor graph includes: Based on maximum a posteriori estimation, the prior factors, the temporal registration factors and the motion factors in the factor graph are jointly solved to optimize the posture positioning data of the UAV within a preset sliding window; Among them, the prior factor is the prior pose and motion model estimation of the drone, the temporal registration factor is the registration residual extracted from the registration result, and the motion factor is the motion residual calculated based on the inertial measurement unit data.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the event camera-based drone visual positioning service method as described in any one of claims 2 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the event camera-based drone visual positioning service method according to any one of claims 2 to 7 is implemented.

10. A computer program product, wherein the computer program product stores a computer program, characterized in that: When the computer program is executed by a processor, the event camera-based drone visual positioning service method according to any one of claims 2 to 7 is implemented.

Images