Unmanned aerial vehicle video geographic information superposition method and device based on deep learning

Abstract

The embodiment of the invention provides an unmanned aerial vehicle video geographic information superposition method and device based on deep learning, and relates to the technical field of unmanned aerial vehicle visual enhancement and geographic information fusion. According to the method, after unmanned aerial vehicle detection data are obtained, a current view field model is constructed according to positioning data and flight state data, and vector map data are retrieved from a map database based on the view field model. And performing cross-modal fusion on the visual semantic features of the video frame image and the map structure features of the vector map data by using a space detection model based on deep learning, and establishing space corresponding information between visual contents and geographic elements based on implicit reasoning. And then, according to the spatial corresponding information, superposing the geographic information to the video frame image to generate a superposed frame image. According to the method, pixel-level accurate alignment of the geographic information in the video frame image can be realized by utilizing visual semantics and the cross-modal fusion capability of map priori, and the result deviation of video geographic information superposition is reduced.

Description

Technical Field

[0001] This application relates to the field of UAV visual enhancement and geographic information fusion technology, and in particular to a method and apparatus for overlaying UAV video geographic information based on deep learning. Background Technology

[0002] Video geographic information overlay refers to an information fusion technology that maps geographic elements such as roads, buildings, and landmarks onto video frames. Through video geographic information overlay, richer visual information can be provided in application scenarios such as inspection, emergency command, surveying and mapping, navigation, and situational awareness.

[0003] When overlaying geographic information into drone video, pose data acquired by airborne sensors such as the Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU) can be used. Geographic features in the map are then mapped to pixel positions in the video frames using geometric projection relationships. To achieve this, geographic information overlay requires first acquiring the drone's latitude, longitude, altitude, heading angle, pitch angle, and roll angle. Then, a projection model is constructed using camera intrinsic parameters, and the world coordinates of the geographic features are mapped to the image coordinate system using collinearity equations or projection matrices. Finally, text or icons are rendered at the corresponding pixel positions to achieve the overlay display of video and map.

[0004] However, geographic information overlay based on geometric projection requires high accuracy in GNSS, IMU, and gimbal attitude. When the GNSS signal is unstable, has multipath error, or the gimbal attitude drifts slightly, the projection result of the video geographic information overlay will have significant deviations, causing the geographic information to drift or be misaligned in the video image. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method and apparatus for overlaying UAV video geographic information based on deep learning, in order to solve the problem of large deviations in the video geographic information overlay results.

[0006] According to a first aspect of this application, a method for overlaying UAV video geographic information based on deep learning is provided, the method comprising: Acquire drone detection data, which includes video streams collected by the drone in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream; A current field-of-view model of the UAV is constructed based on the positioning data and the flight status data; the current field-of-view model is described by at least one parameter among the UAV heading angle, gimbal attitude, camera intrinsic parameters, view frustum range, and projection polygon. Based on the current field of view of the field of view model, vector map data is retrieved from the map database; the vector map data includes at least one geographic element among road vectors, landmark locations, points of interest, feature outlines, and attribute information; The spatial detection model is used to infer the spatial correspondence information of the geographic elements in the video frame image; the spatial detection model is a deep learning model that performs cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and establishes the association relationship between the visual content in the video frame image and the geographic elements based on implicit inference. According to the spatial correspondence information, the geographic information corresponding to the geographic element is superimposed onto the video frame image to generate a superimposed frame image.

[0007] In some embodiments, acquiring drone detection data includes: Receive video streams captured by cameras mounted on drones; The video frame image is extracted from the video stream according to a preset sampling frequency. The preset sampling frequency is set according to influencing factors, including at least one of system load, system configuration information, result output accuracy, and feedback information. Extract the timestamps of the video frame images; The system receives a data stream detected by the pose sensor mounted on the drone, and extracts the positioning data and flight status data from the data stream according to the timestamp.

[0008] In some embodiments, constructing a current field-of-view model of the UAV based on the positioning data and the flight status data includes: The UAV heading angle and the gimbal attitude are extracted from the flight status data, and the gimbal attitude includes pitch angle and roll angle. Calculate the rotation matrix based on the heading angle, the pitch angle, and the roll angle; Obtain the intrinsic parameters of the camera and the transformation parameters between the world coordinate system and the camera coordinate system; A projection matrix is ​​constructed by combining the camera intrinsic parameters and the transformation parameters; The view frustum and projection polygon of the UAV camera are calculated based on the projection matrix. The current geographic location of the UAV is extracted from the positioning data, and field of view information is generated based on the rotation matrix, the view frustum range, and the projection polygon; A current field-of-view model is constructed based on the geographical location and the field-of-view information.

[0009] In some embodiments, retrieving vector map data from a map database based on the current field of view of the field of view model includes: Generate a map request based on the field-of-view model; The map request is sent to the cloud server, so that the cloud server responds to the map request by parsing the projected polygon from the map request; and delineates the data retrieval range based on the projected polygon, and retrieves vector map data from the map database based on the data retrieval range; Receive the vector map data fed back from the cloud server.

[0010] In some embodiments, retrieving vector map data from a map database based on the current field of view of the field of view model includes: Obtain the raw map data; Define graph structure information, which includes a node set and an edge set; the node set is used to represent the landmark locations and / or the points of interest; the edge set is used to represent the road vectors; Construct a preprocessing transformation function based on the graph structure information; The preprocessing transformation function is used to transform the original map data into vector map data that can be read by the spatial detection model.

[0011] In some embodiments, the spatial detection model includes a visual encoder, a map encoder, and a feature fusion inference module; using the spatial detection model to infer the spatial correspondence information of the geographic features in the video frame image includes: The video frame images and the vector map data are input into the spatial detection model; The visual encoder is used to extract visual semantic features from the video frame images; The map encoder is used to extract map structure features from the vector map data; The feature fusion inference module performs cross-modal fusion of the visual semantic features and the map structural features, and generates the geographic information. Based on deep learning algorithms, the association between the visual semantic features and the map structural features is implicitly inferred; the deep learning algorithm includes at least one of cross-modal attention mechanism, feature matching algorithm and contrastive learning algorithm; The overlay position of the geographic information in the video frame image is calculated based on the association relationship; the overlay position includes at least one of pixel coordinates, a set of pixel regions, and an equivalent region of the set of pixel regions; The spatial correspondence information is generated by combining the geographic information and the overlay location.

[0012] In some embodiments, the feature fusion inference module performs cross-modal fusion of the visual semantic features and the map structural features, and generates the geographic information, including: Flatten the visual semantic features into a visual sequence; Obtain the map node feature set and learnable weight matrix; An attention triplet is constructed based on the visual sequence, the map node feature set, and the learnable weight matrix. The attention triplet includes a query matrix, a key matrix, and a value matrix. Based on the attention triplet, the output data of multiple attention heads are calculated using an activation function; The output data of multiple attention heads are concatenated and linearly projected to generate a fused feature representation; The geographic information corresponding to the fused feature representation is calculated using a neural network.

[0013] In some embodiments, according to the spatial correspondence information, the geographic information corresponding to the geographic feature is superimposed onto the video frame image to generate a superimposed frame image, including: The overlay location and confidence level are extracted from the spatial correspondence information. The overlay location includes the normalized coordinates of the geographic features in the video frame image predicted by the fully connected layer of the spatial detection model. The confidence level is the prediction confidence of the fully connected layer of the spatial detection model for the normalized coordinates. Obtain the preset confidence threshold and the image resolution of the video frame; A set of pixel regions is generated based on the confidence threshold and the image resolution. The set of pixel regions includes superimposed pixels. The superimposed pixels are pixels formed by multiplying the normalized coordinates of the pixels with a confidence level higher than the confidence threshold with the image resolution. The geographic information is superimposed onto the video frame image according to the set of pixel regions.

[0014] In some embodiments, superimposing the geographic information onto the video frame image according to the set of pixel regions includes: Based on the set of pixel regions, obtain the coordinates of the overlay positions of consecutive frames; The Kalman filter algorithm is used to smoothly update the superimposed position coordinates of the consecutive frames in order to generate the optimal estimated position; The geographic information is overlaid onto the optimal estimated location in the video frame image to generate the overlaid frame image.

[0015] According to a second aspect of this application, a deep learning-based UAV video geographic information overlay device is provided, the device comprising: The data acquisition module is used to acquire UAV detection data, which includes video streams collected by the UAV in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream. The field of view detection module is used to construct a current field of view model of the UAV based on the positioning data and the flight status data; the current field of view model is described by at least one parameter among the UAV heading angle, gimbal attitude, camera intrinsic parameters, view frustum range, and projection polygon. The map data module is used to retrieve vector map data from the map database based on the current field of view of the field of view model; the vector map data includes at least one geographic element among road vectors, landmark locations, points of interest, feature outlines, and attribute information; The spatial detection module is used to obtain the spatial correspondence information of the geographic elements in the video frame image by using a spatial detection model; the spatial detection model is a deep learning model that performs cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and establishes the association between the visual content in the video frame image and the geographic elements based on implicit reasoning. The image overlay module is used to overlay the geographic information corresponding to the geographic element onto the video frame image according to the spatial correspondence information, so as to generate an overlay frame image.

[0016] According to a third aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described deep learning-based UAV video geographic information overlay method.

[0017] According to a fourth aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described deep learning-based UAV video geographic information overlay method.

[0018] By employing the above technical solutions, embodiments of this application provide a method and apparatus for overlaying UAV video geographic information based on deep learning. After acquiring UAV detection data, the method constructs a current field-of-view model based on the positioning and flight status data within the UAV detection data, and retrieves vector map data from a map database based on the field-of-view model. Then, a deep learning-based spatial detection model is used to perform cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and spatial correspondence information between visual content and geographic elements is established based on implicit inference. Finally, geographic information is overlaid onto the video frame image according to the spatial correspondence information to generate an overlaid frame image. This method can utilize the cross-modal fusion capability of visual semantics and map priors to achieve pixel-level precise alignment of geographic information in the video frame image, reducing the deviation in the video geographic information overlay result.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic flowchart of a deep learning-based UAV video geographic information overlay method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall process of video geographic information overlay provided in the embodiments of this application; Figure 3 This is a schematic diagram of the spatial detection model processing flow provided in the embodiments of this application; Figure 4 This is a schematic diagram of the timing smoothing filtering process provided in the embodiments of this application; Figure 5 A schematic diagram of the structure of a deep learning-based UAV video geographic information overlay device provided in an embodiment of this application. Detailed Implementation

[0021] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0022] In this embodiment, geographic information refers to information used to mark geographic elements such as roads, buildings, and landmarks in a scene. Geographic information can be presented through visualization methods such as text, icons, and display colors, and can also be combined with prompts, animation effects, and voice to achieve multimedia information output. Video geographic information overlay refers to an information fusion technology that maps geographic element information such as roads, buildings, and landmarks onto video frames. Through video geographic information overlay, richer visual information can be provided in application scenarios such as inspection, emergency command, surveying and mapping, navigation, and situational awareness.

[0023] As an aircraft, a drone can carry video recording equipment such as cameras to fly and capture video images during its flight, which are called drone videos. When overlaying geographic information into drone videos, it can rely on pose data obtained by airborne sensors such as the Global Navigation Satellite System (GNSS) and Inertial Measurement Unit (IMU), and map geographic features in the map to the pixel positions in the video frames through geometric projection relationships.

[0024] In some embodiments, geographic information overlay can be based on geometric projection. For example, the latitude, longitude, altitude, heading angle, pitch angle, and roll angle of the UAV are first obtained, and then a projection model is constructed by combining the camera's intrinsic parameters. The world coordinates of the geographic features are then mapped to the image coordinate system through collinearity equations or projection matrices. Text or icons are then rendered at the corresponding pixel positions to achieve the overlay display of video and map.

[0025] However, while the direct sensor projection method is computationally simple, it cannot correct the accumulated errors of the sensor itself, making it difficult to achieve pixel-level precise alignment. Furthermore, it requires high accuracy in GNSS, IMU, and gimbal attitude. When the GNSS signal is unstable, multipath errors exist, or the gimbal attitude experiences slight drift, the projection result of the video geographic information overlay will produce significant deviations, causing the geographic information to drift or misalign in the video image.

[0026] In some embodiments, video geographic information overlay can also be achieved based on Simultaneous Localization and Mapping (SLAM) or 3D map matching. Visual SLAM front-end processing is used for feature extraction and tracking, local map construction, and keyframe selection. Then, geographic information is acquired, and through coordinate system alignment, a rigid transformation is established between the relative coordinate system of SLAM and the geographic coordinate system. Finally, joint pose graph optimization based on visual and geographic constraints is performed to annotate the geographic information in the video frame images.

[0027] While visual SLAM or 3D map matching methods offer high accuracy, they require the prior acquisition of massive amounts of high-precision 3D map data and incur significant computational resource costs, making them difficult to implement in real-time on lightweight drones.

[0028] To address the issue of large deviations in video geographic information overlay results, this application provides a deep learning-based UAV video geographic information overlay method in some embodiments. This method effectively reduces the dependence of geographic information overlay technology on high-precision positioning and attitude information by fusing visual semantic features with cloud-based vector map features. It can achieve accurate alignment and stable overlay of geographic information even under GNSS errors or gimbal attitude drift conditions, reducing deviations in video geographic information overlay results and significantly improving the accuracy and robustness of UAV video geographic information overlay. This method is applicable to scenarios such as inspection, emergency command, surveying, and navigation.

[0029] The method can be applied to unmanned aerial vehicles (UAVs), such as the onboard computing unit of a UAV. It can also be applied to electronic devices that establish communication connections with UAVs and have data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control computers. For ease of description, this application embodiment uses a UAV as the executing subject of the method. It should be understood that the method can also be applied to other types of executing subjects, which are not shown one by one in this application embodiment. Figure 1 As shown, the method includes: S101. Obtain drone detection data.

[0030] When performing geographic information overlay, UAV detection data can be acquired first. This UAV detection data includes video streams collected by the UAV in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream.

[0031] like Figure 2 As shown, to acquire UAV detection data, a data acquisition module can be deployed in the UAV's onboard computing unit. This module is used to collect multi-source heterogeneous data from the UAV in real time and ensure data time synchronization. The data acquisition module can acquire video streams captured by the UAV's onboard camera in real time, and simultaneously acquire UAV positioning data and flight status data corresponding to the time of the video streams. The positioning data includes at least positioning data detected by the Global Navigation Satellite System (GNSS).

[0032] The data acquisition module can connect to the airborne camera and flight control sensor system to acquire the video stream captured by the UAV's airborne camera in real time, and simultaneously acquire UAV positioning data and flight status data that strictly correspond to the timestamp of the current frame of the video stream. The positioning data includes at least latitude, longitude, and altitude information provided by the Global Navigation Satellite System (GNSS), and the flight status data includes the UAV's three-axis attitude angles and gimbal orientation information. Therefore, the data acquisition module is physically connected to the UAV's airborne camera interface and flight control sensor bus.

[0033] In some embodiments, when acquiring UAV detection data, a video stream captured by a camera mounted on the UAV can be received first, and video frame images can be extracted from the video stream according to a preset sampling frequency. The preset sampling frequency is set based on influencing factors, including at least one of system load, system configuration information, result output accuracy, and feedback information. Then, the timestamps of the video frame images are extracted, and a data stream detected by the pose sensor mounted on the UAV is received. Positioning data and flight status data are extracted from the data stream according to the timestamps.

[0034] For example, to acquire drone detection data, the onboard computing unit can receive video streams captured by the camera via the onboard camera interface. Then, through system status monitoring, it can determine influencing factors such as system load, system configuration information, output accuracy, and feedback information. The feedback information can include user-defined feedback settings for video acquisition, as well as feedback generated by the system based on the output quality assessment results.

[0035] After obtaining the information on influencing factors, the sampling frequency can be calculated based on this information. The sampling frequency can have a functional relationship with the quantification results of multiple influencing factors. For example, the sampling frequency can be equal to the weighted sum of the quantification results of multiple influencing factors. This allows for increasing the sampling frequency when the system load is low, the system configuration is high, the output accuracy requirement is high, and the feedback information is of high precision, in order to generate higher precision output results. Conversely, the sampling frequency can be reduced to maintain the stable operation of the system.

[0036] While receiving the video stream, the onboard computing unit can also receive the data stream collected by the attitude sensors on the UAV via the flight control sensor bus. Based on the calculated sampling frequency, the computing unit can extract video frame images from the video stream according to the sampling frequency, using them as keyframes. Then, by reading the timestamps of the video frame images, it can extract time-synchronized sensor monitoring data from the data stream according to the timestamps, obtaining GNSS-detected positioning data and flight status data detected by the IMU and gimbal attitude sensors, etc.

[0037] S102. Construct the current field of view model of the UAV based on the positioning data and flight status data.

[0038] After acquiring drone detection data, the field of view can be determined based on the drone detection data. That is, the current field of view model of the drone can be constructed based on the positioning data and flight status data in the drone detection data to determine the current geographical location and field of view information of the drone.

[0039] The field of view model is specifically represented by a frustum model, which describes the three-dimensional spatial range visible to the UAV's camera and its projected area (footprint) on the ground. The frustum is a truncated pyramid shape formed by the near and far clipping planes of a square pyramid emanating from the camera's optical center. It can be defined by parameters such as the horizontal field of view (HFOV), vertical field of view (VFOV), aspect ratio, and imaging angle (pitch angle). Therefore, the current field of view model can be described by at least one parameter among the following: UAV heading angle, gimbal attitude, camera intrinsic parameters, frustum range, and projected polygon.

[0040] To construct a field-of-view model, in some embodiments, when building the current field-of-view model of the UAV based on positioning data and flight status data, the UAV's heading angle and gimbal attitude can be extracted from the flight status data first. The gimbal attitude includes pitch angle and roll angle. Then, a rotation matrix is ​​calculated based on the heading angle, pitch angle, and roll angle, and finally, camera intrinsic parameters and transformation parameters between the world coordinate system and the camera coordinate system are obtained.

[0041] For example, when constructing the current field of view model, the projection matrix of the camera can be constructed first. This is to determine the field of view. For example, it can be based on the drone's current heading angle. Pitch angle and roll angle Calculate the rotation matrix , is represented as:

[0042] in, Indicates heading angle The fundamental rotation matrix about the z-axis; Indicates pitch angle The basic rotation matrix about the y-axis; Indicates roll angle The basic rotation matrix about the x-axis.

[0043] Then, combining the camera intrinsic parameter matrix K and the transformation parameters from the world coordinate system to the camera coordinate system, i.e. and Thus, the projection matrix is ​​constructed, represented as:

[0044] in, These are heading, pitch, and roll angles, respectively. This is the camera intrinsic parameter matrix. These are the rotation transformation parameters from the world coordinate system to the camera coordinate system; These are the rotation and translation transformation parameters from the world coordinate system to the camera coordinate system.

[0045] The system combines camera intrinsic parameters and transformation parameters to construct a projection matrix, calculates the view frustum range and projection polygon of the drone camera based on the projection matrix, extracts the current location of the drone from the positioning data, and generates field of view information based on the rotation matrix, view frustum range and projection polygon, thereby constructing the current field of view model based on the location and field of view information.

[0046] S103. Based on the field of view model, retrieve vector map data from the map database for the current field of view range.

[0047] After constructing the current field of view model of the UAV, map data requests can be made based on the current field of view model. That is, vector map data can be retrieved from the map database based on the current field of view range of the field of view model. Vector map data includes at least one geographic element among road vectors, landmark locations, points of interest (POIs), terrain outlines, and attribute information.

[0048] The drone can request vector map data from a cloud server, that is, send a map request to the cloud server via a communication network to obtain vector map data corresponding to the geographical location and the drone's current field of view. Therefore, in some embodiments, when retrieving vector map data from the map database based on the current field of view model, a map request can be generated first according to the field of view model, and then the map request can be sent to the cloud server. The cloud server then responds to the map request by parsing the projected polygons from the map request; defining the data retrieval range based on the projected polygons; and retrieving vector map data from the map database based on the data retrieval range. Finally, the vector map data is received from the cloud server.

[0049] For example, the onboard computing unit of a drone can also integrate a communication module to establish a data interaction link between the drone and a cloud server. Based on the positioning and flight status data provided by the data acquisition module, the onboard computing unit can determine the drone's current geographical location and field of view coverage, and then send a map request carrying the field of view parameters to the cloud server via the communication module.

[0050] Therefore, after determining the current geographical location and field of view information of the UAV based on positioning data and flight status data, and constructing a view frustum model, a map request can be sent to the cloud server through the communication network to obtain vector map data corresponding to the geographical location and the current field of view of the UAV.

[0051] The map request carries at least one field-of-view parameter, which includes one or more of the following: UAV heading angle, gimbal attitude, camera intrinsics, view frustum, or ground coverage. This allows the cloud server to respond to the map request by parsing the projected polygons from the request, defining the data retrieval area based on the projected polygons, and retrieving vector map data from the map database based on the data retrieval area. For example, the cloud server may use a projection matrix... Calculate the ground cover polygon And retrieve vector data within that range.

[0052] After retrieving the vector map data, the cloud server can return vector map data that matches the field of view based on the field of view parameters. Therefore, the UAV's onboard computing unit can receive the vector map data returned by the cloud server that matches the current field of view through the communication module.

[0053] After obtaining vector map data, it can be preprocessed before being input into a deep learning model. Preprocessing refers to constructing the vector map into a topological graph structure containing nodes and edges, and encoding the node attributes and edge attributes to obtain map embedding vectors.

[0054] In some embodiments, when retrieving vector map data from a map database based on the current field of view using a field-of-view model, raw map data can be obtained, and graph structure information can be defined. This graph structure information includes a node set and an edge set; the node set represents at least one of landmark locations and points of interest; the edge set represents road vectors. The graph structure information can also be labeled with node information to represent geographic features such as feature outlines and attribute information. Based on the graph structure information, a preprocessing transformation function is constructed, and this function is used to transform the raw map data into vector map data readable by the spatial detection model.

[0055] For example, to make the original vector map data via preprocessing transformation function Convert into a model-readable map feature representation A topological graph structure can be used to define the graph structure information, i.e. , where the node set Represents an intersection or point of interest (POI), edge set This represents road connections. Therefore, the node characteristics can be initialized as follows:

[0056] in, Normalized geographic coordinates For unique thermal coding of land cover categories, Embedding semantic attributes This indicates a splicing operation.

[0057] The vector map is then constructed into a topological graph structure, including: using intersections, POIs or key points of ground features as nodes, road connectivity or spatial adjacency as edges, and using geometric attributes, category attributes or name attributes as node and edge features as input to the graph structure encoder for feature extraction.

[0058] S104. Use spatial detection models to infer spatial correspondence information of geographic features in video frame images.

[0059] After obtaining vector map data, deep learning models can be used to extract features from video frame images and perform cross-modal fusion on the vector map data. In other words, spatial detection models can be used to infer the spatial correspondence information of geographic features in video frame images.

[0060] Among them, the spatial detection model is a deep learning model that performs cross-modal fusion of visual semantic features of video frame images and map structure features of vector map data, and establishes the relationship between visual content and geographic elements in video frame images based on implicit reasoning.

[0061] When performing deep learning feature extraction and cross-modal fusion, video frame images and vector map data from the video stream can be input into the deep learning model. The deep learning model extracts the visual features of the video frame images and the map features of the vector map data through two encoders, and establishes the correlation between the visual features and map features through the cross-modal feature fusion mechanism, thereby inferring geographic information.

[0062] To this end, the onboard computing unit of the UAV can use an embedded AI computing platform. Through the AI ​​computing platform, a deep learning model is called to perform dual-channel feature extraction and cross-modal fusion of video frame images and vector map data, infer the geographic information that needs to be superimposed, and then generate the superimposed UAV video information.

[0063] like Figure 3As shown, in some embodiments, the spatial detection model includes a visual encoder, a map encoder, and a feature fusion inference module. The visual encoder extracts visual semantic features from the video frame images, converting the image data into a high-dimensional visual feature representation to capture the texture of ground features and scene semantic information in the image. The map encoder extracts map structure features from the vector map data received by the communication module or its preprocessed results. The map encoder can process rasterized maps, topological map structures, or hierarchically coded data, converting abstract vector geometric information into machine-readable map semantic features.

[0064] The feature fusion inference module is used to perform cross-modal fusion of visual semantic features and map structural features. The feature fusion inference module can use cross-modal attention mechanisms or feature matching methods to implicitly infer the spatial correspondence between visual features and map features in the absence of high-precision extrinsic parameters, and directly output the precise pixel coordinates, bounding boxes or segmentation masks of landmarks in video frame images.

[0065] The feature fusion reasoning module can use deep learning methods such as cross-modal attention mechanism, feature matching or contrastive learning to implicitly reason about the correspondence between visual features and map features.

[0066] Therefore, when using spatial detection models to infer spatial correspondence information of geographic features in video frame images, the video frame images and vector map data can be input into the spatial detection model first, and visual semantic features can be extracted from the video frame images using a visual encoder, and map structure features can be extracted from the vector map data using a map encoder.

[0067] For example, when extracting visual semantic features, a visual encoder can use a Vision Transformer as the visual encoder. For input video frame images Multi-scale feature extraction is performed to obtain visual feature maps. :

[0068] in, Represents a set of visual features; Indicates a visual encoder; This represents the input video frame image; Indicates the first Layered visual features; Indicates feature level; Indicates the first Layer feature map height; Indicates the first Layer feature map width; Number of channels; This represents the total number of floors.

[0069] Similarly, when extracting map structural features, the map encoder can utilize a graph neural network as the map encoder. For the preprocessed vector map data Perform feature aggregation and updating. Then the map encoder's first... k The layer graph convolution operation is defined as follows:

[0070] in, Represents a node i In the k Layered map structure features; Indicates the activation function; Indicates the first k Layer learnable weights; Represents aggregate functions; Representing neighboring nodes j In the k The map structure features of layer 1; Indicate neighboring nodes; It represents the neighborhood set.

[0071] By fusing the results of multi-layer graph convolution, a set of map node features can be output, namely:

[0072] in, This represents the final set of map node features in the output. Represents a node i In the k Layered map structure features; N Indicates the number of nodes.

[0073] After extracting visual semantic features and map structural features through an encoder, a feature fusion inference module performs cross-modal fusion of the visual semantic features and map structural features to generate geographic information. In some embodiments, during cross-modal fusion, the visual semantic features can first be flattened into a visual sequence, and a set of map node features and a learnable weight matrix can be obtained. Then, an attention triplet is constructed based on the visual sequence, the set of map node features, and the learnable weight matrix. The attention triplet includes a query matrix, a key matrix, and a value matrix.

[0074] Based on the attention triplet, the output data of multiple attention heads are calculated using activation functions. Then, the output data of multiple attention heads are concatenated and linearly projected to generate a fused feature representation. Finally, a neural network is used to calculate the geographic information corresponding to the fused feature representation.

[0075] For example, the feature fusion inference module can employ a multi-head cross-attention mechanism to perform cross-modal fusion of the visual semantic features and the map structural features. The cross-modal fusion process can be abstractly represented as:

[0076] in, This represents the function of the cross-modal fusion inference algorithm. This represents the geographic information generated by the model. Therefore, it allows outputting the precise pixel coordinates of landmarks within video frame images.

[0077] Flatten visual features into visual sequences And calculate the query matrix. Key matrix Sum matrix The attention triplet, that is, the attention triplet is represented as:

[0078] Based on the above attention triples, the first one can be calculated. The output of each attention head, namely:

[0079] Feature representation after fusion For the splicing and linear projection of all heads, that is:

[0080] This process implicitly infers the correspondence between visual features and map features. .

[0081] To obtain the spatial correspondence information of geographic features in video frame images, a deep learning algorithm can be used to implicitly infer the association between visual semantic features and map structural features, and the overlay position of geographic information in the video frame image can be calculated based on the association. The overlay position includes at least one of pixel coordinates, a set of pixel regions, and an equivalent region of the set of pixel regions. Then, by combining the geographic information and the overlay position, spatial correspondence information is generated.

[0082] For example, the output of a deep learning model is digitized landmark information and a set of two-dimensional coordinate points representing the overlay location. The digitized landmark information includes one or more geographic elements such as street names, building names, POI names, directional indicators, or distance information. Taking a landmark as an example, the overlay location of a landmark in a video frame image can be represented as a set of pixel regions, i.e.:

[0083] in, This represents the pixel region of a landmark in the video frame image coordinate system, used to describe the spatial shape and coverage of the landmark in the image. Obviously, other equivalent region representations can also be used.

[0084] S105. Based on the spatial correspondence information, overlay the geographic information corresponding to the geographic elements onto the video frame image to generate an overlaid frame image.

[0085] After obtaining the spatial correspondence information, geographic information and video frame images can be rendered and overlaid. That is, based on the geographic information generation results, the corresponding geographic information content is rendered and overlaid onto the corresponding pixel positions of the video frame images, generating the overlaid output video frame, called the overlaid frame image. Multiple overlaid frame images can form an overlaid video stream.

[0086] To achieve rendering overlay, the onboard computing unit of the drone can be configured with a rendering overlay module, which is used to render the landmark name as text, icon or outline highlight to the corresponding position of the video frame based on the pixel coordinates or region obtained by inference.

[0087] To generate overlay frame images, in some embodiments, when performing the overlay of geographic information corresponding to geographic features onto video frame images according to spatial correspondence information, the overlay position and confidence level can be extracted from the spatial correspondence information first. The overlay position includes the normalized coordinates of geographic features in the video frame image predicted by the fully connected layer of the spatial detection model; the confidence level is the prediction confidence of the fully connected layer of the spatial detection model for the normalized coordinates.

[0088] Next, a preset confidence threshold and the image resolution of the video frame are obtained, and a set of pixel regions is generated based on the confidence threshold and the image resolution. The set of pixel regions includes superimposed pixels; these superimposed pixels are pixels formed by multiplying the normalized coordinates (with confidence levels higher than the confidence threshold) by the image resolution. Then, geographic information is superimposed onto the video frame image according to the set of pixel regions.

[0089] For example, the output of a deep learning model is digitized information of landmarks and a set of two-dimensional coordinate points representing the overlay locations. The model predicts the normalized coordinates of landmarks in the image through fully connected layers. and confidence level ,Right now:

[0090] in, Represents normalized coordinates; Indicates the normalized width; Indicates the normalized height; This represents the feature vector of the i-th landmark; represents a regressive multilayer perceptron; sigmoid represents a sigmoid activation function; t represents a timestamp.

[0091] And perform confidence prediction, that is:

[0092] in, This represents the confidence level of the i-th landmark; This represents the feature vector of the i-th landmark; represents a classification multilayer perceptron; Softmax represents a normalized exponential function.

[0093] The superimposed positions of landmarks in a video frame image are represented as a set of pixel regions. ,Right now:

[0094] in, Represents the set of pixel regions at time t; Represents pixel coordinates; The confidence threshold. Image resolution; This represents the confidence level of the i-th landmark.

[0095] Based on the pixel coordinates or pixel regions output by the deep learning model, the landmark name can be rendered to the corresponding pixel position in the form of text, icon or outline highlight, and the temporal consistency of the superposition result of consecutive frames can be updated according to the timestamp of the video frame image to output a stable superimposed video stream.

[0096] By applying the technical solutions of the above embodiments, the UAV video geographic information overlay method based on deep learning described in the above embodiments can acquire video streams from the UAV's onboard camera in real time, and simultaneously acquire GNSS positioning data and flight status data at the corresponding time. Then, based on the positioning data and flight status data, the UAV's geographic location and field of view information are determined, and vector map data matching the current field of view is requested from the cloud server via the communication network. The video frame image and the vector map data are then input into a deep learning model. Visual features and map features are extracted by a visual encoder and a map encoder, respectively, and a cross-modal feature fusion mechanism is used to establish the correlation between the two, thereby directly inferring the spatial correspondence of geographic information in the image. Based on the inference result, geographic information such as landmark names, road information, or feature outlines is rendered and overlaid onto the corresponding pixel positions of the video frame image, generating the overlaid output video frame or video stream.

[0097] Therefore, the method can achieve pixel-level precise alignment of geographic information in video without relying on high-precision real-time kinematic (RTK) and attitude data, by utilizing the cross-modal fusion capability of visual semantics and map priors. This reduces the deviation of video geographic information overlay results and solves the problems of inaccurate overlay and poor robustness in weak GNSS environments.

[0098] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide a method for overlaying UAV video geographic information based on deep learning. The difference between this method and the above embodiments is that, when overlaying geographic information onto video frame images, temporal smoothing filtering can be performed between multiple consecutive frames, such as... Figure 4 As shown, the method includes: S201. Based on the set of pixel regions, obtain the coordinates of the overlay position of consecutive frames; S202. The Kalman filter algorithm is used to smoothly update the position coordinates of the superimposed consecutive frames in order to generate the optimal estimated position. S203. Overlay the geographic information onto the optimal estimated location in the video frame image to generate an overlaid frame image.

[0099] To ensure temporal stability, when overlaying geographic information onto video frame images, the coordinates of the overlay positions for consecutive frames can first be obtained based on a set of pixel regions. Then, a Kalman filter algorithm is used to smoothly update the coordinates of the overlay positions for consecutive frames to generate the optimal estimated position. The geographic information is then overlaid onto the optimal estimated position in the video frame image to generate the overlay frame image.

[0100] For example, the rendering overlay module of the UAV onboard computing unit can also include smoothing filtering logic. The smoothing filtering logic can use time series information to update the temporal consistency of the overlay results of consecutive frames, eliminate visual jitter, and finally output a stable overlay video stream.

[0101] When using Kalman filtering to smoothly update the coordinates of consecutive frames, the coordinates of the superimposed positions of consecutive frames can be obtained first based on the set of pixel regions. and Then define the state vector, that is:

[0102] in, Represents the state vector; Indicates the overlay position coordinates of the current frame; It represents the superposition coordinates of adjacent frames (previous frame); T represents the transpose symbol.

[0103]

[0104] in, The optimal estimated position after smoothing; This is the optimal estimated position after smoothing from the previous frame; Predict coordinates for the current frame model; Indicates Kalman gain; Let be the observation matrix. Based on the above formula, a stable overlay video stream can be output.

[0105] By applying the technical solutions of the above embodiments, the deep learning-based UAV video geographic information overlay method described in the above embodiments can render landmark names as text, icons, or highlighted outlines to the corresponding positions of video frames based on the inferred pixel coordinates or regions. Furthermore, during the overlay process, smoothing filtering logic is used to update the temporal consistency of the overlay results of consecutive frames using time-series information, eliminating visual jitter and ultimately outputting a stable overlay video stream.

[0106] In some embodiments, as a specific implementation of the deep learning-based UAV video geographic information overlay method described in the above embodiments, some embodiments of this application also provide a deep learning-based UAV video geographic information overlay device, such as... Figure 5 As shown, the device includes: The data acquisition module is used to acquire UAV detection data, which includes video streams collected by the UAV in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream. The field of view detection module is used to construct a current field of view model of the UAV based on the positioning data and the flight status data; the current field of view model is described by at least one parameter among the UAV heading angle, gimbal attitude, camera intrinsic parameters, view frustum range, and projection polygon. The map data module is used to retrieve vector map data from the map database based on the current field of view of the field of view model; the vector map data includes at least one geographic element among road vectors, landmark locations, points of interest, feature outlines, and attribute information; The spatial detection module is used to obtain the spatial correspondence information of the geographic elements in the video frame image by using a spatial detection model; the spatial detection model is a deep learning model that performs cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and establishes the association between the visual content in the video frame image and the geographic elements based on implicit reasoning. The image overlay module is used to overlay the geographic information corresponding to the geographic element onto the video frame image according to the spatial correspondence information, so as to generate an overlay frame image.

[0107] By applying the technical solutions of the above embodiments, the deep learning-based UAV video geographic information overlay device described in the above embodiments, after acquiring UAV detection data, can construct a current field-of-view model based on the positioning data and flight status data in the UAV detection data, and retrieve vector map data from the map database based on the field-of-view model. Then, a deep learning-based spatial detection model is used to perform cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and spatial correspondence information between visual content and geographic elements is established based on implicit reasoning. Then, according to the spatial correspondence information, the geographic information is overlaid onto the video frame image to generate an overlaid frame image. The device can utilize the cross-modal fusion capability of visual semantics and map priors to achieve pixel-level precise alignment of geographic information in the video frame image, reducing the result deviation of video geographic information overlay.

[0108] It should be noted that other corresponding descriptions of the functional units involved in the deep learning-based UAV video geographic information overlay device provided in the embodiments of this application can be found in the corresponding descriptions in the deep learning-based UAV video geographic information overlay method provided in the above embodiments, and will not be repeated here.

[0109] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0110] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0111] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0112] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0114] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0115] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0116] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0117] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely examples of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A method for overlaying UAV video geographic information based on deep learning, characterized in that, The method includes: Acquire drone detection data, which includes video streams collected by the drone in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream; A current field-of-view model of the UAV is constructed based on the positioning data and the flight status data; the current field-of-view model is described by at least one parameter among the UAV heading angle, gimbal attitude, camera intrinsic parameters, view frustum range, and projection polygon. Based on the current field of view of the field of view model, vector map data is retrieved from the map database; the vector map data includes at least one geographic element among road vectors, landmark locations, points of interest, feature outlines, and attribute information; The spatial detection model is used to infer the spatial correspondence information of the geographic elements in the video frame image; the spatial detection model is a deep learning model that performs cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and establishes the association relationship between the visual content in the video frame image and the geographic elements based on implicit inference. According to the spatial correspondence information, the geographic information corresponding to the geographic element is superimposed onto the video frame image to generate a superimposed frame image.

2. The method according to claim 1, characterized in that, Acquire drone detection data, including: Receive video streams captured by cameras mounted on drones; The video frame image is extracted from the video stream according to a preset sampling frequency. The preset sampling frequency is set according to influencing factors, including at least one of system load, system configuration information, result output accuracy, and feedback information. Extract the timestamps of the video frame images; The system receives a data stream detected by the pose sensor mounted on the drone, and extracts the positioning data and flight status data from the data stream according to the timestamp.

3. The method according to claim 1, characterized in that, Constructing a current field-of-view model of the UAV based on the positioning data and the flight status data includes: The UAV heading angle and the gimbal attitude are extracted from the flight status data, and the gimbal attitude includes pitch angle and roll angle. Calculate the rotation matrix based on the heading angle, the pitch angle, and the roll angle; Obtain the intrinsic parameters of the camera and the transformation parameters between the world coordinate system and the camera coordinate system; A projection matrix is ​​constructed by combining the camera intrinsic parameters and the transformation parameters; The view frustum and projection polygon of the UAV camera are calculated based on the projection matrix. The current geographic location of the UAV is extracted from the positioning data, and field of view information is generated based on the rotation matrix, the view frustum range, and the projection polygon; A current field-of-view model is constructed based on the geographical location and the field-of-view information.

4. The method according to claim 1, characterized in that, Based on the current field of view of the aforementioned field of view model, vector map data is retrieved from the map database, including: Generate a map request based on the field-of-view model; The map request is sent to the cloud server, so that the cloud server responds to the map request by parsing the projected polygon from the map request; and delineates the data retrieval range based on the projected polygon, and retrieves vector map data from the map database based on the data retrieval range; Receive the vector map data fed back from the cloud server.

5. The method according to claim 1, characterized in that, Based on the current field of view of the aforementioned field of view model, vector map data is retrieved from the map database, including: Obtain the raw map data; Define graph structure information, which includes a node set and an edge set; the node set is used to represent the landmark locations and / or the points of interest; the edge set is used to represent the road vectors; Construct a preprocessing transformation function based on the graph structure information; The preprocessing transformation function is used to transform the original map data into vector map data that can be read by the spatial detection model.

6. The method according to claim 1, characterized in that, The spatial detection model includes a visual encoder, a map encoder, and a feature fusion inference module. Using a spatial detection model to infer the spatial correspondence information of the geographic features in the video frame image, including: The video frame images and the vector map data are input into the spatial detection model; The visual encoder is used to extract visual semantic features from the video frame images; The map encoder is used to extract map structure features from the vector map data; The feature fusion inference module performs cross-modal fusion of the visual semantic features and the map structural features, and generates the geographic information. Based on deep learning algorithms, the association between the visual semantic features and the map structural features is implicitly inferred; the deep learning algorithm includes at least one of cross-modal attention mechanism, feature matching algorithm and contrastive learning algorithm; The overlay position of the geographic information in the video frame image is calculated based on the association relationship; the overlay position includes at least one of pixel coordinates, a set of pixel regions, and an equivalent region of the set of pixel regions; The spatial correspondence information is generated by combining the geographic information and the overlay location.

7. The method according to claim 6, characterized in that, The feature fusion inference module performs cross-modal fusion of the visual semantic features and the map structural features, and generates the geographic information, including: Flatten the visual semantic features into a visual sequence; Obtain the map node feature set and learnable weight matrix; An attention triplet is constructed based on the visual sequence, the map node feature set, and the learnable weight matrix. The attention triplet includes a query matrix, a key matrix, and a value matrix. Based on the attention triplet, the output data of multiple attention heads are calculated using an activation function; The output data of multiple attention heads are concatenated and linearly projected to generate a fused feature representation; The geographic information corresponding to the fused feature representation is calculated using a neural network.

8. The method according to claim 1, characterized in that, According to the spatial correspondence information, the geographic information corresponding to the geographic features is superimposed onto the video frame image to generate a superimposed frame image, including: The overlay location and confidence level are extracted from the spatial correspondence information. The overlay location includes the normalized coordinates of the geographic features in the video frame image predicted by the fully connected layer of the spatial detection model. The confidence level is the prediction confidence of the fully connected layer of the spatial detection model for the normalized coordinates. Obtain the preset confidence threshold and the image resolution of the video frame; A set of pixel regions is generated based on the confidence threshold and the image resolution. The set of pixel regions includes superimposed pixels. The superimposed pixels are pixels formed by multiplying the normalized coordinates of the pixels with a confidence level higher than the confidence threshold with the image resolution. The geographic information is superimposed onto the video frame image according to the set of pixel regions.

9. The method according to claim 8, characterized in that, According to the set of pixel regions, the geographic information is superimposed onto the video frame image, including: Based on the set of pixel regions, obtain the coordinates of the overlay positions of consecutive frames; The Kalman filter algorithm is used to smoothly update the superimposed position coordinates of the consecutive frames in order to generate the optimal estimated position; The geographic information is overlaid onto the optimal estimated location in the video frame image to generate the overlaid frame image.

10. A deep learning-based UAV video geographic information overlay device, characterized in that, The device includes: The data acquisition module is used to acquire UAV detection data, which includes video streams collected by the UAV in real time, as well as positioning data and flight status data corresponding to the timestamps of video frame images in the video stream. The field of view detection module is used to construct a current field of view model of the UAV based on the positioning data and the flight status data; the current field of view model is described by at least one parameter among the UAV heading angle, gimbal attitude, camera intrinsic parameters, view frustum range, and projection polygon. The map data module is used to retrieve vector map data from the map database based on the current field of view of the field of view model; the vector map data includes at least one geographic element among road vectors, landmark locations, points of interest, feature outlines, and attribute information; The spatial detection module is used to obtain the spatial correspondence information of the geographic elements in the video frame image by using a spatial detection model; the spatial detection model is a deep learning model that performs cross-modal fusion of the visual semantic features of the video frame image and the map structure features of the vector map data, and establishes the association between the visual content in the video frame image and the geographic elements based on implicit reasoning. The image overlay module is used to overlay the geographic information corresponding to the geographic element onto the video frame image according to the spatial correspondence information, so as to generate an overlay frame image.

Images