Unmanned aerial vehicle aerial video playback management system based on space-time fusion

Abstract

The invention provides an unmanned aerial vehicle aerial video playback management system based on space-time fusion, and the system carries out the space-time fusion of the flight parameters and holder parameters of an unmanned aerial vehicle and the information of an aerial video, and carries out the query and playback of the aerial video according to the time and geographic position. The system comprises a data access module, an analysis processing module, a storage management module and a query playback module. According to the method, space-time fusion processing is carried out on the flight parameters, the holder parameters and the video data of the unmanned aerial vehicle, and query and playback of historical videos according to time and geographic positions are supported. Compared with a traditional management system, the system can make full use of aircraft and load parameter information, analyzes a video aerial photography area, and realizes video query and retrieval based on a time axis and a spatial range through association fusion of time and spatial domains.

Description

Technical Field

[0001] The invention relates to a video playback management system, in particular to a drone aerial video playback management system based on time-space fusion. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Unmanned Aerial Vehicle (UAV) can be used for site monitoring, weather detection, highway inspection, surveying and mapping, flood monitoring, aerial photography, traffic management, power line query, forest fire prevention and rescue, etc. As one of the necessary sensors for almost all drones today, visible light sensors can collect and process on-site video information in real time, providing the most basic and important information for flight control, on-site command, and site monitoring.

[0004] With the widespread use of drone aerial videos, how to effectively manage and query and retrieve massive drone aerial videos has become an urgent problem to be solved. Traditional video management systems use video files for storage. According to the video source and the set video length, drone aerial videos are saved as several independent video files in the file system, and the information such as the drone, start time and end time associated with each video file is recorded in the database. When querying historical videos, users can only obtain video files based on the drone name and shooting time, and perform playback analysis, which seriously restricts the query and utilization efficiency of aerial videos.

[0005] While obtaining real-time video, the drone ground station can also obtain the drone flight parameters, including latitude and longitude, altitude, azimuth, pitch angle, roll angle, etc., as well as the status parameters of the visible light sensor, including azimuth, pitch angle, focal length, etc. In the existing video management system, the above information has not been fully utilized.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide a UAV aerial video playback management system based on time-space fusion in view of the shortcomings of the prior art.

[0008] In order to solve the above technical problems, the present invention discloses a UAV aerial video playback management system based on time-space fusion, which performs time-space fusion on the flight parameters, gimbal parameters and information of the aerial video of the UAV, and queries and plays back the aerial video according to time and geographic location.

[0009] Furthermore, the system comprises:

[0010] Data access module, analysis and processing module, storage management module and query playback module, among which,

[0011] The data access module is used to receive data sent by the UAV ground station and forward it to the analysis and processing module. The data includes the flight route data, flight parameters, gimbal parameters and real-time aerial video of the UAV;

[0012] The analysis and processing module is used to analyze and process the data sent by the data access module, perform real-time processing on the flight parameters and gimbal parameters, and generate an image area and a video screenshot according to a preset time interval; wherein the image area is data containing coordinate information of the location of the aerial video, consisting of longitude, latitude and altitude, corresponding to the video screenshot;

[0013] The storage management module stores the generated image areas and video screenshots in a database and a file system;

[0014] The query and playback module provides a visual interactive interface for querying and playback management of single or multiple drone aerial videos based on time and space.

[0015] Furthermore, the data access module receives data sent by the UAV ground station, specifically including:

[0016] Video data reception: the data access module receives the TS video stream sent by the UAV ground station through the UDP protocol, and sends the TS video stream to the analysis and processing module and the storage management module;

[0017] UAV and payload parameter data reception: the data access module receives the UAV and payload parameter information sent by the UAV ground station through the UDP protocol, and sends the above parameter information to the analysis and processing module and the storage management module.

[0018] Furthermore, the drone and payload parameter information specifically includes:

[0019] Timestamp;

[0020] Aircraft longitude, aircraft latitude, aircraft altitude, aircraft azimuth, aircraft pitch angle and aircraft roll angle;

[0021] The relative azimuth angle of the gimbal and the relative pitch angle of the gimbal;

[0022] and sensor focal length, sensor horizontal field of view angle, and sensor vertical field of view angle.

[0023] Furthermore, the analysis and processing module analyzes and processes the data sent by the data access module, specifically including:

[0024] Loading elevation data: loading elevation data of an area corresponding to the route according to the flight route data of the UAV, wherein the elevation data is pre-stored locally;

[0025] Video frame acquisition: taking screenshots of the acquired TS video stream according to a preset time interval to obtain video frames, and sending the video frames to the storage management module;

[0026] Video frame positioning parameter acquisition, matching the video frame with the drone and payload parameter information, obtaining the positioning parameters, and sending them to the storage management module;

[0027] The video frame positioning calculation calculates the longitude and latitude height position information corresponding to the four vertices of the video frame according to the positioning parameters and elevation data, obtains the positioning data, and sends it to the storage management module.

[0028] Furthermore, the positioning parameters specifically include:

[0029] Parameters include ID number, timestamp, aircraft number, mission number, aircraft longitude, aircraft latitude, aircraft altitude, aircraft azimuth, aircraft pitch angle, aircraft roll angle, gimbal relative azimuth, gimbal relative pitch angle, sensor focal length, sensor horizontal field of view angle, and sensor vertical field of view angle.

[0030] Furthermore, the latitude, longitude and altitude position information corresponding to the four vertices of the video frame is calculated by calculating the absolute pitch angle and absolute azimuth angle of the center and four corners of the video frame, and using elevation data to solve the longitude, latitude and altitude information of the intersection.

[0031] Furthermore, the steps of calculating the longitude and latitude position information corresponding to the four vertices of the video frame are as follows:

[0032] Step 1: According to the resolution W×H of the video frame, the pixel coordinates (x, y) in the video frame are defined. The coordinates of the four vertices corresponding to the video frame are expressed as B1=(1, 1), B2=(W, 1), B3=(W, H), and B4=(1, H). The distance L from the CCD panel to the lens of the camera shooting the video is calculated as follows:

[0033]

[0034] Among them, ψ h is the lateral field of view of the camera, ψ v is the longitudinal field of view angle of the camera;

[0035] Step 2: define the three-dimensional coordinates of the center of the lens as ω c =(W / 2,H / 2,L), the coordinate of the upper left corner of the CCD panel is D tl =(0,0,0), the coordinate of the lower right corner is D br =(W,H,0), the center coordinate is D c =(W / 2,H / 2,0), the three-dimensional coordinates of the pixel in the video frame are P c =(x,y,0);

[0036] Step 3: Calculate the line of sight To target sight The rotation matrix

[0037] Step 4: According to the latitude, longitude and altitude (α lon ,α lat ,α alt ), converted to geocentric coordinates β z =(β x ,β y ,β z );

[0038] Let β0 = (β x ,β y ,β z -1000), and obtain the rotation matrix R in the geocentric coordinate system, which rotates the drone from the initial point to the corresponding latitude and longitude. α =R(<(0,0,0),(0,0,1)>,<β0,β z >), and rotate according to the longitude so that the drone is parallel to the horizontal plane in the geocentric coordinate system. The rotation matrix is ​​recorded as R lon ;

[0039] Step 5: According to the aircraft azimuth, aircraft pitch angle and aircraft roll angle (β y ,β p ,β r ), calculate the drone rotation matrix R β =f(β y ,β p ,β r );

[0040] Step 6: According to the relative azimuth angle λ of the gimbal y The relative pitch angle λ of the gimbal p , calculate the gimbal rotation matrix R λ=f(λ y ,λ p ,0);

[0041] Step 7, obtain the elevation data of the UAV flight area and convert it to the geocentric coordinate system, and record the elevation surface as S;

[0042] Step 8: Define the straight line in the geocentric coordinate system Calculate the straight line after rotation according to the drone rotation matrix and the gimbal rotation matrix

[0043] Step 9, calculate the straight line The intersection point P with the elevation surface S s =(P x ,P y ,P z ) and convert it into latitude and longitude P α =(P lon ,P lat ,P alt );

[0044] Step 10, using the methods of steps 1 to 9, calculate the longitude and latitude corresponding to the four vertices B1, B2, B3, and B4 of the video frame.

[0045] Furthermore, the storage management module specifically includes:

[0046] The original video and parameter data are stored, the TS video stream and the UAV and payload parameter information sent by the data access module are received, written into the file system, the video file and the parameter file are obtained, and the URL address of the video file, the URL address of the parameter file and the video start time are recorded in the database; when the video file exceeds the preset limit, the current video file storage is terminated, the video end time is recorded in the database, a new video file and parameter file are created and the recording is restarted;

[0047] Key video frames and positioning data storage, receive the video frames, positioning parameters and positioning data generated by the analysis and processing module, save them to the file system, and save the timestamp of the video frame, the URL address of the video frame, the positioning parameters and positioning data to the database.

[0048] Furthermore, the query playback module specifically includes:

[0049] Time query and review: taking the aerial photography task as the unit, multiple video files corresponding to the single task are combined and reviewed seamlessly on the timeline;

[0050] Spatial query and review, display the aerial photography mission trajectory and the corresponding aerial photography area, and jump the video on the spatial axis; specify the area on the map, query and review the related videos, and jump to the corresponding time to play.

[0051] Beneficial effects:

[0052] The present invention supports querying and replaying historical videos based on time and geographic location by performing spatiotemporal fusion processing on the flight parameters, PTZ parameters and video data of unmanned aerial vehicles. Compared with the traditional management system, the present system can make full use of the aircraft and load parameter information, analyze the video aerial photography area, and realize video query and access based on the time axis and spatial range by associating and fusion of the time and space domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0054] Figure 1 It is a schematic diagram of the system composition.

[0055] Figure 2 It is a schematic diagram of elevation data.

[0056] Figure 3 This is a schematic diagram of the elevation data preloading strategy.

[0057] Figure 4 This is a schematic diagram of the sensor's shooting range.

[0058] Figure 5 This is a diagram showing an example of the sensor's shooting range calculation results.

[0059] Figure 6 This is a schematic diagram of the aerial photography mission video review.

[0060] Figure 7 It is a schematic diagram of video review based on the timeline.

[0061] Figure 8 This is a schematic diagram of video retrieval based on spatial query. DETAILED DESCRIPTION

[0062] The principle of the present invention is that by combining the elevation data of the drone aerial photography area with the drone and sensor parameter data, the video sent back by the drone in real time can be positioned, and the shooting area, target position and status of the sensor can be more accurately grasped. The idea of ​​the present invention is to directly calculate the platform status information of the drone and the status information of the onboard sensors and other equipment, combined with the elevation data information of the flight area, to obtain the geographical location information of the aerial video, and to uniformly manage and store the video data and geographical location data, so as to support users to quickly and conveniently query and perform statistical analysis on single-flight and multi-flight aerial videos based on the two dimensions of time and space.

[0063] The present invention solves the problem of efficient query and review of drone aerial videos. By performing spatiotemporal fusion processing on drone flight parameters, gimbal parameters and video data, the present invention supports query and playback of historical videos according to time and geographic location.

[0064] The technical solution is as follows: A UAV aerial video playback management system based on time-space fusion:

[0065] The system consists of a data access module, an analysis and processing module, a storage management module and a query playback module.

[0066] The data access module supports access to drone flight parameters, gimbal parameters and real-time video data through the drone ground station.

[0067] Step 1-1, video data reception, the system receives the TS video stream sent by the drone ground station through UDP, and sends it to the analysis and processing module and the storage management module;

[0068] Step 1-2, aircraft and payload parameter data reception, the system receives parameter information sent by the UAV ground station through UDP, including the working status of the UAV platform, gimbal, visible light sensor and other equipment, and sends it to the analysis and processing module and the storage management module.

[0069] The analysis and processing module processes the received flight parameters and gimbal parameters in real time in combination with the elevation data, and generates image areas and video screenshots according to preset time intervals;

[0070] Step 2-1, loading elevation data: according to the flight route data, load the elevation data of the corresponding area from the local disk;

[0071] Step 2-2, video frame acquisition, obtain real-time video frame screenshots according to the time interval configured by the system, and send them to the storage management module;

[0072] Step 2-3, video frame positioning parameter acquisition, obtain the aircraft and load parameter data corresponding to the current video frame, and send it to the storage management module;

[0073] Step 2-4, video frame positioning calculation, calculate the longitude and latitude high position information corresponding to the four vertices of the video frame according to the positioning parameters and elevation data, and send it to the storage management module;

[0074] The storage management module stores the generated image area and video screenshots and related parameters in a database and a file system;

[0075] Step 3-1, original video and parameter data storage, the storage management module receives the video data and parameter data sent by the data access module, writes them into the file system, and records the video file URL, parameter file URL, and video start time in the database. When the video file exceeds the time limit, the current file storage is terminated, the video end time is recorded in the database, a new video file and parameter file are created, and recording is restarted;

[0076] Step 3-2, key video frames and positioning data storage, the storage management module receives the video frames, parameter data and positioning data generated by the analysis and processing module, saves them to the file system, and saves the timestamp, image URL, positioning parameter data, and four-vertex geographic coordinates to the database;

[0077] The query and playback module provides a visual interactive interface, supporting query and playback management of single-flight and multi-flight video aerial photography data based on time and space. The module supports time query and space query:

[0078] Time query and review, taking aerial photography tasks as units, shielding the implementation details of a single task corresponding to multiple background video files, supporting users to jump seamlessly on the timeline;

[0079] Spatial query and review supports the display of aerial photography mission trajectories and corresponding aerial photography areas, and supports users to jump videos on the spatial axis; supports selecting areas on the map, querying and obtaining historical related videos, and automatically jumping to the corresponding time for playback.

[0080] Example:

[0081] This embodiment proposes a drone aerial video playback management system based on time-space fusion. The system performs time-space fusion processing on drone flight parameters, gimbal parameters, and video data, and supports querying and playback of historical videos based on time and geographic location. Figure 1 As shown, the system consists of a data access module, an analysis and processing module, a storage management module and a query playback module.

[0082] The data access module supports access to drone flight parameters, gimbal parameters and real-time video data through the drone ground station.

[0083] Step 1-1, video data reception, the system receives the TS video stream sent by the drone ground station through UDP, and sends it to the analysis and processing module and the storage management module;

[0084] Step 1-2, receiving the parameter data of the drone and the payload, the system receives the parameter information sent by the drone ground station through UDP, including the working status of the drone platform, gimbal, visible light sensor and other equipment, and sends it to the analysis and processing module and the storage management module. The specific list of parameter data information is shown in Table 1.

[0085] Table 1 Parameter information list

[0086]

[0087]

[0088] The analysis and processing module processes the received flight parameters and gimbal parameters in real time in combination with the elevation data, and generates image areas and video screenshots according to preset time intervals;

[0089] Step 2-1, loading elevation data, according to the flight route data, load the elevation data of the corresponding area from the local disk. This embodiment uses SRTM elevation data, which covers an area between 60 degrees north latitude and 56 degrees south latitude, with an area of ​​more than 119 million square kilometers, covering more than 80% of the global land surface. Figure 2 The original data exceeds 90G. In order to quickly obtain the elevation information of the specified longitude and latitude in real time during the solution process, we designed a dynamic loading mechanism for elevation data. By slicing the DEM data, loading the DEM slice corresponding to the current position of the drone, and estimating the flight direction and speed, dynamically pre-loading the DEM data of the next slice, up to 4 slices of DEM data can be loaded at the same time, as shown in the figure. Figure 3 As shown, the historical shard data is unloaded, thereby utilizing the limited memory space to ensure a 100% hit rate of the elevation information at the specified longitude and latitude at a limited flight speed;

[0090] Step 2-2, video frame acquisition, obtain real-time video frame screenshots according to the time interval configured by the system, and send them to the storage management module;

[0091] Step 2-3, video frame positioning parameter acquisition, obtain the aircraft and load parameter data corresponding to the current video frame, and send it to the storage management module. The video frame positioning parameters are shown in Table 2;

[0092] Table 2 Video frame positioning parameter description

[0093]

[0094] Step 2-4, video frame positioning calculation, according to the positioning parameters and elevation data, calculate the latitude and longitude high position information corresponding to the four vertices of the video frame, and send it to the storage management module. This technology calculates the absolute pitch angle and absolute azimuth angle of the video center and the four corners, and uses the cached elevation data to solve the intersection point (such as Figure 4 The specific steps are as follows:

[0095] Step 2-4-1, according to the video resolution W×H, define the pixel coordinates (x, y) in the video frame. The coordinates of the four vertices corresponding to the video frame are expressed as B1=(1,1), B2=(W,1), B3=(W,H), B4=(1,H), combined with the camera's horizontal field of view angle ψ h and the longitudinal field angle ψ v , calculate the distance from the CCD panel to the lens

[0096] Step 2-4-2, define the three-dimensional coordinates of the lens center as ω c =(W / 2,H / 2,L), the coordinate of the upper left corner of CCD is D tl =(0,0,0), the coordinate of the lower right corner of CCD is D br =(W,H,0), the center coordinate of CCD is D c =(W / 2,H / 2,0), the three-dimensional coordinates of the video pixel target are P c =(x,y,0);

[0097] Step 2-4-3, calculate the visual axis To target sight The rotation matrix

[0098] Step 2-4-4, according to the latitude, longitude and altitude of the drone (α lon ,α lat ,α alt ), converted to geocentric coordinates β z =(β x ,β y ,β z ), and calculate β0=(β x ,β y ,β z -1000), thereby obtaining the rotation matrix R in the geocentric coordinate system to rotate the drone from the initial point to the corresponding latitude and longitude α =R(<(0,0,0),(0,0,1)>,<β0,β z >), and rotate according to the longitude so that the drone is parallel to the horizontal plane in the geocentric coordinate system. The rotation matrix is ​​recorded as R lon ;

[0099] Step 2-4-5, according to the azimuth, pitch and roll angle gimbal parameters (β y ,β p ,β r ), calculate the rotation matrix R β =f(β y ,β p ,β r );

[0100] Step 2-4-6, according to the gimbal parameters, including azimuth angle λ y and the pitch angle λ p , calculate the rotation matrix R λ =f(λ y ,λ p ,0);

[0101] Step 2-4-7, obtain the elevation data of the UAV flight area and convert it to the geocentric coordinate system, and record the elevation surface as S;

[0102] Step 2-4-8, define the straight line in the geocentric coordinate system Calculate the straight line after rotation according to the parameters of the drone and gimbal

[0103] Step 2-4-9, calculate the straight line The intersection point P with the elevation surface S s =(P x ,P y ,P z ), converted to latitude and longitude P α =(P lon ,P lat ,P alt );

[0104] Step 2-4-10, calculate the latitude and longitude corresponding to the four vertices B1, B2, B3, and B4 of the video frame. The solution result of the quadrilateral area formed by the video frame and its four vertices is as follows: Figure 5 As shown, the solution results are sent to the storage management module, and the result content is shown in Table 3.

[0105] Table 3. Video frame four-vertex solution results description

[0106]

[0107] The storage management module stores the generated image area and video screenshots and related parameters in a database and a file system;

[0108] Step 3-1, original video and parameter data storage, the storage management module receives the video data and parameter data sent by the data access module, writes them into the file system, and records the video file URL, parameter file URL, and video start time in the database. When the video file exceeds the time limit, the current file storage is terminated, and the video end time is recorded in the database. A new video file and parameter file are created and the recording is restarted, as shown in Table 4.

[0109] Table 4 Description of original video and parameter data storage

[0110]

[0111]

[0112] Step 3-2, key video frames and positioning data storage, the storage management module receives the video frames, parameter data and positioning data generated by the analysis and processing module, saves them to the file system, and saves the timestamp, image URL, positioning parameter data, and four-vertex geographic coordinates to the database. The specific storage content is shown in Table 5;

[0113] Table 5 Description of key video frames and positioning data storage

[0114]

[0115]

[0116] The query and playback module provides a visual interactive interface, supporting query and playback management of single-flight and multi-flight video aerial photography data based on time and space. The module supports time query and space query:

[0117] Time query and access, taking the aerial photography task as the unit, shields the implementation details of the background multiple video files corresponding to a single task, and supports users to jump seamlessly on the timeline. The aerial photography task data table is shown in Table 6. The system obtains the aerial photography task number according to the aerial photography task table, and uses the aerial photography task number to query the associated video file and parameter file in the video record database table, and uses the relevant files and data tables to draw the time query and access human-computer interaction interface, as shown in Figure 6 As shown in the figure, it specifically includes ① reading the video record database table, displaying the associated video list in the interface list, obtaining the start and end time of each video segment, and drawing the timeline on the interface; ② reading the task associated parameter file, generating the aircraft trajectory, and plotting the aircraft flight path on the GIS map; ③ reading the video frame positioning database table, and plotting the sensor detection range on the GIS base map; ④ dragging the timeline, the system opens the corresponding video, jumps to the corresponding time to play the video, and simultaneously highlights the drone position and sensor detection position on the GIS map, as shown in the figure. Figure 7 As shown in the figure, the current playing file is FILE-4, and the playing time is 13:55:32. By dragging the timeline, jump to playing FILE-3, and the playing time is 13:05:12. It can be seen that on the corresponding GIS map, the corresponding trajectory segment of the current playing file changes, and the current UAV position and payload detection position on the map also change accordingly.

[0118] Table 6 Description of data storage for aerial photography tasks

[0119]

[0120] Spatial query and review, support display of aerial photography mission trajectory and corresponding aerial photography area, support users to jump videos on the spatial axis, such as Figure 6 As shown in the figure, users can directly click on any point of the aircraft trajectory on the GIS map, and the system will automatically select the corresponding video file and adjust it to the corresponding time point for playback; at the same time, the system supports selecting an area on the map, querying and obtaining historical related videos, and automatically jumping to the corresponding time for playback. Figure 8 As shown in the figure, ① select the search area on the GIS map, ② the system obtains the latitude and longitude quadrilateral corresponding to each key frame in the video frame positioning database table, and determines whether it intersects with the latitude and longitude quadrilateral of the search area, ③ the system sorts the aerial photography tasks according to the number of intersecting key frames, ④ in the regional search results, the aerial photography task list is displayed in reverse order according to the sorting results, ⑤ the user clicks on the task list, the system loads the aerial photography task, and displays the aircraft trajectory and payload aerial photography area on the GIS map, and jumps to the first intersecting key frame time point to play the video.

[0121] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of a drone aerial video playback management system based on time-space fusion provided by the present invention and some or all of the steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0122] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0123] The present invention provides a concept and method for a drone aerial video playback management system based on time-space fusion. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A UAV aerial video playback management system based on time-space fusion, characterized in that: The system performs spatiotemporal fusion of the flight parameters, gimbal parameters and information of the aerial video of the UAV, and queries and replays the aerial video according to time and geographic location.

2. The UAV aerial video playback management system based on time-space fusion according to claim 1 is characterized in that: The system comprises: Data access module, analysis and processing module, storage management module and query playback module, among which, The data access module is used to receive data sent by the UAV ground station and forward it to the analysis and processing module. The data includes the flight route data, flight parameters, gimbal parameters and real-time aerial video of the UAV; The analysis and processing module is used to analyze and process the data sent by the data access module, perform real-time processing on the flight parameters and gimbal parameters, and generate an image area and a video screenshot according to a preset time interval; wherein the image area is data containing coordinate information of the location of the aerial video, consisting of longitude, latitude and altitude, corresponding to the video screenshot; The storage management module stores the generated image areas and video screenshots in a database and a file system; The query and playback module provides a visual interactive interface for querying and playback management of single or multiple drone aerial videos based on time and space.

3. The UAV aerial video playback management system based on time-space fusion according to claim 2 is characterized in that: The data access module receives data sent by the UAV ground station, specifically including: Video data reception: the data access module receives the TS video stream sent by the UAV ground station through the UDP protocol, and sends the TS video stream to the analysis and processing module and the storage management module; UAV and payload parameter data reception: the data access module receives the UAV and payload parameter information sent by the UAV ground station through the UDP protocol, and sends the above parameter information to the analysis and processing module and the storage management module.

4. The UAV aerial video playback management system based on time-space fusion according to claim 3 is characterized in that: The UAV and payload parameter information specifically includes: Timestamp; Aircraft longitude, aircraft latitude, aircraft altitude, aircraft azimuth, aircraft pitch angle and aircraft roll angle; The relative azimuth angle of the gimbal and the relative pitch angle of the gimbal; and sensor focal length, sensor horizontal field of view angle, and sensor vertical field of view angle.

5. The UAV aerial video playback management system based on time-space fusion according to claim 4 is characterized in that: The analysis and processing module analyzes and processes the data sent by the data access module, specifically including: Elevation data loading, according to the flight route data of the UAV, loading the elevation data of the area corresponding to the route, the elevation data is pre-stored locally; Video frame acquisition: taking screenshots of the acquired TS video stream according to a preset time interval to obtain video frames, and sending the video frames to the storage management module; Video frame positioning parameter acquisition, matching the video frame with the drone and payload parameter information, obtaining the positioning parameters, and sending them to the storage management module; The video frame positioning calculation calculates the longitude and latitude height position information corresponding to the four vertices of the video frame according to the positioning parameters and elevation data, obtains the positioning data, and sends it to the storage management module.

6. The UAV aerial video playback management system based on time-space fusion according to claim 5 is characterized in that: The positioning parameters specifically include: Parameters include ID number, timestamp, aircraft number, mission number, aircraft longitude, aircraft latitude, aircraft altitude, aircraft azimuth, aircraft pitch angle, aircraft roll angle, gimbal relative azimuth, gimbal relative pitch angle, sensor focal length, sensor horizontal field of view angle, and sensor vertical field of view angle.

7. The UAV aerial video playback management system based on time-space fusion according to claim 6 is characterized in that: The calculation of the latitude, longitude and altitude position information corresponding to the four vertices of the video frame is performed by calculating the absolute pitch angle and absolute azimuth angle of the center and four corners of the video frame, and using elevation data to solve the longitude, longitude and altitude information of the intersection.

8. The UAV aerial video playback management system based on time-space fusion according to claim 7 is characterized in that: The specific steps of calculating the longitude and latitude position information corresponding to the four vertices of the video frame are as follows: Step 1: According to the resolution W×H of the video frame, the pixel coordinates (x, y) in the video frame are defined. The coordinates of the four vertices corresponding to the video frame are expressed as B1=(1, 1), B2=(W, 1), B3=(W, H), and B4=(1, H). The distance L from the CCD panel to the lens of the camera shooting the video is calculated as follows: Among them, ψ h is the lateral field of view of the camera, ψ v is the longitudinal field of view angle of the camera; Step 2: define the three-dimensional coordinates of the center of the lens as ω c =(W / 2, H / 2, L), the coordinate of the upper left corner of the CCD panel is D tl =(0,0,0), the coordinate of the lower right corner is D br =(W, H, 0), the center coordinate is D c =(W / 2, H / 2, 0), the three-dimensional coordinates of the pixel in the video frame are P c =(x, y, 0); Step 3: Calculate the visual axis To target sight The rotation matrix Step 4: According to the latitude, longitude and altitude (α lon , α lat , α alt ), converted to geocentric coordinates β z =(β x , β y , β z ); Let β0 = (β x , β y , β z -1000), and obtain the rotation matrix R in the geocentric coordinate system, which rotates the drone from the initial point to the corresponding latitude and longitude. α =R(<(0,0,0),(0,0,1)>,<β0,β z >), and rotate according to the longitude to make the drone parallel to the horizontal plane in the geocentric coordinate system. The rotation matrix is ​​recorded as R lon ; Step 5: According to the aircraft azimuth, aircraft pitch angle and aircraft roll angle (β y , β p , β r ), calculate the drone rotation matrix R β =f(β y , β p , β r ); Step 6: According to the relative azimuth angle λ of the gimbal y The relative pitch angle λ of the gimbal p , calculate the gimbal rotation matrix R λ =f(λ y ,λ p , 0); Step 7, obtain the elevation data of the UAV flight area and convert it to the geocentric coordinate system, and record the elevation surface as S; Step 8: Define the straight line in the geocentric coordinate system Calculate the straight line after rotation according to the drone rotation matrix and the gimbal rotation matrix Step 9, calculate the straight line The intersection point P with the elevation surface S s =(P x , P y , P z ) and convert it into latitude and longitude P α =(P lon , P lat , P alt ); Step 10, using the methods of steps 1 to 9, calculate the longitude and latitude corresponding to the four vertices B1, B2, B3, and B4 of the video frame.

9. The UAV aerial video playback management system based on time-space fusion according to claim 8 is characterized in that: The storage management module specifically includes: The original video and parameter data are stored, the TS video stream and the UAV and payload parameter information sent by the data access module are received, written into the file system, the video file and the parameter file are obtained, and the URL address of the video file, the URL address of the parameter file and the video start time are recorded in the database; when the video file exceeds the preset limit, the current video file storage is terminated, the video end time is recorded in the database, a new video file and parameter file are created and the recording is restarted; Key video frames and positioning data storage, receive the video frames, positioning parameters and positioning data generated by the analysis and processing module, save them to the file system, and save the timestamp of the video frame, the URL address of the video frame, the positioning parameters and positioning data to the database.

10. The UAV aerial video playback management system based on time-space fusion according to claim 9, characterized in that: The query playback module specifically includes: Time query and review: taking the aerial photography task as the unit, multiple video files corresponding to the single task are combined and reviewed seamlessly on the timeline; Spatial query and review, display the aerial photography mission trajectory and the corresponding aerial photography area, and jump the video on the spatial axis; specify the area on the map, query and review the related videos, and jump to the corresponding time to play.

Images