FMV full-dynamic video-based forest grass wet and crude digital twinborn application construction method

Abstract

The invention provides an FMV full-dynamic video-based forest grass wet and quarantine digital twinborn application construction method, which is characterized by comprising the following steps of S1, constructing a flight path of an unmanned aerial vehicle according to a preset forest grass wet and quarantine application scene, and obtaining video data; s2, according to the pose information collected by the unmanned aerial vehicle, conversion is carried out according to the MISB metadata specification, and a metadata code conforming to the MISB specification is obtained; s3, through video multiplexing, the metadata conforming to the M ISB specification is coded into the video, and an FMV full-dynamic video with geographic space information and image features is generated; and S4, loading the FMV full-dynamic video and the collaborative multi-source GIS element data to a two-dimensional and three-dimensional integrated digital scene of an ArcG IS Pro platform, executing related editing and analysis applications, and constructing a forest grass wet and crude digital twinborn application scene. The invention aims to solve the problems of difficulty in comparison of video images and map scenes, inaccurate positioning, incapability of bidirectionally collecting data, incapability of executing spatial analysis and the like in the prior art.

Description

Technical Field

[0001] The present invention belongs to the field of digital twins, and specifically relates to an application method for constructing forest, grassland, wetland and wasteland digital twins based on FMV full-motion video. Background Art

[0002] At present, the application of digital twins of forest, grassland, wetland and wasteland mainly focuses on the integration of digital information systems into portal platforms. In terms of visual perception, a two-dimensional digital map is used, combined with a three-dimensional terrain simulation as the base map. Through the Internet of Things, traffic checkpoint videos, high-point monitoring videos, digital infrared cameras, etc. are connected to the perception system and called for viewing as needed. In terms of data application, general GIS vector and raster data are used as the main data model, and spatial analysis is used to meet the daily work needs of forest, grassland, wetland and wasteland. However, there is currently no efficient method for combining perception systems with GIS data, and it is impossible to express the real and intuitive scene status under a unified base map.

[0003] With the rapid development of drone technology, DJI series drones are increasingly used in various fields. As a widely used aerial photography tool, they can shoot videos of business scenes such as forests, grasslands, wetlands, etc. according to preset flight paths, provide real and intuitive image data, and provide important information for decision makers. How to effectively use the video data taken by drones and establish a corresponding relationship with GIS data in a unified scene has become a technical problem that needs to be solved in building digital twins of forests, grasslands, wetlands, and wastelands. Summary of the invention

[0004] In response to the above-mentioned technical problems, the present invention provides a method for constructing a digital twin application of forest, grassland and wetland based on FMV full-motion video, combining drone video data with traditional GIS data to construct a two- and three-dimensional integrated real digital twin application scenario that can be located, edited and easily analyzed.

[0005] The present invention provides a method for constructing a digital twin application of forest, grass, wetland and wasteland based on FMV full-motion video, comprising the following steps:

[0006] S1. According to the preset forest, grass and wetland application scenarios, the flight trajectory of the drone is constructed to obtain video data;

[0007] S2, converting the posture information collected by the drone according to the metadata specification of MISB (Motion Image Standards Committee) to obtain metadata encoding that complies with the MISB specification;

[0008] S3, through video multiplexing, encode metadata that complies with MISB specifications into the video to generate FMV full-motion video with geospatial information and image features;

[0009] S4. Load the FMV full-motion video and collaborative multi-source GIS feature data into the 2D and 3D integrated digital scene of the ArcGIS Pro platform, perform related editing and analysis applications, and build a digital twin application scenario for forest, grassland, wetland and wasteland.

[0010] Furthermore, in step S1, the obtaining of video data includes the following steps:

[0011] S11, determining the scene and target to be photographed, as well as parameters such as the required video quality and resolution;

[0012] S12. Select drone and camera equipment according to shooting requirements;

[0013] S13, taking off the drone, and performing flight operations according to a predetermined flight path and shooting requirements to ensure that video data that meets the requirements is captured;

[0014] S14, transmitting the captured video data to a computer for subsequent processing.

[0015] Furthermore, in step S2, when the DJI Mavic 3 Pro drone is selected to record a video, the video subtitle option should be turned on, and the generated video subtitle SRT file should be re-encoded in accordance with the metadata specification of MISB (Motion Image Standards Committee).

[0016] Furthermore, the SRT file includes the pose information of the drone video, and the re-encoding of the metadata encoding in accordance with the MISB specification includes the following steps:

[0017] S21. Import the SRT file generated by the video recorded by the DJI Mavic 3 Pro drone into the automatic encoding device for converting the SRT video subtitle file to the MISB standard metadata;

[0018] S22, selecting a file output path that complies with the MISB metadata specification;

[0019] S23. Click to start conversion.

[0020] Furthermore, metadata encoding that complies with the MISB specification is generated by the corresponding sensor. The metadata encoding file adopts the comma separated value (CSV) format. The metadata encoding satisfies 12 fields, including the following:

[0021] Precision Time Stamp

[0022] Sensor Latitude

[0023] Sensor Longitude Sensor longitude

[0024] Sensor True Altitude Sensor True Altitude

[0025] Platform Heading Angle

[0026] Platform Pitch Angle

[0027] Platform Roll Angle

[0028] Sensor Relative Roll Angle Sensor Relative Roll Angle

[0029] Sensor Relative Elevation Angle Sensor Relative Elevation Angle

[0030] Sensor Relative Azimuth Angle Sensor Relative Azimuth Angle

[0031] Sensor Horizontal Field of View

[0032] Sensor Vertical Field of View

[0033] Furthermore, in step S2, when a drone without the function of recording SRT files is selected, it is necessary to install the Site Scan App to perform the flight mission, use the Videolog log file generated by each video segment, encode it through a multiplexer, and output the FMV dynamic video.

[0034] Further, in step S3, the video multiplexing includes the following steps:

[0035] S31. Open the video multiplexer on the ArcGIS Pro platform;

[0036] S32, input video file;

[0037] S33, input metadata that complies with MISB specifications;

[0038] S34. Select an output path.

[0039] Furthermore, if the timestamps linking the video and metadata are not precisely synchronized, and if the time shift is observable and consistent, a .csv time shift file can be created that identifies the elapsed time in the video and its associated time difference, and the metadata time can be adjusted by the multiplexer to match the video.

[0040] Furthermore, if the map coordinates of the four corners of the video image frame projected onto the ground are not provided in the metadata file parameters, it is necessary to calculate the video trajectory by inputting the digital elevation model DEM in the video multiplexing tool.

[0041] Furthermore, in step S4, the construction of the digital twin application scenario includes the following steps:

[0042] S41, loading forest, grassland, wetland and wasteland business database;

[0043] S42, converting the scene into 3D mode;

[0044] S43, enable FMV full motion video;

[0045] S44, display elements and annotations;

[0046] S45, zoom to the video and turn on automatic tracking;

[0047] S46. In video and map scenarios, measurement, editing, and spatial analysis can be performed bidirectionally according to business needs.

[0048] Through the above technical solution, a complete digital twin application platform for forest, grassland, wetland and wasteland based on FMV full-motion video has been built, which solves the problems in the existing technology, such as difficulty in comparing video images with map scenes, inaccurate positioning, inability to collect data in both directions, and inability to perform spatial analysis. It significantly improves the integration of video and two- and three-dimensional integrated application scenarios, and meets the technical requirements for digital twin applications of forest, grassland, wetland and wasteland.

[0049] The present invention has at least the following beneficial effects:

[0050] The method of the present invention obtains the posture information and camera parameters of the DJI series drones, uses a homemade encoding device to perform map scene conversion on the video image, and can accurately calculate the position of the video image in the two-dimensional or three-dimensional map scene, thereby achieving high-precision fusion of the video image and GIS data, improving the efficiency of video use, and enabling staff to more intuitively and accurately see the correspondence between the real video and the map scene, thereby improving work efficiency.

[0051] The method of the present invention is different from other video fusion methods. It can collect data in the video, and the collected results can be synchronized with the map scene in real time. Loading or editing business data in the map scene can also be synchronized and displayed in the video in real time. The staff can observe the positional relationship and change differences between the real video and the surrounding geographical environment without delay and error, which improves the real-time and accuracy of decision-making and provides more accurate and reliable technical support for the construction of digital twin applications of forest, grassland, wetland and wasteland.

[0052] The method of the present invention can open and play multiple FMV full-motion videos at the same time, and display them according to different color labels in a two-dimensional and three-dimensional integrated map scene, so that staff can easily obtain multi-angle and multi-viewing remote analysis and interpretation capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flow chart of the forestry digital twin application method based on FMV full-motion video.

[0055] Figure 2 It is a metadata conversion device diagram for converting video subtitle SRT files to MISB specifications.

[0056] Figure 3 It is a digital twin application scenario of forest, grassland and wetland built based on FMV full-motion video. DETAILED DESCRIPTION

[0057] The invention discloses a grounding wire clamp, which can effectively buckle a high-voltage wire and prevent the grounding wire from falling off.

[0058] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the present invention. Obviously, what is described is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 As shown, the present invention provides a method for constructing a digital twin application of forest, grassland, wetland and wasteland based on FMV dynamic video, such as Figure 1 As shown, the following steps are included:

[0060] S1. According to the preset forest, grassland and wetland application scenarios, construct the flight trajectory of the drone and collect video data.

[0061] Specifically, collecting video data includes the following steps:

[0062] S11, determining the scene and target to be photographed, as well as parameters such as the required video quality and resolution;

[0063] S12. Choose the appropriate drone and camera equipment according to your shooting needs;

[0064] S13, take off the drone and perform flight operations according to the predetermined flight path and shooting requirements to ensure that the video that meets the requirements is shot;

[0065] S14, transmitting the captured video data to a computer for subsequent processing.

[0066] S2. According to the posture information collected by the drone, convert it according to the metadata specifications of MISB (Motion Image Standards Committee). If you choose DJI Mavic 3 Pro to record video, you should turn on the video subtitle option and re-encode the video subtitle SRT file in accordance with the metadata specifications of MISB (Motion Image Standards Committee).

[0067] The present invention provides an automatic encoding device for converting SRT video subtitle files into MISB standard metadata, such as Figure 2 As shown, the conversion specifically includes the following steps:

[0068] S21. Import the SRT file generated by the video recorded by DJI Mavic 3 Enterprise Edition into the device;

[0069] S22, selecting a file output path that complies with the MISB metadata specification;

[0070] S23. Click the Start Conversion button to re-encode.

[0071] In step S2, if you are using a drone that does not have the function of recording SRT files, such as the Phantom 4 RTK (development version remote controller), Phantom 4 Pro, Phantom 4 Pro V2, Mavic Pro, Mavic 2 Pro, Matrice 200, Matrice 200 V2, and Inspire 2, you need to install the Site Scan App to perform flight missions, use the Videolog log file generated by each video segment, encode it through a multiplexer, and output FMV dynamic video.

[0072] S3, through the video multiplexer, encodes the metadata that complies with the MISB specification with the drone archive video file to generate FMV full-motion video with geospatial information and image features.

[0073] The performance of generating FMV full motion video files depends on the type and quality of the data contained in the metadata file, and the accuracy of the synchronization of the video data and the metadata file.

[0074] Metadata files that conform to the MISB specification must be in comma-separated value (CSV) format.

[0075] A metadata file that complies with the MISB specification must include 12 fields, including:

[0076] Precision Time Stamp

[0077] Sensor Latitude

[0078] Sensor Longitude Sensor longitude

[0079] Sensor True Altitude Sensor True Altitude

[0080] Platform Heading Angle

[0081] Platform Pitch Angle

[0082] Platform Roll Angle

[0083] Sensor Relative Roll Angle Sensor Relative Roll Angle

[0084] Sensor Relative Elevation Angle Sensor Relative Elevation Angle

[0085] Sensor Relative Azimuth Angle Sensor Relative Azimuth Angle

[0086] Sensor Horizontal Field of View

[0087] Sensor Vertical Field of View

[0088] Specifically, video multiplexing includes the following steps:

[0089] S31. Open the video multiplexer on the ArcGIS Pro platform;

[0090] S32, input video file;

[0091] S33, input metadata that complies with MISB specifications;

[0092] S34. Select an output path.

[0093] If the timestamps linking the video and metadata are not precisely synchronized, a .csv time-shift file identifying the offset position (elapsed time) in the video and its associated time difference (time shift) can be created, and the metadata time can be adjusted by the multiplexer to match the video.

[0094] The metadata file parameters include the map coordinates of the 4 corners of the video image frame projected onto the ground. If the map coordinates of these 4 corners are provided, they will be used to create an FMV-compliant video. If the map coordinates of these 4 corners are not provided, the source of the DEM data needs to be provided in the Digital Elevation Model parameters and the multiplexer tool will calculate the video track based on the required parameters listed above.

[0095] S4. Use the ArcGIS Pro platform's 2D and 3D integrated digital basemap, load multi-source GIS feature data, enable FMV full-motion video, realize the overlay association of geographic space and video in a unified scene, perform complex two-way editing and spatial analysis, and meet the needs of building real, intuitive, and accurate digital twin applications.

[0096] Specifically, Figure 3 As shown in the figure, the steps to build a digital twin application scenario are as follows:

[0097] S41, loading forest, grassland, wetland and wasteland business database;

[0098] S42, converting the scene into 3D mode;

[0099] S43, enable FMV full motion video;

[0100] S44, display elements and annotations;

[0101] S45, zoom to the video and turn on automatic tracking;

[0102] S46. In video and map scenarios, measurement, editing, and spatial analysis can be performed bidirectionally according to business needs.

[0103] It can be seen from the above implementation method description that the present invention constructs a complete set of digital twin application scenarios of forest, grassland and wetland based on drone FMV dynamic video, which solves the problems in the existing technology such as difficulty in comparing video images and map scenes, inaccurate positioning, inability to collect data in both directions, and inability to perform spatial analysis. It significantly improves the integration of video and two- and three-dimensional integrated application scenarios, and meets the technical requirements of digital twin applications of forest, grassland and wetland.

[0104] The present invention provides a metadata encoding device based on automatic conversion of DJI drone video subtitle files into MISB-compliant metadata, which solves the complex position and attitude information and video frame projection calculation process and is suitable for drone systems that can record video subtitle files.

[0105] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for constructing a digital twin of forest, grassland and wetland based on FMV full-motion video, characterized in that: The following steps are involved: S1. According to the preset forest, grass and wetland application scenarios, the flight trajectory of the drone is constructed to obtain video data; S2, converting the posture information collected by the drone according to the metadata specification of MISB (Motion Image Standards Committee) to obtain metadata encoding that complies with the MISB specification; S3, through video multiplexing, encode metadata that complies with MISB specifications into the video to generate FMV full-motion video with geospatial information and image features; S4. Load the FMV full-motion video and multi-source GIS feature data into the 2D and 3D integrated digital scene of the ArcGIS Pro platform, perform related editing and analysis applications, and build a digital twin application scenario for forest, grassland, wetland and wasteland.

2. According to the method for constructing a digital twin of forest, grassland and wetland based on FMV full-motion video according to claim 1, it is characterized in that: In step S1, the obtaining of video data comprises the following steps: S11, determining the scene and target to be photographed, as well as parameters such as the required video quality and resolution; S12. Select drone and camera equipment according to shooting requirements; S13, taking off the drone, and performing flight operations according to a predetermined flight path and shooting requirements to ensure that video data that meets the requirements is captured; S14, transmitting the captured video data to a computer for subsequent processing.

3. According to the method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 1, it is characterized in that: In step S2, when the DJI Mavic 3 Pro drone is selected to record a video, the video subtitle option should be turned on, and the generated video subtitle SRT file should be re-encoded in accordance with the metadata specification of MISB (Motion Image Standards Committee).

4. The method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 3 is characterized in that: The SRT file includes the pose information of the drone video. The re-encoding of the metadata encoding in accordance with the MISB specification includes the following steps: S21. Import the SRT file generated by the video recorded by the DJI Mavic 3 Pro drone into the automatic encoding device for converting the SRT video subtitle file to the MISB standard metadata; S22, selecting a file output path that complies with the MISB metadata specification; S23. Click to start conversion.

5. According to the method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 4, it is characterized in that: The metadata encoding that complies with the MISB specification is generated by the corresponding sensor. The metadata encoding file uses the comma separated value (CSV) format. The metadata encoding satisfies 12 fields, including the following: Precision Time Stamp Sensor Latitude Sensor Longitude Sensor longitude Sensor True Altitude Sensor True Altitude Platform Heading Angle Platform Pitch Angle Platform Roll Angle Sensor Relative Roll Angle Sensor Relative Roll Angle Sensor Relative Elevation Angle Sensor Relative Elevation Angle Sensor Relative Azimuth Angle Sensor Relative Azimuth Angle Sensor Horizontal Field of View Sensor Vertical Field of View 6. According to the method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 1, it is characterized in that: In step S2, when a drone without the function of recording SRT files is selected, it is necessary to install the SiteScan App to perform the flight mission, use the Videolog log file generated by each video segment, encode it through a multiplexer, and output the FMV dynamic video.

7. According to the method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 1, it is characterized in that: In step S3, the video multiplexing comprises the following steps: S31. Open the video multiplexer on the ArcGIS Pro platform; S32, input video file; S33, input metadata that complies with MISB specifications; S34. Select an output path.

8. The method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 7 is characterized in that: If the timestamps linking the video and metadata are not precisely synchronized, and if the time shift is observable and consistent, a .csv time shift file identifying the elapsed time and its associated time difference in the video can be created, and the metadata time can be adjusted to match the video by the multiplexer.

9. The method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 7 is characterized in that: If the map coordinates of the four corners of the video image frame projected onto the ground are not provided in the metadata file parameters, it is necessary to calculate the video trajectory by inputting the digital elevation model DEM in the video multiplexing tool.

10. The method for constructing a digital twin of forest, grassland, wetland and wasteland based on FMV full-motion video according to claim 1 is characterized in that: In step S4, the construction of the digital twin application scenario includes the following steps: S41, loading forest, grassland, wetland and wasteland business database; S42, converting the scene into 3D mode; S43, enable FMV full motion video; S44, display elements and annotations; S45, zoom to the video and turn on automatic tracking; S46. In video and map scenarios, measurement, editing, and spatial analysis can be performed bidirectionally according to business needs.

Images