Reverse video generation method and system for live-action pipeline, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a live-action pipeline reverse video generation method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring an aerial video, video POS (Point Of Sale) data, camera parameters and pipeline geographical coordinates; wherein the aerial video comprises a plurality of video frames; according to the POS data and the camera parameters, constructing a corresponding relation between geographic coordinates and video frame coordinates; calculating to obtain pipeline video frame coordinates corresponding to the pipeline geographical coordinates according to the pipeline geographical coordinates and the corresponding relation; and superposing the pipeline video frame coordinates to the video frames to obtain a live-action pipeline reverse video. According to the embodiment of the invention, the corresponding relation between the geographic coordinate and the video frame coordinate is constructed, and the data with the geographic coordinate attribute of the live-action pipeline is added in the aerial video of the unmanned aerial vehicle, so that the ground condition along the pipeline is more intuitively presented in the aerial video, and pipeline inspection, rescue and relief work and integrity management are more facilitated.

Description

Technical Field

[0001] Embodiments of the present disclosure belong to the technical field of real - scene pipeline reverse - seeking video generation, and particularly relate to a method, a system, an electronic device, and a storage medium for generating a real - scene pipeline reverse - seeking video. Background Art

[0002] Currently, unmanned aerial vehicles (UAVs) are widely used in the field of geographical surveying and mapping, promoting the construction of real - scene three - dimensional models. In the application in the oil and gas pipeline field, real - scene pipelines are proposed. A real - scene pipeline is a virtual pipeline system constructed based on real - world scenes. A real three - dimensional scene is constructed based on UAV aerial photos, and through modeling technology, real scenes and pipeline - related data are converted into virtual digital models with geographical coordinates. Based on UAVs, not only aerial photos can be obtained, but also aerial videos can be obtained. Through aerial videos, the ground conditions along the pipeline can be more intuitively obtained, which are widely used in pipeline inspection, disaster relief, and integrity management.

[0003] The advantage of UAV aerial videos is the intuitive understanding of the ground conditions along the pipeline. However, the current aerial videos have the problem of lacking geographical coordinate attributes and cannot directly superimpose information such as the pipeline center line. Therefore, only preliminary judgments can be made based on the video content, lacking geographical data information. Summary of the Invention

[0004] Embodiments of the present disclosure aim to solve at least one of the technical problems existing in the prior art, and provide a method, a system, an electronic device, and a storage medium for generating a real - scene pipeline reverse - seeking video.

[0005] One aspect of the present disclosure provides a method for generating a real - scene pipeline reverse - seeking video, the method comprising: Obtaining an aerial video, video POS data, camera parameters, and pipeline geographical coordinates; wherein, the aerial video includes a plurality of video frames; Constructing a correspondence between geographical coordinates and video - frame coordinates according to the POS data and the camera parameters; Calculating pipeline video - frame coordinates corresponding to the pipeline geographical coordinates according to the pipeline geographical coordinates and the correspondence; Superimposing the pipeline video - frame coordinates on each of the video frames to obtain a real - scene pipeline reverse - seeking video.

[0006] Further, the video POS data at least includes longitude, latitude, relative altitude, pan angle of the pan - tilt head, roll angle of the pan - tilt head, and yaw angle of the pan - tilt head.

[0007] Further, the camera parameters at least include pixel size and camera focal length.

[0008] Further, the correspondence between the geographical coordinates and the video - frame coordinates is shown as the following formula:

[0009]

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] Wherein, x and y are video frame coordinates, X a and Y a and Z a are geographic coordinates, X s is the latitude, Y s is the longitude, Z s is the relative height, φ is the pan angle of the pan-tilt head, ω is the roll angle of the pan-tilt head, κ is the yaw angle of the pan-tilt head, f is the camera focal length, p size is the pixel size.

[0019] Furthermore, the pipeline geographic coordinates include the pipeline pile geographic coordinates; the pipeline video frame coordinates include the pipeline pile video frame coordinates.

[0020] Furthermore, the step of superimposing the pipeline video frame coordinates on each of the video frames includes: Determine whether the pipeline pile video frame coordinates exist within the video frame image range, and if so, superimpose the pipeline pile video frame coordinates on the video frame.

[0021] Furthermore, after superimposing the pipeline pile video frame coordinates on the video frame, the method further includes: Connect line segments between the video frame coordinates of adjacent pipeline piles to form a pipeline centerline.

[0022] Another embodiment of the present disclosure provides a real-scene pipeline reverse reconstruction video generation system, and the system includes: An acquisition module, configured to acquire an aerial video, video POS data, camera parameters, and pipeline geographical coordinates; wherein, the aerial video includes a plurality of video frames; A construction module, configured to construct a correspondence between geographical coordinates and video frame coordinates according to the POS data and the camera parameters; A calculation module, configured to calculate pipeline video frame coordinates corresponding to the pipeline geographical coordinates according to the pipeline geographical coordinates and the correspondence; An overlay module, configured to overlay the pipeline video frame coordinates on each of the video frames to obtain a real-scene pipeline reverse reconstruction video.

[0023] Another embodiment of the present disclosure provides an electronic device, including: At least one processor; and, A memory communicatively connected to the at least one processor, configured to store one or more programs, and when the one or more programs are executed by the at least one processor, enable the at least one processor to implement the real-scene pipeline reverse reconstruction video generation method described above.

[0024] Another embodiment of the present disclosure provides a computer-readable storage medium, storing a computer program, and when the computer program is executed by a processor, implementing the real-scene pipeline reverse reconstruction video generation method described above.

[0025] A real-scene pipeline reverse reconstruction video generation method, system, electronic device, and storage medium according to an embodiment of the present disclosure, by constructing a correspondence between geographical coordinates and video frame coordinates, add data with real-scene pipeline geographical coordinate attributes to an unmanned aerial vehicle (UAV) aerial video, so as to more intuitively present the ground conditions along the pipeline in the aerial video, which is more conducive to pipeline inspection, emergency rescue, and integrity management. Description of the Drawings

[0026] Figure 1 It is a flowchart of a real-scene pipeline reverse reconstruction video generation method according to an embodiment of the present disclosure; Figure 2 It is a schematic diagram of a real-scene pipeline reverse reconstruction video according to another embodiment of the present disclosure; Figure 3 It is a schematic diagram of the structure of a real-scene pipeline reverse reconstruction video generation system according to another embodiment of the present disclosure; Figure 4 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present disclosure. Detailed Embodiments

[0027] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0028] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or may be implemented using other methods, components, devices, steps, etc. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0029] The flowcharts shown in the accompanying drawings are only illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps may be decomposed, while some operations / steps may be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0030] It should be understood that although terms such as first, second, and third may be used in the present disclosure to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the concept of the present disclosure. As used in the present disclosure, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0031] Those skilled in the art can understand that the drawings are only schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present disclosure. Therefore, they cannot be used to limit the protection scope of the present disclosure.

[0032] As Figure 1 shown, an embodiment of the present disclosure provides a method for generating a real-scene pipeline reverse-engineering video, and the method includes: Step S1, obtaining an aerial video, video POS data, camera parameters, and pipeline geographical coordinates.

[0033] Specifically, a drone is used to collect aerial video data by flying following the terrain and performing orthophoto shooting. After the collection is completed, two types of data files, namely video and SRT, will be generated. The SRT file stores the POS data of the drone's aerial video. The drone's aerial video consists of multiple video frames. It can be understood that each video frame is actually a separate image, and each frame of video corresponds to a series of POS data. The POS data contains the position and attitude information corresponding to each video frame captured by the drone, such as longitude, latitude, relative altitude, gimbal pitch angle, gimbal roll angle, and gimbal yaw angle, etc. Then, obtain the size of the captured video frame image and camera parameters such as the focal length of the camera and the pixel size. Based on this information, the corresponding relationship between the image coordinates and the ground coordinates can be constructed using the collinearity equation in aerial photogrammetry.

[0034] The real-scene platform has rich geographic coordinate data, such as pipeline centerline, pile number, high-consequence area, road, water system data, etc. By parsing the content in the real-scene platform, the geographic coordinates and style information of these data can be extracted. Taking the pipeline centerline and pipeline piles that are most concerned in the embodiments of the present disclosure as examples, the extracted information mainly includes the pipeline pile number and the corresponding geographic coordinates of the pipeline pile. The style information refers to the color and width of the pipeline centerline, the color and size of the pile number font, etc.

[0035] Step S2: Construct the corresponding relationship between the geographic coordinates and the video frame coordinates according to the POS data and the camera parameters.

[0036] Specifically, constructing the corresponding relationship between the geographic coordinates and the video frame coordinates requires 6 parameters in the above POS data, namely longitude, latitude, relative altitude, gimbal pitch angle, gimbal roll angle, and gimbal yaw angle. In addition, the image pixel size and the camera focal length are also required. These two parameters are the same under the same settings of the same model of drone. For example, the video image resolution captured by DJI Mavic3E is 1920×1080, the pixel size is 3.3 microns, and the camera focal length is 12.29 millimeters.

[0037] Based on the above parameters, the corresponding relationship between the geographic coordinates and the video frame coordinates is constructed through the collinearity equation, as shown in the following formula:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] wherein, x and y are video frame coordinates, X a and Y a and Z a are geographic coordinates, X s is the latitude, Y s is the longitude, Z s is the relative height, φ is the pan angle of the pan-tilt head, ω is the roll angle of the pan-tilt head, κ is the yaw angle of the pan-tilt head, f is the camera focal length, p size is the pixel size. The center of the calculated video frame coordinate system is located at the center of the video frame image.

[0048] Step S3: Calculate the pipeline video frame coordinates corresponding to the pipeline geographic coordinates according to the pipeline geographic coordinates and the corresponding relationship.

[0049] Specifically, according to the extracted pipeline geographic coordinates and the corresponding relationship between the geographic coordinates and the video frame coordinates constructed in the previous step S2, calculate the pipeline video frame coordinates corresponding to the pipeline geographic coordinates in the video frame image.

[0050] Taking the pipeline center line and the pipeline piles as an example, the data extracted from the real scene platform can all be regarded as composed of points and lines. The pipeline center line is composed of line segments connected between multiple pipeline piles, and the pipeline piles are individual points. Directly substitute the pipeline pile geographic coordinates into the corresponding relationship between the geographic coordinates and the video frame coordinates constructed in the previous step S2 to calculate the corresponding pipeline pile video frame coordinates.

[0051] Step S4: Superimpose the pipeline video frame coordinates on each video frame to obtain the reverse calculation video of the real scene pipeline.

[0052] Specifically, it is determined whether the coordinates of the pipeline pile video frame exist within the range of the video frame image. If they exist, the coordinates of the pipeline pile video frame are superimposed on the corresponding video frame. The pipeline center line is formed by connecting the line segments between multiple pipeline piles. There are two situations in the display of the pipeline center line in a video frame image: one is that there is only one line segment in the image without pipeline piles; the other is that there are two or more line segments in the image, including one or more pipeline piles. For the pipeline center line, not only the positions of the pipeline piles need to be calculated, but also the intersection points of the line segments and the image boundary need to be calculated. Therefore, first, the positions of the pipeline piles are calculated, then it is determined how many pipeline piles there are in the video frame image, and then the intersection points with the image boundary are calculated according to the above two situations.

[0053] For the first situation, first calculate the positions of the two pipeline pile video frame coordinate points on the image, and then determine the two intersection points of the pipeline center line and the image boundary based on these two coordinate points, and connect these two intersection points to form the pipeline center line on the image; for the second situation, determine the intersection point of the line segment formed by the first pipeline pile in the image and the previous pipeline pile outside the image with the image boundary, then determine the intersection point of the line segment formed by the last pipeline pile in the image and the next pipeline pile outside the image with the image, and finally connect the first image boundary intersection point, the video frame coordinates of each adjacent pipeline pile in the image, and the last image boundary intersection point in sequence with line segments to form the pipeline center line.

[0054] The video frame coordinates of the pipeline piles and the pipeline center line obtained from the above process are superimposed on the video frame by frame according to the style of the data extracted in step S1 or a custom style (the style mainly includes color), and then the video frames with geographical data information superimposed are combined into a new video to obtain the real-scene pipeline inverse solution video. In some embodiments, a frame image in the obtained real-scene pipeline inverse solution video is as Figure 2 shown.

[0055] A method for generating a real-scene pipeline inverse solution video according to an embodiment of the present disclosure presents the ground conditions along the pipeline more intuitively in the aerial video by constructing the correspondence between geographical coordinates and video frame coordinates and adding data with the attributes of real-scene pipeline geographical coordinates to the aerial video of the unmanned aerial vehicle, which is more conducive to pipeline inspection, disaster relief, and integrity management.

[0056] As Figure 3 shown, another embodiment of the present disclosure provides a real-scene pipeline inverse solution video generation system, and the system includes: An acquisition module 310, configured to acquire an aerial video, video POS data, camera parameters, and pipeline geographical coordinates; wherein, the aerial video includes a plurality of video frames; A construction module 320, configured to construct the correspondence between geographical coordinates and video frame coordinates according to the POS data and the camera parameters; A calculation module 330, configured to calculate pipeline video frame coordinates corresponding to the pipeline geographic coordinates according to the pipeline geographic coordinates and the corresponding relationship. An overlay module 340, configured to overlay the pipeline video frame coordinates on each of the video frames to obtain a real-scene pipeline inverse solution video.

[0057] Specifically, a real-scene pipeline inverse solution video generation system according to an embodiment of the present disclosure is used to implement the real-scene pipeline inverse solution video generation method described in the foregoing embodiment. The specific implementation process has been described in detail in the foregoing embodiment and will not be elaborated herein.

[0058] A real-scene pipeline inverse solution video generation system according to an embodiment of the present disclosure constructs a corresponding relationship between geographic coordinates and video frame coordinates, adds data with real-scene pipeline geographic coordinate attributes to an aerial video captured by a drone, so as to more intuitively present the ground conditions along the pipeline in the aerial video, which is more conducive to pipeline inspection, disaster relief, and integrity management.

[0059] As Figure 4 shown, another embodiment of the present disclosure provides an electronic device, including: At least one processor 401; and a memory 402 communicatively connected to the at least one processor 401, configured to store one or more programs, and when the one or more programs are executed by the at least one processor 401, the at least one processor 401 can implement the real-scene pipeline inverse solution video generation method described above.

[0060] Wherein, the memory 402 and the processor 401 are connected in a bus manner. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 401 and the memory 402 together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be an element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium. The data processed by the processor 401 is transmitted on a wireless medium through an antenna. Further, the antenna also receives data and transmits the data to the processor 401.

[0061] The processor 401 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 402 may be used to store data used by the processor 401 when performing operations.

[0062] Another embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the real-scene pipeline reverse engineering video generation method described above.

[0063] Among them, the computer-readable storage medium may be included in the system or electronic device of the present disclosure, or may exist alone.

[0064] The computer-readable storage medium can be any tangible medium that contains or stores a program, which can be an electrical, magnetic, optical, electromagnetic, infrared, semiconductor system, device, equipment. More specific examples include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, an optical fiber, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0065] The computer-readable storage medium may also include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code, and specific examples thereof include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof.

[0066] It can be understood that the above embodiments are merely exemplary embodiments adopted to illustrate the principle of the present disclosure. However, the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present disclosure, and these modifications and improvements are also regarded as the protection scope of the present disclosure.

Claims

1. A method for generating a real-scene pipeline reverse-engineering video, characterized in that, The method includes: Obtaining an aerial video, video POS data, camera parameters, and pipeline geographical coordinates; wherein, the aerial video includes multiple video frames; Constructing a correspondence between geographical coordinates and video frame coordinates according to the POS data and the camera parameters; Calculating pipeline video frame coordinates corresponding to the pipeline geographical coordinates according to the pipeline geographical coordinates and the correspondence; Overlaying the pipeline video frame coordinates on each of the video frames to obtain a real-scene pipeline inverse solution video.

2. The method according to claim 1, wherein The video POS data includes at least longitude, latitude, relative height, pan angle, roll angle, and yaw angle of the pan-tilt head.

3. The method according to claim 2, wherein The camera parameters include at least pixel size and camera focal length.

4. The method according to claim 3, characterized in that, The correspondence between the geographical coordinates and the video frame coordinates is shown by the following formula: Wherein, x , y are video frame coordinates, X a , Y a , Z a are geographical coordinates, X s is the latitude, Y s is the longitude, Z s is the relative height, φ is the pan tilt angle of the pan-tilt head, ω is the roll angle of the pan-tilt head, κ is the yaw angle of the pan-tilt head, f is the camera focal length, p size is the pixel size.

5. The method according to any one of claims 1 to 4, characterized in that The pipeline geographical coordinates include pipeline stake geographical coordinates; the pipeline video frame coordinates include pipeline stake video frame coordinates.

6. The method according to claim 5, wherein The overlaying the pipeline video frame coordinates on each of the video frames includes: Determining whether the pipeline stake video frame coordinates exist within the video frame image range, and if so, overlaying the pipeline stake video frame coordinates on the video frame.

7. The method according to claim 6, wherein After overlaying the pipeline stake video frame coordinates on the video frame, the method further includes: Connecting line segments between adjacent pipeline stake video frame coordinates to form a pipeline center line.

8. A real-scene pipeline reverse-engineering video generation system, characterized in that The system includes: An obtaining module, configured to obtain an aerial video, video POS data, camera parameters, and pipeline geographical coordinates; wherein, the aerial video includes multiple video frames; A constructing module, configured to construct a correspondence between geographical coordinates and video frame coordinates according to the POS data and the camera parameters; A calculating module, configured to calculate pipeline video frame coordinates corresponding to the pipeline geographical coordinates according to the pipeline geographical coordinates and the correspondence; An overlaying module, configured to overlay the pipeline video frame coordinates on each of the video frames to obtain a real-scene pipeline inverse solution video.

9. An electronic device, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor, configured to store one or more programs, and when the one or more programs are executed by the at least one processor, enable the at least one processor to implement the real-scene pipeline inverse solution video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the real-scene pipeline inverse solution video generation method according to any one of claims 1 to 7.

Images