Reverse calculation method for video and three-dimensional model fusion in emergency scene

Abstract

The invention relates to the technical field of video fusion, in particular to a reverse calculation method for fusion of a video and a three-dimensional model in an emergency scene. The reverse calculation method specifically comprises the following steps: S1, data acquisition; s2, creating a three-dimensional model; s3, obtaining the geometric center and the geometric size of the three-dimensional model; s4, calculating the position of the unmanned aerial vehicle; s5, calculating a pan-tilt angle; and S6, video fusion. According to the reverse calculation method for fusion of the video and the three-dimensional model in the emergency scene, the three-dimensional model of the emergency scene can be constructed according to the information acquired by the unmanned aerial vehicle, and the three-dimensional model of the emergency scene can be constructed by calculating the ground clearance of the unmanned aerial vehicle, the distance from the unmanned aerial vehicle to the center point of the three-mode model and the calculated pan-tilt angle. The video collected by the unmanned aerial vehicle is fused to the three-dimensional model, the information (including vision, temperature, gas distribution and other information) of the emergency scene is displayed on the three-dimensional model more visually, accurately and comprehensively, the accuracy and reliability of the information are improved, and the emergency response efficiency and safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video fusion, and particularly relates to a reverse calculation method for video and three-dimensional model fusion in an emergency scenario. Background Art

[0002] Video fusion technology is a branch of virtual reality technology and can also be said to be a development stage of virtual reality. Video fusion technology refers to fusing one or more image sequence videos of a certain scene or model collected by a video acquisition device with a related virtual scene to generate a new virtual scene or model of this scene.

[0003] In an emergency rescue scenario, quickly and accurately obtaining on-site information is crucial for formulating rescue plans and taking effective measures. Existing technologies mostly rely on ground personnel or vehicles to collect on-site information. A common solution is to manually predict the distance and height from the drone to the building and manually adjust the pan-tilt angle of the camera, so as to achieve the fusion effect of video and three-dimensional building model. This method takes a long time and requires manual judgment, resulting in low efficiency or incomplete information acquisition. And manually measuring the height of a building may be risky in some cases. Summary of the Invention

[0004] The present invention provides a reverse calculation method for video and three-dimensional model fusion in an emergency scenario to fuse the video collected by the drone onto the three-dimensional model of the emergency site, so as to quickly and accurately obtain the on-site information of the emergency, which helps to improve the emergency response efficiency and safety.

[0005] To achieve the above object, the technical solution of the present invention is: to provide a reverse calculation method for video and three-dimensional model fusion in an emergency scenario, and its innovation lies in: specifically including the following steps:

[0006] S1. Data collection: When the emergency response is started, control the drone equipped with a camera, a thermal imager, and a gas sensor to fly to the emergency area to collect on-site video, thermal imaging video, and gas distribution video of the emergency site;

[0007] S2. Create a three-dimensional model: Mount five cameras on the drone and use the method of drone oblique photography to create a three-dimensional model corresponding to the emergency site and display it on the Geographic Information System (GIS);

[0008] S3. Obtain the geometric center and geometric dimensions of the three-dimensional model: directly obtain the three-dimensional coordinates of the geometric center and the geometric dimensions of the three-dimensional model in the Geographic Information System (GIS), and the geometric dimensions include the length, width, and depth of the three-dimensional model; convert the three-dimensional coordinates of the geometric center of the three-dimensional model into geographic coordinates through a coordinate conversion function;

[0009] S4. Calculate the position of the UAV: Calculate the position of the UAV based on the flight altitude of the UAV, the geometric center position of the three-dimensional model, and the height dimension of the three-dimensional model. The position of the UAV includes the height H above the ground of the UAV. d and the distance d from the UAV to the center of the three-dimensional model c ;

[0010] S5. Calculate the gimbal angles: Specifically include:

[0011] S51. Calculate the direction vector between the position of the UAV and the center position of the three-dimensional model;

[0012] S52. Normalize the direction vector;

[0013] S53. Calculate the horizontal angle of the direction vector;

[0014] S54. Calculate the gimbal pitch angle;

[0015] S6. Video fusion: Project the on-site video, thermal imaging video, and gas distribution video onto the three-dimensional model based on the geometric center coordinates, geometric dimensions, the height H above the ground of the UAV d , the distance d from the UAV to the center of the three-dimensional model c and the gimbal angles to form a fusion, completing the fusion of the video and the three-dimensional model.

[0016] Further, the height H above the ground of the UAV in step S4 d has the following calculation formula:

[0017] H d = H m + H p

[0018] where H m is the height above the ground at the top of the three-dimensional model, and H p is the distance between the UAV and the top of the three-dimensional model.

[0019] Further, the calculation method for the distance d from the UAV to the center of the three-dimensional model in step S4 c is as follows: Let the three-dimensional coordinates of the center point of the three-dimensional model be: (x m , y m , z m ), and let the three-dimensional coordinates of the UAV position be: (x d , y d , z d ). The calculation formula for the distance d from the UAV to the center of the three-dimensional model c is:

[0020]

[0021] Further, the specific method for calculating the direction vector between the UAV position and the center position of the 3D model in S51 of step S5 is as follows:

[0022] S511. Let the geographical coordinates of the center position of the 3D model be: (A m , B m , H m );

[0023] S512. Let the geographical coordinates of the UAV position be: (A d , B d , H d );

[0024] S513. Convert the geographical coordinates of the center position of the 3D model and the UAV position into Cartesian coordinates in a three-dimensional space coordinate system:

[0025] X m = R·cosB m ·cosA m

[0026] Y m = R·cosB m ·sinA m

[0027] Z m = R·sinB m

[0028] The Cartesian coordinates of the center position of the 3D model are (X m , Y m , Z m ), where R is the radius of the earth;

[0029] X d = R·cosB d ·cosA d

[0030] Y d = R·cosB d ·sinA d

[0031] Z d = R·sinB d

[0032] The Cartesian coordinates of the UAV position are (X d , Y d , Z d ), where R is the radius of the earth;

[0033] S514. Calculate the direction vector The calculation formula of the direction vector is:

[0034]

[0035] This direction vector points to the direction from the center of the three-dimensional model to the position of the drone.

[0036] Furthermore, the specific process of normalizing the direction vector in S52 of step S5 is as follows: Let the direction vector have components:

[0037]

[0038] S521. Calculate the magnitude of the direction vector The calculation formula is:

[0039]

[0040] S522. Normalize the direction vector: The components of the normalized direction vector are D x ′, D y ′, D z ′, and the calculation formula is:

[0041]

[0042]

[0043] The magnitude of the direction vector obtained through the calculation formula of the components of the normalized direction vector is 1, but the direction of the original vector is maintained.

[0044] Furthermore, the specific process of calculating the horizontal angle of the direction vector in S53 of step S5 is as follows: Set the direction vector as Calculate the projection vector of the direction vector on the horizontal plane and the angle with the positive direction of the X-axis. Since it is a projection on the horizontal plane, the D z component is not considered, where

[0045] S531. Calculate the radian θ 弧 of the horizontal angle, and the calculation formula is:

[0046] θ 弧 = arctan2(D y , D x )

[0047] where arctan2 is a two-parameter arctangent function, and the range of the return value is -π to π, that is, -180° to 180°;

[0048] S532. Convert the radian of the horizontal angle to the angle θ of the horizontal angle角 , the conversion formula is:

[0049]

[0050] Further, the specific process of calculating the pan-tilt angle of the pan-tilt head in S54 of step S5 is: according to the set direction vector The calculation method of the pan-tilt angle is:

[0051] S541. Calculate the length of the vector, and the calculation formula is:

[0052]

[0053] S542. Calculate the pan-tilt radian angle φ 弧 , and the calculation formula is:

[0054]

[0055] S543. Convert the pan-tilt radian angle φ 弧 to the pan-tilt angle φ 角 , and the formula is:

[0056]

[0057] Further, in step S6, the on-site video, thermal imaging video, and gas distribution video are projected onto the 3D model through cesium and GIS to form a fusion.

[0058] Compared with the prior art, the beneficial effects produced by the present invention are:

[0059] (1) The reverse calculation method for video and 3D model fusion in an emergency scenario of the present invention can construct a 3D model of the emergency site based on the information collected by the UAV, and fuse the on-site video, thermal imaging video, and gas distribution video collected by the UAV onto the 3D model by calculating the height of the UAV from the ground, the distance from the UAV to the center point of the 3D model, and the calculated pan-tilt angle, more intuitively, accurately, and comprehensively displaying the information of the emergency site (including visual, temperature, and gas distribution information, etc.) on the 3D model, improving the accuracy and reliability of the information, and helping to improve the emergency response efficiency and safety.

[0060] (2) The present invention quickly reaches the emergency site through the UAV, realizes the collection and transmission of real-time on-site data and video, has real-time and high efficiency. At the same time, the use of the UAV reduces the need for rescue personnel to directly enter the dangerous area, reduces the rescue risk, and has safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0062] Figure 1 It is a flowchart of the steps of a reverse calculation method for video and 3D model fusion in an emergency scenario of the present invention.

[0063] Figure 2 It is a simulation display diagram of projection fusion in Embodiment 7 of the present invention.

[0064] Figure 3 It is an emergency scene effect diagram shown after the video and 3D model in Embodiment 7 of the present invention are fused.

[0065] Figure 4 It is an infrared distribution effect diagram of the emergency scene shown after the video and 3D model in Embodiment 7 of the present invention are fused. Detailed implementation manners

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0067] The emergency scene in the present invention includes the scene where an emergency occurs, which is mostly a building in the city.

[0068] Embodiment 1

[0069] This embodiment provides a reverse calculation method for video and 3D model fusion in an emergency scenario. The specific process is as Figure 1 shown, and specifically includes the following steps:

[0070] S1. Data collection: When the emergency response is initiated, the unmanned aerial vehicle (UAV) equipped with a camera, a thermal imager, and a gas sensor is controlled to fly to the emergency area, and on-site videos, thermal imaging videos, and gas distribution videos of the emergency scene are collected;

[0071] S2. Create a 3D model: Five cameras are mounted on the UAV, and the method of UAV oblique photography is used to create a 3D model corresponding to the emergency scene and display it on the geographic information system (GIS);

[0072] S3. Obtain the geometric center and geometric dimensions of the 3D model: directly obtain the 3D coordinates of the geometric center of the 3D model and the geometric dimensions within the Geographic Information System (GIS). The geometric center refers to the center point of the 3D model, and the geometric dimensions include the length, width, and depth of the 3D model; convert the 3D coordinates of the geometric center of the 3D model into geographic coordinates (i.e., coordinates represented by longitude, latitude, and altitude) through a coordinate conversion function; the coordinate conversion function is a prior art and will not be described herein.

[0073] S4. Calculate the position of the UAV: Calculate the position of the UAV based on the flight altitude of the UAV, the position of the geometric center of the 3D model, and the height dimension of the 3D model. The position of the UAV includes the height H above the ground of the UAV d , the distance d from the UAV to the center of the 3D model c ;

[0074] The height H above the ground of the UAV in step S4 d is calculated by the formula:

[0075] H d = H m + H p

[0076] where, H m is the height above the ground at the top of the 3D model, and H p is the distance between the UAV and the top of the 3D model.

[0077] The calculation method of the distance d from the UAV to the center of the 3D model in step S4 c is as follows: Let the 3D coordinates of the center point of the 3D model be: (x m , y m , z m ), and let the 3D coordinates of the UAV position be: (x d , y d , z d ). The calculation formula of the distance d from the UAV to the center of the 3D model c is:

[0078]

[0079] S5. Calculate the gimbal angle: The gimbal pitch angle can be adjusted through the UAV interface, and the UAV interface is the development interface provided by the UAV SDK. Specifically, calculating the gimbal angle includes:

[0080] S51. Calculate the direction vector between the UAV position and the center position of the 3D model;

[0081] S52. Normalize the direction vector;

[0082] S53. Calculate the horizontal angle of the direction vector;

[0083] S54. Calculate the pitching angle of the pan-tilt head;

[0084] S6. Video fusion: Based on the geometric center coordinates, geometric dimensions of the 3D model, the height H of the UAV from the ground d , the distance d from the UAV to the center of the 3D model c and the pan-tilt head angle, project the on-site video, thermal imaging video, and gas distribution video onto the 3D model to form a fusion, completing the fusion of the video and the 3D model.

[0085] Embodiment 2

[0086] Based on Embodiment 1, in this embodiment, the specific method for calculating the direction vector between the UAV position and the center position of the 3D model in S51 of step S5 is as follows:

[0087] S511. Assume that the geographical coordinates of the center position of the 3D model are: (A m , B m , H m );

[0088] S512. Assume that the geographical coordinates of the UAV position are: (A d , B d , H d );

[0089] S513. Convert the geographical coordinates of the center position of the 3D model and the UAV position into Cartesian coordinates in a three-dimensional space coordinate system:

[0090] X m = R·cosB m ·cosA m

[0091] Y m = R·cosB m ·sinA m

[0092] Z m = R·sinB m

[0093] The Cartesian coordinates of the center position of the 3D model are (X m , Y m , Z m ), where R is the radius of the earth;

[0094] X d = R·cosB d ·cosA d

[0095] Y d = R·cosBd ·sinA d

[0096] Z d = R·sinB d

[0097] The Cartesian coordinates of the UAV position are (X d , Y d , Z d ), where R is the radius of the earth, approximately 6371 km;

[0098] S514. Calculate the direction vector The calculation formula of the direction vector is:

[0099]

[0100] This direction vector points to the direction from the center of the 3D model to the UAV position.

[0101] Embodiment 3

[0102] Based on Embodiment 2, in this embodiment, the specific process of normalizing the direction vector in S52 of step S5 is as follows: Let the components of the direction vector be:

[0103]

[0104] S521. Calculate the modulus of the direction vector The calculation formula is:

[0105]

[0106] S522. Normalize the direction vector: The components of the normalized direction vector are D x ′, D y ′, D z ′, and the calculation formula is:

[0107]

[0108] The modulus of the direction vector obtained through the calculation formula of the components of the normalized direction vector is 1, but the direction of the original vector is maintained.

[0109] The principle of the method for normalizing the direction vector is to adjust the length of a vector to 1 while keeping its direction unchanged. This is very common in various fields, such as direction calculation, ray tracing, object rotation, etc., and belongs to the prior art.

[0110] Embodiment 4

[0111] Based on Embodiment 3, the specific process of calculating the horizontal angle of the direction vector in S53 of step S5 in this embodiment is as follows: Set the direction vector as The radian θ of the horizontal included angle 弧 That is, the radian of the included angle between the projection of the vector on the horizontal plane and the due north direction (usually taking due north as 0 degrees and increasing in the clockwise direction). Calculate the projection vector of the direction vector on the horizontal plane The included angle with the positive direction of the X-axis (eastward). Since it is the projection on the horizontal plane, the D z component is not considered, where

[0112] S531. Calculate the radian θ of the horizontal included angle 弧 , and the calculation formula is:

[0113] θ 弧 = arctan2(D y , D x )

[0114] where arctan2 is a two-parameter arctangent function that can correctly handle the situation of vectors in different quadrants. The range of the return value is -π to π, that is, -180° to 180°;

[0115] S532. Convert the radian of the horizontal included angle to the angle θ of the horizontal included angle 角 , and the conversion formula is:

[0116]

[0117] Embodiment 5

[0118] Based on Embodiment 4, the specific process of calculating the pan-tilt angle of the pan-tilt head in S54 of step S5 in this embodiment is as follows: According to the set direction vector The calculation method of the pan-tilt angle is:

[0119] S541. Calculate the length of the vector, and the calculation formula is:

[0120]

[0121] S542. Calculate the pitch radian angle φ 弧 , and the calculation formula is:

[0122]

[0123] S543. Convert the pitch radian angle φ 弧 to the pitch angle φ 角 , and the formula is:

[0124]

[0125] Example 6

[0126] In step S6 of this example based on Example 1, the on-site video, thermal imaging video, and gas distribution video are all projected onto the 3D model through cesium and GIS to form a fusion.

[0127] Example 7

[0128] Based on Examples 1 - 6, this example conducts reverse computational simulation of the video and 3D model fusion in an emergency scenario and performs simulated fusion of the video and 3D model. The emergency scenario in this example is a building. The specific process is as follows:

[0129] (1) Collect on-site video, thermal imaging video, and gas distribution video through a drone;

[0130] (2) Use the method of drone oblique photography to create a 3D model corresponding to the emergency site and display it on the geographic information system GIS;

[0131] (3) Obtain the 3D coordinates of the geometric center of the 3D model as (19.59348562, 15.44390874, 0), the length of the 3D model is 29.20992879m, the width is 10.82505857, and the depth is 19.06055351m;

[0132] (4) The 3D coordinates of the drone are (69.59348562, 15.44390874, 9.53027676);

[0133] (5) Normalize the direction vector from the geometric center of the 3D model to the drone position, and the normalized direction vector is: [-0.9823152 0 -0.18723472];

[0134] (6) The pan angle of the pan-tilt head in this example is 180.00°, and the pitch angle of the pan-tilt head is calculated to be -10.79° through Examples 5 and 4.

[0135] (7) Project the on-site video, thermal imaging video, and gas distribution video collected by the drone onto the 3D model to form a fusion based on the geometric center coordinates of the 3D model, the geometric dimensions of the 3D model, etc. and the pitch angle of the pan-tilt head. Specifically, video fusion is performed through cesium and GIS. As Figure 2 shown, it is a simulated display diagram of projection fusion, where the black dot represents the drone and the rectangular box represents the emergency site (3D model of the building).

[0136] According to the projection fusion in step (7), the on-site video, thermal imaging video, gas distribution video and the three-dimensional model corresponding to the emergency site are fused, realizing the seamless fusion of the UAV video and the three-dimensional model, and providing a more intuitive and accurate display of emergency scene information, such as Figure 3 The on-site video of the emergency scene shown after fusion, such as Figure 4 The infrared image of the emergency scene is shown Figure 4 The whiter the part is, the higher the temperature. In the present invention, the UAV quickly reaches the emergency site, realizing the collection and transmission of real-time on-site data and videos, which has real-time performance and high efficiency. At the same time, the use of the UAV reduces the need for rescue personnel to directly enter the dangerous area, reducing the rescue risk and having safety

[0137] The embodiments described above are only described as the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various variations and improvements made by ordinary engineering and technical personnel in the field to the technical solutions of the present invention should fall within the protection scope of the present invention. The technical content claimed by the present invention has been fully recorded in the claims

Claims

1. A reverse calculation method for fusion of video and three-dimensional model in emergency scenarios, characterized by: The specific steps include: S1. Data collection: When the emergency response is initiated, a drone equipped with a camera, thermal imager and gas sensor is controlled to fly to the emergency area to collect on-site video, thermal imaging video and gas distribution video of the emergency site; S2. Create a 3D model: Equip the drone with five lenses and use the drone’s oblique photography method to create a 3D model corresponding to the emergency scene and display it on the Geographic Information System (GIS); S3. Obtaining the geometric center and geometric dimensions of the three-dimensional model: directly obtaining the three-dimensional coordinates and geometric dimensions of the geometric center of the three-dimensional model in the geographic information system GIS, wherein the geometric dimensions include the length, width and depth of the three-dimensional model; converting the three-dimensional coordinates of the geometric center of the three-dimensional model into geographic coordinates through a coordinate conversion function; S4. Calculate the position of the drone: Calculate the position of the drone according to the flight altitude of the drone, the geometric center position of the three-dimensional model, and the height size of the three-dimensional model. The position of the drone includes the height H of the drone above the ground. d , the distance d from the drone to the center of the three-mode model c ; S5. Calculate the gimbal angle: specifically including: S51, calculating the direction vector between the position of the UAV and the center position of the three-dimensional model; S52, normalized direction vector; S53, calculating the horizontal angle of the direction vector; S54, calculating the pitch angle of the gimbal; S6, video fusion: according to the geometric center coordinates, geometric dimensions, and height H of the drone from the ground d , the distance d from the drone to the center of the three-mode model c The on-site video, thermal imaging video and gas distribution video are projected onto the three-dimensional model from a pan-tilt angle to form a fusion, thus completing the fusion of the video and the three-dimensional model.

2. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 1 is characterized in that: The height H of the drone above the ground in step S4 d The calculation formula is: H d =H m +H p Among them, H m is the height of the top of the 3D model from the ground, H p is the distance between the drone and the top of the 3D model.

3. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 1 is characterized in that: The distance d from the drone to the center of the three-dimensional model in step S4 is c The calculation method is: Let the three-dimensional coordinates of the center point of the three-dimensional model be: (x m ,y m , z m ), let the three-dimensional coordinates of the drone’s position be: (x d ,y d , z d ), the distance d from the drone to the center of the 3D model c The calculation formula is:

4. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 1 is characterized in that: The specific method of calculating the direction vector between the drone position and the center position of the three-dimensional model in S51 of step S5 is: S511, assume that the geographical coordinates of the center position of the three-dimensional model are: (A m , B m , H m ); S512, let the geographic coordinates of the drone's location be: (A d , B d , H d ); S513, converting the geographic coordinates of the center position of the three-dimensional model and the position of the drone into Cartesian coordinates in a three-dimensional space coordinate system: X m =R·cosB m ·What m Y m =R·cosB m · sinA m Z m =R·sinB m The Cartesian coordinates of the center of the 3D model are (X m , Y m , Z m ), where R is the radius of the Earth; X d =R·cosB d ·What d Y d =R·cosB d · sinA d Z d =R·sinB d The Cartesian coordinates of the drone's position are (X d , Y d , Z d ), where R is the radius of the Earth; S514, calculate direction vector The direction vector is calculated as: The direction vector Points in the direction from the center of the 3D model to the drone's position.

5. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 4 is characterized in that: The specific process of normalizing the direction vector in step S52 of step S5 is as follows: assuming that the direction vector The weight is: S521, calculate the modulus of the direction vector The calculation formula is: S522, normalized direction vector: The components of the normalized direction vector are D x ′、D y ′、D z ′, the calculation formula is: The magnitude of the direction vector obtained by the calculation formula of the components of the normalized direction vector is 1, but the direction of the original vector is maintained.

6. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 5 is characterized in that: The specific process of calculating the horizontal angle of the direction vector in step S53 of step S5 is as follows: setting the direction vector to Calculate the projection vector of the direction vector on the horizontal plane The angle with the positive direction of the X-axis is not considered because it is a projection on the horizontal plane. z Quantity, among which, S531, calculate the radian of the horizontal angle θ 弧 , the calculation formula is: θ 弧 =arctan2(D y ,D x ) Among them, arctan2 is a two-parameter inverse tangent function, and the return value range is -π to π, that is, -180° to 180°; S532: Convert the radian of the horizontal angle into the angle θ of the horizontal angle 角 , the conversion formula is:

7. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 6 is characterized in that: The specific process of calculating the pan / tilt angle in step S54 of step S5 is as follows: The pitch angle is calculated as: S541. Calculate the length of the vector using the following formula: S542, calculate the pitch radian angle φ 弧 , the calculation formula is: S543, the pitch arc angle φ 弧 Convert to pitch angle φ 角 , the formula is:

8. The reverse calculation method for fusion of video and three-dimensional model in emergency scenarios according to claim 1 is characterized in that: In step S6, the on-site video, thermal imaging video and gas distribution video are projected onto the three-dimensional model through Cesium and GIS to form a fusion.

Images