Visualization system for images captured by cameras mounted on flying objects

Abstract

This system visualizes images captured by a camera mounted on a flying object. By identifying the position, altitude, sensor size, focal length, azimuth, and tilt of the flying object, it simulates and displays images captured by the camera. Even if the field of view of the simulation system is smaller than the field of view of the camera, it displays a simulated image that closely resembles the image captured by the camera. [Solution] The system for visualizing images captured by a camera mounted on a flying object includes means for inputting flying object position information, means for inputting flying object flight altitude information, means for inputting sensor size information, means for inputting focal length information, means for inputting tilt information, means for inputting azimuth angle information, means for calculating the shooting range directly below, means for calculating the shooting range at an inclination, means for calculating the shooting range at an azimuth angle, means for sending and receiving information to and from a GIS application, and means for displaying a simulated image.

Description

[Technical Field]

[0001] The present invention relates to a system for visualizing images captured by a camera mounted on a flying object, which allows for prior confirmation of the flight position (latitude, longitude, altitude + flight height) from which the target area can be photographed, when considering aerial photography using a drone, helicopter, small airplane, etc. (hereinafter referred to as "flying object") on a desk, by visualizing the images captured by a camera mounted on the flying object. [Background technology]

[0002] In recent years, cameras mounted on flying objects have been used in a variety of fields, including photography for assessing the situation at disaster sites and for security purposes, as well as photography for surveying and inspecting buildings. However, it is difficult to predict, based solely on flat maps or camera angles, what altitude a flying object should fly at and in which direction the camera should be pointed to capture the desired area. Furthermore, it is impossible to confirm what the resulting image will look like without actually taking the photos. Furthermore, especially when using drones or helicopters without people on board for filming, if it is not possible to determine the necessary flight altitude of the aircraft to film the target area, it will be difficult to apply for an exemption from flight restrictions at airports, etc.

[0003] Patent Document 1 (Japanese Patent Publication No. 2000-306084) describes a three-dimensional display device that includes a three-dimensional map generation circuit (1), an image pasting circuit (2), an object pasting circuit (3), a shooting range visualization circuit (19) that visualizes and displays the positional relationship between the shooting range and the background ground surface, a viewpoint extraction circuit (4), and a display (5) in order to clarify whether a target can be photographed along a predetermined flight route and to allow the operator to easily grasp the situation. The device calculates the shooting range on the ground surface from the field of view of the shooting equipment and the position of the aircraft (longitude, latitude, altitude, and bearing), and displays the calculated range on the ground surface as a thin linear shooting range (PZ) (see paragraph 0047 and Figures 20 and 21 in particular). Furthermore, Patent Document 1 describes a three-dimensional display device that, in addition to a three-dimensional map generation circuit (1), an image pasting circuit (2), an object pasting circuit (3), a viewpoint extraction circuit (4), and a display (5), includes a second viewpoint extraction circuit (4b), a viewpoint control circuit (23) that controls which viewpoint from which of the multiple viewpoints the image is extracted, and a video synthesis circuit (22) that synthesizes the images from the viewpoint extraction circuits (4) and (4b), thereby enabling the operator to grasp the flight status of the aircraft from multiple angles by displaying images from multiple viewpoints on the same screen (see paragraphs 0050, 0051 and Figures 26, 27 in particular).

[0004] Furthermore, Patent Document 2 (JP 2024-505370) describes creating or simulating a GIS application environment using an open-source virtual globe application (see paragraph 0064 and Figure 12 in particular) and generating a panoramic view by defining clock tick events that move the camera from point to point, giving a panoramic view (see paragraph 0165 in particular).

[0005] However, as can be seen from Figure 21, the three-dimensional display device described in Patent Document 1 calculates the range of the ground surface that can be photographed from an aircraft and allows for a three-dimensional understanding of that range, and does not simulate and display the images that will be photographed. Furthermore, while Patent Document 2 describes creating or simulating a GIS application environment and generating panoramic views using an open-source virtual globe application, it does not simulate and display images captured by a camera mounted on a flying object. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2000-306084 (Japanese Patent Publication No. 3128549) [Patent Document 2] Special Publication No. 2024-505370 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In view of these circumstances, the present invention aims to provide a first objective: to enable the simulation and display of images captured by a camera mounted on a flying object, based on the location (latitude, longitude) and flight altitude of the flying object, as well as the sensor size, focal length, azimuth angle, and tilt of the camera mounted on the flying object. The second objective is to resolve the problem that images captured by the camera cannot be accurately simulated and displayed when the field of view of the simulation system is smaller than the field of view of the camera, and to enable the display of a simulated image that closely resembles the image captured by the camera. [Means for solving the problem]

[0008] The invention according to claim 1 identifies the position and flight altitude of a flying object, as well as the sensor size, focal length, azimuth angle, and tilt of a camera mounted on the flying object, and displays Device A flying object-mounted camera image visualization system that can simulate and display images captured by the aforementioned camera, A means for inputting the position information of the aforementioned flying object, A means for inputting flight altitude information of the aforementioned flying object, A sensor size information input means for inputting the sensor size information of the aforementioned camera, A focal length information input means for inputting the focal length information of the aforementioned camera, A tilt information input means for inputting tilt information of the aforementioned camera, A means for inputting azimuth information from the aforementioned camera, On the assumption that both the tilt information and the azimuth information are 0°, based on the position information input by the flying object position information input means, the flight altitude information input by the flying object flight altitude information input means, the sensor size information input by the sensor size information input means, and the focal length information input by the focal length information input means, a directly downward shooting range calculation means for calculating the directly downward shooting range with the camera facing directly downward; Based on the directly downward shooting range and the tilt information input by the tilt information input means, an inclined shooting range calculation means for calculating the inclined shooting range after applying the tilt information; Based on the inclined shooting range and the azimuth information input by the azimuth information input means, an azimuth shooting range calculation means for calculating the azimuth shooting range after applying the tilt information and the azimuth information; An information transmission / reception means for transmitting the azimuth shooting range and the position information of the flying object to a GIS application and receiving simulation image information from the GIS application; A simulation image display means for displaying a simulation image on the display device based on the simulation image information received by the information transmission / reception means, characterized by comprising the above.

[0009] The invention according to claim 2 is the flying object-mounted camera shooting image visualization system according to claim 1, wherein the position information is latitude information, longitude information, and altitude information, characterized by this.

[0010] The invention according to claim 3 is the flying object-mounted camera shooting image visualization system according to claim 1, wherein the position information is latitude information and longitude information, further comprising an altitude information acquisition means for acquiring altitude information based on the latitude information and the longitude information, wherein the directly downward shooting range calculation means calculates the directly downward shooting range with the camera facing directly downward based on the position information, the flight altitude information, the sensor size information, the focal length information, and the altitude information acquired by the altitude information acquisition means, characterized by this.

[0011] The invention according to claim 4 is a system for visualizing images captured by a camera mounted on a flying object, as described in any one of claims 1 to 3. The tilt shooting range calculation means calculates the additional tilt shooting range after applying the additional tilt information, based on the tilt shooting range, the directly downward shooting range, and the additional tilt information obtained by adding predetermined tilt information to the tilt information. The azimuth angle shooting range calculation means calculates the tilt information and the additional azimuth angle shooting range after applying the additional azimuth angle information based on the tilt shooting range and the additional azimuth angle information obtained by adding predetermined azimuth angle information to the azimuth angle information, in addition to the azimuth angle shooting range, and based on the additional tilt shooting range and the azimuth angle information, the additional tilt tree The additional tilt azimuth angle shooting range is calculated after applying the information and azimuth angle information, and based on the additional tilt shooting range and the additional azimuth angle information, the additional tilt tree The additional tilt and additional azimuth angle shooting range after applying the information and additional azimuth angle information is calculated. The information transmission means transmits the additional azimuth shooting range, the additional tilt azimuth shooting range, and the additional tilt additional azimuth shooting range to the GIS application, and receives the simulation image information, additional azimuth simulation image information, additional tilt azimuth simulation image information, and additional tilt additional azimuth simulation image information from the GIS application, respectively. The simulation image display means is characterized by displaying a superimposed simulation image on the display device based on the simulation image information, the additional azimuth simulation image information, the additional tilt azimuth simulation image information, and the additional tilt additional azimuth simulation image information.

[0012] The invention according to claim 5 is a system for visualizing images captured by a camera mounted on a flying object as described in claim 4, The simulation image display means is characterized by adding frame lines and auxiliary lines indicating the field of view to the simulation image information, the additional azimuth simulation image information, the additional tilt azimuth simulation image information, and the additional tilt azimuth simulation image information, and by utilizing the characteristic that the frame lines and auxiliary lines act as markers indicating overlapping portions to display the superimposed simulation image on the display device. [Effects of the Invention]

[0013] According to the invention of claim 1 or 2, assuming that the camera's tilt information and azimuth angle information are both 0°, the invention includes a downward shooting range calculation means that calculates the downward shooting range with the camera pointed directly downward based on the position information, flight altitude information, camera sensor size information, and focal length information of the flying object; a tilted shooting range calculation means that calculates the tilted shooting range based on the calculated downward shooting range and the input camera tilt information; an azimuth shooting range calculation means that calculates the azimuth shooting range based on the calculated tilted shooting range and the input camera azimuth angle information; an information transmission and reception means that transmits the calculated azimuth shooting range and the position information of the flying object to a GIS application and receives simulation image information from the GIS application; and a simulation image display means that displays the simulation image on a display device based on the received simulation image information. Therefore, by simply inputting the position information, flight altitude information, camera sensor size information, focal length information, tilt information, and azimuth angle information of the flying object, it is possible to simulate and display images taken in advance by a camera mounted on the flying object and visualize them, and to confirm in advance the flight position of the flying object in which the desired range can be photographed. In particular, when using unmanned flying objects for photography, it becomes possible to consider in advance the necessary flight altitude of the object to photograph the target area, and it becomes easier to apply for the lifting of flight restrictions at airports, etc.

[0014] According to the invention of claim 3, in addition to the effects of the invention of claim 1 or 2, the invention further comprises an altitude information acquisition means for acquiring altitude information based on latitude and longitude information, and the direct-below shooting range calculation means can calculate the direct-below shooting range based on the latitude, longitude and flight altitude information of the flying object, the sensor size information and focal length information of the camera, and the altitude information acquired by the altitude information acquisition means, thus eliminating the need to check and input altitude information corresponding to the latitude and longitude information of the flying object.

[0015] According to the invention of claim 4, in addition to the effects of the invention of any one of claims 1 to 3, the tilt shooting range calculation means calculates an additional tilt shooting range based on the direct downward shooting range and additional tilt information, in addition to the tilt shooting range. The azimuth angle shooting range calculation means calculates an additional azimuth angle shooting range based on the tilt shooting range and additional azimuth angle information in addition to the azimuth angle shooting range, calculates an additional tilt azimuth angle shooting range based on the additional tilt shooting range and azimuth angle information, and calculates an additional tilt additional azimuth angle shooting range based on the additional tilt shooting range and additional azimuth angle information. The information transmission means transmits the azimuth shooting range, additional azimuth shooting range, additional tilt azimuth shooting range, and additional tilt-additional azimuth shooting range to the GIS application, and receives the simulation image information, additional azimuth simulation image information, additional tilt azimuth simulation image information, and additional tilt-additional azimuth simulation image information from the GIS application, respectively. The simulation image display means can display a superimposed simulation image on a display device based on the simulation image information, additional azimuth simulation image information, additional tilt azimuth simulation image information, and additional tilt additional azimuth simulation image information. This solves the problem of not being able to accurately simulate and display the image captured by the camera when the field of view of the GIS application is smaller than the field of view of the camera mounted on the flying object, and allows for the display of a simulation image that is close to the image captured by the camera.

[0016] According to the invention of claim 5, in addition to the effects of the invention of claim 4, the simulation image display means adds frame lines and auxiliary lines indicating the field of view to the simulation image information, additional azimuth simulation image information, additional tilt azimuth simulation image information, and additional tilt additional azimuth simulation image information, and uses the characteristic that the frame lines and auxiliary lines act as markers indicating overlapping portions to display the superimposed simulation image on the display device. Therefore, even if there are few similar parts (feature points) in the overlapping portion of the image information, it is possible to suppress the failure of panoramic synthesis. [Brief explanation of the drawing]

[0017] [Figure 1] An explanatory diagram of the system implemented by the image visualization system using cameras mounted on flying objects. [Figure 2] Block diagram of the image visualization system for images captured by a camera mounted on a flying object according to Example 1. [Figure 3] A flowchart showing the processing flow in Example 1. [Figure 4] A diagram explaining how to calculate the shooting range directly below. [Figure 5] An example of displaying a simulation image according to Example 1. [Figure 6] This figure shows a comparison between an actual captured image and a simulated image from Example 1. [Figure 7] A flowchart showing the processing flow in Example 2. [Figure 8] Examples of simulation images corresponding to the nine shooting ranges in Example 2. [Figure 9] This figure shows a comparison between the superimposed simulation image and the actual captured image from Example 2. [Figure 10] A diagram illustrating the method for determining additional tilt information and additional azimuth angle information. [Figure 11] A diagram illustrating the function of the visualization system according to Example 3. [Modes for carrying out the invention]

[0018] As shown in the explanatory diagram (Figure 1) of the realization content by the flying object-mounted camera shooting image visualization system, when flying the flying object A over the target point and conducting aerial photography with the camera C (hereinafter referred to as "camera C") mounted on the flying object A, the purpose is to display and visualize in advance the image captured by the camera C on a display device. Conventionally, as shown in Figure 1(A), if the position (latitude, longitude) and flight altitude of the flying object A are set, the image captured by the camera C directed downward can be visualized using a two-dimensional map within the range of the viewing angle determined by the sensor size and focal length of the camera C. However, in recent years, it has become common to move the camera C up and down or change the orientation of the flying object A to capture images like bird's-eye views. Therefore, the present invention not only sets the position and flight altitude of the flying object A, but also sets the inclination (0° for directly below, 90° for directly sideways) and azimuth angle (0° for north, 90° for east, 180° for south, 270° for west) of the camera C, and simulates the image captured by the camera C in a GIS (Geographic Information System) application (for example, Google Earth <Google Earth: Google is a registered trademark> or Cesium <Cesium: registered trademark>), so that an image like Figure 1(B) can be displayed and visualized on the display device D. Hereinafter, embodiments of the present invention will be described by way of examples.

Example

[0019] Figure 2 is a block diagram of the flying object-mounted camera shooting image visualization system (hereinafter referred to as "visualization system") according to Example 1. As shown in the figure, the visualization system according to Example 1 includes the following means. (1) Flying object position information input means 1 for inputting the position information of the flying object A. In Example 1, the latitude, longitude, and elevation directly below the flying object A are input as position information. However, when the area where the flying object A flies is over the sea or a flat field with almost no elevation difference, the elevation does not need to be input, and the average value of the elevation can be set in the initial setting, or the average value of the elevation can be uniformly added in the flying object flight altitude information input means 2 described in the next section. Furthermore, depending on the type of GIS application 12 described later, it may be necessary to input the ellipsoidal height (the height from the ellipsoidal surface to mean sea level), which takes geoid height into account, as the elevation. (2) Flight altitude information input means 2 for inputting flight altitude information of flight object A. (3) Sensor size information input means 3 for inputting sensor size information of camera C. (4) Focal length information input means 4 for inputting the focal length information of camera C. (5) Tilt information input means 5 for inputting the tilt information θ of camera C. (6) Azimuth information input means 6 for inputting azimuth information φ from camera C.

[0020] (7) A downward shooting range calculation means 7 that calculates the downward shooting range with camera C pointed directly downwards, based on the position information input means 1 of (1) above, the flight altitude information input means 2 of (2) above, the sensor size information input means 3 of (3) above, and the focal length information input means 4 of (4) above, assuming that both the tilt information θ and the azimuth angle information φ are 0°. (8) An inclined shooting range calculation means 8 calculates an inclined shooting range after applying inclined information, based on the inclined shooting range calculated by the inclined shooting range calculation means 7 and the inclined information θ input by the inclined information input means 5 of (5) above. (9) Azimuth shooting range calculation means 9 calculates the tilt shooting range calculated by the tilt shooting range calculation means 8 and the azimuth angle information φ input by the azimuth angle information input means 6 of (6) above, and calculates the tilt information θ and the azimuth shooting range after applying the azimuth angle information. (10) Information transmission and reception means 10 that transmits the azimuth angle shooting range and position information calculated by the azimuth angle shooting range calculation means 9 to the GIS application 12 via the Internet line, and receives the simulated image information from the GIS application 12 via the Internet line. (11) A simulation image display means 11 that adds a frame line and auxiliary lines indicating the field of view calculated by means of the information transmission means 10 to the simulation image information received by the information transmission means 10 and displays the simulation image on the display device D.

[0021] Figure 3 is a flowchart showing the processing flow in Example 1. Each step in this flowchart will be explained step by step. <Acquisition of various information, coordinate transformation of positional information, calculation of the shooting range directly below> *ST01: An information acquisition step that acquires location information (latitude, longitude, and altitude), flight altitude information, sensor size information (vertical size and horizontal size), focal length information, tilt information θ, and azimuth angle information φ, which are input using each of the input means 1 to 6 above (1) to (6). *ST02: A coordinate transformation step that converts the location information (latitude, longitude, and altitude) obtained in ST01 into X, Y, and Z coordinates in a rectangular coordinate system. *ST03: A direct downward shooting range calculation step that calculates the direct downward shooting range when camera C is pointed directly downwards, based on the X, Y, and Z coordinates converted in ST02, sensor size information (vertical size a1, horizontal size a2), focal length information (focal length b), and flight altitude information (altitude c) acquired in ST01. Figure 4 is a diagram illustrating the method for calculating the direct downward shooting range. In other words, as shown in Figures 4(A) and (B), if the coordinates of the viewpoint position E (center of the camera C lens) are (0,0,0), the distance from the center (directly below the viewpoint) to the vertical edge of the downward shooting range (vertical distance) is d1, and the distance from directly below the viewpoint to the horizontal edge (horizontal distance) is d2, then the coordinates directly below the viewpoint are (0,0,-c), the coordinates of the four corners of the downward shooting range α,β,γ,δ are (d1,-d2,-c), (d1,d2,-c), (-d1,d2,-c), (-d1,-d2,-c), respectively, the vertical shooting range is d1×2, and the horizontal shooting range is d2×2. As shown in Figure 4(A), if we let the sensor size be a, the focal length be b, the distance to the subject (ground surface) (flight altitude) be c, and the distance from directly below the viewpoint to the edge be d, then the distance d can be determined from the ratio of the sensor size a and focal length b to the distance to the subject c. Therefore, the vertical and horizontal shooting ranges can be calculated using the following equations 1 and 2, respectively. Equation 1: Vertical shooting range d1 × 2 = a1 × c ÷ b Equation 2: Horizontal shooting range d2 × 2 = a2 × c ÷ b For the sake of simplicity, the coordinates of the viewpoint E were set to (0,0,0), but the actual X and Y coordinates are the coordinates transformed by ST02, and the Z coordinate is the elevation + flight altitude or ellipsoid height + flight altitude, so the coordinates of the four corners α, β, γ, and δ also need to be shifted in the same way.

[0022] <Calculation of tilt shooting range, calculation of azimuth shooting range> *ST04: A tilt shooting range calculation step that calculates the position based on the camera angle (tilt) derived from the tilt information θ, using the coordinates α, β, γ, δ obtained in ST03. Specifically, a tilt rotation matrix (Equation 3) with respect to the Y-axis is applied to coordinates α, β, γ, and δ, respectively, to obtain the coordinates α', β', γ', and δ' after tilt application.

number

number

[0023] <Transmission of azimuth angle shooting range, reception of image information, display of simulated image> *ST06: Azimuth angle shooting range transmission step that transmits the coordinates α", β", γ", δ" (azimuth angle shooting range) obtained in ST05 and the position information obtained in ST01 to the GIS application 12 via the internet line. *ST07: Image information reception step, which receives simulation image information transmitted from GIS application 12 via the internet. *ST08: A simulation image display step in which a frame and auxiliary lines indicating the calculated field of view are added to the simulation image information received in ST07, and the simulation image is displayed on the display device D. Figure 5 shows an example of a simulation image displayed according to Example 1. As shown in Figure 5, by performing processes ST01 to ST08, the view from a specified viewpoint can be simulated. If the field of view of camera C is smaller than the field of view that can be displayed by the GIS application 12, the entire area captured by camera C (the area of ​​the white line in Figure 5) will be included in the simulated image. In Example 1, as explained in ST04 and ST05, the calculation of the tilt shooting range was performed in the order of calculation of the azimuth shooting range. However, if this process is reversed, and the calculation of the azimuth shooting range is performed in the order of calculation of the tilt shooting range, the calculation result will not point the line of sight in the direction that was originally expected due to the non-commutativity of the product of rotation matrices. [Examples]

[0024] Figure 6 shows a comparison between an actual image captured by camera C and a simulated image from Example 1. As explained in the description of Figure 5, in some cases the entire range captured by camera C is included in the simulated image. However, when comparing the actual image captured by camera C with the simulated image from Example 1, as shown in Figure 6, the range captured by camera C does not fit entirely into the simulated image, making it difficult to confirm in advance whether the desired range can be captured. This is due to the fact that the viewing angle of camera C is larger than the viewing angle that can be displayed on Google Earth <Google Earth: Google is a registered trademark> used in the simulation, and a similar event occurred when using Cesium <Cesium: registered trademark>, another GIS application. In view of the occurrence of such an event, in the visualization system according to Example 2, the functions of the visualization system according to Example 1 are extended so that an additional tilt shooting range based on additional tilt information deviated by a predetermined angle from the directly below shooting range and tilt information θ can be calculated. At the same time, the tilt shooting range and the additional tilt shooting range, and the azimuth shooting range, additional azimuth shooting range, additional tilt azimuth shooting range, and additional tilt additional azimuth shooting range based on the azimuth information φ and the additional azimuth information deviated by a predetermined angle from the azimuth information φ can be calculated. Further, the calculated plurality of shooting ranges and position information are transmitted to the GIS application 12 to receive a plurality of simulation image information, and based on the received plurality of simulation image information, a superimposed simulation image synthesized as a single panoramic image can be displayed on the display device D. According to the visualization system according to Example 2, even if the viewing angle of camera C is larger than the viewing angle that can be displayed in the GIS application 12, the entire range captured by camera C can be included in a single superimposed simulation image.

[0025] The configuration of Example 2 is almost the same as the configuration of Example 1, and only the processing flow in the tilt shooting range calculation means 8, azimuth shooting range calculation means 9, information transmission / reception means 10, and simulation image display means 11 is different. Therefore, the block diagram of the visualization system according to Example 2 and the detailed description of each means are omitted. FIG. 7 is a flowchart showing the processing flow in Example 2. Then, each step shown in FIG. 7 will be described in order. However, the processing of ST01 to ST05 is the same as that in Example 1 and will be omitted, and the processing of ST04' to ST08' and ST07" will be described.

[0026] <Processing in the tilt shooting range calculation means 8> *ST04': For coordinates α, β, γ, δ obtained in ST03, the slope information θ is set to a predetermined slope information θ Δ An additional tilt shooting range calculation step that calculates the position based on the camera angle (tilt) derived from the additional tilt information θa and θb obtained by adding or subtracting these values. Specifically, for coordinates α, β, γ, and δ, a substitution tilt rotation matrix is ​​applied in which the tilt information θ of the rotation matrix with respect to the Y-axis (Equation 3) is replaced with additional tilt information θa and θb, respectively, to obtain the coordinates αa', βa', γa', δa' and αb', βb', γb', δb' after tilt application. In Example 2, the additional tilt information θa and θb are defined as a predetermined tilt information θ corresponding to the difference between the field of view of camera C and the field of view that can be displayed by the GIS application (hereinafter referred to as "field of view difference"). Δ The sum of θa and the slope information θ gives θ Δ We adopted θb after subtracting [a certain factor].

[0027] <Processing in the azimuth angle shooting range calculation means 9> *ST05': For coordinates α', β', γ', δ' obtained in ST04, the azimuth angle information φ is set to a predetermined azimuth angle information φ Δ An additional azimuth angle shooting range calculation step that calculates the position based on the camera direction (heading) based on the additional azimuth angle information φc and φd obtained by adding or subtracting; an additional tilt azimuth angle shooting range calculation step that calculates the position based on the camera direction (heading) based on the azimuth angle information φ for the coordinates αa', βa', γa', δa' and coordinates αb', βb', γb', δb' obtained in ST04'; and an additional tilt additional azimuth angle shooting range calculation step that calculates the position based on the camera direction (heading) based on the additional azimuth angle information φc and φd for the coordinates αa', βa', γa', δa' and coordinates αb', βb', γb', δb' obtained in ST04'. Specifically, the steps are to apply a substitute azimuth rotation matrix to coordinates α', β', γ', δ', where the azimuth information φ of the rotation matrix for azimuth in the Z-axis direction (Equation 4) is replaced with the additional azimuth information φc and φd, respectively, to obtain coordinates αc", βc", γc", δc" and coordinates αd", βd", γd", δd" (additional azimuth shooting range) after applying heading, and to apply the rotation matrix for azimuth in the Z-axis direction (Equation 4) to coordinates αa', βa', γa', δa' and coordinates αb', βb', γb', δb', respectively, to obtain coordinates αa", β The steps are to obtain a”,γa”,δa”, and coordinates αb”,βb”,γb”,δb” (additional tilt azimuth shooting range), and to apply the rotation matrix for the substituted azimuth angle with respect to the Z axis to coordinates αa',βa',γa',δa' and coordinates αb',βb',γb',δb' respectively, to obtain coordinates αac”,βac”,γac”,δac”,αbc”,βbc”,γbc”,δbc”,αad”,βad”,γad”,δad”, and coordinates αbd”,βbd”,γbd”,δbd” (additional tilt additional azimuth shooting range) after applying heading. In Example 2, similar to ST04', the additional azimuth angle information φc and φd are set as azimuth angle information φ and predetermined azimuth angle information φ corresponding to the field of view difference. Δ The sum of φc and the azimuth angle information φ Δ The φd obtained by subtracting was adopted. Therefore, in Example 2, a total of nine shooting ranges are calculated, consisting of the azimuth shooting range obtained in ST05, two additional azimuth shooting ranges obtained in *ST05', two additional tilt azimuth shooting ranges, and four additional tilt azimuth shooting ranges.

[0028] <Processing in the information transmission / reception means 10> *ST06': A azimuth range transmission step that transmits the coordinates α”,β”,γ”,δ” (azimuth range) obtained in ST05, the coordinates αc”,βc”,γc”,δc” and αd”,βd”,γd”,δd” (additional azimuth range), the coordinates αa”,βa”,γa”,δa” and αb”,βb”,γb”,δb” (additional tilt azimuth range), the coordinates αac”,βac”,γac”,δac”, the coordinates αbc”,βbc”,γbc”,δbc”, the coordinates αad”,βad”,γad”,δad” and αbd”,βbd”,γbd”,δbd” (additional tilt additional azimuth range), and the position information obtained in ST01 to the GIS application 12 via the internet. *ST07': Image information reception step, which receives simulation image information corresponding to the azimuth shooting range, additional azimuth simulation image information corresponding to two additional azimuth shooting ranges, additional tilt azimuth simulation image information corresponding to two additional tilt azimuth shooting ranges, and additional tilt additional azimuth simulation image information corresponding to four additional tilt additional azimuth shooting ranges transmitted from GIS application 12 via the internet line.

[0029] <Processing in the simulation image display means 11> *ST07”:ST07” is a superimposed simulation image information creation step in which a frame line and auxiliary lines indicating the calculated field of view are added to the simulation image information received in ST07”, two additional azimuth simulation image information, two additional tilt azimuth simulation image information, and four additional tilt additional azimuth simulation image information to create superimposed simulation image information. Furthermore, the nine image data received by ST07' overlap, making it impossible to simply cut and paste them together into a single image. Therefore, by using the overlapping portions to perform image processing, the images are combined into a single panoramic image, creating superimposed simulation image information. Furthermore, there are two methods for achieving panoramic stitching in a web system: a client-side approach and a server-side approach. The former has the advantage of not being affected by other users, but the disadvantages are that the web page size is large, it takes a long time to load, and it depends on the user's machine specifications. The latter has the advantage of not being dependent on the user's machine specifications, but the disadvantage is that waiting occurs if multiple users request it simultaneously. In Example 2, the latter server-side approach was adopted considering use in outdoor areas with unstable communication conditions and use from mobile PCs with limited machine specifications. *ST08':ST07” is a superimposed simulation image display step in which the superimposed simulation image is displayed on the display device D based on the superimposed simulation image information.

[0030] Figure 8 shows examples of simulation images corresponding to the nine shooting ranges in Example 2. By processing ST01 to ST05 and ST04' to ST07', it is possible to obtain nine simulation image information shifted by a predetermined angle from the specified viewpoint. Furthermore, Figure 9 shows a comparison between the superimposed simulation image and the actual captured image in Example 2. By processing ST07" and ST08', superimposed simulation image information is created, and a single superimposed simulation image can be displayed on the display device D. Even when the field of view of camera C is large, the entire shooting range by camera C (the shooting range simulation frame shown by the frame lines) could be contained within a single superimposed simulation image. Furthermore, comparing the superimposed simulation image shown in Figure 9(A) with the actual photograph shown in Figure 9(B), it can be confirmed that although there is some distortion due to the panoramic stitching, the image within the simulation frame of the shooting range is almost identical to the actual photograph.

[0031] <Method for determining additional tilt information and additional azimuth angle information> Figure 10 is a diagram illustrating the method for determining additional tilt information and additional azimuth angle information. In Example 2, in order to create the superimposed simulation image information, as described in ST04’ and ST05’, as additional inclination information, from the inclination information θ, θa obtained by adding a predetermined inclination information θ corresponding to the angular difference of the viewing angle, and θb obtained by subtracting θ from the inclination information θ Δ were adopted, and as additional azimuth information, from the azimuth information φ, φc obtained by adding a predetermined azimuth information φ corresponding to the angular difference of the viewing angle, and φd obtained by subtracting φ from the azimuth information φ Δ were adopted. Δ And in Example 2, as shown in Fig. 10(A), when the vertical viewing angle of the camera C is θf and the vertical viewing angle that can be displayed in the GIS application is θg, the overlapping angle of adjacent simulation images and the angle by which the simulation images at both ends protrude from the vertical viewing angle of the camera C are made equal to an angle θk. Δ Therefore, θk is obtained by Equation 5, and the predetermined inclination information θ is obtained by Equation 6. Equation 5: θk = (3θg - θf) ÷ 4 Δ Equation 6: θ = θg - θk = (θg + θf) ÷ 4 For example, when θf is 90° and θg is 38°, θk is 6°, and θ Δ is 32°. Similarly, when the horizontal viewing angle of the camera C is φf and the horizontal viewing angle that can be displayed in the GIS application is φg, the overlapping angle of adjacent simulation images and the angle by which the simulation images at both ends protrude from the horizontal viewing angle of the camera C are made equal to an angle φk, so φk is obtained by Equation 7, and the predetermined azimuth information φ Δ is obtained by Equation 8. Equation 7: φk = (3φg - φf) ÷ 4 Δ Equation 8: φ = φg - φk = (φg + φf) ÷ 4

Example

Example

[0032] In the visualization system according to Example 2, ST07' receives simulation image information corresponding to the azimuth shooting range, additional azimuth simulation image information corresponding to two additional azimuth shooting ranges, additional tilt azimuth simulation image information corresponding to two additional tilt azimuth shooting ranges, and additional tilt additional azimuth simulation image information corresponding to four additional tilt additional azimuth shooting ranges from the GIS application 12, and performs panoramic stitching using the WEB system in ST07''. However, if the usage environment (communication status, etc.) is not good, problems may occur when performing panoramic stitching because the timing of acquiring segmented images and updating the rendering does not match. Furthermore, as explained in the ST07" section, the overlapping portions of the nine image information received in ST07' are used for image processing to synthesize them into a single panoramic image and create superimposed simulation image information. However, in image processing using overlapping portions, similar parts (feature points) in the images are superimposed to perform panoramic synthesis. Therefore, in cases of landscapes with few feature points, superimposition may not be possible, resulting in synthesis failure. Therefore, in Example 3, when performing the processing (ST06') in the information transmission / reception means 10 of the visualization system according to Example 2, the user is made able to set the timing (acquisition interval) for acquiring the divided images according to the conditions of the usage environment, and when performing the processing (ST07") in the simulation image display means 11, the user is made able to set the thickness of the frame lines and auxiliary lines indicating the field of view for the nine image information received in ST07'.

[0033] Figure 11 is a diagram illustrating the functions of the visualization system according to Embodiment 3 (in particular, the function that allows the user to set the acquisition interval of the divided images and the thickness of the frame lines and auxiliary lines indicating the field of view). Figure 11(A) is an example of a settings window for setting the acquisition interval of the divided images and the thickness of the frame lines and auxiliary lines in the nine simulation images. Figure 11(B) is an example of each simulation image corresponding to the nine shooting ranges (same as Figure 8). Figure 11(C) is an example of a superimposed simulation image displayed after setting the frame lines and auxiliary lines indicating the field of view to be thicker. As shown in Figure 11(A), the settings window has bars for adjusting the image acquisition interval and the thickness of the border and guide lines. To adjust the image acquisition interval and border thickness, first adjust the length of the bar for the desired item, and then click the OK button in the lower left to confirm the setting. The image acquisition interval can be changed to any of 0.5 seconds to 1.5 seconds, and the thickness of the border and guide lines can be changed to any combination of 3 pixels (px) and 1px, 5px and 3px, or 8px and 5px. The image acquisition interval is normally set to 0.5 seconds, but it is better to switch to 1.5 seconds if the communication conditions are poor. Also, the thicker the border and guide lines, the more reliable the panorama stitching becomes, but if they are too thick, the visibility of the screen will be impaired, so it is better to set them to be thinner for images that are easy to overlay and thicker for images that are difficult to overlay.

[0034] The following are variations of Examples 1-3. (Modification 1) In Examples 1 to 3, the latitude, longitude, and altitude directly below the flying object A were input as position information in ST01. Alternatively, an altitude information acquisition means may be added to input the latitude and longitude directly below the flying object A as position information and acquire altitude information based on the input latitude and longitude information. Based on the input latitude information, longitude information, flight altitude information, sensor size information, focal length information, and altitude information acquired by the added altitude information acquisition means, the downward shooting range with camera C pointed directly downwards may be calculated. (Modification 2) In Examples 1 to 3, ST06 transmitted the azimuth shooting range and other location information to the GIS application 12 via the Internet connection, but the azimuth shooting range and other location information may also be transmitted using other wireless or wired connections. (Modification 3) In Examples 1 to 3, the simulation image was displayed on the display device D in ST08. However, in addition to the simulation image, at least one of the following images may be displayed: an image displaying various input information, a map image of the area in which the flying object A is flying, and an image of arrows indicating the coordinates and azimuth angle directly below the viewpoint.

[0035] (Modification 4) In Example 2, in order to create superimposed simulation image information, as described in ST04' and ST05', the tilt information θ is set to a predetermined tilt information θ corresponding to the field of view difference as additional tilt information. Δ The sum of θa and the slope information θ gives θ Δ The θb obtained by subtracting the angle of view is adopted, and as additional azimuth angle information, a predetermined azimuth angle information φ corresponding to the field of view difference is added to the azimuth angle information φ. Δ The sum of φc and the azimuth angle information φ Δ We adopted φd, which is obtained by subtracting φd. However, if the field of view that can be displayed by the GIS application is more than half of the field of view of camera C, then the tilt information θ will be set to a predetermined tilt information θ corresponding to the difference in field of view as additional tilt information. Δ Only θa, which is the sum of the two, is adopted, and as additional azimuth angle information, a predetermined azimuth angle information φ corresponding to the field of view difference is added to the azimuth angle information φ. Δ Even by simply using φc, which is the sum of the two, the following steps allow us to fit the entire shooting range of camera C into a single superimposed simulation image. <Multiple steps to acquire the shooting range> After obtaining coordinates α',β',γ',δ' (tilt shooting range) using ST04 in Example 1 and coordinates αa',βa',γa',δa' (additional tilt shooting range) using the same method as ST04' in Example 2, coordinates α”,β”,γ”,δ” (azimuth angle shooting range) using ST05 in Example 1 and coordinates αc”,βc”,γc”,δc” (additional azimuth angle shooting range), coordinates αa”,βa”,γa”,δa” (additional tilt azimuth angle shooting range), and coordinates αac”,βac”,γac”,δac” (additional tilt additional azimuth angle shooting range) are obtained using the same method as ST05' in Example 2. <Steps for transmitting azimuth angle, shooting range, etc.> Similar to ST06' in Example 2, the azimuth shooting range, additional azimuth shooting range, additional tilt azimuth shooting range, and additional tilt azimuth shooting range, along with the position information obtained in ST01, are transmitted to the GIS application 12 via the internet. <Image information receiving step> Similar to ST07' in Example 2, the system receives simulation image information corresponding to the azimuth shooting range, additional azimuth simulation image information corresponding to the additional azimuth shooting range, additional tilt azimuth simulation image information corresponding to the additional tilt azimuth shooting range, and additional tilt additional azimuth simulation image information corresponding to the additional tilt additional azimuth shooting range, transmitted from the GIS application 12 via the internet connection. <Steps for creating superimposed simulation image information> Similar to ST07” in Example 2, superimposed simulation image information is created based on the received simulation image information, additional azimuth simulation image information, additional tilt azimuth simulation image information, and additional tilt additional azimuth simulation image information. <Superimposed Simulation Image Display Step> Similar to ST08' in Example 2, the superimposed simulation image is displayed on the display device D based on the created superimposed simulation image information. In modification 4, as shown in Figure 10(B), when the vertical field of view of camera C is θf and the vertical field of view that can be displayed by the GIS application is θg, the overlapping angle of adjacent simulation images and the angle at which the simulation images at both ends extend beyond the vertical field of view of camera C are equal to the angle θk. Therefore, θk is obtained using equation 9, and the predetermined slope information θ Δ This can be found using Equation 10. Equation 9: θk = (2θg - θf) ÷ 3 Equation 10: θ Δ =θg-θk=(θg+θf)÷3 For example, if θf is 90° and θg is 60°, then θk is 10°, θ Δ The angle becomes 50°. Similarly, when the horizontal field of view of camera C is φf and the horizontal field of view that can be displayed in the GIS application is φg, the overlapping angle of adjacent simulation images and the angle at which the simulation images at both ends extend beyond the horizontal field of view of camera C are equal to the angle φk. Therefore, φk can be calculated using Equation 11, and the predetermined azimuth angle information φ ΔThis can be found using Equation 12. Equation 11: φk = (2φg - φf) ÷ 3 Equation 12:φ Δ =φg-φk=(φg+φf)÷3 [Explanation of symbols]

[0036] 1. Means for inputting information on the position of a flying object. 2. Means for inputting information on the altitude of a flying object. 3. Sensor size information input means 4. Focal length information input means 5. Tilt information input means 6. Azimuth angle information input means 7. Direct downward shooting range calculation means 8. Inclined shooting range calculation means 9. Azimuth angle shooting range calculation means 10 Information transmission and reception means 11 Simulation image display means 12 GIS Applications A. Flying object C. Camera D. Display device E. Viewpoint position a. Sensor size b. Focal length c. Distance to subject d Distance from the center to the edge α, β, γ, δ Coordinates of the four corners of the area directly below the camera. α', β', γ', δ': Coordinates of the four corners of the tilted shooting range. α”,β”,γ”,δ” Coordinates of the four corners of the azimuth angle shooting range αa',βa',γa',δa': Coordinates of the four corners of the additional tilt imaging range. αb', βb', γb', δb': Coordinates of the four corners of the additional tilt imaging range. αc”,βc”,γc”,δc” Coordinates of the four corners of the additional azimuth angle imaging range αd”,βd”,γd”,δd” Coordinates of the four corners of the additional azimuth angle shooting range αa”,βa”,γa”,δa” Coordinates of the four corners of the additional tilt azimuth shooting range αb”,βb”,γb”,δb” Coordinates of the four corners of the additional tilt azimuth shooting range αac”,βac”,γac”,δac” Coordinates of the four corners of the shooting range (additional tilt and azimuth angles) αbc”,βbc”,γbc”,δbc” Coordinates of the four corners of the shooting range (additional tilt and azimuth angle) αad”,βad”,γad”,δad” Coordinates of the four corners of the shooting range (additional tilt and azimuth angle) αbd”,βbd”,γbd”,δbd” Additional tilt and azimuth angle coordinates of the four corners of the shooting range θ: Tilt information θa, θb: Additional tilt information θf: Vertical field of view of camera C θg: Vertical field of view that can be displayed in a GIS application. θk is the overlapping angle of adjacent simulation images. Δ Predetermined inclination information φ Azimuth angle information φc,φd Additional azimuth angle information φf Horizontal field of view of camera C φg Horizontal field of view that can be displayed in a GIS application φk is the overlapping angle of adjacent simulation images. Δ Predetermined azimuth angle information

Claims

1. A system for visualizing images captured by a camera mounted on a flying object, which, by identifying the position and altitude of the flying object and the sensor size, focal length, azimuth angle, and tilt of the camera mounted on the flying object, can simulate and display images captured by the camera on a display device, A means for inputting the position information of the aforementioned flying object, A means for inputting flight altitude information of the aforementioned flying object, A sensor size information input means for inputting the sensor size information of the aforementioned camera, A focal length information input means for inputting the focal length information of the aforementioned camera, A tilt information input means for inputting tilt information of the aforementioned camera, A means for inputting azimuth information from the aforementioned camera, Assuming that both the tilt information and the azimuth angle information are 0°, a downward shooting range calculation means calculates the downward shooting range when the camera is pointed directly downwards, based on the position information input by the flying object position information input means, the flight altitude information input by the flying object flight altitude information input means, the sensor size information input by the sensor size information input means, and the focal length information input by the focal length information input means. A tilt shooting range calculation means calculates the tilt shooting range after applying the tilt information, based on the aforementioned direct downward shooting range and the tilt information input means. An azimuth angle shooting range calculation means calculates the azimuth angle shooting range after applying the tilt information and azimuth angle information based on the tilt shooting range and the azimuth angle information input means, Information transmission and reception means for transmitting the aforementioned azimuth angle shooting range and the position information of the flying object to a GIS application and receiving simulation image information from the GIS application, The system includes a simulation image display means that displays a simulation image on the display device based on the simulation image information received by the information transmission and reception means. A system for visualizing images captured by a camera mounted on a flying object, characterized by the following features.

2. The aforementioned location information includes latitude information, longitude information, and elevation information. The image visualization system for images captured by a camera mounted on a flying object, as described in claim 1.

3. The aforementioned location information consists of latitude information and longitude information. The system further comprises an elevation information acquisition means that acquires elevation information based on the latitude information and longitude information, The downward shooting range calculation means calculates the downward shooting range when the camera is pointed directly downwards, based on the position information, the flight altitude information, the sensor size information, the focal length information, and the altitude information acquired by the altitude information acquisition means. The image visualization system for images captured by a camera mounted on a flying object, as described in claim 1.

4. The tilt shooting range calculation means calculates the additional tilt shooting range after applying the additional tilt information, based on the tilt shooting range, the directly downward shooting range, and the additional tilt information obtained by adding predetermined tilt information to the tilt information. The azimuth angle shooting range calculation means calculates the additional azimuth angle shooting range after applying the tilt information and the additional azimuth angle information, based on the azimuth angle shooting range, the tilt shooting range, and the additional azimuth angle information obtained by adding predetermined azimuth angle information to the azimuth angle information; calculates the additional tilt azimuth angle shooting range after applying the additional tilt information and the additional azimuth angle information, based on the additional tilt shooting range and the additional azimuth angle information; and calculates the additional tilt additional azimuth angle shooting range after applying the additional tilt information and the additional azimuth angle information, The information transmission and reception means transmits the additional azimuth angle shooting range, the additional tilt azimuth angle shooting range, and the additional tilt additional azimuth angle shooting range to the GIS application, and receives the simulation image information, additional azimuth angle simulation image information, additional tilt azimuth angle simulation image information, and additional tilt additional azimuth angle simulation image information from the GIS application, respectively. The simulation image display means displays a superimposed simulation image on the display device based on the simulation image information, the additional azimuth simulation image information, the additional tilt azimuth simulation image information, and the additional tilt additional azimuth simulation image information. A system for visualizing images captured by a camera mounted on a flying object, as described in any one of the features of 1 to 3.

5. The simulation image display means adds frame lines and auxiliary lines indicating the field of view to the simulation image information, the additional azimuth simulation image information, the additional tilt azimuth simulation image information, and the additional tilt azimuth simulation image information, and displays the superimposed simulation image on the display device by utilizing the characteristic that the frame lines and auxiliary lines act as markers indicating overlapping portions. The image visualization system for images captured by a camera mounted on a flying object, as described in claim 4.

Images