Gas monitoring method, gas monitoring device, gas monitoring system, and gas monitoring program

Abstract

To provide a gas monitoring method, gas monitoring device, gas monitoring system and gas monitoring program which can properly monitor gas being a monitoring object by using image data acquired by an imaging device mounted on a UAV.SOLUTION: A gas monitoring method comprises: a step of acquiring a plurality of velocity vectors obtained by performing optical flow estimation processing on a plurality of pieces of image data obtained by performing imaging at a plurality of times by an imaging device mounted on a UAV; a step of acquiring a downwash velocity u of the UAV based on at least one variable indicating the flight state of the UAV; and an extraction step of extracting a gas velocity vector having the magnitude within a range determined on the basis of the downwash velocity u from the plurality of velocity vectors.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a gas monitoring method, a gas monitoring device, a gas monitoring system, and a gas monitoring program. [Background technology]

[0002] Gases may be monitored based on image data captured by an imaging device such as a camera.

[0003] Patent Document 1 discloses a black smoke detection system that uses optical flow estimation. This black smoke detection system uses two consecutively captured time-series images of black smoke emitted from a flare stack, and calculates optical flow estimation between the two images to determine local velocity vectors within the images that result from the movement of gas. Of the velocity vectors thus determined, those whose magnitude and direction are within a predetermined range are extracted as velocity vectors that indicate the movement of black smoke. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4266535 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the black smoke detection system of Patent Document 1 uses a fixed imaging device (camera), so the distance between the imaging device and the object being imaged (such as a chimney) is constant, and a scale (such as a chimney) that serves as a reference for velocity calculation exists within the imaging range, so it is possible to calculate the magnitudes (absolute values of velocity) of multiple velocity vectors obtained from optical flow estimation.

[0006] On the other hand, when using an imaging device mounted on a UAV (unmanned aerial vehicle), the height and position of the UAV change, so the distance between the UAV and the object being imaged is not constant. Also, the scale used as a reference for velocity calculation does not necessarily exist within the image. Therefore, the magnitude (absolute value of velocity) of each of the multiple velocity vectors cannot be calculated by optical flow estimation and is unknown. For this reason, it is difficult to appropriately extract a vector indicating the movement of the gas being monitored from the multiple velocity vectors obtained by optical flow estimation.

[0007] In view of the above circumstances, at least one embodiment of the present invention aims to provide a gas monitoring method, a gas monitoring device, a gas monitoring system, and a gas monitoring program that can appropriately monitor the gas being monitored using image data acquired by an imaging device mounted on a UAV. [Means for solving the problem]

[0008] In accordance with at least one embodiment of the present invention, a gas monitoring method includes: A step of acquiring a plurality of velocity vectors obtained by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; Obtaining a downwash velocity u of the UAV based on at least one variable indicative of a flight state of the UAV; an extraction step of extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Equipped with.

[0009] Furthermore, the gas monitoring device according to at least one embodiment of the present invention comprises: a velocity vector acquisition unit that acquires a plurality of velocity vectors by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; A downwash speed acquisition unit that acquires a downwash speed u of the UAV based on at least one variable indicating a flight state of the UAV; an extractor that extracts a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Equipped with.

[0010] In addition, the gas monitoring system according to at least one embodiment of the present invention comprises: A UAV equipped with an imaging device, an optical flow estimation processing unit configured to perform optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times by the imaging device, and calculate a plurality of velocity vectors; a gas monitor as described above configured to extract the gas velocity vector from the plurality of velocity vectors; Equipped with.

[0011] In addition, the gas monitoring program according to at least one embodiment of the present invention includes: On the computer, A procedure for acquiring a plurality of velocity vectors obtained by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; obtaining a downwash velocity u of the UAV based on at least one variable indicative of a flight state of the UAV; extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Let it run. [Effects of the Invention]

[0012] According to at least one embodiment of the present invention, there are provided a gas monitoring method, a gas monitoring device, a gas monitoring system, and a gas monitoring program that can appropriately monitor the gas to be monitored using image data acquired by an imaging device mounted on a UAV. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a schematic diagram of a UAV constituting a gas monitoring system according to one embodiment. FIG. [Figure 2] FIG. 1 is a schematic diagram of a gas monitoring system according to one embodiment. [Figure 3] 1 is a flowchart of a gas monitoring method according to one embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of an image captured by an imaging device. [Figure 5] 10 is an example of an image in which a plurality of velocity vectors calculated by optical flow estimation are superimposed on an image captured by an imaging device. [Figure 6] 1 is a graph showing an example of the correlation between the flight height H and flight speed v of a UAV and the downwash speed of the UAV. [Figure 7] 1 is a chart showing an example of a correlation between the flight height H and flight speed v of a UAV and the downwash speed of the UAV. [Figure 8] FIG. 10 is a diagram for explaining the procedure for acquiring the downwash speed of a UAV. [Figure 9] FIG. 10 is a diagram for explaining the procedure for obtaining the corrected downwash speed of a UAV. [Figure 10] 1 is a graph showing an example of the correlation between height from the ground surface and wind speed. [Figure 11] 10 is an example of an image in which the gas velocity vector VG extracted in the extraction step is superimposed on an image captured by an imaging device. [Figure 12] 10 is a graph for explaining the procedure for extracting a gas velocity vector VG. [Figure 13] 10 is a histogram showing an example of a frequency distribution of the magnitudes of a plurality of velocity vectors. [Figure 14] 10 is a histogram showing an example of a frequency distribution of the directions of a plurality of velocity vectors. [Figure 15] FIG. 10 is a diagram for explaining a reframe process of image data. [Figure 16] FIG. 10 is a diagram for explaining a reframe process of image data. [Figure 17] FIG. 10 is a diagram for explaining a reframe process of image data. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, several embodiments of the present invention will be described with reference to the accompanying drawings. However, the dimensions, materials, shapes, relative arrangements, etc. of components described as embodiments or shown in the drawings are merely illustrative examples and are not intended to limit the scope of the present invention.

[0015] (Gas monitoring system / gas monitoring device configuration) FIG. 1 is a schematic diagram of a UAV that constitutes a gas monitoring system according to some embodiments, and FIG. 2 is a schematic diagram of a gas monitoring system according to some embodiments.

[0016] Gas monitoring devices and gas monitoring systems according to some embodiments are devices and systems capable of identifying a gas region (a region where a monitored gas is flowing) within an image based on image data obtained by photographing a field using an imaging device (such as a camera) mounted on a UAV (unmanned aerial vehicle). This gas monitoring device / gas monitoring system can detect and monitor the monitored gas (e.g., gas leak detection from a facility, gas outflow monitoring, etc.) within the field region being photographed. The type of monitored gas is not particularly limited. The following description will be given of a case where carbon dioxide (CO2) gas is the monitored gas, as an example.

[0017] As shown in Figures 1 and 2, a gas monitoring system 100 according to one embodiment includes a UAV 10, an imaging device 20 mounted on the UAV 10, and a gas monitoring device 50 for processing signals representing measurement data and image data acquired by the UAV 10 and / or the imaging device 20, etc.

[0018] As shown in Figure 1, the UAV 10 includes a UAV body 12 and multiple propellers 14 (typically three or more propellers 14) attached to the UAV body 12. Each of the multiple propellers 14 is configured to be rotationally driven by a motor. The flight height, flight speed, flight direction, etc. of the UAV 10 can be controlled by adjusting the current value of each motor that drives the multiple propellers 14. A signal indicating the current value of each motor may be sent to the gas monitoring device 50 via wireless communication.

[0019] When the UAV 10 flies, downwash D (a downward airflow) is generated by the multiple propellers 14. The downwash D flowing downward from the UAV 10 collides with the ground surface G, is turned horizontally, and flows radially along the ground surface G from a position Pc directly below the UAV 10 on the ground surface.

[0020] The UAV 10 may be provided with an altimeter (not shown) for measuring the flight height H of the UAV 10, or a speedometer (not shown) for measuring the flight speed v of the UAV 10. A signal indicating the flight height H of the UAV 10 obtained by the altimeter and / or a signal indicating the flight speed v of the UAV 10 obtained by the speedometer may be transmitted to the gas monitoring device 50 via wireless communication.

[0021] The imaging device 20 is capable of continuously capturing images of the field below the UAV 10 in a time series while the UAV 10 is flying. Image data representing the images captured by the imaging device 20 may be transmitted to the gas monitoring device 50 via wireless communication.

[0022] The imaging device 20 may be an infrared camera equipped with a filter that selectively transmits infrared light of a wavelength absorbed by the gas being monitored (4.3 μm for CO2 gas). By using an infrared camera that selectively filters and images specific wavelengths in this way, only the specific gas in the fluid (gas) in the field of view is imaged in the image data.

[0023] The gas monitoring device 50 is configured to process measurement data such as flight height H or flight speed v sent from the UAV 10 or image data sent from the imaging device 20. As shown in FIG. 2 , the gas monitoring device 50 according to one embodiment includes an image acquisition unit 52, a reframe processing unit 54, an optical flow processing unit 56, a velocity vector acquisition unit 58, a noise component removal unit 60, a flight state acquisition unit 62, a downwash speed acquisition unit 64, an extraction unit 66, and a display image generation unit 68.

[0024] The gas monitoring device 50 includes a computer equipped with a processor (such as a CPU or GPU), a storage device (such as a memory device; RAM), an auxiliary storage unit, and an interface. The gas monitoring device 50 receives signals from the UAV 10 or the imaging device 20 via the interface. The processor is configured to process the signals received in this manner. The processor is also configured to process programs loaded into the storage device. This realizes the functions of the above-mentioned functional units (image acquisition unit 52 to display image generation unit 68).

[0025] The processing contents of the gas monitoring device 50 are implemented as programs executed by a processor. The programs may be stored in an auxiliary storage unit. When the programs are executed, they are loaded into a storage device. The processor reads the programs from the storage device and executes the instructions contained in the programs.

[0026] The image acquisition unit 52 is configured to acquire a plurality of image data obtained by capturing images at a plurality of times using the imaging device 20 mounted on the UAV 10.

[0027] The reframe processing unit 54 is configured to perform reframe processing on the plurality of image data acquired by the image acquisition unit 52 as necessary.

[0028] The optical flow processing unit 56 is configured to perform optical flow estimation processing on a plurality of image data acquired by the image acquisition unit 52 and subjected to reframe processing as necessary.

[0029] The velocity vector acquisition unit 58 is configured to acquire a plurality of velocity vectors calculated by the optical flow processing in the optical flow processing unit 56 .

[0030] The noise component removal unit 60 is configured to remove noise components from the plurality of velocity vectors acquired by the velocity vector acquisition unit 58 as necessary.

[0031] The flight state acquisition unit 62 is configured to acquire a measurement value of at least one variable indicating a flight state of the UAV 10. In one embodiment, the flight state acquisition unit 62 is configured to acquire a measurement value of a flight height H of the UAV 10 and / or a flight speed v of the UAV 10 as a variable indicating a flight state of the UAV 10.

[0032] The downwash speed acquisition unit 64 is configured to acquire the downwash speed u of the UAV 10 based on at least one variable indicating the flight state of the UAV 10. This downwash speed u is the speed of the downwash flow in a direction along the ground surface G in the vicinity of a position Pc directly below the UAV 10 on the ground surface.

[0033] The extraction unit 66 is configured to extract, from the plurality of velocity vectors acquired by the velocity vector acquisition unit 58, a gas velocity vector having a magnitude within a range determined based on the downwash velocity u.

[0034] The display image generating unit 68 is configured to generate image data (monitoring image data) showing information about the gas to be monitored (for example, information about a gas region where the gas exists, the distribution of gas velocity vectors, etc.) based on the gas velocity vector extracted by the extracting unit 66. The image data generated by the display image generating unit 68 may be output to a display unit 70 for displaying an image.

[0035] The gas monitoring device 50 may also include a memory unit 69 for storing the correlation between a variable indicating the flight state of the UAV 10 and the downwash speed of the UAV 10, the correlation between the height from the ground and the wind speed, etc. The memory unit 69 may include a storage device (memory device; RAM, etc.) or auxiliary storage device of a computer constituting the gas monitoring device 50, or may include a storage device connected to the computer via a network.

[0036] (Gas monitoring method flow) The flow of a gas monitoring method according to several embodiments will be described below. Note that, although the following describes a case where CO2 gas, which is a gas to be monitored, is monitored using the gas monitoring device 50 described above, some or all of the procedures described below may be performed using other devices or manually, and the gas to be monitored may be a gas other than CO2.

[0037] FIG. 3 is a flowchart of a gas monitoring method according to one embodiment.

[0038] In one embodiment, first, an imaging device 20 mounted on the UAV 10 captures images at multiple times (S100). The imaging device 20 may continuously capture images of the field below the UAV 10 in time series while the UAV 10 is flying. FIG. 4 is a diagram showing an example of an image captured by the imaging device 20. Image I0 shown in FIG. 4 captures the field below the UAV 10 during flight, and shows the ground G and CO2 gas cylinders B1 and B2 installed on the ground G. In this embodiment, the imaging device 20 is an infrared camera equipped with a filter that selectively transmits infrared light of a wavelength (4.3 μm) absorbed by the CO2 gas being monitored.

[0039] The image acquisition unit 52 acquires data of a plurality of images (a plurality of image data) captured by the imaging device 20 in this manner.

[0040] Next, the reframe processing unit 54 may perform reframe processing on the multiple image data acquired by the image acquisition unit 52 in step S100, as necessary (S200). The reframe processing can be performed to remove noise due to the wobbling of the UAV 10 before performing the optical flow estimation processing in step S300, which will be described later. This can further improve the accuracy of extracting the gas velocity vector in step S800, which will be described later. The reframe processing in step S200 will be described later.

[0041] Next, the optical flow processing unit 56 performs optical flow estimation processing (S300) on the multiple image data acquired by the image acquisition unit 52 in step S100 or on the multiple image data that have been reframed in step S200. In optical flow estimation, two time-series image data that are continuous in time are used to calculate multiple velocity vectors that respectively indicate the movement of multiple pixels in the image between the two sets of image data. The optical flow estimation processing can be performed, for example, using the method described in Patent Document 1. Image I1 shown in FIG. 5 is an example of an image in which multiple velocity vectors calculated by optical flow estimation are superimposed on an image (captured by an imaging device) that was the subject of the optical flow estimation processing in step S300.

[0042] In this embodiment, an infrared camera is used that selectively filters wavelengths absorbed by CO2 gas for imaging, so that only CO2 gas is imaged in the image data of the fluid (gas) in the field of view and is also the subject of calculation for optical flow estimation.

[0043] Next, the velocity vector acquisition unit 58 acquires the multiple velocity vectors calculated in the optical flow estimation process in step S300 (S400).

[0044] Next, the noise component removal unit 60 removes noise components from the multiple velocity vectors acquired in step S400 as necessary (S500). This improves the accuracy of extracting gas velocity vectors in step S800, which will be described later. The removal of noise components in step S500 will be described later.

[0045] Next, the flight state acquisition unit 62 acquires (S600) a measurement value of at least one variable indicating the flight state of the UAV 10. In step S600, the measurement value of the flight height H of the UAV 10 and / or the flight speed v of the UAV 10 may be acquired as a variable indicating the flight state of the UAV 10.

[0046] Next, the downwash speed acquisition unit 64 acquires the downwash speed u of the UAV 10 based on at least one variable indicating the flight state of the UAV 10 acquired in step S600 (S700). The correlation between the at least one variable indicating the flight state of the UAV 10 and the downwash speed u of the UAV 10 may be acquired in advance and stored in the storage unit 69. In step S700, the downwash speed acquisition unit 64 may acquire the downwash speed u corresponding to the measurement value based on the measurement value related to the at least one variable indicating the flight state of the UAV 10 acquired in step S600 and the above correlation acquired from the storage unit 69.

[0047] The at least one variable indicating the flight state of the UAV 10 may include the flight height H of the UAV 10 and the flight speed v of the UAV 10. Here, Figures 6 and 7 are examples of graphs or charts showing the correlation between the flight height H of the UAV 10, the flight speed v of the UAV 10, and the downwash speed u of the UAV 10, respectively.

[0048] In one embodiment, the downdraft velocity u may be obtained based on the correlation shown in FIG. 6. That is, the flight height H of the UAV 10, the flight velocity v of the UAV 10, and the downdraft velocity u of the UAV 10 have a predetermined correlation as shown in FIG. 6, for example. According to this correlation, at the same flight velocity v, the higher the flight height, the greater the downdraft velocity u, and the greater the flight velocity, the greater the downdraft velocity u (however, in FIG. 6, the flight velocities v are v0 < v1 < v2). This correlation is obtained in advance by a test using the UAV 10 and stored in the storage unit 69. Then, the downdraft velocity acquisition unit 64 can obtain the downdraft velocity u of the UAV 10 by applying the measured values of the flight height H and flight velocity v of the UAV 10 obtained in step S600 to the correlation obtained from the storage unit 69.

[0049] In one embodiment, the downdraft velocity u at the position P1 may be obtained based on the correlation shown in FIG. 7. The position P1 is a position near the position Pc directly below the UAV 10. That is, as shown in FIG. 7, for example, in addition to the flight height H of the UAV 10 and the flight velocity v of the UAV 10, the downdraft velocity u has a predetermined correlation with the distance r from the position Pc directly below the UAV 10 to P1 on the ground surface and the angle θ from the reference azimuth of the position P1 when centered on the position Pc on the ground surface (see FIG. 8). This correlation is obtained in advance by a test using the UAV 10 and stored in the storage unit 69. Then, the downdraft velocity acquisition unit 64 can obtain the downdraft velocity u of the UAV 10 by applying the measured values of the flight height H and flight velocity v of the UAV 10 obtained in step S600, and the measured values of the above-mentioned distance r and angle θ, to the correlation obtained from the storage unit 69. The measured values of the above-mentioned distance r and angle θ may be input to the gas monitoring device 50 by the measurer via an input device (such as a keyboard, mouse, or touch panel).

[0050] Figure 8 is a diagram for explaining the procedure for obtaining the downwash speed u, and shows the distance r from position Pc directly below UAV10 on the ground surface to P1, and the angle θ from the reference direction of position P1 when position Pcwo is the center on the ground surface.

[0051] In some embodiments, in step S700, a corrected downwash speed ug may be obtained, which is a downwash speed that takes into account the wind speed Wg at the ground surface G. The corrected downwash speed ug may be calculated based on the downwash speed u obtained as described above and the wind speed Wg at the ground surface.

[0052] The corrected downwash speed ug (vector) can be obtained as a resultant vector of the downwash speed u (vector) calculated as described above and the wind speed Wg (vector) at the ground surface (see FIG. 9). Here, FIG. 9 is a diagram for explaining the procedure for obtaining the corrected downwash speed ug.

[0053] The wind speed Wg (vector) at the ground surface may be obtained based on the wind speed W1 (vector) at the flight height of the UAV 10 and the correlation between the height from the ground surface and the wind speed. FIG. 10 is a graph showing an example of the correlation between the height from the ground surface and the wind speed. The wind speed and the height from the ground surface have a correlation, for example, as shown in FIG. 10 , via a wind speed correction coefficient α. That is, in the example shown in FIG. 10 , the wind speed correction coefficient α (where the wind speed correction coefficient α at the ground surface is set to zero) is expressed as a linear function with respect to the flight height of the UAV 10. From this function, the relationship between the wind speed Wg at the ground surface and the wind speed W1 at flight height H1 can be expressed as Wg = α1 × W1, where α1 is the wind speed correction coefficient at flight height H1. The above-described correlation between the height from the ground surface and the wind speed may be obtained in advance and stored in the storage unit 69.

[0054] The above correlation between height from the ground surface and wind speed is based on the assumption that the wind direction on the ground surface (field) is the same as the wind direction at the flight height of the UAV 10, and the rate of change in wind speed in the height direction is the same regardless of the location on the ground surface (field). It may also be obtained by acquiring wind speed W at multiple points at any point in the field at flight height H of UAV10.

[0055] The wind speed W1 (vector) at the flight height of the UAV 10 may be obtained from the motor current value (the current value of the motor for rotating the propellers 14) of the UAV 10. The UAV 10 can maintain a constant flight speed by controlling the motor current value of each propeller 14 for a specified flight speed (and direction), and can maintain a constant flight speed even when there is wind in the surroundings by controlling the motor load of each propeller 14. Therefore, the wind speed (magnitude of the wind speed W1 (vector)) and wind direction (direction of the wind speed W1 (vector)) can be determined from the motor current value of each propeller 14 of the UAV 10.

[0056] Alternatively, the wind speed W1 at the flight height of the UAV 10 may be obtained based on the measurement results of an anemometer (not shown) and a wind vane (not shown) configured to measure wind speed and wind direction and provided on the UAV 10. Note that signals indicating the measurement results of the anemometer and the wind vane may be sent to the gas monitoring device 50.

[0057] That is, the downwash speed acquisition unit 64 may acquire a motor current value of the UAV 10 and calculate the wind speed W1 (vector) at the flight height of the UAV 10 based on the motor current value. Alternatively, the downwash speed acquisition unit 64 may acquire measurement results from an anemometer and a wind vane provided on the UAV 10 and calculate the wind speed W1 (vector) at the flight height of the UAV 10 based on the measurement values. The downwash speed acquisition unit 64 may then acquire a correlation between height from the ground surface and wind speed from the storage unit 69 and acquire the wind speed Wg at the ground surface based on the correlation and the wind speed W1 at the flight height of the UAV 10. The downwash speed acquisition unit 64 may acquire a corrected downwash speed ug based on the wind speed Wg at the ground surface acquired in this manner and the downwash speed u described above.

[0058] Next, the extraction unit 66 extracts (S800) a gas velocity vector V G having a magnitude within a range determined based on the downwash velocity u (or the corrected downwash velocity ug) obtained in step S700 from the multiple velocity vectors obtained in step S400 (multiple velocity vectors obtained as a result of the optical flow estimation process; see FIG. 5). Note that if noise components have been removed in step S500, in step S800, a gas velocity vector V G having a magnitude within a range determined based on the downwash velocity u (or the corrected downwash velocity ug) obtained in step S700 is extracted from the multiple velocity vectors from which noise components have been removed in step S500. Here, image I2 shown in FIG. 11 is an example of an image in which the gas velocity vector V G extracted in step S800 is superimposed on the image (captured by an imaging device) that was the target of the optical flow estimation process in step S300.

[0059] In one embodiment, in step S800, for example, the velocity vector at the position of the downwash velocity u (or the corrected downwash velocity ug) acquired in step S700 among the multiple velocity vectors acquired in step S400 is considered to have a velocity magnitude equal to that of the downwash velocity u (or the corrected downwash velocity ug). Then, among the multiple velocity vectors, those having a velocity magnitude close to that of the downwash velocity u (or the corrected downwash velocity ug) are extracted.

[0060] As an example of how to extract the gas velocity vector VG, a case will be described where a gas velocity vector VG having a magnitude within a range determined based on the corrected downwash speed ug is extracted. In one example, a velocity vector having a magnitude between a velocity vector (VG_max) having the same magnitude as the corrected downwash speed ug and a velocity vector (VG_min) having the same magnitude as a velocity umin, which is smaller than the corrected downwash speed ug, is extracted as the gas velocity vector VG (see FIG. 12). For example, from the multiple velocity vectors, those having a magnitude equal to or greater than 0.25×ug and equal to or less than 1×ug are extracted. Note that FIG. 12 is a graph for explaining the procedure for extracting the gas velocity vector VG.

[0061] 11 shows the distribution of the extracted gas velocity vectors V and the gas region A G where the extracted gas velocity vectors V exist. The gas velocity vectors V G are vectors that indicate the flow of CO2 gas ejected from the CO2 gas cylinder.

[0062] Based on the gas velocity vector V G extracted in step S800, the display image generating unit 68 may generate image data (monitoring image data; for example, data of image I2 shown in FIG. 11 ) showing information about the gas to be monitored (for example, information about the gas region where the gas exists, the distribution of the gas velocity vector, etc.) based on the gas velocity vector extracted by the extracting unit 66. The image data generated in this manner may be sent to the display unit 70 and output (displayed) on the display unit 70 (S900).

[0063] In the optical flow estimation process in step S300, the direction and relative magnitude of multiple velocity vectors can be calculated, but the absolute magnitude (absolute value of velocity) cannot be calculated. This is because the distance between the image capture device 20 mounted on the UAV 10 and the subject is not constant, and the scale that serves as the basis for velocity calculation does not necessarily exist in the image. In this regard, in the above-described embodiment, the downwash velocity u of the UAV 10 can be obtained based on at least one variable indicating the flight state of the UAV 10 (e.g., flight height H or flight speed v). Therefore, among the multiple velocity vectors having unknown magnitudes obtained by optical flow estimation, a velocity vector having a magnitude within a range determined based on the downwash velocity u (or a magnitude within a range determined based on a corrected downwash velocity ug obtained from the downwash velocity u) can be identified as a gas velocity vector VG (gas region) indicating the movement speed and direction of the gas to be monitored, based on the magnitude of the vector indicating the downwash velocity u of the UAV 10. The gas velocity vector VG extracted in this manner can be used to appropriately monitor the gas.

[0064] In one embodiment, as described above, the downwash speed u of the UAV 10 has a correlation with the flight height H and flight speed v of the UAV. Therefore, based on the flight height H of the UAV 10 and the flight speed v of the UVA, the downwash speed u of the UAV 10 can be appropriately acquired from the correlation therewith.

[0065] Furthermore, if a wind separate from the downwash is blowing on the ground, the downwash flow on the ground may be affected by that wind. In this regard, in one embodiment, as described above, a corrected downwash speed ug is calculated based on the downwash speed u obtained based on variables indicating the flight state of the UAV 10 (flight height H and flight speed v) and the wind speed Wg at the ground, taking the wind speed Wg into consideration. Then, from the multiple vectors obtained by optical flow estimation, a gas velocity vector VG having a magnitude within a range determined based on the corrected downwash speed ug is extracted. This allows for more appropriate extraction of the gas velocity vector indicating the movement speed and direction of the gas being monitored.

[0066] (Regarding noise component removal) The noise component removal process in step S500 will be described. In some embodiments, in step S500, noise component velocity vectors may be removed from the multiple vectors obtained in step S400 (multiple vectors calculated in the optical flow estimation process in step S300) based on the frequency distribution of the magnitudes of the multiple velocity vectors or the frequency distribution of the directions of the multiple velocity vectors.

[0067] The velocity vectors of noise components tend to have a predetermined pattern in terms of magnitude or orientation. In the above-described embodiment, the noise components can be removed from the multiple velocity vectors obtained by optical flow estimation based on the frequency distribution of the magnitude or orientation of the multiple vectors. Furthermore, in step S800, the gas velocity vector is extracted based on the downwash velocity u from the multiple vectors from which the noise components have been removed, thereby improving the accuracy of extracting the gas velocity vector V.

[0068] Here, FIG. 13 is a histogram showing an example of the frequency distribution of the magnitudes of the multiple velocity vectors obtained in step S400, and FIG. 14 is a histogram showing an example of the frequency distribution of the directions of the multiple velocity vectors obtained in step S400.

[0069] In one embodiment, in step S500, velocity vectors whose frequency in the frequency distribution of the magnitudes of the multiple velocity vectors (see FIG. 13) falls within a range of magnitudes equal to or greater than a first threshold (threshold 1 in FIG. 13) and whose frequency in the frequency distribution of the orientations of the multiple velocity vectors (see FIG. 14) falls within a range of orientations equal to or less than a second threshold (threshold 2 in FIG. 14) are removed as noise components from the multiple velocity vectors obtained in step S400.

[0070] The velocity vectors of noise components (e.g., velocity vectors indicating the wobble of the UAV 10) may have a pattern in which they have approximately the same magnitude and vary in orientation. In the above-described embodiment, vectors that belong to a relatively large frequency range in the magnitude frequency distribution and a relatively small frequency range in the orientation frequency distribution (i.e., vectors that match the above-described pattern) can be appropriately removed as noise components.

[0071] The average value or median value on the vertical axis of each histogram may be used as the first or second threshold value.

[0072] In addition, in both the frequency distribution of the magnitude (see Figure 13) and the frequency distribution of the direction (see Figure 14) of multiple velocity vectors, a velocity vector whose frequency is greater than a threshold (for example, a vector included in both the peak indicated by P1 in Figure 13 and the peak indicated by P2 in Figure 14) is likely to be a velocity vector of the gas being monitored.

[0073] (About reframe processing) The reframe processing in step S200 will now be described. Here, FIGS. 15 to 17 are diagrams for explaining the reframe processing of a plurality of image data. Image I shown in FIGS. 15 to 17 11 ~I 13are images captured in time series by the imaging device 20 mounted on the UAV 10. In the reframe process, the aspect ratio (a:b) of the image is adjusted (i.e., the area of the image outside of frame F is trimmed) so that the reference point P0 is located at a predetermined position (for example, the center position of frame F) within frame F (display area of the image) for the image data of each image captured in time series. In the illustrated example, the positions of the CO2 gas cylinders B1 and B2 are used as the reference point P0, and each image I 11 ~I 13 The image positions are adjusted so that the aspect ratios of the images are a1:b1, a2:b2, and a3:b3, respectively, so that the reference point P0 is located at the center of the frame.

[0074] Because the UAV 10 moves (e.g., wobbles) while capturing images using the imaging device 20, the capturing positions of the multiple image data captured continuously by the imaging device 20 mounted on the UAV 10 may shift. In this regard, according to the above-described embodiment, the reframing process is performed on the multiple image data, so that the position of the captured object included in the multiple image data in the frame F can be fixed (so-called shake correction). In this way, by performing the reframing process, noise due to the wobbling of the UAV 10 is removed in advance, and then the optical flow estimation process is performed in step S300, thereby further improving the extraction accuracy of the gas velocity vector in step S800.

[0075] The contents described in each of the above embodiments can be understood, for example, as follows.

[0076] (1) A gas monitoring method according to at least one embodiment of the present invention includes: A step (S400) of acquiring a plurality of velocity vectors obtained by subjecting a plurality of image data obtained by capturing images at a plurality of times using an imaging device (20) mounted on a UAV (10) to optical flow estimation processing; (S700) obtaining a downwash speed u of the UAV based on at least one variable indicating a flight state of the UAV; An extraction step (S800) of extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Equipped with.

[0077] In the method (1) above, the multiple velocity vectors obtained by optical flow estimation of time-series image data captured by an imaging device mounted on a UAV include not only the velocity vector of the gas being detected (gas velocity vector) but also other velocity vectors (velocity vectors indicating the movement of objects other than the gas being detected, noise, etc.). Furthermore, because the distance between the UAV and the captured object is not constant and the scale used as a reference for velocity calculation is not necessarily present in the image, the magnitude (absolute value of velocity) of each of the multiple velocity vectors cannot be calculated by optical flow estimation. On the other hand, in the configuration (1) above, the downwash velocity u of the UAV can be obtained based on at least one variable indicating the flight state of the UAV. Therefore, among the multiple velocity vectors with unknown magnitudes obtained by optical flow estimation, a velocity vector with a magnitude within a range determined based on the downwash velocity u can be identified as a gas velocity vector (gas region) indicating the movement speed and direction of the gas being monitored, using the magnitude of the vector indicating the UAV's downwash velocity as a reference. Gas can be appropriately monitored using the gas velocity vectors extracted in this manner.

[0078] (2) In some embodiments, in the method (1), The at least one variable indicating the flight state of the UAV includes a flight height H of the UAV and a flight speed v of the UAV.

[0079] The downwash velocity u of the UAV is correlated with the flight height H and flight speed v of the UAV. According to the method (2) above, the downwash velocity u of the UAV can be appropriately obtained based on the correlation between the flight height H of the UAV and the flight speed v of the UVA.

[0080] (3) In some embodiments, in the method (1) or (2) above, The gas monitoring method includes: A step (S700) of acquiring wind speed Wg at the ground surface; and (S700) calculating a corrected downwash speed ug, which is a downwash speed taking the wind speed Wg into consideration, based on the downwash speed u and the wind speed Wg, In the extracting step, a gas velocity vector having a magnitude within a range determined based on the corrected downwash velocity ug is extracted from the plurality of velocity vectors.

[0081] When a wind separate from the downwash is blowing on the ground, the downwash flow on the ground may be affected by that wind. According to the method (3), a corrected downwash speed ug is calculated based on the downwash speed u obtained from variables indicating the UAV's flight state and the wind speed Wg at the ground, taking the wind speed Wg into account. A gas velocity vector having a magnitude within a range determined based on the corrected downwash speed ug is extracted from the multiple vectors obtained by optical flow estimation. This allows for more accurate extraction of the gas velocity vector indicating the movement speed and direction of the monitored gas.

[0082] (4) In some embodiments, in the method (3), The vector of the downwash velocity u and the vector of the wind velocity Wg at the ground surface are combined to calculate the vector of the corrected downwash velocity ug.

[0083] According to the above method (4), the corrected downwash speed ug can be appropriately calculated as a resultant vector of the downwash speed u and the wind speed Wg.

[0084] (5) In some embodiments, in the method (3) or (4) above, The gas monitoring method includes: A step (S700) of obtaining a correlation between height from the ground surface and wind speed; and (S700) acquiring a wind speed W1 at the flight height of the UAV; In the step of acquiring the wind speed Wg at the ground surface, the wind speed Wg at the ground surface is acquired based on the wind speed W1 and the correlation.

[0085] According to the method (5), the wind speed Wg at the ground surface is obtained from the wind speed W1 at the UAV flight height based on the correlation between the height above the ground surface and the wind speed. Based on the wind speed Wg at the ground surface thus obtained, the corrected downwash speed ug can be appropriately calculated.

[0086] (6) In some embodiments, in the method of (5), In the step of acquiring the wind speed W1, the wind speed W1 at the flight height of the UAV is calculated based on the motor current value of the UAV.

[0087] There is a predetermined correlation between the wind speed W1 at the flight height of the UAV and the current value of the motor that drives the propeller of the UAV. According to the method (6) above, the wind speed W1 at the flight height of the UAV can be appropriately obtained based on the motor current value of the UAV, without using, for example, an anemometer.

[0088] (7) In some embodiments, in any of the methods (1) to (6) above, The imaging device includes an infrared camera equipped with a filter that selectively transmits infrared light of wavelengths absorbed by the gas being monitored.

[0089] According to the method (7) above, an infrared camera equipped with a filter that selectively transmits infrared light of a wavelength absorbed by the gas to be monitored is used as the imaging device, so that the gas to be monitored can be photographed. Then, by performing optical flow processing on the time-series image data obtained in this way, a velocity vector indicating the movement of the gas to be monitored can be obtained.

[0090] (8) In some embodiments, in any of the methods (1) to (7) above, The gas monitoring method includes: a step (S500) of removing noise components from the plurality of velocity vectors obtained by subjecting the plurality of image data to optical flow estimation processing, based on a frequency distribution of the magnitudes of the plurality of velocity vectors or a frequency distribution of the directions of the plurality of velocity vectors; In the extracting step, the gas velocity vector is extracted from the plurality of velocity vectors from which the noise components have been removed.

[0091] The velocity vectors of noise components tend to have a predetermined pattern in terms of magnitude or direction. According to the method (8) above, noise components can be removed from multiple velocity vectors obtained by optical flow estimation based on the frequency distribution of the magnitude or direction of the vectors. Furthermore, since the gas velocity vector is extracted based on the downwash velocity u from the multiple vectors from which the noise components have been removed, the accuracy of gas velocity vector extraction can be improved.

[0092] (9) In some embodiments, in the method of (8), In the frequency distribution of the magnitudes of the plurality of velocity vectors, velocity vectors whose frequency falls within a range of magnitudes equal to or greater than a first threshold value and whose frequency falls within a range of orientations equal to or less than a second threshold value in the frequency distribution of the orientations of the plurality of velocity vectors are removed from the plurality of velocity vectors as the noise components.

[0093] The velocity vectors of noise components may have a pattern in which they have approximately the same magnitude but vary in direction. According to the method of (9) above, vectors that belong to a relatively high frequency range in the magnitude frequency distribution and a relatively low frequency range in the direction frequency distribution (i.e., vectors that match the above pattern) can be appropriately removed as noise components.

[0094] (10) In some embodiments, in any of the methods (1) to (9) above, The gas monitoring method includes: A step (S100) of acquiring a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; A step (S200) of performing a reframe process on the plurality of image data; Steps (S300, S400) of performing optical flow estimation processing on the plurality of image data that have been subjected to the reframe processing to obtain the plurality of velocity vectors; Equipped with.

[0095] Because the UAV moves (e.g., wobbles) while capturing images with the imaging device, the capturing positions of multiple image data captured continuously by the imaging device mounted on the UAV may shift. In this regard, according to the method (10) above, reframing is performed on the multiple image data, so the position of the captured object included in the multiple image data can be fixed in the frame (so-called blur correction). In this way, by performing reframing, noise caused by UAV wobble is removed in advance before optical flow estimation processing is performed, thereby further improving the extraction accuracy of the gas velocity vector.

[0096] (11) At least one embodiment of the gas monitoring device (50) of the present invention comprises: a velocity vector acquisition unit (58) that acquires a plurality of velocity vectors obtained by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; a downwash speed acquisition unit (64) that acquires a downwash speed u of the UAV based on at least one variable indicating a flight state of the UAV; an extraction unit (66) that extracts, from the plurality of velocity vectors, a gas velocity vector having a magnitude within a range determined based on the downwash velocity u; Equipped with.

[0097] In the configuration (11) above, the multiple velocity vectors obtained by optical flow estimation of time-series image data captured by an imaging device mounted on a UAV include not only the velocity vector of the gas being detected (gas velocity vector) but also other velocity vectors (velocity vectors indicating the movement of objects other than the gas being detected, noise, etc.). Furthermore, because the distance between the UAV and the captured object is not constant and the scale used as a reference for velocity calculation is not necessarily present in the image, the magnitude (absolute value of velocity) of each of the multiple velocity vectors cannot be calculated by optical flow estimation. On the other hand, in the configuration (11) above, the downwash velocity u of the UAV can be obtained based on at least one variable indicating the flight state of the UAV. Therefore, among the multiple velocity vectors with unknown magnitudes obtained by optical flow estimation, a velocity vector with a magnitude within a range determined based on the downwash velocity u can be identified as a gas velocity vector (gas region) indicating the movement speed and direction of the gas being monitored, using the magnitude of the vector indicating the UAV's downwash velocity as a reference. Gas can be appropriately monitored using the gas velocity vectors extracted in this manner.

[0098] (12) At least one embodiment of the gas monitoring system (100) of the present invention comprises: A UAV (10) equipped with an imaging device (20); an optical flow estimation processing unit (56) configured to perform optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times by the imaging device, and calculate a plurality of velocity vectors; a gas monitoring device (50) according to claim 11 configured to extract the gas velocity vector from the plurality of velocity vectors; Equipped with.

[0099] In the configuration (12) above, the multiple velocity vectors obtained by optical flow estimation of time-series image data captured by an imaging device mounted on a UAV include not only the velocity vector of the gas being detected (gas velocity vector) but also other velocity vectors (velocity vectors indicating the movement of objects other than the gas being detected, noise, etc.). Furthermore, because the distance between the UAV and the captured object is not constant and the scale used as a reference for velocity calculation is not necessarily present in the image, the magnitude (absolute value of velocity) of each of the multiple velocity vectors cannot be calculated by optical flow estimation. On the other hand, in the configuration (12) above, the downwash velocity u of the UAV can be obtained based on at least one variable indicating the flight state of the UAV. Therefore, among the multiple velocity vectors with unknown magnitudes obtained by optical flow estimation, a velocity vector with a magnitude within a range determined based on the downwash velocity u can be identified as a gas velocity vector (gas region) indicating the movement speed and direction of the gas being monitored, using the magnitude of the vector indicating the UAV's downwash velocity as a reference. Gas can be appropriately monitored using the gas velocity vectors extracted in this manner.

[0100] (13) A gas monitoring program according to at least one embodiment of the present invention includes: On the computer, A procedure for acquiring a plurality of velocity vectors obtained by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; obtaining a downwash velocity u of the UAV based on at least one variable indicative of a flight state of the UAV; extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Let it run.

[0101] In the configuration (13) above, the multiple velocity vectors obtained by optical flow estimation of time-series image data captured by an imaging device mounted on a UAV include not only the velocity vector of the gas being detected (gas velocity vector) but also other velocity vectors (velocity vectors indicating the movement of objects other than the gas being detected, noise, etc.). Furthermore, because the distance between the UAV and the captured object is not constant and the scale used as a reference for velocity calculation is not necessarily present in the image, the magnitude (absolute value of velocity) of each of the multiple velocity vectors cannot be calculated by optical flow estimation. On the other hand, in the configuration (13) above, the downwash velocity u of the UAV can be obtained based on at least one variable indicating the flight state of the UAV. Therefore, among the multiple velocity vectors with unknown magnitudes obtained by optical flow estimation, a velocity vector with a magnitude within a range determined based on the downwash velocity u can be identified as a gas velocity vector (gas region) indicating the movement speed and direction of the gas being monitored, using the magnitude of the vector indicating the UAV's downwash velocity as a reference. Gas can be appropriately monitored using the gas velocity vectors extracted in this manner.

[0102] The above describes an embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and also includes forms in which the above-described embodiment is modified, or forms in which these forms are appropriately combined.

[0103] In this specification, expressions expressing relative or absolute arrangement such as "in a certain direction," "along a certain direction," "parallel," "orthogonal," "center," "concentric," or "coaxial" not only express such an arrangement strictly, but also express a state in which there is a relative displacement with a tolerance or an angle or distance to the extent that the same function is obtained. For example, expressions such as "identical," "equal," and "homogeneous" that indicate that something is in an equal state not only indicate a state of strict equality, but also indicate a state in which there is a tolerance or a difference to the extent that the same function is obtained. Furthermore, in this specification, expressions representing shapes such as a rectangular shape or a cylindrical shape not only represent rectangular shapes or cylindrical shapes in the strict geometric sense, but also represent shapes including uneven portions, chamfered portions, etc., to the extent that the same effect can be obtained. Furthermore, in this specification, the expressions "comprise," "include," or "have" a component are not exclusive expressions that exclude the presence of other components. [Explanation of symbols]

[0104] 10 UAV 12 UAV body 14 propellers 20 Imaging device 50 Gas monitoring equipment 52 Image acquisition unit 54 Reframe processing section 56 Optical flow processing section 58 Velocity vector acquisition unit 60 Noise component removal section 62 Flight status acquisition unit 64 Downwash speed acquisition unit 66 Extraction part 68 Display image generation unit 69 Memory section 70 Display section 100 Gas Monitoring System AG Gas field B1 CO2 gas cylinder B2 CO2 gas cylinder D Down Wash F Frame G surface P0 reference point VG gas velocity vector

Claims

1. A step of acquiring a plurality of velocity vectors obtained by subjecting a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV to optical flow estimation processing; Obtaining a downwash velocity u of the UAV based on at least one variable indicative of a flight state of the UAV; an extraction step of extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; A gas monitoring method comprising:

2. The at least one variable indicating the flight state of the UAV includes a flight height H of the UAV and a flight speed v of the UAV.

10. The gas monitoring method of claim 1.

3. Obtaining a wind speed Wg at the ground surface; and calculating a corrected downwash speed ug, which is a downwash speed taking the wind speed Wg into consideration, based on the downwash speed u and the wind speed Wg, In the extracting step, a gas velocity vector having a magnitude within a range determined based on the corrected downwash velocity ug is extracted from the plurality of velocity vectors.

3. A gas monitoring method according to claim 1 or 2.

4. The vector of the downwash velocity u and the vector of the wind velocity Wg at the ground surface are combined to calculate the vector of the corrected downwash velocity ug.

4. The gas monitoring method of claim 3.

5. obtaining a correlation between height above ground and wind speed; and acquiring a wind speed W1 at the flight height of the UAV; In the step of acquiring the wind speed Wg at the ground surface, the wind speed Wg at the ground surface is acquired based on the wind speed W1 and the correlation.

5. A gas monitoring method according to claim 3 or 4.

6. In the step of acquiring the wind speed W1, the wind speed W1 at the flight height of the UAV is calculated based on a motor current value of the UAV.

6. The gas monitoring method of claim 5.

7. The imaging device includes an infrared camera equipped with a filter that selectively transmits infrared light of a wavelength absorbed by the gas being monitored. A gas monitoring method according to any one of claims 1 to 6.

8. removing noise components from the plurality of velocity vectors obtained by subjecting the plurality of image data to optical flow estimation processing based on a frequency distribution of the magnitudes of the plurality of velocity vectors or a frequency distribution of the directions of the plurality of velocity vectors; In the extraction step, the gas velocity vector is extracted from the plurality of velocity vectors from which the noise components have been removed. A gas monitoring method according to any one of claims 1 to 7.

9. Velocity vectors whose frequency in a frequency distribution of the magnitudes of the plurality of velocity vectors is in a range of magnitudes equal to or greater than a first threshold value and whose frequency in a frequency distribution of the orientations of the plurality of velocity vectors is in a range of orientations equal to or less than a second threshold value are removed from the plurality of velocity vectors as noise components.

9. The gas monitoring method of claim 8.

10. Acquiring a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; performing a reframe process on the plurality of image data; performing an optical flow estimation process on the plurality of image data that have been subjected to the reframe process to obtain the plurality of velocity vectors; 10. A method of gas monitoring according to any one of claims 1 to 9, comprising:

11. a velocity vector acquisition unit that acquires a plurality of velocity vectors by performing optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV; A downwash speed acquisition unit that acquires a downwash speed u of the UAV based on at least one variable indicating a flight state of the UAV; an extractor that extracts a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; A gas monitoring device comprising:

12. A UAV equipped with an imaging device, an optical flow estimation processing unit configured to perform optical flow estimation processing on a plurality of image data obtained by capturing images at a plurality of times by the imaging device, and calculate a plurality of velocity vectors; 12. The gas monitor of claim 11 configured to extract the gas velocity vector from the plurality of velocity vectors; A gas monitoring system comprising:

13. On the computer, A procedure for acquiring a plurality of velocity vectors by subjecting a plurality of image data obtained by capturing images at a plurality of times using an imaging device mounted on the UAV to optical flow estimation processing; Obtaining a downwash velocity u of the UAV based on at least one variable indicative of a flight state of the UAV; extracting a gas velocity vector having a magnitude within a range determined based on the downwash velocity u from the plurality of velocity vectors; Gas monitoring program to run.

Images