Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program capable of easily detecting a detection object such as a gas leakage.SOLUTION: An information processing device comprises: an acquisition unit which acquires a first image and a second image of different frames captured by an imaging unit loaded on a mobile body; an alignment unit which aligns the first and second images; and a difference calculation unit which calculates a difference between the aligned first and second images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Patent Document 1 describes a gas leakage monitoring apparatus for enabling easy monitoring of leakage of gas or the like in a relatively wide range of monitoring targets. Specifically, the gas leakage monitoring apparatus described in Patent Document 1 includes a visible image camera and an infrared image camera mounted on an aircraft, and an image processing unit that synthesizes a difference image, in which pixels having no difference between frames in an infrared image are made transparent, with a visible image and causes the synthesized image to be displayed on a display unit. According to the gas leakage monitoring apparatus described in Patent Document 1, even leakage of transparent gas having a temperature equivalent to the ambient temperature can be visualized as an image corresponding to the infrared absorption spectrum of the gas, and by obtaining the luminance difference for each pixel between frames, it becomes easy to grasp the leakage situation according to the ever-changing ejection situation and the flow situation caused by wind vinegar . Furthermore, according to the gas leakage monitoring apparatus described in Patent Document 1, even when the visual field shifts between frames due to the flight movement and vibration of the aircraft, and thus contours of non-gas subjects are likely to be emphasized as differences, a visible image is synthesized. As a result, the emphasized portion is likely to be buried in the contour portion of the visible image, or becomes inconspicuous when it is recognized as a contour portion. This improves visibility, and thus gas leakage can be easily monitored.

[0003] Note that Non-Patent Document 1 describes an example of a technique related to image alignment.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005] [Non-Patent Document 1] Yao Xi Li, Hiroshi Fujimoto, "Design of a Robust Image Registration Method Based on FFT," Transactions of the Institute of Electrical Engineers of Japan (JETRO), Vol. 139, No. 1, pp. 22-29, January 2019. [Overview of the project] [Problems that the invention aims to solve]

[0006] However, the gas leak monitoring device described in Patent Document 1 attempts to improve visibility by utilizing the ease of recognizing the outlines of objects such as structures in visible images. Therefore, it has the problem that it may not be able to improve visibility if, for example, the outlines of objects in visible images are not easily recognizable.

[0007] This disclosure was made to solve the above-mentioned problems and aims to provide an information processing device, an information processing method, and a program that can easily detect detection targets such as gas leaks. [Means for solving the problem]

[0008] To solve the above problems, the information processing device according to this disclosure includes an acquisition unit that acquires a first image and a second image, each with a different frame, captured by an imaging unit mounted on a moving object; an alignment unit that aligns the first image and the second image; and a difference calculation unit that calculates the difference between the aligned first image and the second image.

[0009] The information processing method relating to this disclosure includes the steps of acquiring a first image and a second image, each having different frames, captured by an imaging unit mounted on a mobile body; aligning the first image and the second image; and calculating the difference between the aligned first image and the second image.

[0010] The program relating to this disclosure causes a computer to perform the following steps: acquire a first image and a second image, each with a different frame, captured by an imaging unit mounted on a mobile body; align the first image and the second image; and calculate the difference between the aligned first image and the second image. [Effects of the Invention]

[0011] According to the information processing device, information processing method, and program of this disclosure, detection targets such as gas leaks can be easily detected. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example configuration of an information processing device according to the first embodiment of this disclosure. [Figure 2] This is a flowchart showing an example of the operation of the information processing device according to the first embodiment of this disclosure. [Figure 3] A flowchart illustrating an example of alignment processing according to the first embodiment of this disclosure. [Figure 4] This is a diagram illustrating an example of the operation of an information processing apparatus according to the first embodiment of this disclosure. [Figure 5] This figure shows an example configuration of an information processing device according to the second embodiment of this disclosure. [Figure 6] This flowchart shows an example of the operation of the information processing device according to the second embodiment of this disclosure. [Figure 7] This is a schematic block diagram showing the configuration of a computer according to at least one embodiment. [Modes for carrying out the invention]

[0013] Hereinafter, an information processing apparatus, information processing method, and program according to the embodiments of this disclosure will be described with reference to the drawings. In each drawing, the same or corresponding components are given the same reference numerals, and their descriptions are omitted as appropriate.

[0014] <First Embodiment> Figure 1 is a diagram showing an example configuration of an information processing apparatus according to the first embodiment of this disclosure. Figure 2 is a flowchart showing an example of operation of the information processing apparatus according to the first embodiment of this disclosure. Figure 3 is a flowchart showing an example of alignment processing according to the first embodiment of this disclosure. Figure 4 is a diagram illustrating an example of operation of the information processing apparatus according to the first embodiment of this disclosure.

[0015] As shown in Figure 1, the information processing device 1 according to this embodiment processes images captured by an imaging unit 21 mounted on a mobile body 2 within the information processing system 100, and is a device that performs predetermined information processing for detecting predetermined detection targets such as gas leaks. The information processing system 100 comprises the information processing device 1 and the mobile body 2. In the example shown in Figure 1, the mobile body 2 is a drone. However, the mobile body 2 is not limited to a drone (unmanned aerial vehicle), and may be, for example, a manned fixed-wing aircraft, a rotary-wing aircraft, a satellite, a mobile body that moves in contact with the ground or water surface, or a mobile body that moves underwater. Furthermore, the detection target is not limited to gas leaks, and may be, for example, any object whose presence or movement is reflected in the difference between frames of multiple captured images (for example, the difference in brightness value or pixel value for each pixel). Furthermore, the imaging unit 21 may capture visible light images, infrared images, or a combination thereof. Furthermore, examples of detection targets include, but are not limited to, colorless and transparent gases such as carbon dioxide, methane, and ammonia, smoke, and colored gases.

[0016] The information processing apparatus 1 is configured using a computer, and includes an acquisition unit 11, an alignment unit 12, a difference calculation unit 13, and a storage unit 14 as a functional configuration constituted by a combination of hardware such as a computer and its peripheral devices, and software such as programs executed by the computer. The information processing apparatus 1 may receive image data captured by the imaging unit 21 via wireless communication and process the data in substantially real time, or may store the image data in a predetermined storage device and process the data later. In addition, the information processing apparatus 1 may be mounted on a moving body 2 together with the imaging unit 21. The information processing apparatus 1 sequentially performs processing for detecting a detection target based on inter-frame difference information in a plurality of frames of images captured by the imaging unit 21. Hereinafter, one-time image processing for two images (two frames) will be mainly described. In addition, in the present embodiment, one of the two images is referred to as an original image, and the other is referred to as a background image. The background image is an image that serves as a reference when calculating a difference, and is the image on the side where the position is aligned (the side that is changed) with respect to the original image (based on the original image) in the alignment processing described later. Further, for example, when the original image is an n-th frame image, the background image is an (n-x)-th frame image. n is a natural number (or an integer of 0 or more) representing the frame number of an image captured by the imaging unit 21, and x is an integer of 1 or more. The order of the first image and the second image according to the present disclosure is relative, one of the original image and the background image corresponds to the first image, and the other corresponds to the second image. Further, the value of x (the frame interval between the two images) can be set according to, for example, the moving speed of the moving body 2.

[0017] The acquisition unit 11 included in the information processing apparatus 1 acquires an original image and a background image of different frames captured by the imaging unit 21 mounted on the moving body 2. The acquisition by the acquisition unit 11 includes acquiring the images from the imaging unit 21 via wireless communication, and acquiring the images from the imaging unit 21 via wireless communication, storing the images in, for example, the storage unit 14, and acquiring the images from the storage unit 14.

[0018] The alignment unit 12 aligns an original image and a background image. In the present embodiment, the alignment unit 12 aligns the background image with respect to the original image. Alignment is a process of moving, rotating and deforming one image to match the other image such that the position, size and shape of the same subject between the two images are made identical, and is also called image alignment (image registration) or the like. There is no limitation on the alignment method. For example, a method of moving an image in horizontal and vertical directions such that the sum (total) of luminance values of respective pixels of two images is minimized as described later, the method described in Non-Patent Document 1, or the like can be used. In the method of Non-Patent Document 1, results obtained by estimating scaling and rotation amounts between two images are used for alignment.

[0019] The difference calculation unit 13 calculates a difference between the background image aligned by the alignment unit 12 and the original image for each mutually corresponding pixel (pixel at the same position). For example, if the background image and the original image are color images, a difference in pixel values may be calculated for each pixel and for each component, or a difference in luminance values may be calculated for each pixel after conversion into luminance values, for example. Further, in the case of a grayscale image, a difference in luminance values (pixel values) is calculated for each pixel. The difference calculation unit 13 causes the calculation result of the difference to be stored in the storage unit 14 or displayed on a predetermined display unit. In the present embodiment, storing or displaying the calculation result by the difference calculation unit 13 is referred to as outputting.

[0020] The storage unit 14 stores image data captured by the imaging unit 21 and stores a calculation result obtained by the difference calculation unit 13.

[0021] Next, an example of the operation of the information processing apparatus 1 will be described with reference to FIG. 2 and FIG. 3. The process illustrated in FIG. 2 is an example of a process for one original image and one background image. The process illustrated in FIG. 3 is an example of the alignment process executed in step S13 of FIG. 2.

[0022] In the process shown in Figure 2, first, the acquisition unit 11 acquires the original image (nth frame) (step S11). Next, the acquisition unit 11 acquires the background image ((nx)th frame) (step S12). Then, the alignment unit 12 performs the process of aligning the position of the background image with that of the original image (step S13).

[0023] In the alignment process performed in step S13, as shown in Figure 3, the alignment unit 12 first stores -XL in the variable dx (step S21). The value of the variable dx represents how many pixels the background image will be moved in the X direction (horizontal direction). -XL represents the initial number of pixels to move, and in this example, the background image is moved in the X direction from -XL pixels to +XL pixels.

[0024] Next, the alignment unit 12 creates an image by shifting the background image by dx pixels in the X direction (step S22). In step S22, the pixels at the end of the source image are padded with zeros. Next, the alignment unit 12 calculates the sum Sx_dx of the differences in brightness values ​​for each pixel between the original image and the image shifted in step S22 (step S23). The "dx" in sum Sx_dx is the subscript of the variable "sum Sx", and a numerical value representing the amount of shift is set. Next, the alignment unit 12 increments the variable dx by 1 (step S24). Next, the alignment unit 12 determines whether the variable dx is greater than the constant XL (step S25). If it is not greater (step S25: NO), the alignment unit 12 executes the process from step S22 onwards again.

[0025] On the other hand, if the value is large (step S25: YES), the alignment unit 12 stores -YL in the variable dy (step S26). The value of the variable dy represents how many pixels the background image will be moved in the Y direction (vertical direction). -YL represents the number of pixels to be moved first, and in this example, the background image will be moved in the Y direction from -YL pixels to +YL pixels.

[0026] Next, the alignment unit 12 creates an image by shifting the background image in the Y direction by dy pixels (step S27). In step S27, the pixels at the end of the source image are padded with zero. Next, the alignment unit 12 calculates the sum Sy_dy, which is the difference in brightness values ​​for each pixel between the original image and the image shifted in step S27 (step S28). The "dy" in the sum Sy_dy is the subscript of the variable "sumSy", and a numerical value representing the amount of movement is set. Next, the alignment unit 12 increments the variable dy by 1 (step S29). Next, the alignment unit 12 determines whether the variable dy is greater than the constant YL (step S30). If it is not greater (step S30: NO), the alignment unit 12 executes the process from step S27 onwards again.

[0027] On the other hand, if the value is large (step S30: YES), the alignment unit 12 determines that the subscript "dx" of the smallest sum (Sx_dx) is the amount of movement in the X direction dx, and the subscript "dy" of the smallest sum (Sy_dy) is the amount of movement in the Y direction dy (step S31). Next, the alignment unit 12 shifts the background image by an amount of movement dx in the X direction and an amount of movement dy in the Y direction to create a background image that is aligned (step S32).

[0028] Next, in step S14 of Figure 2, the difference calculation unit 13 calculates the difference between the original image and the background image that the alignment unit 12 aligned in step S13 (step S32 of Figure 3) (step S14). Next, the difference calculation unit 13 outputs the calculation result (step S15).

[0029] Figure 4 shows examples of images representing the original image, an image showing the difference between the original image and a aligned background image (hereinafter referred to as the "aligned + frame difference" image), and an image showing the difference between the original image and a background image that has not been aligned (hereinafter referred to as the "simple frame difference" image). In the image showing the difference calculation result, the brightness value decreases (approaches black) when the difference is small, and increases (approaches white) when the difference is large. The original image contains carbon dioxide gas experimentally ejected in the area enclosed by the dashed circle in the "aligned + frame difference" image. The shooting location is a wasteland. The "aligned + frame difference" image captures the movement of carbon dioxide gas because the difference is calculated by aligning the shooting position. On the other hand, in the "simple frame difference" image, the difference information corresponding to the carbon dioxide gas cannot be extracted because the shooting position changes. The alignment process uses the method described in Non-Patent Literature 1.

[0030] (Effects of this embodiment) According to this embodiment, since the difference processing is performed after correcting the position of the background (background image), the influence of movement and vibration of the moving object 2 can be suppressed compared to the case where position correction is not performed, and the target to be detected can be easily detected.

[0031] <Second Embodiment> Figure 5 is a diagram showing an example configuration of an information processing apparatus according to the second embodiment of this disclosure. Figure 6 is a flowchart showing an example of operation of the information processing apparatus according to the second embodiment of this disclosure.

[0032] As shown in Figure 5, the information processing device 1a in the information processing system 100a of the second embodiment differs from the information processing device 1 of the first embodiment shown in Figure 1 in that it newly includes a focus correction unit 15, the alignment unit 12a processes the original image and background image processed by the focus correction unit 15, and the difference calculation unit 13a processes the original image and background image processed by the focus correction unit 15.

[0033] The focus correction unit 15 shown in Figure 5 corrects the focus blur of the background image and the original image. The alignment unit 12a aligns the background image with the original image with the focus blur correction. The difference calculation unit 13a calculates the difference between the aligned, focus-corrected background image and the original image with the focus blur correction.

[0034] Out of focus refers to a state where the subject is not in focus. There are no limitations on the method of out-of-focus processing performed by the out-of-focus correction unit 15. The out-of-focus correction processing may, for example, be a process that enhances the outlines. Alternatively, the out-of-focus correction processing may be a method that estimates the degradation trajectory (PSF: point image distribution function) of an image whose image quality has deteriorated using a method such as blind deconvolution, and restores the image based on the estimated PSF.

[0035] Next, an example of the operation of the information processing device 1a will be described with reference to Figure 6. In the process shown in Figure 6, first, the acquisition unit 11 acquires the original image (nth frame) (step S41). Next, the acquisition unit 11 acquires the background image ((nx)th frame) (step S42). Next, the focus correction unit 15 performs focus correction processing on the original image and the background image (step S43). Next, the alignment unit 12a performs the process of aligning the position of the focus-corrected background image with the focus-corrected original image (step S44).

[0036] Next, the difference calculation unit 13a calculates the difference between the original image with focus correction and the background image whose position was aligned by the alignment unit 12a in step S44 (step S45). Then, the difference calculation unit 13a outputs the calculation result (step S46).

[0037] (Effects of this embodiment) According to this embodiment, the detection sensitivity of the target can be improved even for out-of-focus images.

[0038] (Effects and Benefits) In the information processing device, information processing method, and program configured above, the imaging unit mounted on the mobile body performs alignment between a background image and the original image, which have different frames captured, and then calculates the difference between the frames. Therefore, according to the information processing device, information processing method, and program of this embodiment, detection targets such as gas leaks can be easily detected.

[0039] (Other embodiments) Although embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and may include design changes and the like that do not depart from the gist of this disclosure.

[0040] <Computer Configuration> Figure 7 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, main memory 92, storage 93, and an interface 94. The information processing devices 1 and 1a described above are implemented in the computer 90. The operation of each processing unit described above is stored in storage 93 in the form of a program. The processor 91 reads the program from storage 93, loads it into main memory 92, and executes the above processing according to the program. The processor 91 also allocates storage areas in main memory 92 corresponding to each of the storage units described above, according to the program.

[0041] The program may be for implementing some of the functions that the computer 90 is to perform. For example, the program may perform functions in combination with other programs already stored in storage, or in combination with other programs implemented in other devices. In other embodiments, the computer may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to, or instead of, the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), FPGA (Field Programmable Gate Array), etc. In this case, some or all of the functions implemented by the processor may be implemented by the integrated circuit.

[0042] Examples of storage 93 include HDDs (Hard Disk Drives), SSDs (Solid State Drives), magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read Only Memory), DVD-ROMs (Digital Versatile Disc Read Only Memory), and semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of the computer 90, or an external medium connected to the computer 90 via an interface 94 or a communication line. Furthermore, if this program is distributed to the computer 90 via a communication line, the computer 90 that receives the program may expand it into main memory 92 and execute the above processing. In at least one embodiment, storage 93 is a tangible storage medium that is not temporary.

[0043] <Note> The information processing devices 1 and 1a described in each embodiment can be understood, for example, as follows.

[0044] (1) The information processing device 1 and 1a according to the first embodiment includes an acquisition unit 11 that acquires a first image and a second image with different frames captured by an imaging unit 21 mounted on a mobile body 2, an alignment unit 12 or 12a that aligns the first image and the second image, and a difference calculation unit 13 or 13a that calculates the difference between the aligned first image and the second image. According to this embodiment and the following embodiments, detection targets such as gas leaks can be easily detected.

[0045] (2) The information processing device 1a according to the second embodiment is the information processing device of (1), further comprising a focus correction unit 15 for correcting the focus blur of the first image and the second image, and the alignment unit 12a performs alignment of the first image and the second image after the focus blur has been corrected. According to this embodiment, the detection sensitivity for out-of-focus images can be improved.

[0046] (3) The information processing devices 1 and 1a according to the third embodiment are the information processing devices of (1) or (2), wherein the frame interval between the first image and the second image is set according to the movement speed of the moving body. According to this embodiment, the frame interval between the first image and the second image can be appropriately changed according to the speed of the moving body. [Explanation of Symbols]

[0047] 1, 1a... Information Processing Device 2… Mobile 11…Acquisition part 12, 12a... Alignment section 13, 13a...Difference calculation section 14...Storage section 15...Focus correction section

Claims

1. An acquisition unit that acquires a first image and a second image, each with a different frame, captured by an imaging unit mounted on a mobile device, A positioning unit that aligns the first image and the second image, A difference calculation unit that calculates the difference between the aligned first image and the second image, Equipped with, The alignment unit generates a plurality of third images by shifting the second image only in the first direction, and calculates a value for each third image by summing the differences in pixel-wise brightness values ​​between the first image and the third image. The alignment unit generates a plurality of fourth images obtained by shifting the second image only in the second direction, and calculates a value for each fourth image by summing the differences in the brightness values ​​of each pixel between the first image and the fourth image. The alignment unit sets the amount of movement corresponding to the minimum value of the values ​​calculated for each third image as the first amount of movement. The alignment unit sets the amount of movement corresponding to the minimum value of the values ​​calculated for each fourth image as the second amount of movement. The alignment unit performs alignment by moving the second image by a first movement amount relative to the first image, and moving the second image by a second movement amount relative to the first image. Information processing device.

2. The system further includes a focus correction unit that corrects the focus blur of the first and second images, The alignment unit aligns the first image, which has been corrected for out-of-focus, with the second image. The information processing apparatus according to claim 1.

3. The frame interval between the first image and the second image is set according to the movement speed of the moving object. The information processing apparatus according to claim 1 or 2.

4. The steps include acquiring a first image and a second image, each with a different frame, captured by an imaging unit mounted on a mobile device, The steps include aligning the first image and the second image, A step of calculating the difference between the aligned first image and the second image, Includes, In the alignment step, a plurality of third images are generated by shifting the second image only in the first direction, and a value is calculated for each third image by summing the differences in pixel-wise brightness values ​​between the first image and the third image. In the alignment step, a plurality of fourth images are generated by shifting the second image only in the second direction, and a value is calculated for each fourth image by summing the differences in pixel-wise brightness values ​​between the first image and the fourth image. In the step of performing the alignment, the amount of movement corresponding to the minimum value of the value calculated for each third image is set as the first amount of movement. In the step of performing the alignment, the amount of movement corresponding to the minimum value of the values ​​calculated for each fourth image is set as the second amount of movement. In the step of performing the alignment, the second image is moved relative to the first image by a first movement amount, and the second image is moved relative to the first image by a second movement amount to perform the alignment. Information processing methods.

5. The steps include acquiring a first image and a second image, each with a different frame, captured by an imaging unit mounted on a mobile device, The steps include aligning the first image and the second image, A step of calculating the difference between the aligned first image and the second image, A program that causes a computer to execute, In the alignment step, a plurality of third images are generated by shifting the second image only in the first direction, and a value is calculated for each third image by summing the differences in pixel-wise brightness values ​​between the first image and the third image. In the alignment step, a plurality of fourth images are generated by shifting the second image only in the second direction, and a value is calculated for each fourth image by summing the differences in pixel-wise brightness values ​​between the first image and the fourth image. In the step of performing the alignment, the amount of movement corresponding to the minimum value of the value calculated for each third image is set as the first amount of movement. In the step of performing the alignment, the amount of movement corresponding to the minimum value of the values ​​calculated for each fourth image is set as the second amount of movement. In the step of performing the alignment, the second image is moved relative to the first image by a first movement amount, and the second image is moved relative to the first image by a second movement amount to perform the alignment. program.

Images