Information processing device, information processing method, and program

Abstract

An information processing device is provided with: an acquisition unit that acquires a first image and a second image having different frames captured by an imaging unit mounted on a moving body; an alignment unit that aligns the first image and the second image; and a difference calculation unit that calculates the difference between the aligned first image and second image.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method and a program.

[0002] This application claims priority based on Japanese Patent Application No. 2022-135686 filed in Japan on August 29, 2022, and the contents are incorporated herein by reference. Background Art

[0003] Patent document 1 describes a gas leakage monitoring device for easily monitoring leakage of gas or the like in a relatively wide range of monitoring objects. Specifically, the gas leakage monitoring device described in Patent document 1 includes: a visible image camera and an infrared image camera, which are mounted on a flying object; and an image processing unit, which synthesizes a difference image in which pixels in the infrared image that have no difference between frames are made transparent with a visible image and displays the result on a display unit. According to the gas leakage monitoring device described in Patent document 1, even leakage of a transparent gas having the same temperature as the ambient air temperature can be visualized as an image corresponding to the infrared absorption spectrum of the gas, and by obtaining the difference in brightness of each pixel between frames, it is easy to grasp the leakage status corresponding to the constantly changing ejection status or the flow status caused by the wind. Furthermore, according to the gas leakage monitoring device described in Patent document 1, even if the field of view shifts between frames due to the flight movement or vibration of the aircraft, and the contour of a non-gas object is easily emphasized as a difference, since the visible image is synthesized, the emphasized portion is easily buried in the contour portion of the visible image, or becomes inconspicuous by being recognized as the contour portion. This improves visibility and thus makes it easier to monitor gas leakage.

[0004] In addition, Non-Patent Document 1 describes an example of a technique related to image alignment.

[0005] Previous technical literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2022-13126

[0008] Non-patent literature

[0009] Non-patent document 1: Lee Yoshiki and Fujimoto Hiroshi, “Design of an FFT-based enhanced image registration method,” Proceedings of the Institute of Electrical Engineering D (Industrial Applications Section), The Institute of Electrical Engineering, January 2019, Vol. 139, No. 1, pp. 22-29. Summary of the invention

[0010] Technical issues to be solved by the invention

[0011] However, the gas leakage monitoring device described in Patent Document 1 improves visibility by utilizing the ease of recognizing the outline of a subject such as a structure in a visible image. Therefore, there is a problem that, for example, when the outline of a subject in a visible image is difficult to recognize, visibility cannot be improved.

[0012] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide an information processing device, an information processing method, and a program that can easily detect a detection target such as a gas leak.

[0013] Means for solving technical problems

[0014] In order to solve the above-mentioned problems, the information processing device involved in the present invention comprises: an acquisition unit, which acquires a first image and a second image with different frames taken by a camera unit mounted on a mobile body; a positioning unit, which aligns the first image and the second image; and a difference calculation unit, which calculates the difference between the aligned first image and the second image.

[0015] The information processing method involved in the present invention includes the following steps: acquiring a first image and a second image with different frames taken by a camera unit mounted on a mobile body; aligning the first image and the second image; and calculating the difference between the aligned first image and the second image.

[0016] The program involved in the present invention enables a computer to execute the following steps: obtaining a first image and a second image with different frames taken by a camera unit mounted on a mobile body; aligning the first image and the second image; and calculating the difference between the aligned first image and the second image.

[0017] Effects of the Invention

[0018] According to the information processing device, information processing method, and program of the present invention, it is possible to easily detect a detection target such as gas leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing a configuration example of an information processing device according to the first embodiment of the present invention.

[0020] Figure 2 This is a flowchart showing an operation example of the information processing device according to the first embodiment of the present invention.

[0021] Figure 3 This is a flowchart showing an example of the positioning process according to the first embodiment of the present invention.

[0022] Figure 4 It is a diagram for explaining an operation example of the information processing device according to the first embodiment of the present invention.

[0023] Figure 5 It is a diagram showing a configuration example of an information processing device according to a second embodiment of the present invention.

[0024] Figure 6 This is a flowchart showing an operation example of the information processing device according to the second embodiment of the present invention.

[0025] Figure 7 This is a schematic block diagram showing the structure of a computer involved in at least one embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, the information processing device, information processing method and program according to the embodiment of the present invention will be described with reference to the accompanying drawings. In addition, in each figure, the same reference numerals are used for the same or corresponding structures, and the description is appropriately omitted.

[0027] <First Embodiment>

[0028] Figure 1 It is a diagram showing a configuration example of an information processing device according to the first embodiment of the present invention. Figure 2 This is a flowchart showing an operation example of the information processing device according to the first embodiment of the present invention. Figure 3 This is a flowchart showing an example of the positioning process according to the first embodiment of the present invention. Figure 4 It is a diagram for explaining an operation example of the information processing device according to the first embodiment of the present invention.

[0029] like Figure 1 As shown, the information processing device 1 according to this embodiment processes images captured by the camera unit 21 mounted on the mobile body 2 in the information processing system 100, and is a device for performing predetermined information processing for detecting a predetermined detection object such as gas leakage. The information processing system 100 includes the information processing device 1 and the mobile body 2. Figure 1 In the example shown, the mobile body 2 is a drone. However, the mobile body 2 is not limited to drones (unmanned aerial vehicles), and may be, for example, manned fixed-wing aircraft or rotorcraft, satellites, mobile bodies that move in contact with the ground or water, mobile bodies that move in water, etc. Furthermore, the detection object is not limited to gas leakage, for example, as long as the presence or absence or movement of the detection object appears in the inter-frame difference of the captured multiple frames of images (for example, the brightness value of each pixel or the difference in pixel value). Furthermore, the camera unit 21 can capture visible light images, infrared images, or a combination thereof. Furthermore, as detection objects, for example, colorless and transparent gases such as carbon dioxide, methane, and ammonia, smoke, and colored gases can be cited, but are not limited to these.

[0030] The information processing device 1 is configured using a computer, and as a functional structure composed of a combination of hardware such as a computer, its peripheral devices, and software such as a program executed by the computer, the information processing device 1 includes an acquisition unit 11, an alignment unit 12, a difference calculation unit 13, and a storage unit 14. In addition, the information processing device 1 can receive the image data captured by the camera unit 21 through wireless communication and process it in almost real time, or store it in a predetermined storage device and process it later. In addition, the information processing device 1 can be mounted on the mobile body 2 together with the camera unit 21. In addition, the information processing device 1 sequentially performs processing for detecting the detection object based on the inter-frame difference information in the multiple frames of images captured by the camera unit 21, but the following mainly describes the image processing for one time for two images (two frames). In addition, in this embodiment, one of the two images is called the original image, and the other is called the background image. The background image is an image that becomes a reference when calculating the difference, and is an image on the side (the side that is changed) that is aligned with respect to the original image (with the original image as a reference) in the alignment processing described later. Furthermore, for example, when the original image is set to the nth frame image, the background image is the (nx)th frame image. n represents a natural number (or an integer greater than 0) of the frame number of the image captured by the camera unit 21, and x is an integer greater than 1. In addition, the order of the first image and the second image involved in the present invention is relative, and one of the original image or the background image corresponds to the first image, and the other corresponds to the second image. Furthermore, the x value (frame interval between two images) can be set, for example, according to the moving speed of the moving body 2.

[0031] The acquisition unit 11 of the information processing device 1 acquires the original image and the background image of different frames captured by the camera unit 21 mounted on the mobile body 2. The acquisition by the acquisition unit 11 includes acquiring from the camera unit 21 via wireless communication, acquiring from the camera unit 21 via wireless communication and storing in the storage unit 14, and acquiring from the storage unit 14.

[0032] The alignment unit 12 performs alignment of the original image and the background image. In the present embodiment, the alignment unit 12 performs alignment of the background image relative to the original image. Alignment is a process of moving, rotating, and deforming one image to align it with another image in order to make the position, size, and shape of the same subject the same between two images. It is also called image alignment (image registration), etc. The alignment method is not limited. For example, as described later, a method of moving the images horizontally and vertically in order to minimize the sum (total) of the brightness values ​​of each pixel of the two images, a method described in non-patent document 1, etc. can be used. In the method of non-patent document 1, the results of the enlargement, reduction, and rotation amounts inferred between the two images are used for alignment.

[0033] The difference calculation unit 13 calculates the difference between the background image and the original image aligned by the alignment unit 12 for each corresponding pixel (pixel at the same position). For example, if the background image and the original image are color images, the difference in pixel value can be calculated for each pixel and for each component, for example, after conversion to a brightness value, the brightness value difference can be calculated for each pixel. And, in the case of a grayscale image, the difference in brightness value (pixel value) is calculated for each pixel. In addition, the difference calculation unit 13 stores the calculation result of the difference in the storage unit 14, or displays it on a specified display unit. In addition, in the present embodiment, the situation where the difference calculation unit 13 stores or displays the calculation result is referred to as output.

[0034] The storage unit 14 stores image data captured by the imaging unit 21 or stores calculation results by the difference calculation unit 13 .

[0035] Next, refer to Figure 2 and Figure 3 , an operation example of the information processing device 1 is described. Figure 2 The processing shown is an example of processing on each of the original image and the background image. Figure 3 The treatment shown is in Figure 2 This is an example of the alignment processing performed in step S13.

[0036] exist Figure 2 In the process shown, first, the acquisition unit 11 acquires the original image (nth frame) (step S11). Next, the acquisition unit 11 acquires the background image ((nx)th frame) (step S12). Next, the alignment unit 12 performs the process of aligning the background image with respect to the original image (step S13).

[0037] In the alignment process performed in step S13, Figure 3 As shown, first, the alignment unit 12 stores -XL in the variable dx (step S21). The value of the variable dx is a value indicating how many pixels the background image is moved in the X direction (horizontal direction). -XL indicates the number of pixels to be moved first. In this example, the background image is moved from -XL pixels to +XL pixels in the X direction.

[0038] Next, the alignment unit 12 creates an image in which the background image is shifted by dx pixels in the X direction (step S22). In addition, in step S22, the pixels at the end of the moving source are filled with zeros. Next, the alignment unit 12 calculates the sum Sx_dx of the differences in the brightness values ​​of each pixel between the original image and the image shifted in step S22 (step S23). The "dx" in sum Sx_dx is the suffix of the variable "sum Sx", and a numerical value representing the amount of movement is set. Next, the alignment unit 12 increments the variable dx by only 1 (step S24). The alignment unit 12 determines whether the variable dx is greater than the constant XL (step S25). If it is not greater (step S25: No), the alignment unit 12 performs the processing after step S22 again.

[0039] On the other hand, if the value is large (step S25: Yes), the alignment unit 12 stores -YL in the variable dy (step S26). The value of the variable dy is a value indicating how many pixels the background image is moved in the Y direction (vertical direction). -YL indicates the number of pixels to be moved first. In this example, the background image is moved from -YL pixels to +YL pixels in the Y direction.

[0040] Next, the alignment unit 12 creates an image in which the background image is shifted by dy pixels in the Y direction (step S27). In addition, in step S27, the pixels at the end of the moving source are filled with zeros. Next, the alignment unit 12 calculates the sum Sy_dy of the differences in the brightness values ​​of each pixel between the original image and the image shifted in step S27 (step S28). The "dy" in sum Sy_dy is the suffix of the variable "sum Sy", which sets a numerical value representing the amount of movement. Next, the alignment unit 12 increments the variable dy by only 1 (step S29). Next, the alignment unit 12 determines whether the variable dy is greater than the constant YL (step S30). If it is not greater (step S30: No), the alignment unit 12 performs step S27 and subsequent processing again.

[0041] On the other hand, if the distance is large (step S30: Yes), the alignment unit 12 determines the suffix "dx" of the minimum sum (Sx_dx) as the movement amount dx in the X direction, and determines the suffix "dy" of the minimum sum (Sy_dy) as the movement amount dy in the Y direction (step S31). Next, the alignment unit 12 shifts the background image by the movement amount dx in the X direction and by the movement amount dy in the Y direction to create the aligned background image (step S32).

[0042] Then, in Figure 2 In step S14, the difference calculation unit 13 calculates the difference between the original image and the image in step S13 ( Figure 3 The difference calculation unit 13 outputs the calculation result (step S15).

[0043] Figure 4 Examples of an original image, an image showing the calculation result of the difference between the background image after alignment and the original image (hereinafter referred to as an image of "alignment + frame difference"), and an image showing the calculation result of the difference between the background image and the original image without alignment (hereinafter referred to as an image of "simple frame difference"). In addition, the image showing the calculation result of the difference has a smaller brightness value (closer to black) when the difference is small, and has a larger brightness value (closer to white) when the difference is large. In addition, the original image contains carbon dioxide gas that was experimentally ejected within the range surrounded by a dotted circle in the image of "alignment + frame difference". In addition, the shooting location is a wasteland. The image of "alignment + frame difference" captures the movement of carbon dioxide gas because the difference is calculated based on the shooting position. On the other hand, in the image of "simple frame difference", the difference information corresponding to the carbon dioxide gas is not extracted because the shooting position changes. In addition, the alignment process uses the method described in non-patent document 1.

[0044] (Effects of this embodiment)

[0045] According to the present embodiment, since the difference processing is performed after the position of the background (background image) is corrected, the influence caused by the movement or vibration of the moving body 2 can be suppressed and the detection object can be easily detected compared to the case where the position correction is not performed.

[0046] <Second Embodiment>

[0047] Figure 5 It is a diagram showing a configuration example of an information processing device according to a second embodiment of the present invention. Figure 6 This is a flowchart showing an operation example of the information processing device according to the second embodiment of the present invention.

[0048] like Figure 5 As shown, Figure 1 Compared with the information processing device 1 of the first embodiment shown in the figure, the information processing device 1a in the information processing system 100a of the second embodiment is different in that: it is provided with a newly added defocus blur correction unit 15; the alignment unit 12a takes the original image and the background image processed by the defocus blur correction unit 15 as processing objects; and the difference calculation unit 13a takes the original image and the background image processed by the defocus blur correction unit 15 as processing objects.

[0049] Figure 5 The defocus blur correction unit 15 shown corrects the defocus blur of the background image and the original image. Furthermore, the alignment unit 12a aligns the defocus blur corrected background image with the defocus blur corrected original image. Furthermore, the difference calculation unit 13a calculates the difference between the aligned defocus blur corrected background image and the defocus blur corrected original image.

[0050] In addition, out of focus refers to a state where the subject is not in focus. It is not limited to the method of out of focus correction processing based on the out of focus correction unit 15. The out of focus correction processing can be set as a process of emphasizing contours, for example. Furthermore, the out of focus correction processing can be set as a method of inferring the degradation trajectory (PSF: point image distribution function) of an image with degraded image quality by a blind deconvolution method, etc., and restoring the image based on the inferred PSF.

[0051] Next, refer to Figure 6 , an operation example of the information processing device 1a is described. Figure 6 In the process shown, first, the acquisition unit 11 acquires the original image (nth frame) (step S41). Next, the acquisition unit 11 acquires the background image ((nx)th frame) (step S42). Next, the defocus blur correction unit 15 performs defocus blur correction processing on the original image and the background image (step S43). Next, the alignment unit 12a performs a process of aligning the background image after defocus blur correction with the original image after defocus blur correction (step S44).

[0052] Next, the difference calculation unit 13a calculates the difference between the original image after the defocus blur correction and the background image aligned by the alignment unit 12a in step S44 (step S45). Next, the difference calculation unit 13a outputs the calculation result (step S46).

[0053] (Effects of this embodiment)

[0054] According to the present embodiment, even for an out-of-focus and blurred image, the detection sensitivity of the detection object can be improved.

[0055] (Effect)

[0056] In the information processing device, information processing method and program of the above structure, after the background image and the original image with different frames taken by the camera unit mounted on the mobile body are aligned, the difference between the frames is calculated. Therefore, according to the information processing device, information processing method and program of the embodiment, it is possible to easily detect the detection object such as gas leakage.

[0057] (Other embodiments)

[0058] As mentioned above, although the embodiment of the present invention is described in detail with reference to the drawings, the specific structure is not limited to this embodiment, and design changes and the like that do not depart from the scope of the gist of the present invention are also included.

[0059] <Computer Architecture>

[0060] Figure 7 This is a schematic block diagram showing the structure of a computer involved in at least one embodiment.

[0061] The computer 90 includes a processor 91 , a main memory 92 , a storage device (storage) 93 , and an interface 94 .

[0062] The information processing devices 1 and 1a are installed in a computer 90. The operations of the processing units are stored in a storage device 93 in the form of a program. The processor 91 reads the program from the storage device 93 and expands it into the main memory 92, and executes the processing according to the program. The processor 91 reserves storage areas corresponding to the storage units in the main memory 92 according to the program.

[0063] The program can also be used to implement a part of the function of the computer 90. For example, the program can function by combining with other programs stored in a storage device, or by combining with other programs installed in other devices. In addition, in other embodiments, the computer can also have a customized LSI (Large Scale Integrated Circuit) such as PLD (Programmable Logic Device) in addition to the above structure, or instead of the above structure. Examples of PLD include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), FPGA (Field Programmable Gate Array), etc. In this case, part or all of the functions implemented by the processor can be implemented by the integrated circuit.

[0064] Examples of the storage device 93 include a HDD (Hard Disk Drive), an SSD (Solid State Drive), a magnetic disk, an optical magnetic disk, a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a semiconductor memory. The storage device 93 may be an internal medium directly connected to the bus of the computer 90, or an external medium connected to the computer 90 via an interface 94 or a communication line. Furthermore, when the program is transmitted to the computer 90 via a communication line, the computer 90 receiving the transmission may expand the program to the main memory 92 and execute the above-mentioned processing. In at least one embodiment, the storage device 93 is a non-temporary tangible storage medium.

[0065] <Notes>

[0066] The information processing devices 1 and 1 a described in each embodiment can be understood, for example, as follows.

[0067] (1) The information processing device 1 or 1a according to the first embodiment comprises: an acquisition unit 11 for acquiring a first image and a second image of different frames captured by an imaging unit 21 mounted on a mobile body 2; an alignment unit 12 or 12a for aligning the first image and the second image; and a difference calculation unit 13 or 13a for calculating the difference between the aligned first image and the second image. According to this embodiment and the following embodiments, a detection object such as a gas leak can be easily detected.

[0068] (2) The information processing device 1a involved in the second embodiment is the information processing device of (1), which further includes a defocus blur correction unit 15 for correcting the defocus blur of the first image and the second image, and the alignment unit 12a aligns the first image and the second image after the defocus blur correction. According to this embodiment, the detection sensitivity of the defocus blur image can be improved.

[0069] (3) The information processing device 1 or 1a according to the third aspect is the information processing device according to (1) or (2), wherein the frame interval between the first image and the second image is set according to the moving speed of the moving object. According to this aspect, the frame interval between the first image and the second image can be appropriately changed according to the speed of the moving object.

[0070] Industrial Applicability

[0071] According to the above-mentioned one aspect, according to the information processing device, the information processing method, and the program, it is possible to easily detect a detection target such as gas leakage.

[0072] Explanation of symbols

[0073] 1, 1a-information processing device, 2-moving body, 11-acquisition unit, 12, 12a-positioning unit, 13, 13a-difference calculation unit, 14-storage unit, 15-out-of-focus blur correction unit.

Claims

1. An information processing device comprising: an acquisition unit that acquires a first image and a second image of different frames captured by an imaging unit mounted on the mobile object; a positioning unit that performs positioning of the first image and the second image; and The difference calculation unit calculates a difference between the aligned first image and the second image.

2. The information processing device according to claim 1, further comprising a focus blur correction unit for correcting focus blur of the first image and the second image. The positioning unit performs positioning between the first image and the second image after defocus blur correction.

3. The information processing device according to claim 1 or 2, wherein: The frame interval between the first image and the second image is set according to the moving speed of the moving object.

4. An information processing method comprising the following steps: acquiring a first image and a second image having different frames and captured by an imaging unit mounted on a mobile object; Performing alignment between the first image and the second image; and The difference between the aligned first image and the second image is calculated.

5. A program that causes a computer to execute the following steps: acquiring a first image and a second image having different frames and captured by an imaging unit mounted on a mobile object; Performing alignment between the first image and the second image; and The difference between the aligned first image and the second image is calculated.

Images