Image processing system and image processing method

Abstract

To reduce a calculation load by suppressing an increase in the number of photographed images used for inspection.SOLUTION: A first acquisition unit configured to acquire, from a first camera that generates a first image used for an inspection record by imaging a structure, the first image, a second acquisition unit configured to acquire, from a second camera having a fixed relative position with respect to the first camera and configured to generate a second image having a resolution lower than that of the first image and including at least a part of an outer periphery of the first image, the second image, and a relative position between the first image and the second image captured in the first posture is maintained; And an image arrangement unit that arranges a reference region that is a partial region of the second image captured in the first attitude and a similar region that is a region similar to the reference region in the second image captured in the second attitude so that a relative position between the first image and the second image captured in the second attitude is maintained.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing system and an image processing method. [Background technology]

[0002] When conducting a visual inspection of a structure, a large number of digital images must be taken to document the entire area, and these images must be properly edited to document the size, location, and characteristics of any damage within the structure. When these captured images are used as inspection images, multiple images are taken of the area of ​​the structure to be inspected by slightly changing the imaging range so that some areas overlap (overlap).The similarity of each image in the group of images of the structure is then mechanically calculated and compared by a computer to match identical parts between the images.For example, in the case of two-dimensional images, similar parts of the images are used as clues to align the images, and multiple images are arranged side by side, and a combined image is created by combining each of the arranged images to create a wide-area two-dimensional image. Furthermore, when creating 3D information from images, computer vision techniques such as Structure from Motion are used. In this case, the position of the camera that captured the image is estimated by comparing the similarity of each image in the image group, and the shape of the object is determined using the principle of triangulation. Since these calculations are performed using the feature values ​​(similarity) of the pixel groups in the image as an index, it is necessary to capture images with a certain amount of overlap (lap) between them to ensure that identical parts are captured. On the other hand, in order to capture tiny damage such as cracks in concrete (on the order of a few tenths of a millimeter, such as 0.1 mm or 0.2 mm wide) in a digital image, it is necessary to set the angle of view (shooting range) so that the size of each pixel in the image is as small as possible. Here, for example, there is a technique for evaluating the state of a crack using image data of the crack (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-090548 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in order to detect minute structural damage (for example, concrete cracks as small as 0.1 to 0.2 mm wide), it is necessary to capture images with a narrow angle of view (zoomed in on the object). As mentioned above, when capturing images with a narrow shooting range, the overlap rate must be taken into consideration. Therefore, the larger the overlap rate, the more images must be captured, which leads to problems such as increased on-site work time and a larger amount of data to be managed. Furthermore, because the image has a narrow angle of view, it is relatively difficult to grasp the features in the image, and it may be difficult to match images by mechanical calculation.

[0005] The present invention has been made in consideration of these circumstances, and its purpose is to provide an image processing system and an image processing method that can suppress an increase in the number of images taken for inspection and reduce the calculation load. [Means for solving the problem]

[0006] In order to solve the above-described problems, one aspect of the present invention includes a first acquisition unit that acquires a first image from a first camera that captures an image of a structure to generate a first image used for an inspection record, a second camera that is fixed in a position relative to the first camera and generates a second image that has a lower resolution than the first image and is captured so as to include at least a part of the periphery of the first image, and a second acquisition unit that acquires a first image and a second image captured by the first camera and the second camera in a first orientation, and a second image captured by the second camera in a second orientation that is different from the first orientation. and a second image processing system including an image arrangement unit that arranges the first image captured in the first orientation and the second image captured in the second orientation by using the first image and the second image captured in the first orientation so that the relative positions of the first image and the second image captured in the first orientation are maintained and the relative positions of the first image and the second image captured in the second orientation are maintained, and by arranging the first image captured in the first orientation and the second image captured in the second orientation so that a reference area, which is a part of the second image captured in the first orientation, and a similar area, which is a region of the second image captured in the second orientation that is similar to the reference area, overlap each other.

[0007] Another aspect of the present invention is an image processing method executed by a computer that functions as an image processing device, the image processing method comprising: acquiring a first image from a first camera that captures an image of a structure to generate a first image used for inspection records; acquiring a second image from a second camera that is fixed in a position relative to the first camera and generates a second image that has a lower resolution than the first image and is captured so as to include at least a portion of the periphery of the first image; using the first and second images captured by the first and second cameras in a first orientation and the first and second images captured in a second orientation that is different from the first orientation, the method maintains the relative positions of the first and second images captured in the first orientation and the relative positions of the first and second images captured in the second orientation; and arranging a reference area that is a portion of the second image captured in the first orientation and a similar area that is a region of the second image captured in the second orientation that is similar to the reference area so that they overlap, thereby arranging the first image captured in the first orientation and the second image captured in the second orientation. [Effects of the Invention]

[0008] As described above, according to the present invention, by using two cameras with different angles of view, a low-resolution camera and a high-resolution camera in combination, it is possible to suppress an increase in the number of images taken for inspection and reduce the calculation load. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic external view showing the configuration of an image processing system S according to an embodiment of the present invention. [Figure 2] 10 is a diagram showing the relationship between the imaging areas of a first camera C1 and a second camera C2. FIG. [Figure 3] FIG. 2 is a functional block diagram showing the general functions of the image processing system S. [Figure 4] 10 is a flowchart illustrating the operation of the image processing device G. [Figure 5] FIG. 10 is a diagram showing an example of a case where alignment is performed on a second image. [Figure 6] FIG. 10 is a diagram showing an example of an image in which a first image is arranged relative to a second image. DETAILED DESCRIPTION OF THE INVENTION

[0010] An image processing system S according to an embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic external view showing the configuration of an image processing system S according to an embodiment of the present invention, and FIG. 2 is a diagram showing the relationship between the imaging areas of a first camera C1 and a second camera C2. The image processing system S includes a camera system CS and an image processing device G. The camera system CS includes a first camera C1, a second camera C2, and a fixed member F. The first camera C1 and the second camera C2 are each attached to a fixed member F. The first camera C1 and the second camera C2 are attached aligned vertically to a common fixed member F, and thus their relative positional relationship is fixed. Here, the first camera C1 and the second camera C2 are aligned vertically and fixed, but they may also be aligned horizontally and fixed.

[0011] The first camera C1 and the second camera C2 are fixed to a fixed member F and adjusted so that the approximate center position of the shooting range Rc2 shot by the second camera C2 overlaps with the center position of the shooting range Rc1 shot by the first camera C1. For example, as shown in Fig. 2, the positional relationship between the two cameras (first camera C1 and second camera C2) is fixed, and the angle of view of the first camera C1 is adjusted to fit in the center of the imaging range of the second camera C2. The second camera C2 can capture an image that includes at least a portion of the periphery of the first image captured by the first camera C1. For example, the second image captured by the second camera C2 can be set to have an area that is larger than the periphery of the first image captured by the first camera C1 and that surrounds the first image, and an image can be obtained based on such settings. The imaging directions of the first camera C1 and the second camera C2 are generally the same. The first camera C1 and the second camera C2 capture images of a structure K. The structure K is a structure to be inspected. The structure K may be, for example, any of a building, a road, a tunnel, a bridge, etc.

[0012] The resolution of the first camera C1 is higher than the resolution of the second camera C2. Therefore, the image obtained from the first camera C1 is used as an inspection image of the structure K, and the image obtained from the second camera C2 can be used as a guide image for determining the position to place the inspection image obtained from the first camera C1. The inspection image can be used for at least one of the following purposes: an image for detecting damage to the structure, an image for recording in a report reporting the inspection results, etc.

[0013] The image processing device G acquires an image captured by the first camera C1 and an image captured by the second camera C2, and uses these images to generate an array image in which images captured by the first camera C1 at different postures are arranged.

[0014] FIG. 3 is a functional block diagram showing an outline of the functions of the image processing system S. As shown in FIG. The first camera C1 outputs the captured image data to the image processing device G. The second camera C2 outputs the captured image data to the image processing device G. The first camera C1 and the second camera C2 may be communicably connected to the image processing device G via a communication cable and may transmit imaging data to the image processing device G. Alternatively, the first camera C1 and the second camera C2 may be communicably connected to the image processing device G wirelessly and may transmit imaging data to the image processing device G.

[0015] The image processing device G includes a communication unit 101, a storage unit 102, a first acquisition unit 103, a second acquisition unit 104, an image arrangement unit 105, an input unit 106, an output unit 107, and a control unit . The communication unit 101 communicates with the first camera C1 and the second camera C2, and transmits and receives various data. The storage unit 102 stores various types of data. For example, the storage unit 102 stores configuration parameter data that represents the relative positional relationship between the first image and the second image. The storage unit 102 is configured by a storage medium, such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), random access read / write memory (RAM), read-only memory (ROM), or any combination of these storage media. The storage unit 102 may be, for example, a nonvolatile memory.

[0016] The first acquisition unit 103 acquires, via the communication unit 101, a first image from a first camera C1 that captures an image of the structure K to generate a first image to be used for inspection recording. The second acquisition unit 104 acquires the second image from a second camera C2, which is a second camera whose position relative to the first camera C1 is fixed, has a lower resolution than the first camera C1, and generates a second image that is captured so as to include at least a portion of the outer periphery of the first image, via the communication unit 101.

[0017] The image arrangement unit 105 uses the first image and the second image taken by the first camera C1 and the second camera C2 in a first attitude, and the first image and the second image taken in a second attitude that is different from the first attitude, so that the relative positions of the first image and the second image taken in the first attitude are maintained and the relative positions of the first image and the second image taken in the second attitude are maintained, and arranges the first image taken in the first attitude and the second image taken in the second attitude by arranging a reference area, which is a part of the area of ​​the second image taken in the first attitude, and a similar area, which is an area of ​​the second image taken in the second attitude that is similar to the reference area, so that they overlap.

[0018] Furthermore, the image arrangement unit 105 arranges the second image captured in the second orientation relative to the second image captured in the first orientation so that the reference area and the similar area overlap, and arranges the first image captured in the second orientation relative to the arranged second image captured in the second orientation based on the configuration parameter data.

[0019] The input unit 106 acquires various data from the outside by taking in the data and accepting operation contents from an external input device (keyboard, mouse, etc.). The output unit 107 outputs various data to an external display device or electronic device. The control unit 108 controls each unit of the image processing device G.

[0020] The communication unit 101, first acquisition unit 103, second acquisition unit 104, image placement unit 105, input unit 106, output unit 107, and control unit 108 of the image processing device G described above may be configured as a processing device such as a CPU (central processing unit) or a dedicated electronic circuit. Furthermore, the image processing device G may be a physical server or a cloud server provided by a cloud computing service.

[0021] Next, the operation of the image processing system S described above will be described. FIG. 4 is a flowchart illustrating the operation of the image processing device G of the image processing system S. The user attaches the first camera C1 and the second camera C2 to the fixed member F and adjusts the attachment positions etc. so that the imaging range has the relationship shown in Fig. 2. Here, the outer periphery of the second image generated by capturing an image with the second camera C2 is set to be larger than the outer periphery of the first image generated by capturing an image with the first camera C1, and the first image is adjusted so that it is included in the ends of each side (top end, bottom end, left end, right end) of the second image.

[0022] After various adjustments have been made, the input unit 106 of the image processing device G acquires configuration parameters based on the results of the calibration (step S101). The control unit 108 stores the acquired configuration parameters by writing them into the storage unit 102 (step S102). The configuration parameters may be input by the user via an input device such as a keyboard. The configuration parameters may be, for example, data including at least one of the following (a), (b), and (c): (a) Lens distortion correction parameters for the first camera (inspection camera) (b) Lens distortion correction parameters for the second camera (guide camera) (c) External parameters (relative orientation) of the first camera (inspection camera) relative to the second camera (guide camera)

[0023] Using the camera system CS configured as above, a wide-area 2D image for detecting defects in a structure can be generated by the following steps. The following algorithm can also be extended to 3D calculations using Structure from Motion.

[0024] Next, in the first posture, the user uses the camera system CS to capture an image of the structure so that the area to be inspected is included. When the user presses the capture button, the camera system CS synchronizes the shutters of the first camera C1 and the second camera C2. For example, when the capture button is pressed for each of the first camera C1 and the second camera C2, the cameras capture images at synchronized timing (step S103). By capturing images in a synchronized manner, as shown in FIG. 2, an image is generated in which the center of the capture range Rc1 of the image data captured by the first camera C1 overlaps with the approximate center of the capture range Rc2 of the image data captured by the second camera C2. The first camera C1 and the second camera C2 then generate image data captured according to their respective capture timings and output the data to the image processing device G.

[0025] The first acquisition unit 103 of the image processing device G acquires image data transmitted from the first camera C1 via the communication unit 101, and the second acquisition unit 104 acquires image data transmitted from the second camera C2 via the communication unit 101. The control unit 108 associates the image data acquired by the first acquisition unit 103 with the image data acquired by the second acquisition unit 104 and stores them in the storage unit 102 (step S104). Here, the images stored are defined such that the center of the image capture range Rc1 captured by the first camera C1 overlaps with the approximate center of the image capture range Rc2 captured by the second camera C2. The positional relationship between these two sets of image data is determined by configuration parameters (for example, the above-mentioned (c) external parameters).

[0026] Next, the user changes the imaging direction (imaging attitude) of the camera system CS so that the imaging direction is different from the first attitude (step S105). The amount of change in the imaging direction from the first attitude to the second attitude may be such that the next image captured by the first camera C1 is adjacent to the image data captured by the first camera C1 in step S103. Here, the imaging target area of ​​the second image is set to include the periphery (upper end, lower end, left end, right end) of the first image. Therefore, the imaging direction to which the camera system CS changes the imaging direction may be any direction in which the first image can be positioned using the second image as a clue, and may be either the right or left side in the horizontal direction, or either the upper or lower side in the vertical direction. After the postures of the first camera C1 and the second camera C2 are changed, the first camera C1 and the second camera C2 each capture images according to synchronized capture timing (step S106), generate captured image data, and output it to the image processing device G. The control unit 108 associates the image data acquired by the first acquisition unit 103 with the image data acquired by the second acquisition unit 104 and stores them in the storage unit 102 (step S107). Thereafter, the posture of the structure to be inspected is changed and images are sequentially taken by the camera system CS, and multiple sets of image data for combinations of the first image and the second image are stored in the storage unit 102.

[0027] When the first image and the second image captured in the first and second orientations are obtained, respectively, the image arrangement unit 105 determines the feature amounts of the second image captured in the first orientation and the second image captured in the second orientation, and aligns the images by matching these feature amounts (step S108). The feature amount of the image may be, for example, at least one of the brightness distribution, color distribution, color appearance rate, etc. included in the image data, or other evaluation items may be used. The image arrangement unit 105 performs the alignment so that the features indicated by the feature amount data of the second image in the first orientation overlap with the features indicated by the feature amount data of the second image in the second orientation.

[0028] Here, FIG. 5 is a diagram showing an example of a case where the second image is aligned. 5 shows a state in which second image 522 captured in the second orientation is superimposed by aligning its position with second image 512 captured in the first orientation. Here, image region 555, in which partial regions of second image 512 and partial regions of second image 522 have similar feature amounts, is superimposed, so that second image 522 is aligned horizontally with second image 512 and arranged so that partial regions overlap. The second image (image from the guide camera) has a wider capture range and a smaller image resolution (number of pixels) than the first image (image from the inspection camera), allowing for fast and stable alignment calculations.

[0029] Next, the image arrangement unit 105 performs a process of projecting the first image (image of the inspection camera) onto the second image (image of the guide camera) determined by the calculation in step S108 (step S109). The image arrangement unit 105 can determine the projective transformation matrix between the two cameras, the first camera C1 and the second camera C2, from the configuration parameters (lens parameters and external parameters of each camera) stored in the storage unit 102. In other words, once the position of the second image (image of the guide camera) is determined, the position of the first image (image of the inspection camera) that is captured simultaneously when the second image is captured is also determined.

[0030] FIG. 6 is a diagram showing an example of an image in which a first image is arranged relative to a second image. In this Figure 6, in an image in which a second image 522 captured in a second posture is aligned with a second image 512 captured in a first posture, a first image 511 captured in the first posture is positioned with respect to the second image 512 captured in the first posture based on the configuration parameters of the first camera C1, and a first image 521 captured in the second posture is positioned with respect to the second image 522 captured in the second posture based on the configuration parameters of the second camera C2.

[0031] The image arrangement unit 105 performs the processes of steps S108 and S109 on the remaining image data stored in the storage unit 102. For example, of the combination of the first image and the second image, the first image or the second image obtained in either the first or second orientation for which the process of step S109 was performed may be set as the image obtained in the first orientation, and the image stored in the storage unit 102 for which the alignment process has not yet been performed may be read out as the image in the second orientation, and the processes of steps S108 and S109 may be performed. This makes it possible to obtain wide-area image data (array image) in which a plurality of first images, which are images of the structure to be inspected taken in different postures, are arranged. Furthermore, by performing the above calculations on a group of photographs taken of the same plane, it is possible to quickly align high-resolution photographs using a small number of photographs.

[0032] According to the embodiment described above, the first image is positioned relative to the second image after alignment is performed based on the feature data of each second image captured in different postures. This eliminates the need to consider the overlap rate of the first image (image for inspection). This makes it possible to reduce the number of high-resolution images captured, for example, to about 1 / 5, thereby reducing the shooting time and simplifying the shooting work.

[0033] In addition, since the image capture range used for alignment is wider, it is possible to make the method more robust against the shape of the structure. Also, since the resolution of the image used for alignment calculation can be smaller than that of the inspection image, calculation time can be reduced.

[0034] According to the embodiment described above, by combining and using two cameras with different angles of view (the second camera C2 with low resolution and a wide angle of view and the first camera C1 with high resolution and a narrow angle of view), it is possible to eliminate the overlap ratio of images and improve the accuracy of alignment using feature quantities. The overlap ratio is the ratio of areas where similar image portions overlap in a first image captured in a first posture and a first image captured in a second posture. In this embodiment, instead of generating an array image by overlapping a first image captured in the first posture with a first image captured in the second posture, a second image captured in the first posture with a second image captured in the second posture is overlapped, and then a corresponding first image is positioned for each second image. This eliminates the need to ensure an overlap ratio for overlapping a first image captured in the first posture with a first image captured in the second posture to generate an array image. Therefore, it is possible to reduce the overlap ratio (reducing overlap), and a smaller number of first images can be generated when generating an array image, resulting in more efficient generation.

[0035] Concrete inspection records can be created from images taken with a digital camera. By reducing the time required to acquire and process data, the cost of infrastructure inspection can be reduced.

[0036] The image processing device G in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may be for implementing only a portion of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0037] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0038] 101 Communications Department 102 Storage section 103 First acquisition part 104 Second acquisition part 105 Image placement section 106 Input section 107 Output section 108 Control Unit 511,521 First image 512,522 2nd image 555 Image Area C1 First Camera C2 Second Camera CS Camera System F Fixing member G Image Processing Device K structure Rc1, Rc2 shooting range S Image Processing System

Claims

1. a first acquisition unit that acquires a first image from a first camera that captures an image of a structure to generate a first image used for inspection recording; a second acquisition unit that acquires a second image from a second camera that is fixed in position relative to the first camera and generates a second image that has a lower resolution than the first image and is captured so as to include at least a portion of the periphery of the first image; and a first image and a second image captured by the first camera and the second camera in a first attitude; using a first image and a second image captured in a second posture that is different from the first posture, a relative position of the first image and the second image captured in the first attitude is maintained, and a relative position of the first image and the second image captured in the second attitude is maintained; a reference area that is a part of the second image captured in the first orientation and a similar area that is a part of the second image captured in the second orientation and is similar to the reference area are arranged so as to overlap each other; an image arrangement unit that arranges a first image captured in the first attitude and a second image captured in the second attitude; An image processing system having:

2. The outer periphery of the second image is larger than the outer periphery of the first image and is disposed so as to surround the outer periphery of the first image. The image processing system according to claim 1 .

3. a storage unit that stores configuration parameter data that represents a relative positional relationship between the first image and the second image; The image placement unit disposing the second image captured in the second orientation relative to the second image captured in the first orientation so that the reference region and the similar region overlap; and aligning a first image captured in the second orientation with a second image captured in the aligned second orientation based on the configuration parameter data.

3. The image processing system according to claim 1.

4. An image processing method executed by a computer functioning as an image processing device, comprising: acquiring a first image from a first camera that captures an image of the structure to generate a first image used for inspection recording; acquiring a second image from a second camera whose position relative to the first camera is fixed, the second image having a lower resolution than the first image and captured so as to include at least a portion of the periphery of the first image; a first image and a second image captured by the first camera and the second camera in a first attitude; using a first image and a second image captured in a second posture that is different from the first posture, a relative position of the first image and the second image captured in the first attitude is maintained, and a relative position of the first image and the second image captured in the second attitude is maintained; a reference area that is a part of the second image captured in the first orientation and a similar area that is a part of the second image captured in the second orientation and is similar to the reference area are arranged so as to overlap each other; A first image captured in the first orientation and a second image captured in the second orientation are arranged. Image processing methods.

Images