Image processing device, method, and program

Abstract

Provided are an image processing device, a method, and a program with which it is possible to generate an image that is effective in the summary of an object to be grasped, even when a normal composite image cannot be generated. The image processing apparatus includes a processor. The processor performs the following processing: acquiring information of an image group and shooting positions of images constituting the image group; performing a first synthesis process on the image group to generate a first synthesized image; determining whether the quality of the first composite image satisfies a first reference; and when the quality of the first composite image does not satisfy the first reference, generating a second composite image by performing a second composite process on the group of images or the first composite image based on the information of the imaging position.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, method, and program, and more particularly to an image processing apparatus, method, and program for connecting multiple images to generate a single composite image. Background Technology

[0002] As a technique for generating a single image that captures a large area, stitching is known. In stitching, an object is divided into multiple parts for photography, and the obtained multiple images are joined together to generate a single composite image (e.g., Patent Documents 1-4, etc.).

[0003] Previous technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2023-7662 Patent Document 2: Japanese Patent Application Publication No. 2005-30961 Patent Document 3: Japanese Patent Application Publication No. 2003-111073 Patent Document 4: Japanese Patent Application Publication No. 2002-188998 Summary of the Invention

[0004] One embodiment of the present invention provides an image processing apparatus, method, and program that can generate an image that is effective in summarizing the subject matter, even when a normal synthetic image cannot be generated.

[0005] means for solving technical problems (1) An image processing apparatus comprising a processor, wherein the processor performs the following processing: Obtain information about the image group and the photographic locations of the images that make up the image group; The first composite image is generated by performing a first composite process on the image group; Determine whether the quality of the first composite image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the photographic location.

[0006] (2) The image processing apparatus according to (1), wherein, The processor, as the first synthesis process, performs the following: feature point matching between the images constituting the image group, and generates the first synthesized image based on the feature point matching results.

[0007] (3) The image processing apparatus according to (1) or (2), wherein, The processor performs the following processing: determines the degree of fragmentation of the first synthesized image, thereby determining whether the quality of the first synthesized image meets the first benchmark.

[0008] (4) The image processing apparatus according to (3), wherein, The processor, as the second compositing process, performs the following: based on the information of the photographic location, it configures at least a portion of the fragments of the first composite image as fragments different from the fragments and performs compositing, and generates an image with a less fragmented degree than the first composite image as the second composite image.

[0009] (5) The image processing apparatus according to (4), wherein, The processor further performs the following processing: before performing the second synthesis process, it extracts fragments with distortions exceeding the specified range from multiple fragments and corrects the distortions of the extracted fragments with distortions.

[0010] (6) The image processing apparatus according to any one of (1) to (5), wherein, The processor performs the following processing: Determine whether the quality of the second composite image meets the second criterion; and If the quality of the second composite image does not meet the second benchmark, an image is generated that configures the images constituting the image group according to the information of the shooting location as the second composite image.

[0011] (7) The image processing apparatus according to any one of (1) to (6), wherein, The processor performs the following processing: determining whether the degree of distortion of at least the first synthesized image is within the allowable range, thereby determining whether the quality of the first synthesized image meets the first benchmark.

[0012] (8) The image processing apparatus according to (7), wherein, The processor, as the second compositing process, performs the following steps: it configures the images constituting the image group according to the information of the shooting location to generate a second composite image.

[0013] (9) The image processing apparatus according to (7) or (8), wherein, The processor performs the following processing: Determine the maximum and minimum values ​​of the width in the first direction of the first composite image; Calculate the difference between the maximum and minimum width in the first direction; and The difference is determined to be below a threshold, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

[0014] (10) The image processing apparatus according to any one of (7) to (9), wherein, The processor performs the following processing: Obtain information about the photographic conditions of the images that make up the image group; Estimate the width of the first composite image in the second direction based on information about the photographic conditions; Measure the width of the first composite image in the second direction; Calculate the difference between the estimated and measured values ​​of the width in the second direction; and The difference is determined to be below a threshold, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

[0015] (11) The image processing apparatus according to any one of (7) to (10), wherein, The processor performs the following processing: Calculate the deviation of the synthesis parameters determined based on the feature point matching results performed between adjacent images; and The presence or absence of images with a deviation exceeding the threshold is determined, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

[0016] (12) The image processing apparatus according to any one of (1) to (11), wherein, The processor performs the following processing: based on the information of the shooting location, it performs feature point matching between adjacent images.

[0017] (13) The image processing apparatus according to any one of (1) to (12), wherein, The image group consists of images taken by a photographic device equipped with multiple cameras while changing position. The information on the shooting position consists of the configuration position of the camera in the shooting device and the position where the shooting device performs the shooting.

[0018] (14) The image processing apparatus according to any one of (1) to (13), wherein, The processor performs the following processing: For an object that is divided into multiple segments along its length, information on the image group and the photographic positions of the images constituting the image group is obtained for each segment; Generate either the first composite image or the second composite image for each segment; and Generate a first output image, which will be either a first composite image or a second composite image generated for each segment, configured according to the arrangement of the segments.

[0019] (15) The image processing apparatus according to any one of (1) to (14), wherein, The processor performs the following processing: Analyze the images constituting the image group, the first composite image, or the second composite image, and detect damage to the surface of the object; and A second output image is generated, which contains the detection results of damage superimposed on the first or second composite image.

[0020] (16) An image processing method, comprising the following steps: Obtain information about the image group and the photographic locations of the images that make up the image group; The first composite image is generated by performing a first composite process on the image group; Determine whether the quality of the first composite image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the photographic location.

[0021] (17) An image processing program that enables a computer to perform the following functions: Obtain information about the image group and the photographic locations of the images that make up the image group; The first composite image is generated by performing a first composite process on the image group; Determine whether the quality of the first composite image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the photographic location. Attached Figure Description

[0022] Figure 1 It is a diagram showing the general structure of a photographic system.

[0023] Figure 2 It is a three-dimensional diagram showing the structure of a multi-view camera device.

[0024] Figure 3 This is the front view showing the structure of the multi-view camera device.

[0025] Figure 4 This is a side view showing the structure of a multi-view camera device.

[0026] Figure 5 This is a block diagram representing the electrical structure of a multi-view camera device.

[0027] Figure 6 This is a diagram illustrating an example of the hardware structure of a control device.

[0028] Figure 7 This is a functional block diagram of the photographic control function of the control device.

[0029] Figure 8 This is a block diagram illustrating the main functions of the camera control unit.

[0030] Figure 9 This is an example of a live view display.

[0031] Figure 10 This is a functional block diagram representing the image processing capabilities of the control device.

[0032] Figure 11 This is a conceptual diagram of the synthesis process based on feature point matching.

[0033] Figure 12 This is a diagram representing an example of a fragmented synthetic image.

[0034] Figure 13 This is an example of a composite image that has been distorted.

[0035] Figure 14 This is an example of a composite image that has been distorted.

[0036] Figure 15 This is a conceptual diagram of the synthesis process in the second synthesis processing unit.

[0037] Figure 16 This is a diagram representing an example of the synthesis result.

[0038] Figure 17 This is a concept diagram of the generation of composite images based on the third composite image processing unit.

[0039] Figure 18 This is a diagram showing an example of a composite image generated by the third composite processing unit.

[0040] Figure 19 It is a flowchart showing the sequence of processes used to generate the composite image.

[0041] Figure 20 This is a conceptual diagram of a method for determining images from repeated photographs.

[0042] Figure 21 This is a diagram illustrating an example of a fragmented original image in a distorted state.

[0043] Figure 22 This is a function block diagram representing an example of generating a secondary composite image by correcting distortion.

[0044] Figure 23 This is an example of generating a secondary composite image by performing distortion correction.

[0045] Figure 24 This is an example of a diagram showing the composite result when a composite image is generated by segmenting the image into multiple segments.

[0046] Figure 25 This is an example of a magnified display of the entire image.

[0047] Figure 26 This is an example of a diagram showing the results of damage detection.

[0048] Figure 27 This is an example of photographing an object using a single camera. Detailed Implementation

[0049] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0050] Here, we will take the case of so-called feature point matching (correspondence point search) synthesis as an example to illustrate the splicing-based synthesis process.

[0051] In feature point matching-based image synthesis, feature points are extracted from each image, similar feature points are matched, synthesis parameters are generated based on their correspondence, and the synthesized image is processed. Therefore, if there are missing or omitted features in the photograph, insufficient repetition between adjacent images, or jitter or blurring in the image, synthesis may sometimes fail. These are due to photographic factors, but sometimes synthesis failure can also be caused by the photographic subject. For example, if the surface has significant unevenness, the entry of shadows or the shape of the shadows may change depending on the photographic position, sometimes feature point matching may fail, leading to synthesis failure. Furthermore, if the shape of the photographic subject deviates significantly from the intended shape model, it can also cause synthesis failure. If synthesis fails, the image will be fragmented or a severely distorted image will be generated. "Image fragmentation" refers to the image being split into multiple parts.

[0052] In cases where compositing fails, re-photographing is necessary to obtain a normal composite image. However, depending on the subject matter, re-photographing can sometimes be difficult. On the other hand, depending on the intended use, sometimes only a general outline of the subject matter is required.

[0053] In this embodiment, the objective is to provide an image processing apparatus that can generate an image that is effective in summarizing the subject matter, even when it is impossible to generate a normal composite image.

[0054] [Photography System] Here, we will take the application of the present invention to a photographic system for inspecting tunnel structures as an example.

[0055] Tunnel structures such as water diversion channels for hydroelectric power generation facilities and subway tunnels are regularly inspected to ensure their safety. In recent years, visual inspection has been gradually replaced by image-based inspection. Image-based inspection is carried out by photographing the walls of the tunnel structure with a camera and then detecting damage such as cracks from the obtained images through visual inspection or image processing.

[0056] Regarding photography, the area to be photographed is divided into multiple parts (so-called segmented photography). Furthermore, in order to generate a composite image from the photographed images (so-called panoramic composite image), each image is photographed by partially repeating (overlapping) between adjacent images.

[0057] [Structure of a photographic system] Figure 1 It is a diagram showing the general structure of a photographic system.

[0058] As described above, the photographic system 1 of this embodiment is configured as the system of the inner wall surface of the photographic tunnel structure TS. The tunnel structure TS, which is the photographic object (subject), has an arc-shaped cross-sectional shape (semi-circular).

[0059] like Figure 1 As shown, the photography system 1 of this embodiment includes a multi-camera photography device 10 that uses multiple cameras to photograph the inner wall of the tunnel structure TS, and a control device 100 that controls the multi-camera photography device 10 and processes the images photographed by the multi-camera photography device 10.

[0060] The multi-camera camera 10, for example, is mounted on a trolley Tr and takes pictures while moving within the tunnel structure TS (taking pictures while changing position). The trolley Tr may be equipped with an electric auxiliary function (the function of assisting human operation through an electric motor) as needed.

[0061] [Multi-view camera device] Figure 2 It is a three-dimensional diagram showing the structure of a multi-view camera device. Figure 3 This is the front view showing the structure of the multi-view camera device. Figure 4 This is a side view showing the structure of a multi-view camera setup. Figures 2 to 4 In this diagram, the x-axis, y-axis, and z-axis are three mutually orthogonal axes. The plane containing the x-axis and y-axis is considered the horizontal plane, and the direction of the z-axis is considered the vertical direction. Furthermore, the direction of the x-axis is set as the direction of travel of the trolley Tr, and the + direction of the x-axis (…) is… Figure 4 The right direction of the x-axis is set as the direction of travel during photography. Therefore, the + direction of the x-axis ( Figure 4 The left direction) is the forward direction (forward direction) of the trolley Tr and the multi-eye camera device 10, - direction ( Figure 4 The left direction is the rear direction (reverse direction) of the trolley Tr and the multi-eye camera device 10.

[0062] The multi-view camera system 10 is composed of multiple cameras and multiple lighting devices. The number of cameras and lighting devices is adjusted according to the subject being photographed. Here, we will explain the case where the multi-view camera system 10 is composed of 5 cameras C1 to C5 and 5 lighting devices L1 to L5 as an example.

[0063] The multi-camera device 10 has a frame 11 for mounting multiple cameras C1 to C5 and multiple lighting devices L1 to L5.

[0064] The frame 11 mainly consists of a flat base plate 12, prism-shaped columns 13 mounted on the base plate 12, and a circular mounting base 14 mounted on the columns 13. The base plate 12 functions as a mounting section on the trolley Tr. The columns 13 and the mounting base 14 are perpendicular to the base plate 12. An axis Ax passing through the center of the mounting base 14 and parallel to the x-axis is defined as the axis of the multi-view camera device 10.

[0065] Cameras C1 to C5 and lighting devices L1 to L5 are mounted on mounting base 14 via brackets B1 to B5. Hereinafter, as needed, camera C1 will be referred to as "first camera C1", camera C2 as "second camera C2", camera C3 as "third camera C3", camera C4 as "fourth camera C4", and camera C5 as "fifth camera C5" to distinguish each camera C1 to C5.

[0066] Furthermore, lighting device L1 is referred to as "first lighting device L1", lighting device L2 as "second lighting device L2", lighting device L3 as "third lighting device L3", lighting device L4 as "fourth lighting device L4", and lighting device L5 as "fifth lighting device L5" to distinguish each lighting device L1 to L5.

[0067] Furthermore, stent B1 is referred to as "Stent B1 No. 1", stent B2 as "Stent B2 No. 2", stent B3 as "Stent B3 No. 3", stent B4 as "Stent B4 No. 4", and stent B5 as "Stent B5" to distinguish each stent B1 to B5.

[0068] The first camera C1 and the first lighting device L1 are mounted on the mounting base 14 via the first bracket B1. The second camera C2 and the second lighting device L2 are mounted on the mounting base 14 via the second bracket B2. The third camera C3 and the third lighting device L3 are mounted on the mounting base 14 via the third bracket B3. The fourth camera C4 and the fourth lighting device L4 are mounted on the mounting base 14 via the fourth bracket B4. The fifth camera C5 and the fifth lighting device L5 are mounted on the mounting base 14 via the fifth bracket B5.

[0069] Each bracket B1 to B5 is positioned relative to the mounting base 14 on the same circumference centered on axis Ax. Furthermore, each bracket B1 to B5 is movably mounted relative to the mounting base 14 along the circumferential direction with respect to axis Ax within a specified angular range (e.g., 30°). Each bracket B1 to B5 is fixed to the mounting base 14 by a clamp (e.g., a hinge clamp) CL. Therefore, the circumferential position of each bracket B1 to B5 can be adjusted by releasing the clamp CL.

[0070] Each camera C1 to C5 is mounted on the camera mounting section of the brackets B1 to B5. Similarly, each lighting device L1 to L5 is mounted on the lighting mounting section of the brackets B1 to B5. Each camera C1 to C5 is mounted on the camera mounting section, for example, using a three-pronged threaded hole. Furthermore, each lighting device L1 to L5 is mounted on the lighting mounting section by bolts securing the arm portion.

[0071] Cameras C1-C5 and lighting devices L1-L5, mounted on mounting base 14 via brackets B1-B5, are arranged in a predetermined posture on frame 11. Specifically, they are arranged radially (normally) outward from the axis Ax of the multi-eye camera device 10 in a plane orthogonal to the axis Ax (in the zy plane). More specifically, cameras C1-C5 are arranged such that their optical axes are radially (normally) outward from the axis Ax of the multi-eye camera device 10. Furthermore, cameras C1-C5 are mounted with their main bodies parallel to mounting base 14 (parallel to the zy plane) (the bottom edge of the image sensor is mounted parallel to the zy plane). Thus, each camera C1-C5 is arranged circumferentially at predetermined intervals in the zy plane with the axis Ax of the multi-eye camera device 10 as its center. Lighting devices L1-L5 are arranged with their illumination direction facing radially (normally) outward from the axis Ax of the multi-eye camera device 10. As a result, the cameras C1 to C5 and the lighting devices L1 to L5 are arranged radially in the zy plane with the axis Ax of the multi-eye camera device 10 as the center.

[0072] Here, as described above, brackets B1 to B5 are movably installed circumferentially within a specified angle range with the axis Ax of the multi-view camera device 10 as the center. Figure 3 and Figure 4 This shows the state with each bracket B1 to B5 fixed in the reference position. Viewed from the front by fixing each bracket B1 to B5 in the reference position ( Figure 3When the camera is positioned, the first camera C1 and the first lighting device L1 are positioned at 330° (-30°). The second camera C2 and the second lighting device L2 are positioned at 30°. The third camera C3 and the third lighting device L3 are positioned at 90°. The fourth camera C4 and the fourth lighting device L4 are positioned at 150°. The fifth camera C5 and the fifth lighting device L5 are positioned at 210°.

[0073] Each bracket B1 to B5 is movable within a range of ±15° along the circumference from the reference position. Therefore, the positions of each camera C1 to C5 and the lighting devices L1 to L5 can be adjusted within a range of ±15° along the circumference from the reference position.

[0074] The multi-camera imaging device 10 configured as described above includes five cameras C1 to C5 and lighting devices L1 to L5 arranged on a circumference centered on the device's axis Ax. The positions of each camera C1 to C5 are adjusted so that the photographic area overlaps between adjacent cameras. Preferably, the positions of each camera C1 to C5 are adjusted to ensure a repetition rate of at least 10%. The repetition rate refers to the proportion of overlapping photographic areas between adjacent cameras (the proportion of overlapping images captured).

[0075] The cameras C1 to C5 used are digital video cameras. There is no particular limitation on the type of digital video camera. As long as it has the function of electronically recording images (still images or moving images), it is acceptable. As an example, an interchangeable-lens digital video camera is used. In this embodiment, cameras C1 to C5 have a storage device (storage medium) and store the captured images in the storage device. The storage device can be a built-in memory or a replaceable memory card.

[0076] The lighting devices L1 to L5 used are not particularly limited. Halogen lamps are used as an example. Other examples include LED (light emitting diode) lamps and xenon lamps. In this embodiment, lighting devices with an adjustable illumination angle (illumination direction) are used. Each lighting device L1 to L5 adjusts its illumination angle (illumination direction) by rotating (swinging in the back-and-forth direction) around an axis orthogonal to the optical axis of cameras C1 to C5. Each lighting device L1 to L5 has an illumination range capable of covering the photographic range of its corresponding cameras C1 to C5.

[0077] [Electrical Structure of a Multi-Eye Camera] Figure 5 This is a block diagram representing the electrical structure of a multi-view camera device.

[0078] like Figure 5As shown, the multi-eye camera device 10 has a relay device 20, which is communicatively connected to the control device 100.

[0079] The relay device 20 is, for example, a computer with communication capabilities. Each camera C1-C5 and each lighting device L1-L5 is connected to the relay device 20. The connection method between each camera C1-C5 and the relay device 20 is not particularly limited. It can be connected via wired communication or wireless communication.

[0080] The communication method between the control device 100 and the relay device 20 is not particularly limited. It can be wired or wireless communication. As an example, in this embodiment, the control device 100 and the relay device 20 are connected via a wireless LAN (local area network).

[0081] [Control Device] Figure 6 This is a diagram illustrating an example of the hardware structure of a control device.

[0082] like Figure 6 As shown, the control device 100 includes a CPU (central processing unit) 111, a ROM (read-only memory) 112, a RAM (random access memory) 113, an auxiliary storage device 114, an input device 115, a display device 116, and a communication interface (I / F) 117. Typically, this structure can be implemented using a computer. As an example, in this embodiment, the control device 100 is composed of a notebook computer. The control device 100 is an example of a processing device.

[0083] The control device 100 functions as a control device by executing a predetermined program through the CPU 111, which acts as a processor. The program executed by the CPU 111 is stored in the ROM 112 or the auxiliary storage device 114.

[0084] The auxiliary storage device 114 constitutes the storage unit of the control device 100. The auxiliary storage device 114 may be, for example, an HDD (hard disk drive) or an SSD (solid state drive).

[0085] The input device 115 constitutes the operation unit of the control device 100. The input device 115 may be, for example, a keyboard, a mouse, a touch panel, etc.

[0086] The display device 116 constitutes the display section of the control device 100. The display device 116 may be, for example, an LCD (liquid crystal display) or an OLED (organic light-emitting diode) display.

[0087] The communication interface 117 constitutes the communication unit of the control device 100. The communication interface 117 is configured to enable communication with the relay device 20 via a predetermined communication method. As an example, in this embodiment, it is configured to enable communication via a wireless LAN.

[0088] [Functions of the control device] The control device 100 has the function of controlling the multi-eye camera device 10 and the function of processing images captured by the multi-eye camera device 10 (image processing function). The function of controlling the multi-eye camera device 10 includes the function of controlling the photography based on the multi-eye camera device 10 (photography control function).

[0089] [Photography control functions] Figure 7 This is a functional block diagram of the photographic control function of the control device.

[0090] like Figure 7 As shown, the control device 100 has functions such as a camera control unit 111A and a lighting control unit 111B as part of the photography control function. The functions of the camera control unit 111A and the lighting control unit 111B are implemented by the CPU 111 executing a predetermined program.

[0091] Figure 8 This is a block diagram showing the main functions of the camera control unit.

[0092] like Figure 8 As shown, the camera control unit 111A mainly has functions such as a photography control unit 111A1, a photography image acquisition unit 111A2, a photography image display control unit 111A3, and a photography image recording control unit 111A4.

[0093] The camera control unit 111A1 controls the cameras C1 to C5 mounted on the multi-camera camera device 10, causing each camera C1 to C5 to perform photography. Photography includes both still image photography and moving image photography. Furthermore, still image photography includes so-called interval photography. Interval photography is a function that repeatedly performs still image photography at constant intervals. The camera control unit 111A1 causes each camera C1 to C5 to perform photography based on the operation input (instruction to perform photography) from the input device 115. In the case of moving image photography and interval photography, photography begins according to the instruction to start photography and ends according to the instruction to end photography.

[0094] Each camera, C1 through C5, records images simultaneously. Therefore, in still image recording, each camera, C1 through C5, records images simultaneously (including those considered to be almost simultaneous). Furthermore, in moving image recording, each camera, C1 through C5, starts recording simultaneously and stops recording simultaneously.

[0095] The photographic image acquisition unit 111A2 acquires images captured by each camera C1 to C5. These images include not only the actual photographed images (images obtained according to photographic instructions) but also images captured in real-time view.

[0096] The photographic image display control unit 111A3 controls the display of images (including live view images) captured by each camera C1 to C5.

[0097] Figure 9 This is an example of a live view display.

[0098] like Figure 9 As shown, the live-view images of each camera C1 to C5 are displayed in five image display areas IDA1 to IDA5 set on the display screen of the display device 116. The image display areas IDA1 to IDA5 are arranged in a layout corresponding to the configuration of the cameras C1 to C5 in the multi-camera device 10. Therefore, in this embodiment, the image display areas IDA1 to IDA5 are arranged in an arc shape. The image of the first camera C1 is displayed in the first image display area IDA1. The image of the second camera C2 is displayed in the second image display area IDA2. The image of the third camera C3 is displayed in the third image display area IDA3. The image of the fourth camera C4 is displayed in the fourth image display area IDA4. The image of the fifth camera C5 is displayed in the fifth image display area IDA5.

[0099] The photographic image recording control unit 111A4 controls the recording of images captured by each camera C1 to C5. In this embodiment, the photographic image recording control unit 111A4 creates an image database (DB) 120 in the auxiliary storage device 114 and records the images captured by each camera C1 to C5 in the image database 120. The images of each camera C1 to C5 are recorded in photographic units. Regarding photography, a photograph that generates one composite image is defined as one unit. Therefore, for example, in the case of generating a composite image of the entire length of the tunnel, the image of the entire length of the tunnel structure is recorded as a whole (image group) and distinguishable from other images. The photographic image recording control unit 111A4 establishes an association between the information of the camera that took the photograph and the information of the photographing sequence, and records the images of each camera C1 to C5. That is, it is possible to distinguish which camera took which image and record the image of each camera C1 to C5. By recording images in this way, the photographing position of each image can be estimated. That is, the relative positional relationship of each camera C1 to C5 (the configuration of each camera C1 to C5) is known, and they take pictures at almost constant distance intervals. Therefore, as long as it is possible to distinguish which camera took which image, the approximate shooting position can be determined. Furthermore, for the same reason, the relative positional relationship between each image can be determined. That is, adjacent images can be identified. Therefore, in this embodiment, the information of the cameras taking pictures and the information of the shooting order constitute the information of the image's shooting position.

[0100] In addition, the information on "photograph location" does not require precise geographical location; it is sufficient to determine the relative positional relationship between the images that make up the image group, or at least the information on adjacent images.

[0101] Furthermore, the "information about the camera being photographed" refers to information provided that the camera's configuration is known. Therefore, the "information about the camera being photographed" has the same meaning as the information about the camera's configuration location (configuration location on a multi-view camera setup).

[0102] There are no particular limitations on the method of establishing the association. It is sufficient that the camera that took the photos can be identified in each image, and the order in which the photos were taken can be determined. As an example, in this embodiment, information about the camera that took the photos and information about the order in which the photos were taken are added to the images as supplementary information (e.g., metadata), and the images taken by each camera C1 to C5 are recorded.

[0103] The lighting control unit 111B controls the lighting devices L1 to L5 mounted on the multi-lens camera device 10. That is, it controls the on / off state of the illumination light emitted from the lighting devices L1 to L5. The lighting control unit 111B illuminates the light according to the operation input (light-on indicator and light-off indicator) from the input device 115.

[0104] [Image Processing Functions] Figure 10 This is a functional block diagram of the image processing function of the control device.

[0105] The control device 100, as an image processing function, has the ability to generate a composite image from a group of images captured by the multi-eye camera device 10. In this embodiment, the control device 100 is an example of an image processing device.

[0106] like Figure 10 As shown, the control device 100, in order to generate composite images, includes a processing object image acquisition unit 111C, a first composite processing unit 111D, a first pass / fail determination unit 111E, a second composite processing unit 111F, a second pass / fail determination unit 111G, a third composite processing unit 111H, a composite image recording control unit 111J, and a composite image display control unit 111K. The functions of each unit are implemented by the CPU 111 executing a predetermined program (image processing program).

[0107] [Image Acquisition Unit] The processing object image acquisition unit 111C acquires a set of images that are the processing objects for the compositing process. That is, it acquires the image set used to generate the composite image. The processing object image acquisition unit 111C acquires the image set of the processing objects from the image database 120.

[0108] As described above, information about the camera that took the photos and the order of the photos are added to the images recorded in the image database 120. Based on this information, information about the camera positions can be obtained. Therefore, by acquiring a group of images of the object being processed, information about the camera positions (approximate camera positions within the tunnel) of each image in the group can also be obtained.

[0109] [First Synthesis Processing Department] The first compositing processing unit 111D performs a prescribed compositing process on the image group acquired by the processing target image acquisition unit 111C to generate a composite image. In this embodiment, a so-called feature point matching (correspondence point search) compositing process is performed. Feature point matching is a process that matches feature points with high similarity between images. Typically, feature points are detected and feature descriptors are calculated for two images to match feature points with high similarity.

[0110] In feature point matching-based synthesis processing, the parameters required for the synthesis process (synthesis parameters) are determined based on the results of feature point matching, and the synthesis process is performed according to the determined synthesis parameters. More specifically, based on the determined synthesis parameters, the image is projected onto a shape model to generate a synthesized image.

[0111] Figure 11 This is a conceptual diagram of the synthesis process based on feature point matching.

[0112] In feature point matching-based synthesis processing, the pose parameters of the shape model, the pose parameters (rotation matrix, translation vector) of each camera corresponding to each image, and the lens distortion parameters of each camera corresponding to each image are determined as synthesis parameters based on the feature point matching results between images.

[0113] The shape model is selected based on the photographic object (subject). If the subject is planar, a planar model is selected. With a planar model, the rotation matrix and translation vector are determined as pose parameters. If the subject is curved, a cylindrical model is selected. With a cylindrical model, the rotation matrix, translation vector, and the radius of the cylinder are determined as pose parameters.

[0114] In the case of a tunnel structure with an arc-shaped inner wall (curved tunnel), a cylindrical model is selected as the shape model. Therefore, in this case, the rotation matrix, translation vector, and radius of the cylinder are determined as the pose parameters of the shape model.

[0115] The first compositing processing unit 111D performs feature point matching between the image groups acquired by the processing object image acquisition unit 111C, and determines compositing parameters based on the result of the feature point matching. Furthermore, the first compositing processing unit 111D projects the image onto a shape model to generate a composite image based on the determined compositing parameters. Moreover, when a cylindrical model is selected as the shape model, the first compositing processing unit 111D unfolds the image projected onto the shape model to generate a composite image.

[0116] Hereinafter, as needed, the composite image generated by the first composite processing unit 111D will be referred to as a "first composite image" to distinguish it from other composite images.

[0117] In this embodiment, the feature point matching-based compositing process performed by the first compositing processing unit 111D is an example of the first compositing process. Furthermore, in this embodiment, the composite image (a single-stage composite image) generated by the first compositing processing unit 111D is an example of the first composite image.

[0118] [First Qualification / Failure Determination Department] The first pass / fail determination unit 111E determines whether the composite image (one-time composite image) generated by the first compositing processing unit 111D is pass / fail. In this embodiment, pass / fail is determined based on the quality of the one-time composite image. Specifically, if the quality of the one-time composite image meets the prescribed quality standard, it is determined to be passable. Therefore, if the quality of the one-time composite image does not meet the prescribed quality standard, it is determined to be failable.

[0119] In this embodiment, firstly, the degree of fragmentation of the generated single-image composite is determined, thereby determining whether the quality of the single-image composite meets the prescribed quality standard. Secondly, the degree of distortion of the generated single-image composite is determined, thereby determining whether the quality of the single-image composite meets the prescribed quality standard.

[0120] Here, "fragmentation of a composite image" refers to the situation where a composite image is generated by splitting it into multiple images (fragments).

[0121] Figure 12 This is a diagram representing an example of a fragmented synthetic image.

[0122] Figure 12 An example is shown where an image that should have been generated as a single image is split (fragmented) into three images (fragments) IF1 to IF3.

[0123] Figure 13 and Figure 14 This is an example of a composite image that has been distorted.

[0124] Figure 13 and Figure 14 This shows an example of how an image that should have been synthesized as a roughly rectangular image is distorted and synthesized instead. Figure 13 An example is shown where the lower central portion of an image is distorted and then composited. Figure 14 An example is shown where the upper right corner of the image is distorted and then composited.

[0125] Fragmentation and distortion in synthesized images arise from failed compositing processes. In feature-point matching-based compositing, compositing failures can occur due to lost or omitted images, insufficient repetition between adjacent images, image jitter, or blurring, resulting in fragmentation and distortion. Furthermore, differences in the imaging of the same area between images in overlapping regions (e.g., differences in shadow rendering) can prevent feature point matching, leading to compositing failure. Additionally, significant deviations between the object's (subject's) shape and the intended shape model can also cause compositing failure.

[0126] (1) Qualification judgment based on the degree of fragmentation The first pass / fail determination unit 111E determines the degree of fragmentation of the generated single-image composite, thereby determining whether the quality of the single-image composite meets the prescribed quality standards. That is, it determines whether the degree of fragmentation meets the prescribed standards, and thus determines whether the quality of the single-image composite meets the prescribed quality standards. If the degree of fragmentation meets the prescribed standards, the quality of the single-image composite is determined to meet the prescribed quality standards, and it is thus determined to be passable. On the other hand, if the degree of fragmentation does not meet the prescribed standards, the quality of the single-image composite is determined to not meet the prescribed quality standards, and it is thus determined to be failable.

[0127] As an example, in this embodiment, the presence or absence of fragmentation is used as the criterion for determining the "degree of fragmentation." That is, the presence or absence of fragmentation in a single composite image is determined to determine whether the quality of the single composite image meets the prescribed quality standard. Therefore, if a single composite image is generated through fragmentation, it is determined that the quality of the single composite image does not meet the prescribed quality standard, and thus it is deemed unqualified. On the other hand, if a single composite image is generated without fragmentation, it is determined that the quality of the single composite image meets the prescribed quality standard, and thus it is deemed qualified.

[0128] In this embodiment, the criterion for determining the degree of fragmentation of the generated single-layer composite image is an example of the first criterion.

[0129] (2) Determination of passability based on the degree of distortion The first pass / fail determination unit 111E determines the degree of distortion in the generated single-image composite, thereby determining whether the quality of the single-image composite meets the prescribed quality standards. If the degree of distortion is within the allowable range, it is determined to meet the prescribed quality standards and is thus deemed passable. On the other hand, if the degree of distortion exceeds the allowable range, it is determined to fail to meet the prescribed quality standards and is thus deemed failable.

[0130] As an example, in this embodiment, the degree of deviation from the shape of the image that should have been generated is determined, thereby assessing the degree of distortion. "The shape of the image that should have been generated" refers to the shape of the image generated under normal compositing processing, which is determined by the photographic range. In photography aimed at inspecting structures, the photographic range is typically set as the area cut into a rectangle (in the case of tunnel structures, the area that becomes a rectangle when the plane is unfolded is set as the photographic range).

[0131] When a tunnel structure is photographed using the multi-view camera device 10, the resulting composite image (a normally synthesized composite image) generated by unfolding the plane becomes an approximately rectangular image. Therefore, the degree of distortion can be determined by calculating the degree of deviation from the rectangle. The degree of deviation from the rectangle can be calculated based on the difference between the maximum and minimum values ​​of the horizontal or vertical width of the image. That is, since the greater the distortion from the rectangle, the greater the difference, the degree of distortion can be determined based on the difference. In this embodiment, the difference between the maximum and minimum values ​​of the horizontal width and the vertical width of the image are calculated and compared with a threshold. Furthermore, if the difference between at least one of them exceeds the threshold, it is determined that the distortion is large (distortion exceeds the allowable range). That is, it is determined that the degree of distortion exceeds the allowable range. Therefore, in this embodiment, if the difference between the maximum and minimum values ​​of the width in both the horizontal and vertical directions of the image is below the threshold, it is determined that the degree of distortion is within the allowable range.

[0132] Figure 13 This is an example of vertical distortion occurring in a portion (the central part) of an image. In this case, the difference between the maximum and minimum vertical width of the image exceeds a threshold. Furthermore, Figure 14 This is an example of a portion of an image (the top right corner) that is distorted horizontally. In this case, the difference between the maximum and minimum values ​​of the horizontal width exceeds a threshold.

[0133] The width of an image is calculated, for example, based on the number of pixels. In this case, the difference between the maximum and minimum horizontal width of the image is calculated based on the difference between the maximum and minimum horizontal pixel count. Similarly, the difference between the maximum and minimum vertical width of the image is calculated based on the difference between the maximum and minimum vertical pixel count.

[0134] Furthermore, in the typical generation of composite images, the composite result is shaped into a rectangular image and output. That is, even in the case of distortion-based composites, the image is shaped into a rectangular image and output through so-called padding. Padding refers to the process of filling in meaningless pixels. For example, in Figure 13 and Figure 14 In the image, the surrounding black area represents the filled area. Figure 13 and Figure 14 This is an example of an image shaped into a rectangle by filling it with black pixels.

[0135] The "image width" mentioned here does not refer to the width of the padded image, but rather to the width (vertical and / or horizontal) of the image itself generated through compositing. Therefore, for example, regarding the padded composite image, it refers to the width (vertical and / or horizontal) of the area excluding the padded region. Figure 13 and Figure 14In the example, it is the width of the image area excluding the surrounding black area.

[0136] In this embodiment, the determination criterion for judging the degree of distortion of the generated single-image composite is another example of the first criterion. Furthermore, the measurement direction (horizontal and / or vertical) of the image width when calculating the degree of distortion is an example of the first direction.

[0137] [Second Synthesis Processing Department] The second compositing processing unit 111F performs compositing processing on the fragmented primary composite image to generate a composite image with less fragmentation. Since the fragmented primary composite image is the processing target, the image processed in the second compositing processing unit 111F is the primary composite image that was determined to be unqualified due to fragmentation in the first pass / fail determination unit 111E.

[0138] In this embodiment, the fragmented single-image composite is synthesized using the information of the shooting position of each image. That is, by using the information of the shooting position (in this embodiment, the information of the camera and the shooting order), the configuration positions of multiple fragments are adjusted to generate a composite image with less fragmentation, and more preferably, a single composite image (1 composite image) is generated.

[0139] Figure 15 This is a conceptual diagram of the synthesis process in the second synthesis processing unit.

[0140] Figure 15 This example illustrates the generation of a composite image by separating (fragmenting) it into three images (fragments) IF1 to IF3. Hereinafter, "Image IF1" will be referred to as "Fragment 1 Image IF1", "Image IF2" as "Fragment 2 Image IF2", and "Image IF3" as "Fragment 3 Image IF3" to distinguish each image.

[0141] As described above, in an image group used to generate a composite image, each image contains information about the camera used for the shot and the shooting order, serving as information about its shooting location. Using this information about the shooting location, the images constituting the image group can identify their respective adjacent images. This relationship also applies to the composite image.

[0142] First, consider the relationship between the first fragment image IF1 and the second fragment image IF2.

[0143] In the first fragment image IF1, the image Im(1, n) that constitutes a part of it is the nth image captured by the first camera C1. And in the second fragment image IF2, the image Im(1, n+1) that constitutes a part of it is the (n+1)th image captured by the first camera C1.

[0144] The first fragment image IF1, image Im(1, n), and the second fragment image IF2, image Im(1, n+1), are both images captured by the same first camera C1. Furthermore, the second fragment image IF2, image Im(1, n+1), is an image captured chronologically after the first fragment image IF1, image Im(1, n). Therefore, it can be concluded that the first fragment image IF1, image Im(1, n), should be positioned to the left of the second fragment image IF2, image Im(1, n+1).

[0145] Therefore, by arranging the image Im(1, n) of the first fragment image IF1 to the left of the image Im(1, n+1) of the second fragment image IF2 and then compositing them, it is possible to eliminate the separation state between the first fragment image IF1 and the second fragment image IF2.

[0146] Next, consider the relationship between the second fragment image IF2 and the third fragment image IF3.

[0147] In the second fragment image IF2, the image Im(5, m) that constitutes a part of it is the m-th image captured by the fifth camera C5. And in the third fragment image IF3, the image Im(5, m+1) that constitutes a part of it is the (m+1)-th image captured by the fifth camera C5.

[0148] Image Im(5, m) of the second fragment image IF2 and image Im(5, m+1) of the third fragment image IF3 are both images captured by the same fifth camera C5. Furthermore, image Im(5, m+1) of the third fragment image IF3 is an image captured chronologically after image Im(5, m) of the second fragment image IF2. Therefore, it can be concluded that image Im(5, m) of the second fragment image IF2 should be positioned to the left of image Im(5, m+1) of the third fragment image IF3.

[0149] Therefore, by arranging the image Im(5, m) of the second fragment image IF2 to the left of the image Im(5, m+1) of the third fragment image IF3 and then synthesizing them, it is possible to eliminate the separation state between the second fragment image IF2 and the third fragment image IF3.

[0150] Figure 16 This is a diagram representing an example of the synthesis result.

[0151] exist Figure 16In the example shown, images Im(1, n) and Im(1, n+1) are arranged adjacently to synthesize the first fragment image IF1 and the second fragment image IF2. Furthermore, images Im(5, m) and Im(5, m+1) are arranged adjacently to synthesize the second fragment image IF2 and the third fragment image IF3. The resulting composite image CI2 is of lower quality than a normally generated single-layer composite image, but it is sufficient to capture the state of the object (subject).

[0152] In addition, in the above example, the structure of adjacent images is determined by using information about the shooting order (information about the position of the moving direction of the multi-eye camera 10), but it is also possible to determine the structure of adjacent images by using information about the cameras being photographed (information about the camera's configuration position) or both.

[0153] Furthermore, in the above example, the structure is set to focus on only one image to determine the adjacent images and decide the configuration position, but it is also possible to determine the adjacency relationship and decide the configuration position through multiple images.

[0154] Furthermore, regarding the synthesized image, it is preferable to generate it by processing the fragment images as needed, such as enlarging, reducing, or rotating them, rather than simply arranging the fragment images in a predetermined position. These processing steps are facilitated by determining the adjacency relationships of multiple images and utilizing the results. Therefore, a structure that generates the synthesized image by determining the adjacency relationships of multiple images is preferred.

[0155] Thus, the second compositing processing unit 111F uses information about the image's photographic location to adjust the arrangement of each fragmented image and composites them, thereby eliminating fragmentation. By eliminating all fragmentation, a single composite image is generated.

[0156] In this embodiment, the compositing process performed by the second compositing processing unit 111F is an example of the second compositing process. Furthermore, in this embodiment, the composite image generated by the second compositing processing unit 111F is an example of the second composite image. Hereinafter, as needed, the composite image generated by the second compositing processing unit 111F will be referred to as a "secondary composite image" to distinguish it from other composite images.

[0157] [Second Pass / Fail Judgment Department] The second pass / fail determination unit 111G determines whether the composite image (secondary composite image) generated by the second compositing processing unit 111F is pass / fail. In this embodiment, similar to the first pass / fail determination unit 111E, pass / fail is determined based on the quality of the secondary composite image. Therefore, it is only deemed passable if the quality of the secondary composite image meets the prescribed quality standards (if the quality of the secondary composite image does not meet the prescribed quality standards, it is deemed failable).

[0158] Similar to the first pass / fail determination unit 111E, the second pass / fail determination unit 111G first determines the degree of fragmentation of the generated secondary composite image, thereby determining whether the quality of the secondary composite image meets the prescribed quality standards. Secondly, it determines the degree of distortion of the generated secondary composite image, thereby determining whether the quality of the secondary composite image meets the prescribed quality standards.

[0159] (1) Qualification judgment based on the degree of fragmentation In this embodiment, whether the prescribed quality standard is met is determined based on whether fragmentation can be eliminated. That is, whether the prescribed quality standard is met is determined based on whether a single composite image can be generated. In the second compositing processing unit 111F, if a single composite image can be generated, it is determined that the prescribed quality standard is met, and thus it is deemed acceptable. On the other hand, in the second compositing processing unit 111F, if a single composite image cannot be generated, that is, if compositing fails, it is determined that the prescribed quality standard is not met, and thus it is deemed unacceptable.

[0160] In this embodiment, the criterion for determining the degree of fragmentation of the generated secondary composite image is an example of the second criterion.

[0161] (2) Determination of passability based on the degree of distortion The second pass / fail determination unit 111G determines the degree of distortion in the generated secondary composite image, thereby determining whether the quality of the secondary composite image meets the prescribed quality standards. If the degree of distortion is within the allowable range, the quality is determined to meet the prescribed quality standards, and the image is thus deemed passable. On the other hand, if the degree of distortion exceeds the allowable range, the quality is determined to fail to meet the prescribed quality standards, and the image is thus deemed failable.

[0162] Similar to the first pass / fail determination unit 111E, in this embodiment, the degree of deviation from the shape of the image to be generated is calculated to determine the degree of distortion. Therefore, the difference between the maximum and minimum values ​​of the horizontal width and the vertical width of the generated secondary composite image is calculated to determine the degree of distortion. If the difference between at least one of them exceeds a threshold, it is determined that the distortion is large (distortion exceeds the allowable range), and thus it is determined to be unqualified.

[0163] Furthermore, the "image width" here does not refer to the width of the padded image, but rather to the width of the image itself generated through compositing. That is, it is the width of the area excluding the padded region.

[0164] [Third Synthesis Processing Department] In cases where the first composite image is deemed unqualified due to the degree of distortion not meeting the prescribed benchmark, and in cases where the second composite image is deemed unqualified, the third composite processing unit 111H generates a composite image of a prescribed format from the image group. The cases where the second composite image is deemed unqualified include, in addition to the case where the degree of distortion does not meet the prescribed benchmark, two other cases: cases where fragmentation cannot be eliminated (cases where a single composite image cannot be generated).

[0165] Figure 17 This is a concept diagram of the generation of composite images based on the third composite image processing unit.

[0166] The third compositing processing unit 111H generates a composite image by arranging the images constituting the image group according to their shooting positions. In this embodiment, based on the information of the cameras that took the photos and the shooting order, each image is arranged in a predetermined position to generate one composite image. Specifically, the vertical axis is set to the arrangement of the cameras that took the photos, and the horizontal axis is set to the shooting order of each camera. The images are arranged in a matrix to generate one composite image.

[0167] like Figure 17As shown, the images captured by camera C1 are designated in chronological order (shooting order) as Im(1,1), Im(1,2), Im(1,3), ..., Im(1,n). Similarly, the images captured by camera C2 are designated in chronological order as Im(2,1), Im(2,2), Im(2,3), ..., Im(2,n). The images captured by camera C3 are designated in chronological order as Im(3,1), Im(3,2), Im(3,3), ..., Im(3,n). The images captured by camera C4 are designated in chronological order as Im(4,1), Im(4,2), Im(4,3), ..., Im(4,n). Finally, the images captured by camera C5 are designated in chronological order as Im(5,1), Im(5,2), Im(5,3), ..., Im(5,n).

[0168] In the composite image IC3, images captured by each camera C1 to C5 in each row are generated in chronological order (shooting order). Furthermore, images captured by each camera C1 to C5 at the same time point are arranged in each column. That is, in the multi-eye camera device 10 of this embodiment, each camera C1 to C5 captures images synchronously; therefore, when the images captured by each camera C1 to C5 are arranged in shooting order, images captured at the same moment (including substantially the same moment) are arranged in each column.

[0169] Figure 18 This is a diagram showing an example of a composite image generated by the third composite processing unit.

[0170] Figure 18 An example of a composite image CI3 is shown, generated from images obtained during interval photography while the image moves at a nearly constant speed within a tunnel structure. In particular, Figure 18 An example is shown where five cameras, C1 to C5, take nine images. In this case, a single composite image, CI3, is generated from the image set consisting of 5 × 9 = 45 images.

[0171] like Figure 18 As shown, the generated composite image CI3 has images taken by cameras C1 to C5 arranged in the order of photography in each row. Furthermore, each column contains images taken by cameras C1 to C5 in the same order (images taken at the same time point).

[0172] Thus, the third compositing processing unit 111H generates a composite image CI3 according to the information of the shooting positions of each image constituting the image group (in this embodiment, the information of the camera that took the photos and the shooting order) in a prescribed arrangement. Hereinafter, the composite image CI3 generated by the third compositing processing unit 111H will be referred to as a "juxtaposed composite image" to distinguish it from other composite images. The juxtaposed composite image CI3 is of lower quality than a normally generated primary composite image, but it is sufficient to obtain an image that captures the state of the object (subject). In this embodiment, the "juxtaposed composite image" is an example of the second composite image.

[0173] In this embodiment, one of the following is generated from the image group: a first-synthesized image, a second-synthesized image, or a juxtaposed synthesized image.

[0174] [Synthetic Image Recording Control Unit] The composite image recording control unit 111J controls the recording of generated composite images (first-order composite images, second-order composite images, and juxtaposed composite images). The composite image recording control unit 111J associates the generated composite images with the image group of the generation source and records them in the image database 120. The composite image recording control unit 111J automatically records the generated composite images in the image database 120, or records the generated composite images in the image database 120 according to recording instructions from the user.

[0175] [Synthetic Image Display Control Unit] The composite image display control unit 111K controls the display of the generated composite images (first-order composite images, second-order composite images, and juxtaposed composite images). The composite image display control unit 111K displays the generated composite images on the display device 116. According to instructions from the user, the composite image display control unit 111K zooms in, zooms out, moves, etc., the composite images displayed on the display device 116.

[0176] [The role of the camera system] Here, we will take the case of using a multi-view camera device 10 to segment the inner wall of a photographic tunnel structure and generate a composite image of the inner wall of the tunnel structure from the obtained image set as an example.

[0177] Photography is performed using a trolley Tr to move the multi-eye camera device 10. As an example, the trolley Tr moves at a nearly constant speed and performs interval photography to photograph the inner wall of the tunnel structure. The travel speed of the trolley Tr and the photography interval are set such that the repetition rate (overlap ratio) of the images in the direction of trolley Tr's movement meets a predetermined condition. The repetition rate condition is determined from the viewpoint of composite processing. As an example, the repetition rate condition is 10% or more. Therefore, the travel speed of the trolley Tr and the photography interval are set such that the repetition rate is at least 10%.

[0178] The image groups obtained through photography are recorded in the image database 120 by photographic unit so that they can be distinguished from image groups obtained through other photography. Furthermore, each image constituting an image group is associated with information about the photographing location (in this embodiment, information about the camera used for the photograph and the photographing order) and recorded in the image database 120.

[0179] After photography, a composite image is generated according to instructions from the user. In the photography system 1 of this embodiment, the composite image generation process is performed in the control device 100. More specifically, the CPU 111 of the control device 100 performs the process of generating a composite image from the image set obtained by photography.

[0180] [Generation and Processing of Composite Images] Figure 19 It is a flowchart showing the sequence of processing steps (image processing method) for generating a composite image.

[0181] First, the image set of the processing target is acquired (step S11). As described above, in the photography system 1 of this embodiment, the image set acquired by photography is recorded in the image database 120. The CPU 111 acquires the image set as the processing target from the image database 120.

[0182] Next, a process for generating a composite image is performed (step S12). In this embodiment, the acquired image group is subjected to a synthesis process based on feature point matching to generate a composite image.

[0183] Next, the generated composite image is processed to determine its pass / fail status (first pass / fail determination process) (step S13). In this embodiment, it is determined whether the quality of the composite image meets the prescribed quality standard, thereby determining whether it is pass / fail. Specifically, firstly, the degree of fragmentation is determined, and the image quality is determined, thereby determining whether it is pass / fail. Secondly, the degree of distortion is determined, thereby determining the image quality, and determining whether it is pass / fail. In this embodiment, if no fragmentation occurs and the distortion is within the allowable range, it is determined to be pass / fail.

[0184] Next, based on the result of the pass / fail determination, it is determined whether the synthesized image is qualified (step S14). That is, the success or failure of the synthesis process (synthesis process based on feature point matching) is determined. In the result of the pass / fail determination, if the synthesized image is determined to be "qualified", the synthesis process ends. In this case, the synthesized image becomes the image of the synthesis process result.

[0185] On the other hand, in the result of the pass / fail determination, if a synthesized image is determined to be "unqualified", the reason is determined. That is, it is determined whether the failure is due to fragmentation (step S15).

[0186] If an image is deemed unqualified due to fragmentation, a process is performed to generate a secondary composite image from the fragmented primary composite image (step S16). Specifically, using the photographic position information of each image, the configuration positions of multiple fragments are adjusted to generate a composite image with less fragmentation (preferably a single composite image) (see reference). Figure 16 ).

[0187] When a secondary composite image is generated, a pass / fail determination process is performed on the generated secondary composite image (second pass / fail determination process) (step S17). In this embodiment, similar to the pass / fail determination process for the primary composite image (first pass / fail determination process), it is determined whether the quality of the generated secondary composite image meets the prescribed quality standard, thereby determining whether it is pass / fail. Then, based on the pass / fail determination result, it is determined whether the secondary composite image is pass / fail (step S18). That is, it is determined whether the compositing process (compositing process that adjusts the configuration position based on the information of the shooting position) is successful or not. If the secondary composite image is determined to be "pass" in the pass / fail determination result, the compositing process ends. In this case, the secondary composite image becomes the image of the compositing process result.

[0188] If the secondary composite image is deemed unqualified (step S18 is "No"), or if the primary composite image is deemed unqualified due to distortion (step S15 is "No"), a process of generating and juxtaposing a composite image is performed (step S19). That is, the images are arranged according to the photographic position information of each image constituting the image group, thereby generating one composite image (see reference). Figure 18 In the case of generating a juxtaposed composite image, the juxtaposed composite image becomes the image of the composite processing result.

[0189] Through the series of steps described above, the generation of the composite image is completed. As mentioned above, if the generation of a composite image is successful (assuming the first composite image is acceptable), the first composite image becomes the image of the composite processing result. Furthermore, if the generation of a first composite image fails but the generation of a second composite image is successful (assuming the second composite image is acceptable), the second composite image becomes the image of the composite processing result. If the generation of all composite images fails, the composite image is then set as the image of the composite processing result. Therefore, one composite image is always generated.

[0190] Thus, according to the control device 100 of this embodiment, even if the generation of a single composite image fails, a composite image will still be generated. The composite image generated when the generation of a single composite image fails (a secondary composite image and a juxtaposed composite image) is of lower quality than a normally generated single composite image (a successfully generated composite image), but it still provides an image sufficient to capture the state of the object (subject) (see reference). Figure 16 and Figure 18 ).

[0191] The generated composite images (first composite image, second composite image, and juxtaposed composite image) are displayed on the display device 116. Furthermore, the generated composite images are automatically recorded in the image database 120, or recorded in the image database 120 according to a recording instruction from the user.

[0192] [Variation Example] [Generation of a single composite image] In the above embodiment, the structure is set to generate a single synthesized image through a synthesis process based on feature point matching, but the method for generating a single synthesized image is not limited to this. It can also be set to generate the structure using other methods (so-called panoramic synthesis methods).

[0193] Furthermore, the compositing parameters are preferably set appropriately according to the photographic object, etc. For example, in the case of planar compositing with a plane as the object, the projection transformation matrix of each image can be used as the compositing parameters.

[0194] [Information on the photography location] In the above embodiments, the structure is set to obtain information about the camera being photographed (information about the configuration position on the multi-eye camera device) and information about the shooting order as information about the shooting position, but the information obtained as information about the shooting position is not limited to this.

[0195] As mentioned above, the information of "photograph location" is sufficient to determine the relative positional relationship between the images that make up the image group or to determine the information of adjacent images.

[0196] For example, it can also be configured such that cameras C1 to C5 have GPS (global positioning systems) functionality or IMES (indoor messaging system) functionality as indoor GPS, and the structure utilizes their GPS or IMES functions to obtain information about the photographing location. In this case, the GPS or IMES location information (latitude, longitude, altitude) is appended to the photographed image for recording (e.g., recorded as tag information).

[0197] GPS or IMES functionality can be configured on the multi-camera system 10 or the trolley Tr. When each camera has GPS or IMES functionality, the camera's location information can be omitted. This is because the location where photography was performed can be determined based on GPS or IMES location information.

[0198] Furthermore, when using the information about the shooting order as information about the shooting location, the information about the shooting order can be structured as being indirectly obtained from other information. For example, when images are recorded with the shooting date and time (a so-called timestamp) attached as supplementary information (such as tag information), the shooting order can be determined based on the shooting date and time information, thereby obtaining the shooting order information. Also, for example, when images are recorded by assigning consecutive numbers to filenames (incrementing the number contained in the filename by 1 each time an image is recorded), the shooting order can be determined based on the filename, thereby obtaining the shooting order information.

[0199] Furthermore, for example, it can be configured such that, when the trolley Tr is equipped with a distance meter (odometer, etc.), the distance information (distance information from the reference point) obtained from its distance meter is used to acquire the information of the photographing position (distance from the reference point or position coordinates). In this case, the distance information of the photographing time point is obtained from the distance meter, associated with the image, and recorded. The reference point is, for example, the position where the photographing begins (the position where the trolley begins to move).

[0200] [Feature Point Matching] As described above, in the synthesis process that utilizes feature point matching, feature point matching is performed as a preprocessing step (the process required to determine the synthesis parameters).

[0201] Typically, feature point matching is performed across all images in a group of images. However, if feature point matching is performed across all images, the number of potential match points increases, raising the probability of mismatches involving similar feature points.

[0202] Basically, feature point matching is performed between images taken in repeated photographs (images with repetitive regions). Then, by limiting the objects to which feature point matching is performed, the chance of false matches can be reduced.

[0203] In the photographic system 1 described above, since each image constituting the image group has information about its photographic position, it is possible to identify images that have been photographed repeatedly. Therefore, when performing feature point matching, the information about the photographic position is used to identify images that have been photographed repeatedly. Then, feature point matching is performed between the identified images. As a result, false matching can be suppressed. Furthermore, by reducing the number of processing objects, the computational load can also be reduced.

[0204] Figure 20 This is a conceptual diagram of a method for determining images from repeated photographs.

[0205] Figure 20 This diagram shows a portion of an image sequence captured by a multi-camera system, arranged according to their positions. The vertical arrangement corresponds to the position of each camera. The horizontal arrangement corresponds to the time sequence (shooting order) of the images.

[0206] exist Figure 20 In this context, the image of interest is denoted as image Im(i,j). Image Im(i,j) is the j-th image captured by the i-th camera. Assuming that each image is captured normally, at least the following four images are images captured repeatedly with image Im(i,j): The first image is image Im(i,j-1) captured by the i-th camera immediately preceding image Im(i,j). The second image is image Im(i,j+1) captured by the i-th camera immediately following image Im(i,j). The third image is image Im(i-1,j) captured by the (i-1)-th camera adjacent to the i-th camera at the same time point as image Im(i,j). The fourth image is image Im(i,j+1) captured by the (i+1)-th camera adjacent to the i-th camera at the same time point as image Im(i,j). These four images are adjacent to image Im(i,j).

[0207] Since each image in a group of images contains information about its photographic location, adjacent images can be identified by utilizing this information.

[0208] In this way, by using information about the camera's location to identify repeatedly photographed images, the processing targets for feature point matching are limited. This helps to suppress false matches.

[0209] Furthermore, the range of images captured in repeated shots varies depending on the camera configuration and shooting conditions (movement speed, shooting interval, etc.). Therefore, it is preferable to determine the range of images captured in repeated shots (the range of adjacent images) based on the camera configuration, shooting conditions, etc. For example, images captured two times prior to the images of interest in chronological order (…). Figure 20 Image Im(i,j-2) and images taken 2 times later ( Figure 20 If there are also repeating regions (repeating regions capable of being synthesized) between images Im(i,j+2), it is preferable to also use these images as objects for feature point matching. Furthermore, for example, in adjacent cameras, images captured in chronological order immediately before and immediately after each other (…) Figure 20 If there are also more than a specified number of overlapping regions between images Im(i-1, j-1), Im(i-1, j+1), Im(i+1, j-1), and Im(i+1, j+1), these images are preferably also used as objects for feature point matching.

[0210] [Pass / Fail Determination Based on Distortion] In the above embodiments, a method for determining the degree of distortion is to measure the maximum and minimum values ​​of the vertical and / or horizontal widths of the generated composite images (first-stage composite images and second-stage composite images) and determine whether the degree of distortion is within the allowable range based on whether the difference is below a threshold. The method for determining the degree of distortion is not limited to this.

[0211] (1) Methods using information from photographic conditions The shape of the generated composite image can be estimated based on the photographic conditions of the image group from the composite source. For example, in the photographic system 1 of the above embodiment, when the multi-eye photographic device 10 is moved by the trolley Tr to photograph the object (subject), the generated composite image becomes rectangular. Furthermore, the size of the image (vertical width and horizontal width) can also be estimated based on the photographic conditions (photographic resolution, photographic interval, etc.).

[0212] Therefore, information about the photographic conditions is used to estimate the vertical width and / or horizontal width of the generated synthetic image, and the degree of distortion is determined by comparing it with the estimated value.

[0213] Specifically, the vertical width and / or horizontal width of the generated composite image are measured and compared with estimated values ​​of the vertical width and / or horizontal width of the composite image based on photographic conditions. For example, this can be done by calculating the difference between the measured and estimated values. Distortion below a threshold calculated from this difference is considered acceptable. That is, a difference below the threshold means that the difference from the estimated shape or width is small, and therefore can be judged as small distortion.

[0214] Preferably, the vertical and horizontal widths of the generated composite image are measured and compared with estimated values ​​of the vertical and horizontal widths of the composite image based on the photographic conditions. Then, if the difference between the measured and estimated widths is below a threshold, the image is deemed acceptable. For example, the number of pixels can be used to determine the width.

[0215] Additionally, in this example, information about the shooting conditions is required. This information can be configured to be input by the user via input device 115, or it can be configured to be automatically acquired by the camera.

[0216] In this example, the width in the longitudinal and / or transverse directions is an example of the width in the second direction.

[0217] (2) Using the method of synthesizing parameters In synthesis processing based on synthesis parameters, the degree of distortion can also be determined based on these parameters. Generally, if the synthesis parameters (camera pose parameters, camera lens distortion parameters) deviate significantly from the surrounding images, distortion will occur in the generated synthesized image. Therefore, by focusing on the synthesis parameters, the presence or absence of distortion in the synthesized image (whether the distortion exceeds the allowable range) can be determined. Specifically, after determining the synthesis parameters based on the results of feature point matching, it is determined whether there are images where the synthesis parameters deviate significantly from the surrounding images. If there are no images where the synthesis parameters deviate significantly from the surrounding images, the distortion is considered to be within the allowable range. On the other hand, if there are images where the synthesis parameters deviate significantly from the surrounding images, the distortion is considered to exceed the allowable range. To determine whether an image has significantly deviated from the surrounding images, the degree of deviation (deviation degree) of the synthesis parameters from the surrounding images (e.g., adjacent images) is calculated, and the calculated deviation degree is compared with a threshold. If the deviation degree is above the threshold, it is determined to be an image where the synthesis parameters deviate significantly from the surrounding images.

[0218] (3) Other ingredients The degree of distortion can also be determined by combining the above-mentioned determination methods. For example, it can be determined by combining a method of determining the distortion by measuring the width of the generated composite image and a method of determining the distortion by using composite parameters. In this case, for example, if it is determined to be acceptable in all determinations, it is determined to be acceptable.

[0219] [Generation of secondary composite images] A synthesized image can sometimes be fragmented in a distorted state. Figure 21 This is a diagram illustrating an example of a fragmented original image in a distorted state. Figure 21 An example is shown when the image is fragmented into three images IF1, IF2, and IF3. Image IF1 is designated as the first fragment image IF1, image IF2 as the second fragment image IF2, and image IF3 as the third fragment image IF3. Among the three fragment images IF1 to IF3, the first fragment image IF1 is generated due to significant distortion. Therefore, even if a secondary image is generated by configuring a primary composite image containing the significantly distorted image based solely on the information of the shooting position, a normal secondary composite image (a secondary composite image that can be determined as acceptable in the acceptance / disaccharification process) cannot be generated. Therefore, when generating a secondary composite image from a primary composite image containing a significantly distorted image (fragment), it is preferable to generate the secondary composite image after correcting the distortion.

[0220] Figure 22 This is a function block diagram representing an example of generating a secondary composite image by correcting distortion.

[0221] like Figure 22 As shown, a preprocessing unit 111L is provided in front of the second composite processing unit 111F. Before generating the secondary composite image, the preprocessing unit 111L preprocesses the primary composite image (fragmented primary composite image) of the processing object. The preprocessing unit 111L has the functions of a distortion image extraction unit 111L1 and a distortion correction unit 111L2. Each function is implemented by the CPU 111 executing a predetermined program.

[0222] The distortion image extraction unit 111L1 detects fragments with distortion exceeding the allowable range from multiple fragments in a fragmented, synthesized image. As an example, the distortion image extraction unit 111L1 determines the degree of distortion of each fragment based on the synthesis parameters. Specifically, it detects the presence of images whose synthesis parameters deviate significantly from those of the surrounding images among the images constituting each fragment. If an image with synthesis parameters significantly deviating from its surrounding images is detected, it is determined to have large distortion, and thus exceeds the allowable range. In this example, the fragment with distortion exceeding the allowable range is an example of a fragment with distortion exceeding a predetermined range.

[0223] When the distortion correction unit 111L2 extracts fragments with distortion exceeding the allowable range from the distortion image extraction unit 111L1, it performs image processing on the fragments and corrects the distortion. As an example, the distortion is corrected in the following order: First, an image whose synthesis parameters deviate significantly from those of the surrounding images is identified. Then, the synthesis parameters of the identified image are corrected to be the same as those of the surrounding images with less distortion. This reduces the distortion generated in the fragment's image. In addition to this, known image processing techniques related to distortion correction can be employed for distortion correction.

[0224] After distortion correction, the second synthesis processing unit 111F performs synthesis processing on the corrected fragment image to generate a secondary synthesized image.

[0225] Figure 23 This is an example of generating a secondary composite image by performing distortion correction.

[0226] Figure 23 It shows the Figure 21 The example shown is generated by correcting the distortion of the first fragment image IF1 out of the three fragment images IF1 to IF3 to produce the secondary composite image CI2. Figure 23 In the image, IF1+ is the first fragment image after distortion correction.

[0227] Thus, by correcting the distortion of fragmented images generated by distortion, a high-quality secondary composite image can be generated.

[0228] [Generation of Juxtaposed Composite Images (1)] In the above embodiments, the images constituting the image group are arranged in a non-overlapping configuration to generate a juxtaposed composite image (see reference). Figure 17 and Figure 18 However, it can also be configured to overlap images according to a specified configuration rule to generate a single juxtaposed composite image. For example, it can be configured to overlap adjacent images at a specified overlap rate to generate a single juxtaposed composite image. In this case, the overlap rate is set as follows, for example.

[0229] (1) Set to a preset value. In this case, for example, set to the same as the repetition rate setting during photography. That is, set to the same as the repetition rate setting between cameras and the repetition rate setting in the direction of movement. For example, when photography is performed with a repetition rate of 10% for both, images are overlaid with an overlap rate of 10% in the top, bottom, left, and right sides to generate a juxtaposed composite image.

[0230] (2) The user can set the overlap rate arbitrarily. In this case, the user's specified overlap rate is accepted and a juxtaposed composite image is generated.

[0231] (3) The overlap rate is determined based on the information of the shooting position of each image. For example, the overlap rate is determined based on the distance from the reference point or the position coordinates.

[0232] (4) The overlap rate is determined by using the size of the generated juxtaposed composite image as the target size. For example, the overlap rate is determined by using the number of pixels in the vertical and horizontal directions of the generated juxtaposed composite image as the target number of pixels. In this case, the user can arbitrarily set the target size. Furthermore, as described later, when a composite image is generated by dividing the photographic object into multiple segments, the target size can be set as the size of the composite image (first composite image) of the successfully composited segment. For example, the target size can be set as the same size as the first composite image of the segment that was successfully composited at the nearest point. Furthermore, the target size can be set as the same size as the first composite image of the segment that was successfully composited in an adjacent segment.

[0233] [Generation of Juxtaposed Composite Images (2)] In the above embodiments, the structure for generating a juxtaposed composite image is set to occur when the distortion of the first composite image exceeds the allowable range and the second composite image is unqualified. However, it can also be set to generate a juxtaposed composite image when the first composite image is unqualified. In this case, if the generation of the first composite image fails (if it is unqualified in the qualification determination of the first composite image), the generation of the second composite image is not performed, and the juxtaposed composite image is generated directly.

[0234] [Generation of composite images from multiple segments] In the case of large infrastructure structures such as tunnels and bridges, it is impractical to represent the entire structure with a single composite image. Therefore, it is preferable to divide the structure into multiple segments and generate a composite image for each segment. In particular, in the case of large infrastructure structures such as tunnels and bridges, it is preferable to divide the structure into multiple segments along its length and generate a composite image for each segment. In this case, for example, the infrastructure structure is divided into multiple segments along its length at constant intervals, and a composite image is generated for each segment.

[0235] When an object is divided into multiple segments along its length, and a composite image is generated for each segment, the photographic job itself can be completed in one go. That is, it is possible to photograph images of all segments in a single photographic job. In this case, for example, image groups of each segment can be extracted from the image group that photographed all segments, and composite images of each segment (single-segment composite image, double-segment composite image, or juxtaposed composite image) can be generated based on the extracted image groups. The image groups of each segment can be extracted, for example, using information about the photographic location.

[0236] Thus, when an object is divided into multiple segments and a composite image is generated for each segment, an image is generated that arranges the composite images (first-order composite image, second-order composite image, or juxtaposed composite image) generated for each segment according to the arrangement of the segments, and this image is used as the overall composite image. For example, when a tunnel structure is divided into multiple segments along its length and a composite image is generated for each segment, an image is generated that arranges the composite images (first-order composite image, second-order composite image, or juxtaposed composite image) generated for each segment in a horizontal row, and this image is used as the overall composite image. That is, a composite image of the tunnel structure as a whole.

[0237] Figure 24 This is an example of a diagram showing the composite result when a composite image is generated by dividing the image into multiple segments.

[0238] Figure 24 An example is shown where a tunnel structure is divided into multiple segments (10 segments) along its length at constant intervals, and composite images SI1–SI10 are generated for each segment. In this case, as... Figure 24 As shown, images SI are generated by arranging the composite images SI1 to SI10 generated for each segment into a horizontal row. The generated images SI are displayed on the display device 116 as an image of the tunnel structure as a whole (overall image).

[0239] Composite images SI1 to SI10 are composed of primary composite images, secondary composite images, or juxtaposed composite images. Therefore, even if a primary composite image cannot be generated in all sections, at least a secondary composite image or a juxtaposed composite image will be displayed. Thus, for example, if photographic loss occurs in some sections and a normal composite image (primary composite image) cannot be generated, at least a secondary composite image or a juxtaposed composite image will be displayed. Therefore, even for large infrastructure structures such as tunnels and bridges, it is possible to establish an environment where the overall condition of the infrastructure can be confirmed through images.

[0240] In this example, the image (overall image) SI generated by arranging the composite images SI1 to SI10 in a horizontal row is an example of the first output image.

[0241] Preferably, the overall image SI can be enlarged or reduced for display based on user instructions.

[0242] Figure 25 This is an example of a magnified display of the entire image.

[0243] Zooming in or out can be indicated, for example, by specifying the center point of zoom on the overall image SI and entering the magnification. Alternatively, the center point of zoom can be specified on the overall image SI and zoomed in or out using the mouse wheel.

[0244] When the display is magnified, such as Figure 25 As shown, it is preferable to display a scaled-down version of the overall image SI on the screen as a map image MI, so that the regions of the magnified image in the map image MI can be distinguished.

[0245] Damage Detection It can automatically detect damage (cracks, peeling, corrosion, etc.) on the surface of an object from an image and display the results together with the synthesized image.

[0246] It can be configured to detect damage to the image before synthesis (each image in the image group constituting the synthesis source) or to detect damage to the synthesized image (synthetic image). The synthesized image includes not only the primary synthesized image but also the secondary synthesized image and the juxtaposed synthesized image.

[0247] Techniques for detecting damage from images are well known, therefore a detailed description thereof is omitted. Typically, damage to the surface of an object is detected by analyzing an image of the object being photographed. In cases where damage is chalking, chalking-based lines (e.g., powdery lines drawn along cracks) are also included in the detected object.

[0248] Image-based damage detection includes damage detection using what is known as artificial intelligence. For example, a learned model that detects damage from images can be used for damage detection.

[0249] Figure 26 This is an example of a diagram showing the results of damage detection.

[0250] An image IX is generated that overlays the detection results of the damage onto the synthetic image and outputs it to the display device 116. Figure 26 An example is shown where a crack is detected as damage. In this case, image IX is generated and overlaid on the composite image as a result of the damage detection, with the line tracing the detected crack as the result.

[0251] The display of damage detection results can be turned on or off at will. For example, based on user instructions, it is possible to switch between displaying the damage detection results overlaid on the composite image (display on) and not displaying them (display off).

[0252] Furthermore, when multiple types of damage are detected, the display can be switched according to the type of damage.

[0253] Furthermore, to differentiate the size of damage on the screen, the display method can be changed according to the size of the damage. For example, for cracks, the color and / or line type can be changed according to the width of the crack. In this way, the size (width) of each crack can be clearly seen on the screen.

[0254] In this example, image IX is an example of the second output image.

[0255] [photography] In the above embodiments, the example described is the use of a multi-camera imaging device equipped with multiple cameras to photograph an object and generate a composite image of it. However, the method for photographing the object is not limited to this. For example, a single camera can be used to segment and photograph the object to obtain a group of images for synthesis.

[0256] Figure 27 This is an example of photographing an object using a single camera.

[0257] Figure 27 This illustrates an example of segmenting and photographing a rectangular area (Ob) on a plane using a single camera. The symbol Sa in the diagram represents the camera's photographic area Sa. For example... Figure 27 As shown, photography is performed while changing positions, with the photographic area Sa repeated at a specified repetition rate (e.g., 10% or more) along both the shorter side (vertical, y-direction) and the longer side (horizontal, x-direction) of the rectangle. Figure 27 The image shows an example of photographing a vertical column sequentially, one by one.

[0258] Information about the shooting location can be obtained using methods such as GPS. Furthermore, information about the shooting sequence in both the longitudinal (y-direction) and lateral (x-direction) directions can be obtained and used as information about the shooting location.

[0259] Regarding photography, it doesn't necessarily have to be done directly by a person; it can also be structured with a mobile robot (a self-moving photography robot) performing the photography. For example, it can also be structured with a drone (unmanned aerial vehicle) equipped with a camera performing the photography. In the case of photography by drones, it can also be structured with a device that moves along a pre-set route and automatically performs photography.

[0260] [Photography System] In the above embodiment, image processing functions such as generating composite images are integrated into the control device 100. However, the image processing functions can also be implemented by a device different from the control device 100. For example, a computer (server) on a network can also have image processing capabilities. In this case, the image set of the processing target is sent to the computer on the network, and the computer on the network performs image processing such as composite processing.

[0261] [Hardware Structure] The hardware of the image processing apparatus implementing this invention can be composed of various processors. Among these processors, there are general-purpose processors that include a CPU (Central Processing Unit) that executes programs to function as various processing units, processors such as FPGAs (Field Programmable Gate Arrays) that allow for changes in circuit structure after manufacturing (i.e., Programmable Logic Devices; PLDs), and processors such as ASICs (Application Specific Integrated Circuits) that have circuit structures specifically designed for performing specific processes (i.e., dedicated circuits). A processing unit constituting the inspection support device can be composed of one of the aforementioned processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs or a combination of a CPU and an FPGA. Furthermore, multiple processing units can be composed of a single processor. As an example of multiple processing units composed of a single processor, firstly, represented by a computer such as a client or server, a processor can be composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, systems-on-a-chip (SoC) are represented by the following approach: a processor that implements the overall system functionality, including multiple processing units, using a single integrated circuit (IC) chip. Thus, the various processing units are configured using one or more of these processors as the hardware structure. More specifically, the hardware structure of these processors is a circuit composed of combined semiconductor elements and other circuitry.

[0262] Symbol Explanation 1-Photography system, 10-Multi-eye camera device, 11-Frame, 12-Base plate, 13-Column, 14-Mounting base, 20-Relay device, 100-Control device, 111-CPU, 111A-Camera control unit, 111A1-Photography control unit, 111A2-Photography image acquisition unit, 111A3-Photography image display control unit, 111A4-Photography image recording control unit, 111B-Lighting control unit, 111C-Image acquisition unit for processed objects, 111D-First compositing processing unit, 111E-First pass / fail determination unit, 111F-Second compositing processing unit, 111G-Second pass / fail determination unit, 111H-Third compositing processing unit, 111J-Composited image The components include: 111K - Recording Control Unit, 111L - Composite Image Display Control Unit, 111L - Preprocessing Unit, 111L1 - Distorted Image Extraction Unit, 111L2 - Distortion Correction Unit, 112 - ROM, 113 - RAM, 114 - Auxiliary Storage Device, 115 - Input Device, 116 - Display Device, 117 - Communication Interface, 120 - Image Database, Ax - Axis, B1 - Bracket (1st Bracket), B2 - Bracket (2nd Bracket), B3 - Bracket (3rd Bracket), B4 - Bracket (4th Bracket), B5 - Bracket (5th Bracket), C1 - Camera (1st Camera), C2 - Camera (2nd Camera), C3 - Camera (3rd Camera), C4 - Camera (4th Camera) C5 - Camera (5th camera), CI2 - Composite image (secondary composite image), CI3 - Composite image (juxtaposed composite image), CL - Fixture, IC3 - Composite image, IDA1 - Image display area (1st image display area), IDA2 - Image display area (2nd image display area), IDA3 - Image display area (3rd image display area), IDA4 - Image display area (4th image display area), IDA5 - Image display area (5th image display area), IF1 - Image (1st fragment image), IF2 - Image (2nd fragment image), IF3 - Image (3rd fragment image), IX - Image (damage detection overlaid on composite image). The resulting image), L1 - Illumination device (first illumination device), L2 - Illumination device (second illumination device), L3 - Illumination device (third illumination device), L4 - Illumination device (fourth illumination device), L5 - Illumination device (fifth illumination device), MI - Map image, SI - Image (overall image), SI1 - Composite image, SI2 - Composite image, SI3 - Composite image, SI4 - Composite image, SI5 - Composite image, SI6 - Composite image, SI7 - Composite image, SI8 - Composite image, SI9 - Composite image, SI10 - Composite image, Sa - Photographic area, TS - Tunnel structure, Tr - Trolley, S11~S19 - Sequence of composite image generation processing.

Claims

1. An image processing apparatus comprising a processor, The processor performs the following processing: Obtain information about the image group and the photographic locations of the images constituting the image group; The image group is subjected to a first composite processing to generate a first composite image; Determine whether the quality of the first synthesized image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the shooting location.

2. The image processing apparatus according to claim 1, wherein, The processor performs the following processing as the first synthesis process: performing feature point matching between the images constituting the image group, and generating the first synthesized image based on the result of the feature point matching.

3. The image processing apparatus according to claim 2, wherein, The processor performs the following processing: determining the degree of fragmentation of the first synthesized image, thereby determining whether the quality of the first synthesized image meets the first benchmark.

4. The image processing apparatus according to claim 3, wherein, The processor performs the following processing as the second compositing process: based on the information of the shooting location, at least a portion of the fragments of the first composite image are configured as fragments different from the fragments and then composited, and an image with a less fragmented degree than the first composite image is generated as the second composite image.

5. The image processing apparatus according to claim 4, wherein, The processor further performs the following processing: before performing the second synthesis process, it extracts fragments with distortions exceeding a specified range from the plurality of fragments and corrects the distortions of the extracted fragments with the distortions.

6. The image processing apparatus according to any one of claims 1 to 5, wherein, The processor performs the following processing: Determine whether the quality of the second synthesized image meets the second criterion; and If the quality of the second composite image does not meet the second benchmark, an image is generated that configures the images constituting the image group according to the information of the shooting location as the second composite image.

7. The image processing apparatus according to claim 2, wherein, The processor performs the following processing: determining whether the degree of distortion of at least the first synthesized image is within the allowable range, thereby determining whether the quality of the first synthesized image meets the first benchmark.

8. The image processing apparatus according to claim 7, wherein, The processor performs the following processing as part of the second compositing process: configuring the images constituting the image group according to the information of the shooting location to generate the second composite image.

9. The image processing apparatus according to claim 7 or 8, wherein, The processor performs the following processing: The maximum and minimum values ​​of the width in the first direction of the first synthesized image are determined; Calculate the difference between the maximum and minimum width in the first direction; and Determine whether the difference is below a threshold, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

10. The image processing apparatus according to claim 7 or 8, wherein, The processor performs the following processing: Obtain information about the photographic conditions of the images constituting the image group; The width of the first composite image in the second direction is estimated based on the information of the photographic conditions; Measure the width of the first synthesized image in the second direction; Calculate the difference between the estimated and measured values ​​of the width in the second direction; and Determine whether the difference is below a threshold, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

11. The image processing apparatus according to claim 7 or 8, wherein, The processor performs the following processing: Calculate the deviation of the synthesis parameters determined based on the results of the feature point matching performed between adjacent images; and The presence or absence of the image with a deviation exceeding the threshold is determined, thereby determining whether the degree of distortion of the first synthesized image is within the allowable range.

12. The image processing apparatus according to claim 2, 3, 4, 5, 7 or 8, wherein, The processor performs the following processing: based on the information of the shooting location, it performs feature point matching between adjacent images.

13. The image processing apparatus according to claim 1, 2, 3, 4, 5, 7 or 8, wherein, The image set consists of images captured by a photographic device equipped with multiple cameras while changing position. The information about the shooting location consists of the configuration location information of the camera in the shooting device and the location information of the shooting location implemented by the shooting device.

14. The image processing apparatus according to claim 1, 2, 3, 4, 5, 7 or 8, wherein, The processor performs the following processing: For an object that is divided into multiple segments along its length, information on the image group and the photographic position of the images constituting the image group is obtained for each segment; Generate the first composite image or the second composite image according to each of the aforementioned segments; and A first output image is generated, which is configured according to the arrangement of the segments, based on the first composite image or the second composite image generated for each segment.

15. The image processing apparatus according to claim 1, 2, 3, 4, 5, 7 or 8, wherein, The processor performs the following processing: Analyze the images constituting the image group, the first composite image, or the second composite image, and detect damage to the surface of the object; and A second output image is generated, which superimposes the detection results of the damage onto the first composite image or the second composite image.

16. An image processing method, comprising the following steps: Obtain information about the image group and the photographic locations of the images constituting the image group; The image group is subjected to a first composite processing to generate a first composite image; Determine whether the quality of the first synthesized image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the shooting location.

17. An image processing program that enables a computer to perform the following functions: Obtain information about the image group and the photographic locations of the images constituting the image group; The image group is subjected to a first composite processing to generate a first composite image; Determine whether the quality of the first synthesized image meets the first criterion; and If the quality of the first composite image does not meet the first benchmark, a second composite image is generated by performing a second composite processing on the image group or the first composite image based on the information of the shooting location.

18. A non-transitory, computer-readable recording medium having recorded the program of claim 17.

Images