Panoramic image generation system and panoramic image generation method

Abstract

To provide a panoramic image generation system capable of achieving both reduction in generation time of a panoramic image and improvement in composition accuracy.SOLUTION: The panoramic image generation system 1 includes the imaging unit 50 that images the outer wall surface 2a of the vessel 2, the wheel unit 20 that moves the imaging unit 50 along the outer wall surface 2a, and the image synthesis unit 105. The image combining section 105 includes a first combining section 105a that combines adjacent images using a first combining method that requires a relatively short time and has relatively low accuracy, and a second combining section 105b that combines adjacent images using a second combining method that requires a relatively long time and has relatively high accuracy. The image combining unit 105 matches adjacent images by the first combining unit 105a, combines the adjacent images by the matching when errors in the matching are smaller than thresholds, and combines the adjacent images by the second combining unit 105b when the errors in the matching are equal to or larger than the thresholds.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for generating a panoramic image of the exterior wall of a ship. [Background technology]

[0002] In recent years, there has been a demand for preventing the movement of marine organisms (algae, shellfish, etc.) attached to the outer wall surfaces of ships. Patent Document 1 describes a robot that cleans the outer wall surfaces of ships while they are sailing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-535681 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to determine whether a ship can enter a port, it is desirable to check whether the exterior wall surface of the ship is dirty before the ship enters the port, and there is a need to obtain clear images of the exterior wall surface. To meet this requirement, the applicant of the present application is developing an inspection method and inspection device for capturing images of the exterior wall surface of a ship (unpublished at the time of filing this application). In such an inspection method and inspection device, it is desirable to generate a panoramic image by combining captured images in order to determine the location of the dirt on the exterior wall surface of the ship. However, because the exterior wall surface of a ship has few feature points, manually combining images takes a long time and the accuracy of the image combination varies depending on the skill level of the operator.

[0005] The present invention has been made in consideration of the above points, and its objective is to provide a panoramic image generation system and a panoramic image generation method that can simultaneously shorten the time required to generate a panoramic image and improve the synthesis accuracy. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the panoramic image generation system of the present invention comprises an imaging unit that images the outer wall surface of an object, a moving unit that moves the imaging unit along the outer wall surface, and an image synthesis unit that generates a panoramic image by synthesizing images captured by the imaging unit, wherein the image synthesis unit comprises a first synthesis unit that synthesizes adjacent images using a first synthesis method that is relatively short and relatively low-accuracy, and a second synthesis unit that synthesizes adjacent images using a second synthesis method that is relatively long and relatively high-accuracy, and the image synthesis unit matches adjacent images using the first synthesis unit, and if the error in matching is less than a threshold, synthesizes the adjacent images by said matching, and if the error in matching is equal to or greater than the threshold, synthesizes the adjacent images by the second synthesis unit. [Effects of the Invention]

[0007] According to the present invention, a panoramic image is generated primarily using a first synthesis method, and when the error is large, a panoramic image is generated using a second synthesis method, thereby achieving both a reduction in the time required to generate a panoramic image and an improvement in synthesis accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a perspective view schematically illustrating an imaging device of a panoramic image system according to an embodiment of the present invention. [Figure 2] 1 is a front view schematically showing an imaging device of a panoramic image generation system according to an embodiment of the present invention. [Figure 3] 1 is a side view schematically showing an imaging device of a panoramic image generation system according to an embodiment of the present invention. [Figure 4] 1 is a block diagram schematically illustrating a panoramic image generation system according to an embodiment of the present invention. [Figure 5] 10 is a table schematically illustrating an example of a database for determining the degree of soiling. [Figure 6]1 is a schematic diagram for explaining a panoramic image generation method by a panoramic image generation system according to an embodiment of the present invention. FIG. [Figure 7] 10A and 10B are schematic diagrams showing examples of panoramic images generated from imaging results of the vicinity of a draft mark on the exterior wall surface of a ship, where (a) is a schematic diagram showing a manually generated panoramic image, (b) is a schematic diagram showing a panoramic image generated using the ECC method, and (c) is a schematic diagram showing a panoramic image generated using the LoFTR method. [Figure 8] 10A and 10B are schematic diagrams showing examples of panoramic images generated from imaging results of a portion of the exterior wall surface of a ship with few feature points, where (a) is a schematic diagram showing a manually generated panoramic image, (b) is a schematic diagram showing a panoramic image generated using the ECC method, and (c) is a schematic diagram showing a panoramic image generated using the LoFTR method. [Figure 9] Schematic diagrams showing examples of panoramic images generated from imaging results of a black-painted portion of the exterior wall surface of a ship, where (a) is a schematic diagram showing a manually generated panoramic image, (b) is a schematic diagram showing a panoramic image generated using the ECC method, and (c) is a schematic diagram showing a panoramic image generated using the LoFTR method. [Figure 10] 1 is a flowchart illustrating an example of the operation of the panoramic image generation system according to an embodiment of the present invention. [Figure 11] 1 is a flowchart illustrating an example of the operation of the panoramic image generation system according to an embodiment of the present invention. [Figure 12] 1 is a flowchart illustrating an example of the operation of the panoramic image generation system according to an embodiment of the present invention. [Figure 13] 10A and 10B are schematic diagrams for explaining a method for extracting contaminated areas on the exterior wall surface of a ship and determining the degree of contamination, in which (a) is a schematic diagram showing an image (cropped image) captured by an imaging unit, (b) is a schematic diagram showing an image obtained by dividing the cropped image into multiple parts, (c) is a schematic diagram showing an image from which contaminated areas have been extracted, (d) is a schematic diagram showing a combined image, and (e) is a schematic diagram showing a panoramic image from the images from which contaminated areas have been extracted. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment of the present invention will be described in detail with reference to the drawings, taking an example of imaging the outer wall surface of a ship while it is sailing. In the description, the same elements are given the same symbols, and duplicate explanations will be omitted. In the following description, with regard to expressions indicating the attitude and direction of the imaging device, of directions perpendicular to the up-down direction (vertical direction), the direction toward the outer wall surface of the ship is defined as "front," and the direction away from the outer wall surface of the ship is defined as "rear," and directions perpendicular to the up-down direction (vertical direction) and the fore-aft direction are defined as "left and right."

[0010] 1, 2, and 3, an imaging system 1 according to an embodiment of the present invention is a system for capturing images of an outer wall surface 2a (see FIG. 3) of a ship 2, and includes an imaging device 3 and a cable unit 70. Furthermore, as shown in FIG. 4, the imaging system 1 includes an operation unit 80, a display unit 90, and a control unit 100.

[0011] <Imaging device> As shown in FIGS. 1 to 3, the imaging device 3 is a device that images the outer wall surface 2a of the ship 2 while moving vertically along the outer wall surface 2a (while ascending in this embodiment). The imaging device 3 includes a frame unit 10, a plurality of (four in this embodiment) wheel units 20, a housing unit 30, a mirror unit 40, an imaging unit 50, and a plurality of (two in this embodiment) lighting units 60. The imaging device 3 can be moved up and down in the sea by a winch 4 (see FIG. 4) arranged on the deck of the ship 2. The winch 4 is an electric or hydraulic traction mechanism that is operated based on manual operation by an operator.

[0012] <Frame> The frame 10 is a metal member having a substantially rectangular parallelepiped frame shape. A plurality of wheel units 20, a housing unit 30, and a plurality of lighting units 60 are attached to the frame 10, and the tip of a wire 4a of a winch 4 (see FIG. 4) for moving the imaging device is also attached to the frame 10. The winch 4 is an example of an operating unit that operates the wheel units 20 as a moving unit.

[0013] <Wheel section> The wheel unit 20 is attached to the four corners of the front side (the outer wall surface 2a side of the ship 2) of the frame unit 10. The wheel unit 20 is an example of a moving unit that travels passively in one direction from bottom to top on the outer wall surface 2a of the ship 2 as the imaging device 3 is pulled up in the sea. In this embodiment, the wheel unit 20 is formed of a magnet, and rotates and travels on the metal outer wall surface 2a while adhering to the outer wall surface 2a, while maintaining a constant distance between the mirror unit 40 and the imaging unit 50 and the outer wall surface 2a of the ship 2.

[0014] <Housing> The housing 30 is a member that has a substantially equilateral triangular shape when viewed from the front. The housing 30 includes a metal member integrally including a substantially equilateral triangular main wall 31 and a peripheral wall 32 extending forward from the periphery of the main wall 31, and a transparent resin wall 33 that faces the main wall 31 and closes the opening of the metal member. The housing 30 is attached to the frame 10 with the transparent wall 33 of the housing 30 facing forward. The housing 30 covers the space between the imaging unit 50 and the imaging portion on the outer wall surface 2a of the ship 2, thereby preventing ocean currents (seawater resistance) and external light from affecting the mirror unit 40 and the imaging unit 50 (imaging results). Fresh water is poured into the housing 30 to prevent deformation of the housing 30 due to water pressure.

[0015] ≪Mirror Section≫ The mirror unit 40 is disposed on the substantially equilateral triangular bottom surface of the housing 30. The mirror unit 40 has a horizontally long rectangular shape with a relatively small vertical dimension and a relatively large horizontal dimension, and is intended to ensure a large imaging range by the imaging unit 50 by increasing the distance between the imaging unit 50 and the outer wall surface 2a of the ship 2.

[0016] <Imaging unit> The imaging unit 50 is disposed at the apex of the peripheral wall 32 of the housing 30, with the axial direction (imaging direction) of the imaging element and optical system (lens, etc.) of the imaging unit parallel to the outer wall surface 2a of the vessel 2. In this embodiment, the imaging unit 50 is attached to the outside of the housing 30, with the optical system of the imaging unit 50 inserted downward into a hole formed at the apex of the peripheral wall 32 of the housing 30. The imaging unit 50 periodically images the outer wall surface 2a of the vessel 2 via the mirror unit 40 and outputs the imaging results to the control unit 100 via the cable unit 70. The imaging unit 50 images the outer wall surface 2a of the vessel 2 from a position close to the outer wall surface 2a, thereby enabling continuous acquisition of images that are less affected by seawater noise such as marine snow.

[0017] The imaging timing of the imaging unit 50 and the moving speed (rising speed in this embodiment) of the cable unit 70 are set so that the imaging areas of adjacent images overlap partly (preferably half or more) in the vertical direction.

[0018] <Lighting Department> The lighting units 60 are attached to both left and right ends of the frame unit 10, and irradiate light onto the area of ​​the outer wall surface 2a of the ship 2 that is to be imaged by the imaging unit 50.

[0019] <Cable section> The cable unit 70 connects the imaging unit 50 and the control unit 100 (see FIG. 4) so ​​that they can communicate with each other. The cable unit 70 is extended as the imaging device 3 is lowered vertically along the outer wall surface 2a of the ship 2 by the winch 4 (see FIG. 4) and the wire 4a.

[0020] <Operation section> As shown in FIG. 4, the operation unit 80 is configured with a keyboard, a mouse, a touch panel, etc., and outputs to the control unit 100 a signal based on the result of an operation of the operation unit 80 by an operator.

[0021] <Display section> The display unit 90 is configured with a monitor or the like, and displays an image (a panoramic image or the like) based on a signal from the control unit 100.

[0022] <Control unit> The control unit 100 is configured with a CPU (Central Processing Unit), a ROM (Read-Only Memory), a RAM (Random Access Memory), an input / output circuit, etc. The control unit 100 includes, as functional units, a storage unit 101, an image acquisition unit 102, a distortion correction unit 103, a cropping unit 104, an image synthesis unit 105, a soiled area extraction unit 106, a soiling degree determination unit 107, and an image output unit 108.

[0023] <Storage section> The storage unit 101 stores images acquired by the image acquisition unit 102. The storage unit 101 also stores images processed by the distortion correction unit 103 and the cropping unit 104. The storage unit 101 also stores a panoramic image generated by the image synthesis unit 105. The storage unit 101 also stores a soiling degree determination database 101a.

[0024] The contamination level determination database 101a is a database in which the type of contamination and the acceptable range of contamination level are associated for each part of the exterior wall surface 2a of the ship 2 (see FIG. 5). In this embodiment, the contamination level determination database 101a is generated using a convolutional neural network (CNN). As shown in FIG. 5, parts of the exterior wall surface 2a include, for example, all hull surfaces (exterior wall surfaces), areas around the waterline (exterior wall surfaces around the waterline), and hull surfaces (exterior wall surfaces other than those around the waterline). Types of contamination include slime layers, shellfish (such as scallops and barnacles), algae, brown algae, red algae, tube worms, and bryozoans.

[0025] <Image acquisition section> As shown in FIG. 4, the image acquisition unit 102 acquires the image captured by the imaging unit 50 via the cable unit 70 and stores the acquired image in the storage unit 101.

[0026] Distortion correction section The distortion correction unit 103 reads out the image captured by the imaging unit 50 and stored in the memory unit 101, corrects the distortion of the read out image, and outputs the image with the corrected distortion (distortion-corrected image) to the crop unit 104.

[0027] <Crop section> The cropping unit 104 acquires the distortion-corrected image output from the distortion correction unit 103 and generates a cropped image by cutting out a rectangular area at the center of the acquired distortion-corrected image (the area in which the outer wall surface 2a of the ship 2 is captured by reflection from the mirror unit 40) (i.e., removing unnecessary peripheral areas in which images other than the outer wall surface 2a of the ship 2 are captured). The cropping unit 104 stores the generated cropped image in the memory unit 101. The cropped image, which is the area captured through the mirror unit 40, is a localized image with a narrow viewing angle that is elongated in the left-right direction.

[0028] <Image synthesis section> The image composition unit 105 generates a panoramic image representing the exterior wall surface 2a of the ship 2 by matching and synthesizing multiple images captured by the imaging unit 50 (in this embodiment, multiple cropped images processed by the distortion correction unit 103 and the cropping unit 104). The image composition unit 105 also stores the generated panoramic image in the storage unit 101. The image composition unit 105 can also generate a panoramic image using multiple cropped images in which stained areas have been extracted and the degree of staining determined by the stained area extraction unit 106 and the stain degree determination unit 107 (described later). In this embodiment, the image composition unit 105 generates a panoramic image by assuming that the imaging device 3 moves parallel (vertically downward) to the exterior wall surface 2a of the ship 2 and taking into account only the vertical (up-down) movement component between adjacent images. This is because the imaging device 3 is set to move linearly up and down along the exterior wall surface 2a of the ship 2 using the wheel unit 20, and movement in the left-right direction and in a direction perpendicular to the exterior wall surface 2a can be ignored. The panorama synthesis unit 105 includes a first synthesis unit 105a and a second synthesis unit 105b.

[0029] <<First Synthesis Section>> The first composition unit 105a estimates the amount of movement between adjacent images using a first composition method, and combines the adjacent images based on the estimated amount of movement. The first composition method is a method for combining images in a relatively short time and with relatively low accuracy compared to the second composition method, and in this embodiment is the ECC (Enhanced Correlation Coefficient) method, which is a type of homography estimation algorithm.

[0030] The ECC method is a method in which the brightness of neighboring pixels is used as a feature value, and the amount of movement of pixels having the same feature value in adjacent images is used as a parameter of a nonlinear function, and the parameter is calculated by solving a linear optimization problem of the nonlinear function. In this embodiment, the ECC method is a method in which a transformation matrix between adjacent images is calculated by maximizing the correlation value of the images, as shown in FIG. 6, and the feature points of adjacent images 5A and 5B (the feature point P of image 5A) are calculated. A1 ,P A2 ,P A3 ,P A4 , feature point P in image 5B B1 ,P B2 ,P B3 ,P B4 ) and calculates a homography transformation matrix (corresponding to the amount of movement based on the feature amount) between adjacent images from the feature amount (brightness, etc.) of the extracted feature points, and then uses the homography transformation matrix to A1 and P B1 , P A2 and P B2 , P A3 and P B3 , P A4 and P B4 ) (see the bottom two images 5A and 5B in Figure 6).

[0031] The first composition unit 105a converts one of the adjacent images using the ECC method and calculates a correlation function of the luminance values ​​of the converted image and the other image on a pixel-by-pixel basis. This correlation function is the photometric error, which is 1 if the luminance values ​​of the adjacent images on a pixel-by-pixel basis perfectly match, and approaches 0 as the degree of match decreases. The first composition unit 105a can determine that composition is possible if the photometric error is equal to or greater than a predetermined value, and can determine that composition is not possible if the photometric error is less than the predetermined value.

[0032] The first composition unit 105a predicts the amount of movement between adjacent images using a state space model and calculates the error between the amount of movement estimated by the first composition method and the amount of movement predicted by the state space model. If the calculated error is less than a preset threshold, the first composition unit 105a combines the adjacent images based on the amount of movement estimated by the first composition method. If the calculated error is equal to or greater than the threshold, the second composition unit 105 combines the adjacent images.

[0033] <<Second Synthesis Section>> 4, when the error in the panoramic synthesis performed by the first synthesis unit 105a is large, the second synthesis unit 105b uses a second synthesis method to estimate the amount of movement between adjacent images and synthesize the adjacent images based on the estimated amount of movement. The second synthesis method is a method for synthesizing images in a relatively long time and with relatively high accuracy compared to the first synthesis method, and in this embodiment, is the LoFTR (Detector-Free Local Feature Matching with Transformers) method, which is a type of homography estimation algorithm.

[0034] As shown in Figure 6, the LoFTR method calculates the feature values ​​of the entire image, performs correspondence for each patch of the image based on the calculated feature values, extracts feature points, and calculates a homography transformation matrix between adjacent images from the feature values ​​(luminance, etc.) of the feature points, and then uses the homography transformation matrix to directly construct a close matching between pixels (area X1 in image 5B and area X2 in image 5C) (see the two images 5B and 5C at the top of Figure 6).

[0035] 4, the second composition unit 105b resizes each of two adjacent images into a low-quality image and a high-quality image. Then, the second composition unit 105b extracts feature maps of the adjacent low-quality images using CNN, and also extracts feature maps of the adjacent high-quality images using CNN.

[0036] Next, the second synthesis unit 105b calculates spatially long-distance dependencies for each of the two low-quality images from which feature maps have been extracted using a self-attention mechanism that takes into account the dependencies between each pixel within an image, and adds the calculation results to the feature maps.Similarly, the second synthesis unit 105b calculates spatially long-distance dependencies for each of the two low-quality images from which feature maps have been extracted using a cross-attention mechanism that takes into account the dependencies between each pixel between the two images, and adds the calculation results to the feature maps.

[0037] Next, the second composition unit 105b matches the adjacent low-quality images using the feature maps (corresponding to the amount of movement) of the low-quality images. Next, the second composition unit 105b matches the adjacent high-quality images by applying the matching results of the adjacent low-quality images to the feature maps of the adjacent high-quality images, and generates a panoramic image using the high-quality images.

[0038] <Example of generating a panoramic image> Here, with reference to FIG. 7, a case will be described in which a panoramic image is generated from an image of the vicinity of a draft mark (painted red) that has many characteristic points as the exterior wall surface 2a of a ship 2. Panoramic images generated from such an area are substantially the same when generated manually as shown in FIG. 7(a), when using the ECC method shown in FIG. 7(b), and when using the LoFTR method shown in FIG. 7(c). Note that manual generation takes longer than the ECC method and the LoFTR method, and accuracy depends on the skill level of the operator. Therefore, when generating a panoramic image of an area with many characteristic points, the ECC method, which can be executed in a short time, is preferred.

[0039] Next, we will explain the case where a panoramic image is generated from an image of an area with few feature points (a clean area painted red) on the exterior wall surface 2a of the ship 2. The panoramic image generated from such an area is more accurate when generated manually as shown in FIG. 8(a) and when using the LoFTR method as shown in FIG. 8(c) than when using the ECC method as shown in FIG. 8(b). Note that manual generation takes longer than the ECC method and the LoFTR method, and the accuracy depends on the skill level of the operator. Therefore, when generating a panoramic image of an area with few feature points, the LoFTR method is preferable, as it takes a relatively long time but can be performed with high accuracy.

[0040] Next, a case where a panoramic image is generated from a portion of the outer wall surface 2a of the ship 2 that is painted black will be described. The panoramic image generated from such a portion is much more accurate when generated manually as shown in FIG. 9(a) and when using the LoFTR method as shown in FIG. 9(c) than when using the ECC method as shown in FIG. 9(b). Note that the manual method requires a longer time than the ECC method and the LoFTR method, and the accuracy depends on the skill level of the operator. Therefore, when generating a panoramic image of a portion that is painted black, the LoFTR method is preferable, as it takes a relatively long time but can be performed with high accuracy.

[0041] <Stained area extraction section> As shown in Figure 4, the soiled area extraction unit 106 reads out the image (cropped image) stored in the memory unit 101, extracts the soiled areas from the read out image, and outputs the image from which the soiled areas have been extracted to the soiling degree determination unit 107.

[0042] The contaminated area extraction unit 106 is a functional unit that performs object recognition using CNN, and extracts contaminated areas in the image by comparing the learning results of pre-learned contaminated images with the image captured by the imaging unit 50. The contaminated area extraction unit 106 can also identify the type of contaminated area (algae, shellfish, etc.). For each type of contaminated area, the contaminated area extraction unit 106 calculates the size of the contaminated area (width, area, area ratio of the contaminated area to the exterior wall surface in the image, etc.) and includes the type and size of the contaminated area in the extraction result of the contaminated area.

[0043] <<Damage Degree Judgment Unit>> The contamination level determination unit 107 acquires an image in which a contamination portion has been extracted, and determines the contamination level of the exterior wall surface 2a within the image range based on the acquired image and the contamination level determination database 101a. The contamination level determination unit 107 determines the contamination level for each image and type of contamination by referring to the contamination level determination database 101a, and outputs the determination result to the storage unit 101, the image composition unit 105, and / or the display unit 90. In this embodiment, the contamination level determination unit 107 determines whether each image (e.g., a divided image) is contamination (NG) or not contamination (OK). Note that the method of determining the contamination level is not limited to two levels, and may be three or more levels, a score system, or the like. The contamination level determination unit 107 may also be configured to highlight images determined to be contamination.

[0044] As a method for identifying which part of the ship 2 the contamination determination part is, the contamination level determination unit 107 may be configured to set, for each image (or divided image), which part of the ship 2 the image represents, based on, for example, the result of an operator operating the operation unit 80. The contamination level determination unit 107 may also be configured to determine which part of the ship 2 the image represents, based on the mold depth of the ship 2 pre-stored in the memory unit 101 and the amount of wire 4a pulled out of the winch 4 detected by a detection unit or the like for each timing at which the image is captured.

[0045] <Image output section> The image output unit 108 reads out the panoramic image generated by the image synthesis unit 105 and causes the display unit 90 to display the read out panoramic image.

[0046] <Example of operation> Next, an example of the operation of the panoramic image generation system according to the embodiment of the present invention, that is, a panoramic image generation method, will be described with reference to the flowcharts of FIGS. 10, 11 and 12 (see FIG. 3 as appropriate).

[0047] First, the imaging unit 50 images the exterior wall surface 2a while moving in one direction from bottom to top by the moving unit (wheel unit 20) as the winch 4 pulls (imaging step). The captured images are stored in the storage unit 101. Next, the image synthesis unit 105 generates a panoramic image based on the captured images (panoramic image generation step). The panoramic image generation step will be described in detail below.

[0048] <<Example of operation: Image pre-processing>> 10, distortion correction unit 103 reads out the nth (first, the first) image stored in storage unit 101 (steps S1 and S2). Subsequently, distortion correction unit 103 performs distortion correction on the read out nth image, and outputs the distortion-corrected image to crop unit 104 (step S3).

[0049] Next, the cropping unit 104 acquires the distortion-corrected image output from the distortion correction unit 103 and generates a cropped image from the acquired distortion-corrected image (step S4). Next, the cropping unit 104 stores the generated cropped image in the storage unit 101 (step S5).

[0050] If the processed image is the final frame (Nth) image (Yes in step S6), control unit 100 ends this flow. If the processed image is not the final frame image (No in step S6), distortion correction unit 103 and cropping unit 104 execute steps S2 to S5 on the (n+1)th (first, second) image.

[0051] <<Example of operation: Panorama image generation>> As shown in FIG. 11, the first composition unit 105a reads the (n-1)th and nth (first, the 1st and 2nd) cropped images from the storage unit 101 and estimates the amount of movement of the nth image relative to the (n-1)th image using the ECC method (steps S11 and S12). Next, the first composition unit 105a predicts the amount of movement of the nth image relative to the (n-1)th image using the information space model (step S13). The order of the movement amount estimation using the ECC method in step S12 and the movement amount prediction using the information space model in step S13 is not limited to the above, and step S13 may be performed first, or may be performed simultaneously. Next, the first composition unit 105a calculates the error between the movement amount estimated using the ECC method and the movement amount predicted using the information space model, and determines whether the calculated error is less than a threshold (step S14).

[0052] If the calculated error is equal to or greater than the threshold value (No in step S14), the second synthesis unit 105b estimates the amount of movement of the n-th image relative to the n-1-th image using the LoFTR method (step S15).

[0053] Next, the first composition unit 105a or the second composition unit 105b generates a panoramic image in which the (n-1)th and n-th cropped images are panoramic-combined based on the estimated movement amount (step S16). Specifically, if the answer is Yes in step S14, the first composition unit 105a generates a panoramic image in which the (n-1)th and n-th cropped images are panoramic-combined based on the movement amount estimated by the ECC method. Also, if the answer is No in step S14, the second composition unit 105b generates a panoramic image in which the (n-1)th and n-th cropped images are panoramic-combined based on the movement amount estimated by the LoFTR method.

[0054] If one of the combined images is the final frame (Nth) image (Yes in step S17), the control unit 100 ends this flow. If one of the combined images is not the final frame image (No in step S17), the first combining unit 105a and the second combining unit 105b increment n by 1 (step S18) and execute steps S12 to S16 for the nth and n+1th (first, second, and third) images.

[0055] <Example of operation: Extracting soiled areas and determining the degree of soiling> As shown in FIG. 12, the soiled area extraction unit 106 reads out the n-th cropped image 6E (see FIG. 13(a)) stored in the storage unit 101 in step S5 from the storage unit 101 (steps S21 and S22).

[0056] Next, the soiled area extraction unit 106 divides the read cropped image 6E in the left-right direction into multiple images 6E1, 6E2, and 6E3 (see FIG. 13(b)) (step S22). Next, the soiled area extraction unit 106 extracts soiled areas from the divided cropped images 6E1, 6E2, and 6E3 (step S23).

[0057] Next, the contamination degree determination unit 107 determines the contamination degree of the exterior wall surface 2a for each of the images 7E1, 7E2, and 7E3 from which the contamination areas have been extracted (see FIG. 13(c)) (step S24). Next, the contamination degree determination unit 107 combines the images 7E1, 7E2, and 7E3 from which the contamination areas have been extracted and the contamination degrees have been determined in the same manner as the original cropped image (see FIG. 13(d)) (step S25).

[0058] Next, the first composition unit 105a or the second composition unit 105b uses the same method as in step S16 to compose the (n-1)th image 7D and the nth image 7E, in which the soiled areas have been extracted and the degree of soiling has been determined, to generate a panoramic image (see FIG. 13(e)) (step S26). In the panoramic image shown in FIG. 13(e), the extracted soiled areas are depicted, and the areas corresponding to image 7E2, which have been determined to be soiled, are highlighted (displayed with high brightness, lit or flashing, displayed in a different color, displayed in a frame, etc.).

[0059] If one of the combined images is the final frame (Nth) image (Yes in step S27), control unit 100 ends this flow. If one of the combined images is not the final frame image (No in step S27), image combination unit 105, soiled area extraction unit 106, and soiling level determination unit 107 increment n by 1 (step S28) and execute steps S22 to S26 for the nth and n+1th (first, second, and third) images.

[0060] A panoramic image generation system 1 according to an embodiment of the present invention includes an imaging unit 50 that captures images of an outer wall surface 2a of an object (ship 2), a moving unit (wheel unit 20) that moves the imaging unit 50 along the outer wall surface 2a, and an image synthesis unit 105 that generates a panoramic image by synthesizing the images captured by the imaging unit 50. The image synthesis unit 105 includes a first synthesis unit 105a that synthesizes adjacent images using a first synthesis method that is relatively short and relatively low-accuracy, and a second synthesis unit 105b that synthesizes adjacent images using a second synthesis method that is relatively long and relatively high-accuracy. The image synthesis unit 105 matches adjacent images using the first synthesis unit 105a, and if the error in matching is less than a threshold, synthesizes the adjacent images using the matching. If the error in matching is equal to or greater than the threshold, synthesizes the adjacent images using the second synthesis unit 105b. Therefore, the panoramic image generation system 1 uses the first synthesis method and the second synthesis method to synthesize adjacent images, thereby achieving both a reduction in the time required to generate a panoramic image of an exterior wall surface 2a with few feature points and an improvement in synthesis accuracy.

[0061] In the panoramic image generation system 1, the first synthesis method is the ECC method, and the second synthesis method is the LoFTR method. Therefore, the panoramic image generation system 1 uses the ECC method and the LoFTR method to synthesize adjacent images, thereby more effectively achieving both a shorter generation time for a panoramic image of an exterior wall surface 2a with few feature points and improved synthesis accuracy.

[0062] In the panoramic image generation system 1, the first synthesis unit 105a estimates the amount of movement between adjacent images using the ECC method and predicts it using an information space model, and calculates the error based on the amount of movement estimated using the ECC method and the amount of movement predicted using the information space model. Therefore, the panoramic image generation system 1 can suitably calculate the error in the amount of movement estimated by the ECC method, and suitably determine whether to use the ECC method or the LoFTR method.

[0063] In the panoramic image generation system 1, the image synthesis unit 105 synthesizes adjacent images based only on the movement component in the movement direction of the imaging unit 50 caused by the moving unit. Therefore, the panoramic image generation system 1 can improve the processing speed of image synthesis.

[0064] The panoramic image generating system 1 includes a soiled portion extracting unit 106 that extracts soiled portions of the exterior wall surface 2a from the images, and the image combining unit 105 combines the images from which the soiled portions have been extracted. Therefore, the panoramic image synthesis system 1 can improve the visibility of a soiled area in a panoramic image.

[0065] The panoramic image generation system 1 includes a contamination level determination unit 107 that determines the contamination level of the exterior wall surface 2 a based on the image in which the contamination area is extracted, Therefore, the panoramic image synthesis system 1 can suitably determine the degree of contamination of the exterior wall surface 2a of the ship 2, and can suitably inspect the presence or absence of contamination on the exterior wall surface 2a of the ship 2. Furthermore, the panoramic image generation system 1 can improve the visibility of the degree of contamination of each part of the exterior wall surface 2a in the panoramic image.

[0066] In addition, the panoramic image generation method by the panoramic image generation system 1 according to an embodiment of the present invention includes an imaging step in which the imaging unit 50 images the exterior wall surface 2a while moving using the moving unit, and a panoramic image generation step in which the image synthesis unit 105 generates the panoramic image. Therefore, according to the panoramic image generation method, adjacent images are synthesized by selectively using the first synthesis method and the second synthesis method, thereby achieving both a reduction in the time required to generate a panoramic image of an exterior wall surface 2a with few feature points and an improvement in synthesis accuracy.

[0067] In the panoramic image generation method, the panoramic image generation step includes the steps of: the first synthesis unit 105a estimating the amount of movement of the adjacent images by an ECC method and predicting the amount of movement of the adjacent images by an information space model; if an error between the amount of movement estimated by the ECC method and the amount of movement predicted by the information space model is less than a threshold, the first synthesis unit 105a synthesizing the adjacent images based on the amount of movement estimated by the ECC method; and if an error between the amount of movement estimated by the ECC method and the amount of movement predicted by the information space model is equal to or greater than the threshold, the second synthesis unit 105b estimating the amount of movement of the adjacent images by a LoFTR method and synthesizing the adjacent images based on the amount of movement estimated by the LoFTR method. Therefore, according to the panoramic image generation method, the ECC method is used when the error in the amount of movement estimated by the ECC method is relatively small, and processing using LoFTR is performed only when the amount of movement estimated by the ECC method is relatively large, thereby making it possible to advantageously shorten the generation time of a panoramic image for an exterior wall surface 2a with few feature points and improve synthesis accuracy at the same time.

[0068] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments and can be modified as appropriate without departing from the gist of the present invention. For example, the present invention may be configured to generate a panoramic image of an outer wall surface of an object other than the outer wall surface 2a of a ship 2 while sailing. Furthermore, the order of panoramic synthesis of adjacent images is not limited to being performed in the order from the first to the final frame (Nth), and can be changed as appropriate. [Explanation of symbols]

[0069] 1. Panoramic image generation system 2. Ships (objects) 2a External wall surface 4 Winch (operating part) 20 Wheel section (moving section) 50 Imaging unit 100 control section 105 Panorama synthesis section 105a First composite part 105b Second synthesis part 106 Contaminated area extraction unit 107 Degree of Contamination Judgment Unit

Claims

1. an imaging unit that captures an image of an outer wall surface of an object; a moving unit that moves the imaging unit along the outer wall surface; an image synthesis unit that generates a panoramic image by synthesizing the images captured by the imaging unit; Equipped with The image synthesis unit a first synthesis unit that synthesizes the adjacent images using a first synthesis method that is relatively short in time and relatively low in accuracy; a second synthesis unit that synthesizes the adjacent images using a second synthesis method that takes a relatively long time and has a relatively high accuracy; Equipped with The image synthesis unit Matching adjacent images by the first synthesis unit; If the error in the matching is less than a threshold, the adjacent images are combined by the matching; If the error in the matching is equal to or greater than the threshold, the second combining unit combines the adjacent images. A panoramic image generation system.

2. the first synthesis method is an ECC method, The second synthesis method is the LoFTR method.

2. The panoramic image generation system according to claim 1.

3. The first synthesis unit is The amount of movement of the adjacent images is estimated by the ECC method and predicted by an information space model; The error is calculated based on the amount of movement estimated by the ECC method and the amount of movement predicted by the information space model.

3. The panoramic image generating system according to claim 2.

4. The image synthesis unit synthesizes adjacent images based only on a movement component in a movement direction of the imaging unit caused by the movement unit.

2. The panoramic image generation system according to claim 1.

5. a soiled area extraction unit that extracts soiled areas on the exterior wall surface from the image; The image synthesis unit synthesizes the images from which the stained portion is extracted.

5. The panoramic image generation system according to claim 1, wherein the panoramic image generation system is a panoramic image generation system.

6. a contamination level determination unit that determines the contamination level of the exterior wall surface based on the image from which the contamination area is extracted, The image composition unit composes the images for which the degree of soiling has been determined.

6. The panoramic image generation system according to claim 5.

7. A panoramic image generation method using the panoramic image generation system according to claim 1, an imaging step in which the imaging unit images the outer wall surface while moving by the moving unit; a panoramic image generating step in which the image synthesis unit generates the panoramic image; A panoramic image generating method comprising:

8. The panoramic image generating step includes: the first synthesis unit estimating the amount of movement between the adjacent images by an ECC method and predicting the amount of movement between the adjacent images by an information space model; When an error between the amount of movement estimated by the ECC method and the amount of movement predicted by an information space model is less than a threshold, the first composition unit composes the adjacent images based on the amount of movement estimated by the ECC method; when an error between the amount of movement estimated by the ECC method and the amount of movement predicted by an information space model is equal to or greater than the threshold, the second composition unit estimates the amount of movement between the adjacent images by the LoFTR method, and composes the adjacent images based on the amount of movement estimated by the LoFTR method; The panoramic image generating method according to claim 7, further comprising:

Images