Image processing system and image processing method for visual inspection

Abstract

This suppresses the cumulative errors that occur when generating visual inspection images by stitching together multiple images. [Solution] The visual inspection image processing system 1 includes a camera 11 that photographs an object to be inspected and acquires multiple images, a shooting position estimation unit 21 that estimates the shooting position of the camera 11 when the camera 11 acquires an image from the information contained in the image, and a shooting position correction unit 22 that corrects the shooting position so that the shooting position corresponding to the image is optimal when three or more images are stitched together to generate a visual inspection image.

Description

Technical Field

[0001] Embodiments of the present invention relate to image processing system technology for visual inspection.

Background Art

[0002] For visual inspection (VT) inside a nuclear reactor, a camera with a narrow shooting range is used to avoid missing small defects, such as defects on the order of 0.025 mm, in accordance with inspection standards such as VT-1. This camera is suspended from an operation pole from the opening and closing flange, and the method is to manually operate the camera at the tip of the operation pole. Therefore, although the approximate inspection position can be grasped, it is difficult to accurately grasp the position. The inspector manually stitches together inspection images after the fact to confirm the inspection position, and confirms it based on landmarks inside the reactor, but it is difficult to accurately stitch the images manually. In addition, cracks that do not fit within one shooting range are sized based on the stitched images and the dimensions of the landmarks inside the reactor. However, if the stitching is not accurate, sizing errors occur, and the burden of post-work to manually correct these errors is also large.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to suppress the errors that accumulate when generating an image for visual inspection by stitching together a large number of images. [Means for solving the problem]

[0005] An image processing system for visual inspection according to an embodiment of the present invention comprises: a camera that photographs an object to be inspected and acquires a plurality of captured images; a shooting position estimation unit that estimates the shooting position of the camera when the camera acquires the captured images from the information contained in the captured images; and a shooting position correction unit that corrects the shooting position so that the shooting position corresponding to the captured image is optimal when three or more of the captured images are stitched together to generate a visual inspection image. [Effects of the Invention]

[0006] According to the embodiments of the present invention, it is possible to suppress the cumulative errors that occur when generating an image for visual inspection by stitching together a large number of images. [Brief explanation of the drawing]

[0007] [Figure 1] A cross-sectional view showing a nuclear reactor being inspected by an image processing system for visual inspection. [Figure 2] A block diagram showing an image processing system for visual inspection. [Figure 3] A functional block diagram showing the processing flow of an image processing method for visual inspection. [Figure 4] An explanatory diagram showing images for visual inspection. [Figure 5] An explanatory diagram showing three captured images. [Figure 6] An explanatory diagram showing the first correction process when two captured images are stitched together. [Figure 7] An explanatory diagram showing the second correction process when three captured images are stitched together. [Figure 8] A side view showing the camera's shooting position. [Figure 9] An explanatory diagram showing the captured image. [Figure 10] An explanatory diagram showing the mapping image. [Modes for carrying out the invention]

[0008] The embodiments of the visual inspection image processing system and visual inspection image processing method will be described in detail below with reference to the drawings.

[0009] Reference numeral 1 in Figure 1 denotes the visual inspection image processing system of this embodiment. The visual inspection image processing method is carried out using this visual inspection image processing system 1.

[0010] The visual inspection image processing system 1 comprises an inspection computer 2, a scanning device 10, and a camera 11. The scanning device 10 has a drive mechanism 12. The drive mechanism 12 is, for example, a motor. The scanning device 10 is equipped with a camera 11, and the drive mechanism 12 drives the camera 11 along the object to be inspected. The camera 11 photographs the object to be inspected and acquires multiple images. The camera 11 may also be equipped with lighting equipment (not shown). Furthermore, the camera 11 may be either a visible light camera or an infrared camera. In addition, the images acquired by the camera 11 may be either still images or moving images. The following description uses still images as an example.

[0011] This visual inspection image processing system 1 takes a picture of a predetermined inspection target with a camera 11 and performs a visual inspection. The inspection target is, for example, a nuclear reactor 13 installed in a nuclear power plant. The visual inspection image processing system 1 generates a visual inspection image, which is an image used for the visual inspection of the nuclear reactor 13.

[0012] Furthermore, the inspection is not limited to reactor 13; it may also include equipment installed in other designated facilities such as power plants, chemical plants, and large-scale factories.

[0013] For example, the scanning device 10 is mounted on a crane 15 located on the operation floor 14. The camera 11 is suspended from an operating pole 16 extending downward from the operation floor 14. This camera 11 photographs predetermined equipment and in-reactor structures inside the reactor 13. The movement and position of the camera 11 are controlled by the drive mechanism 12 of the scanning device 10.

[0014] In addition, an operator may manually lower the operation pole 16 and perform imaging inside the reactor 13 by manual operation. That is, in the visual inspection image processing system 1, the configuration of the scanning device 10 may be omitted.

[0015] Next, the system configuration of the visual inspection image processing system 1 will be described with reference to the block diagram shown in FIG. 2.

[0016] The visual inspection image processing system 1 includes hardware resources such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and an SSD (Solid State Drive). The CPU executes various programs, and information processing by software is realized using the hardware resources, which is composed of a computer. Furthermore, the visual inspection image processing method of the present embodiment is realized by causing a computer to execute various programs.

[0017] The inspection computer 2 of the present embodiment includes an input unit 3, an output unit 4, a communication unit 5, a processing circuit 6, and a storage unit 7.

[0018] Predetermined information is input to the input unit 3 according to the operation of a user who uses the inspection computer 2. The input unit 3 includes input devices such as a mouse, a keyboard, and a touch panel. That is, predetermined information is input to the inspection computer 2 according to the operation of these input devices.

[0019] Output unit 4 outputs predetermined information. The inspection computer 2 includes a device for displaying images, such as a display that outputs the analysis results. In other words, output unit 4 controls the images displayed on the display. The display may be separate from the computer body or integrated with it. Additionally or alternatively, output unit 4 may also control images displayed on displays of other computers connected via the network.

[0020] In this embodiment, a display is exemplified as a device for displaying images, but other embodiments are also possible. For example, images may be displayed using a head-mounted display or a projector. Furthermore, a printer that prints information on paper may be used instead of a display. In other words, the objects controlled by the output unit 4 may include a head-mounted display, a projector, or a printer.

[0021] The communication unit 5 communicates with the scanning device 10. The communication unit 5 also communicates with other computers via a communication line such as a predetermined network. Furthermore, each device may be connected to each other via a bus. The network includes the Internet, LAN (Local Area Network), WAN (Wide Area Network), and mobile communication networks.

[0022] The processing circuit 6 is, for example, a circuit equipped with a CPU, GPU, or a dedicated or general-purpose processor. This processor realizes various functions by executing various programs stored in the memory unit 7. The processing circuit 6 may also be composed of hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Various functions can also be realized by this hardware. Furthermore, the processing circuit 6 can realize various functions by combining software processing by the processor and programs with hardware processing.

[0023] The memory unit 7 stores a predetermined program to be executed by the processing circuit 6. The memory unit 7 also stores various information necessary for the inspection computer 2 to perform visual inspection image processing. Furthermore, the memory unit 7 includes a predetermined database. This database is a collection of information organized to be stored in memory, HDD, and cloud computing resources, and to be searchable or stored.

[0024] Next, the processing flow of the visual inspection image processing system 1 will be explained with reference to the functional block diagram shown in Figure 3. This visual inspection image processing system 1 may include components other than those shown in Figure 3, and some of the components shown in Figure 3 may be omitted.

[0025] Note that the arrows in the functional block diagram are just one example of a processing flow, and there may be other processing flows besides those indicated by the arrows. Also, the order of each process is not necessarily fixed, and the order of some processes may be reversed. Furthermore, some processes may be executed in parallel with other processes.

[0026] The inspection computer 2 comprises a shooting position calculation unit 20, a shooting position estimation unit 21, a shooting position correction unit 22, a mapping image generation unit 23, an image correction unit 24, an image selection unit 25, an image synthesis unit 26, a learning unit 27, a defect inference unit 28, and a defect analysis unit 29. These functions are realized by the CPU executing a program stored in memory or on the HDD. Furthermore, the inspection computer 2 includes a learning model 30.

[0027] Each component of the visual inspection image processing system 1 does not necessarily have to be installed on a single computer. For example, one visual inspection image processing system 1 may be implemented using multiple computers connected to each other via a network. For instance, each component of the inspection computer 2 shown in Figure 3 may be installed on a separate computer.

[0028] Furthermore, the configuration of the visual inspection image processing system 1 may be implemented as a cloud service. In other words, the computers that make up the visual inspection image processing system 1 may be servers on the cloud. For example, not only the configuration that processes memory, but the entire configuration that processes the main processing may reside on the cloud, and the user may only configure the visual inspection image processing system 1 and check its input / output via an API (Application Programming Interface) or a web browser.

[0029] The shooting position calculation unit 20 acquires an image of the object to be inspected from the camera 11. Furthermore, the shooting position calculation unit 20 acquires drive information from the drive mechanism 12 of the scanning device 10, indicating the result of the drive mechanism 12's operation. The shooting position calculation unit 20 calculates the shooting position of the camera 11 from the drive information. In this way, the shooting position of the camera 11 can be obtained using the drive information of the drive mechanism 12 of the scanning device 10. The shooting position is a three-dimensional coordinate system with an arbitrary position on the object to be inspected as the origin.

[0030] The shooting position estimation unit 21 estimates the shooting position of the camera 11 at the time the camera 11 acquired each captured image, based on the information contained in the captured image. In this way, even if the drive information of the drive mechanism 12 of the scanning device 10 cannot be obtained, the shooting position can be estimated from the information contained in the captured image. Furthermore, even if there is an error in the shooting position of the camera 11 based on the drive information, the shooting position can be estimated from the information contained in the captured image. In addition, even when taking pictures by manually operating the operation pole 16 without using the scanning device 10, the shooting position can be estimated.

[0031] If there is a discrepancy between the shooting position of camera 11 based on the drive information and the shooting position of camera 11 based on the information contained in the captured image, the shooting position of camera 11 based on the information contained in the captured image may be prioritized.

[0032] Furthermore, the shooting position estimation unit 21 performs at least one of the following processes: movement amount processing, number of images processing, consistency processing, and measurement area processing. In this way, the shooting position of the camera 11 can be accurately estimated.

[0033] The movement processing involves extracting multiple feature points of the subject in the captured image and measuring the amount of movement of the camera 11. Here, feature points are, for example, the edges or corners of the rectangular prism if the subject is a rectangular prism. When the camera 11 photographs a stationary subject while moving, the positions of the feature points included in the captured image change. By measuring the change in the position of the feature points for each captured image, the amount of movement of the camera 11 can be calculated in reverse.

[0034] The image counting process extracts multiple feature points of the subject in the captured image and controls the number of feature points so that it is equal to or greater than a predetermined number. The number of feature points can be set arbitrarily by the user or automatically by the shooting position estimation unit 21. In this way, even if the camera 11 moves, the target subject can continue to be captured in the image.

[0035] The consistency process determines the consistency between the direction of movement of camera 11 and the amount of movement of camera 11. This improves the accuracy of determining the shooting position of camera 11. For example, if the direction of movement and the amount of movement of camera 11 are consistent, the shooting position estimated from the captured image can also be determined to be appropriate.

[0036] The measurement area processing is a process that calculates the overlap area between captured images based on at least one of the camera 11's shooting position, camera 11's shooting direction, or camera 11's field of view, and limits the measurement area of ​​the camera 11's movement. In this way, if there is an overlap area between captured images, it can be determined that the camera 11 has not moved significantly. Based on this, the range of the camera 11's movement can be limited. The camera 11's shooting direction is the orientation of the camera 11 with the shooting position as the origin.

[0037] The shooting position correction unit 22 corrects the shooting position so that the shooting position of each camera 11 corresponding to each captured image is optimal when generating images for visual inspection. Here, an image for visual inspection is an image created by stitching together three or more captured images.

[0038] As shown in Figure 4, the visual inspection image 40 is a panoramic or omnidirectional image generated by stitching together multiple captured images 41. For example, when photographing the reactor 13 (Figure 1), the visual inspection image 40 is an image taken with the center of the reactor 13 as a virtual origin, covering a range of 360° in the circumferential direction and 90° upwards and downwards. If multiple objects 42 (subjects) are arranged in the circumferential and horizontal directions inside the reactor 13, these objects 42 will be captured in the visual inspection image 40 at coordinates corresponding to their horizontal positions. The positions of these objects 42 are captured in the visual inspection image 40 to match the actual positions of the reactor 13, due to the correction of the camera 11's shooting position.

[0039] Furthermore, "correcting the shooting position" means adjusting the actual shooting position of camera 11 to a virtual shooting position so that the shooting direction and field of view are appropriate for obtaining a suitable image. In other words, the "correcting the shooting position" process is performed after the actual shooting by camera 11 has finished, and does not change the actual shooting position of camera 11, that is, it does not move camera 11. Of course, if the actual shooting position of camera 11 is appropriate, there is no need to "correct the shooting position".

[0040] Furthermore, "optimal" refers to the state in which the cumulative error of multiple captured images is suppressed to the greatest extent possible, and where the numerical value of the cumulative error is the smallest.

[0041] Furthermore, the shooting position correction unit 22 performs a first correction process and a second correction process. The first correction process corrects the shooting positions of two adjacent images that are joined together. The second correction process corrects the shooting positions of the images again when three or more images are joined together after the first correction process to generate an image for visual inspection. In this way, the overall error is corrected by performing two stages of correction, and an image suitable for visual inspection can be generated.

[0042] For example, as shown in Figure 5, suppose three images are acquired: a first captured image 41A, a second captured image 41B, and a third captured image 41C. Here, the three images are taken from different positions, and their shooting directions, angles of view, and pixels are also different. However, the same subject is captured in all three images. First, the shooting position correction unit 22 extracts multiple feature points 43 of the subject captured in the first captured image 41A, the second captured image 41B, and the third captured image 41C.

[0043] As shown in Figure 6, the shooting position correction unit 22 combines the first captured image 41A and the second captured image 41B in the first correction process. Here, the first captured image 41A and the second captured image 41B are corrected so that multiple feature points 43 of the first captured image 41A and the second captured image 41B match. For example, the shooting position correction unit 22 performs corrections that change the size of the image, change the orientation of the image, and change the shape of the image. Then, the shooting position correction unit 22 combines the first captured image 41A and the second captured image 41B with the multiple feature points 43 matching.

[0044] As shown in Figure 7, the shooting position correction unit 22 combines the first captured image 41A, the second captured image 41B, and the third captured image 41C in the second correction process. Here, the shooting position correction unit 22 corrects the second captured image 41B and the third captured image 41C so that multiple feature points 43 of the second captured image 41B and the third captured image 41C match. Then, the shooting position correction unit 22 combines the second captured image 41B and the third captured image 41C with the multiple feature points 43 matching. However, there are cases where multiple feature points 43 of the first captured image 41A and the third captured image 41C do not match. Here, the shooting position correction unit 22 corrects the first captured image 41A, the second captured image 41B, and the third captured image 41C so that multiple feature points 43 of the first captured image 41A, the second captured image 41B, and the third captured image 41C match. In other words, the second correction process corrects each captured image after all the captured images have been stitched together. This method suppresses the cumulative errors that occur when stitching together multiple images to generate the visual inspection image 40.

[0045] Here, the position of the subject in the captured image is the 3D coordinate of the feature points of the subject, with an arbitrary origin. The shooting position correction unit 22 corrects three or more feature points of at least one subject included in a single captured image. The captured image is corrected in accordance with the correction of these feature points. The shooting position correction unit 22 corrects multiple captured images that constitute the visual inspection image. In other words, the shooting position correction unit 22 optimizes the entire visual inspection image.

[0046] As shown in Figure 3, the shooting position correction unit 22 determines whether the correction of the camera 11's shooting position is appropriate based on at least one of either a landmark of the object being inspected or the dimensions and shape of the CAD (Computer Aided Design) data. In this way, it is possible to automatically determine whether the correction of the camera 11's shooting position is appropriate or not.

[0047] Landmarks include, for example, piping, valves, and openings in the furnace structure. CAD data dimensions and shapes are, for example, the dimensions and shapes of parts that make up an object in a captured image.

[0048] As shown in Figure 3, the image generation unit 23 generates a mapped image based on the information of the camera 11's shooting position, correcting at least one captured image to appear as if it were taken from an arbitrary viewpoint position. The arbitrary viewpoint position may be set arbitrarily by the user, or it may be set automatically by the image generation unit 23.

[0049] A mapping image is an image obtained by transforming a captured image. For example, as shown in Figure 8, the camera 11 photographs a predetermined object 44 (subject) from a predetermined shooting position 45 and acquires a captured image 41 (Figure 9). Here, the mapping image generation unit 23 corrects the captured image 41 so that it becomes an image taken from a set arbitrary viewpoint position 46. This corrected image is the mapping image 47 (Figure 10). The mapping image generation unit 23 may also convert at least a portion of the visual inspection image 40 into the mapping image 47.

[0050] The mapping image generation unit 23 generates a mapping image 47 (Figure 10) of the object 44 taken from directly above, for example, when the captured image 41 (Figure 9) is an image of the object 44 taken from an oblique direction. If the object 44 has a predetermined defect such as a crack 48, the width dimension W (Figure 9) of the crack 48 cannot be accurately obtained from the captured image 41. However, the mapping image 47 can accurately obtain the width dimension W' (Figure 10) of the crack 48. Furthermore, by converting to an image taken from a fixed viewpoint position, variations in image scale and the appearance of defects can be reduced, thereby improving and stabilizing the accuracy of the defect inference unit 28, which will be described later, in determining the presence or absence of defects.

[0051] As shown in Figure 3, the image correction unit 24 corrects at least one of the illumination unevenness or the white balance of the captured image. In this way, the captured image can be corrected to an image suitable for visual inspection.

[0052] The image selection unit 25 selects the most appropriate image for inspection from multiple images of the same subject based on at least one of the following: the area of ​​the overlap region between captured images, the movement of the camera 11, or the presence or absence of blur due to camera shake. In this way, an image suitable for visual inspection can be generated.

[0053] The image synthesis unit 26 generates a visual inspection image by stitching together multiple captured images based on the shooting position corrected by the shooting position correction unit 22. In this way, the process of generating the visual inspection image can be performed automatically.

[0054] Various methods are possible for the image synthesis unit 26 to generate an image for visual inspection. For example, the image synthesis unit 26 calculates the brightness of the overlapping region using a coefficient corresponding to the distance from the center of each captured image as the starting point, and generates an image for visual inspection.

[0055] Furthermore, the image synthesis unit 26 uses areas with high similarity in the overlapping regions between captured images as boundaries between the captured images and stitches them together.

[0056] Furthermore, the image synthesis unit 26 uses regions with fewer frequency components, that is, regions with less texture, as boundaries to stitch together the captured images.

[0057] Furthermore, the image synthesis unit 26 averages only a specified area size from the boundary and superimposes the captured images.

[0058] Furthermore, the image synthesis unit 26 superimposes the images by varying the weights for each frequency component in an area of ​​arbitrary size from the boundary.

[0059] The learning unit 27 performs machine learning on the learning model 30 based on the mapped image. The defect inference unit 28 performs inference using the trained learning model 30 to determine whether or not a defect exists in the object being inspected. In this way, the learning model 30 can be made to determine whether or not a defect exists.

[0060] The memory unit 7 stores various information necessary for generating the learning model 30. For example, the memory unit 7 stores the mapping image or visual inspection image sent from the mapping image generation unit 23 or the image synthesis unit 26. The image stored in the memory unit 7 may be only a part of the mapping image or visual inspection image.

[0061] The defect inference unit 28 divides at least one mapping image into multiple local regions. The defect inference unit 28 performs inference for each local region using a machine learning model 30. Based on the shooting position corrected by the shooting position correction unit 22, the defect inference unit 28 determines whether or not a defect exists in the object under inspection from the inference results of multiple local regions corresponding to the same shooting position. By determining the presence or absence of a defect from multiple inference results, it is possible to suppress undetected or falsely detected defects, thereby improving and stabilizing the judgment accuracy. In addition, the dimensions of defects, etc., can be accurately grasped from the mapping image. Furthermore, by dividing the image into local regions, the defect inference unit 28 makes it easier for the machine learning model 30 to determine the presence or absence of a defect. A "local region" is an image obtained by dividing a single mapping image into multiple images, and is an image that shows a local part of the object under inspection.

[0062] Furthermore, the defect inference unit 28 extracts candidate local regions containing defects using a machine learning model 30 that has been trained using only normal data. The defect inference unit 28 then infers defects from the candidate local regions using a machine learning model 30 that has been trained using both normal and abnormal data. In this way, the accuracy of the processing can be improved by using a machine learning model 30 that is appropriate for each process.

[0063] For example, the defect inference unit 28 extracts candidate local regions from an autoencoder and infers defects using semantic segmentation.

[0064] The learning model 30 comprises an input layer, a hidden layer, and an output layer. Input data is input to the input layer. The hidden layer's parameters are pre-machine-trained using training data. The output layer outputs output data that shows the results processed by the hidden layer in response to the input data input to the input layer.

[0065] The learning model 30 is machine-trained using training data in which at least one of the input data or data that mimics it is input, and at least one of the output data or data that mimics it is output.

[0066] The learning unit 27 retrains the learning model 30 using images of the relevant local region when the inference result of the defect inference unit 28 differs from reality. For example, the learning unit 27 extracts images in which the user has checked the inference result of the defect inference unit 28 and the defect inference unit 28 has determined that a defect exists, but in reality there is no defect. Or, for example, the learning unit 27 extracts images in which the user has checked the inference result of the defect inference unit 28 and the defect inference unit 28 has determined that no defect exists, but in reality there is a defect. Then, the learning unit 27 retrains the learning model 30 using these extracted images. In this way, the accuracy of distinguishing between images with and without defects can be improved. Furthermore, the accuracy of the learning model 30 can be improved each time a visual inspection is performed.

[0067] The defect analysis unit 29 evaluates at least one of the following: the distribution of defects at each location, or the degree of defect progression compared to the past. In this way, the distribution of defects at each location and the degree of defect progression can be automatically evaluated.

[0068] Furthermore, the defect analysis unit 29 determines at least one of the following based on the defect occurrence distribution or the degree of defect progression: a method for addressing the defect or the timing of the next inspection. As a method of addressing the defect, the defect analysis unit 29 may, for example, predict the size of the defect at the time of the next inspection based on the degree of defect progression and decide whether to remove the defect now or to wait for the next inspection, i.e., to continue monitoring it. Alternatively, the defect analysis unit 29 may make the aforementioned determination for each location based on the defect density using the defect occurrence distribution. Alternatively, the defect analysis unit 29 may predict the size of the defect over time based on the degree of defect progression and set the next inspection at the time when it reaches a certain size. In this way, the method of addressing the defect and the timing of the next inspection can be determined automatically.

[0069] In the example described above, the computer constituting the visual inspection image processing system 1 performs various processes (including various processes such as setting, judgment, evaluation, estimation, and inference), but other configurations are also possible. For example, the user may perform some of the aforementioned processes, and the computer may receive the input of the processing results and use them in its own processing.

[0070] The aforementioned visual inspection image processing system 1 comprises a control device, a storage device, an output device, an input device, and a communication interface. Here, the control device includes a highly integrated processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), or dedicated chip. The storage device includes ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. The output device includes a display panel, head-mounted display, projector, printer, etc. The input device includes a mouse, keyboard, touch panel, etc. This visual inspection image processing system 1 can be implemented with a hardware configuration using a standard computer.

[0071] The program or learning model 30 executed by the aforementioned visual inspection image processing system 1 is provided pre-installed in ROM or the like. Alternatively, this program or learning model 30 may be provided as an installable or executable file stored on a computer-readable non-temporary storage medium. This storage medium includes CD-ROMs, CD-Rs, memory cards, DVDs, flexible disks (FDs), and the like.

[0072] Furthermore, the program or learning model 30 executed by this visual inspection image processing system 1 may be stored on a computer connected to a network such as the Internet and provided for download via the network. In other words, the program or learning model 30 may be provided from cloud computing resources. Alternatively, a server on the cloud may execute the program or learning model 30, and only the processing results may be provided via the cloud. In addition, this visual inspection image processing system 1 can also be configured by connecting and combining separate modules, each independently performing the function of its constituent elements, via a network or dedicated line.

[0073] According to the embodiments described above, the visual inspection image processing system 1 includes a shooting position correction unit 22 that corrects the shooting position so that the shooting position corresponding to each shooting image is optimized when three or more shooting images are stitched together to generate a visual inspection image. This makes it possible to suppress the cumulative error when generating a visual inspection image by stitching together a large number of images.

[0074] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, modifications, and combinations are possible without departing from the spirit of the invention. These embodiments or their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0075] 1…Visual inspection image processing system, 2…Inspection computer, 3…Input unit, 4…Output unit, 5…Communication unit, 6…Processing circuit, 7…Storage unit, 10…Scanning device, 11…Camera, 12…Drive mechanism, 13…Nuclear reactor, 14…Operation floor, 15…Crane, 16…Operation pole, 20…Shooting position calculation unit, 21…Shooting position estimation unit, 22…Shooting position correction unit, 23…Image generation unit, 24… Image correction unit, 25... Image selection unit, 26... Image synthesis unit, 27... Learning unit, 28... Defect inference unit, 29... Defect analysis unit, 30... Learning model, 40... Image for visual inspection, 41... Captured image, 41A... First captured image, 41B... Second captured image, 41C... Third captured image, 42... Object, 43... Feature points, 44... Object, 45... Shooting position, 46... Viewpoint position, 47... Mapping image, 48... Crack, W, W'... Width dimensions.

Claims

1. A camera that photographs the object to be inspected and acquires multiple images, A shooting position estimation unit that estimates the shooting position of the camera when the camera acquires the captured image from the information contained in the captured image, A shooting position correction unit corrects the shooting position so that the shooting position corresponding to the captured image is optimal when three or more of the aforementioned captured images are stitched together to generate an image for visual inspection, Equipped with, Image processing system for visual inspection.

2. A scanning device equipped with the aforementioned camera, which scans the camera along the object to be inspected by driving a drive mechanism, A shooting position calculation unit calculates the shooting position from drive information indicating the result of the drive mechanism being driven, Equipped with, The image processing system for visual inspection according to claim 1.

3. The device includes an image correction unit that corrects at least one of the lighting unevenness or white balance of the captured image. The image processing system for visual inspection according to claim 1 or claim 2.

4. The system includes an image selection unit that selects the most appropriate image for inspection from a plurality of images of the same subject, based on at least one of the following: the area of ​​the overlap region between the captured images, the movement of the camera, or the presence or absence of blur due to camera shake. The image processing system for visual inspection according to claim 1 or claim 2.

5. The aforementioned shooting position estimation unit, A process to extract multiple feature points of the subject captured in the aforementioned image and measure the amount of movement of the camera. A process that extracts multiple feature points of the subject captured in the aforementioned image and controls the number of feature points to be equal to or greater than a predetermined number. A process for determining the consistency between the direction of movement of the camera and the amount of movement of the camera. A process to calculate the overlap area between the captured images based on at least one of the shooting position, the shooting direction of the camera, and the angle of view of the camera, and to limit the measurement area of ​​the camera's movement amount. Perform at least one of the following processes: The image processing system for visual inspection according to claim 1 or claim 2.

6. The system includes an image synthesis unit that generates the visual inspection image by stitching together a plurality of the captured images based on the shooting position corrected by the shooting position correction unit. The image processing system for visual inspection according to claim 1 or claim 2.

7. A mapping image generation unit generates a mapping image that corrects at least one of the captured images to the state of being captured from an arbitrary viewpoint position, based on the information of the shooting position. A defect inference unit divides at least one of the aforementioned mapping images into a plurality of local regions, performs inference for each of the local regions using a machine learning model, and determines whether or not a defect exists in the object to be inspected based on the inference results of the plurality of local regions corresponding to the same shooting position, based on the shooting position corrected by the shooting position correction unit. Equipped with, The image processing system for visual inspection according to claim 1 or claim 2.

8. The defect reasoning unit, Using the aforementioned trained model, which has been trained using a method that learns only normal data, candidate local regions in which the defect is present are extracted. The defects are inferred from the local regions extracted as candidates using the machine learning model that has been trained using the method described above for learning normal and abnormal data. The image processing system for visual inspection according to claim 7.

9. The system includes a learning unit that retrains the learning model using images of the relevant local region if the inference result of the defect inference unit differs from the actual result. The image processing system for visual inspection according to claim 7.

10. The system includes a defect analysis unit that evaluates at least one of the occurrence distribution of the defects at each location, or the degree of defect progression compared to the past. The image processing system for visual inspection according to claim 7.

11. The defect analysis unit determines, based on at least one of the occurrence distribution or the degree of progression, at least one of the treatment method for the defect or the timing of the next inspection. The image processing system for visual inspection according to claim 10.

12. The shooting position correction unit determines whether the result of the shooting position correction is appropriate based on at least one of the landmarks of the object to be inspected, or the dimensions and shape of the CAD data. The image processing system for visual inspection according to claim 1 or claim 2.

13. A mapping image generation unit generates a mapping image that corrects the captured image to a state as if it were taken from an arbitrary viewpoint position, based on the information of the shooting position. A learning unit that performs machine learning of a learning model based on the aforementioned mapping image, A defect inference unit that uses the machine learning model described above to infer whether or not a defect exists in the object to be inspected, Equipped with, The image processing system for visual inspection according to claim 1 or claim 2.

14. The aforementioned shooting position correction unit is A first correction process for correcting the shooting position of two adjacent images that are joined together, After the first correction process, a second correction process is performed to correct the shooting position of the captured image again when generating the visual inspection image, This is what will be done. The image processing system for visual inspection according to claim 1 or claim 2.

15. The camera takes pictures of the object being inspected and acquires multiple images. The shooting position estimation unit estimates the shooting position of the camera when the camera acquired the captured image from the information contained in the captured image. When three or more of the aforementioned captured images are stitched together to generate an image for visual inspection, the shooting position correction unit corrects the shooting position so that the shooting position corresponding to the captured image is optimal. Image processing method for visual inspection.

Images