Visual Inspection System

Abstract

To surely perform an appearance inspection on a weld zone in a structure at high speed.SOLUTION: An appearance inspection system comprises: an imaging unit which is arranged in a structure; a position measurement unit which measures the position of the imaging unit; a coordinate information acquisition unit which acquires the position and the imaging direction in the structure of the imaging unit; a video acquisition unit which acquires an image from the imaging unit; a weld zone coordinate collation unit which creates a weld zone point cloud in the structure from the position of the imaging unit, the imaging direction and pixel coordinates of the image; and an image analysis unit which determines success / failure of the weld zone from the image from the imaging unit and weld zone inspection specification information corresponding to the weld zone point cloud.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present invention relates to a visual inspection system for welded parts of a structure. [Background technology]

[0002] Until now, inspection of welds in large welded structures, such as railway vehicle bodies, has been difficult to automate because there are more than 1,000 welds to inspect, and multiple check items, such as bead shape and dimensions and weld defects, have been carried out visually. For this reason, it has been considered essential to provide training to visual inspectors of welds so that they can perform accurate inspections. However, due to the recent social issues of a declining birthrate and aging population, as well as the difficulty in securing workers, there is a demand for an automatic inspection mechanism for welds in large welded structures that does not affect the skills of inspectors.

[0003] Under these circumstances, an automatic inspection mechanism for welds using a non-contact three-dimensional shape measuring device, such as a 3D laser scanner as exemplified by Patent Document 1, can address the above issues by enabling automatic inspection of welds, making it possible to inspect welds without relying on skill.

[0004] Meanwhile, a mechanism that acquires 3D shapes by attaching a 2D laser displacement meter to a robot arm and scanning the 2D laser displacement meter as the robot arm moves, as typified by Patent Document 2, also enables automatic inspection of welds, making it possible to inspect welds without relying on skill. Also, by acquiring the coordinate system of the tip of the robot arm, it is possible to determine the position within a structure of the weld for which shape information is being acquired. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-68580 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-253221 Summary of the Invention [Problem to be solved by the invention]

[0006] The method of acquiring shape information of a weld using a fixed 3D scanner as in Patent Document 1 has the following problems.

[0007] (1) Obtaining external shape information of welds using a 3D scanner for large welded structures exceeding 10 m requires high-density point cloud acquisition, which takes more than 12 hours to measure. Therefore, an inspection method with an even shorter lead time is required.

[0008] (2) Depending on the location of the 3D scanner, the welded part may be hidden by the laser irradiation, making it impossible to acquire its shape.

[0009] Furthermore, the method of acquiring the shape of the welded portion by scanning with a 2D laser displacement meter using a robot as in Patent Document 2 has the following problems.

[0010] (3) Many of the welds in railway vehicles are used to attach small parts, and many of these locations are inaccessible to robots for inspection, making it impossible to perform visual inspections of all welds.

[0011] An object of the present invention is to perform visual inspection of welds in a structure quickly and reliably. [Means for solving the problem]

[0012] The present invention is an appearance inspection system having an imaging unit arranged within a structure, a position measurement unit that measures the position of the imaging unit, a coordinate information acquisition unit that acquires the position of the imaging unit within the structure and the imaging direction, an image acquisition unit that acquires images from the imaging unit, a weld coordinate matching unit that creates a weld point cloud within the structure from the position of the imaging unit, the imaging direction, and pixel coordinates of the image, and an image analysis unit that determines the pass / fail of welds based on the images from the imaging unit and weld inspection specification information corresponding to the weld point cloud. [Effects of the Invention]

[0013] According to the present invention, it is possible to quickly and reliably perform visual inspection of welds in a structure. [Brief explanation of the drawings]

[0014] [Figure 1A] 1 is an overall view of a high-speed visual inspection system for structural welds. [Figure 1B] FIG. 2 shows a functional block diagram of a processing computer. [Figure 2] 10 is a flowchart showing the operation of a processing computer. [Figure 3] FIG. 10 is a diagram showing three-dimensional coordinates of a weld point cloud within a structure. [Figure 4] FIG. 10 shows pixel coordinates of welds in an image corresponding to the camera position and shooting direction. [Figure 5] FIG. 10 shows an image of a weld in a camera image. [Figure 6A] FIG. 1 is a conceptual diagram of the inside of a structure 7 where an actual appearance inspection is performed. [Figure 6B] FIG. 10 is a conceptual diagram of the inside of a structure 7 when data stored in a CAD system is used. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0016] 1A is an overall view of a high-speed visual inspection system for welds in a structure. The visual inspection system includes a weld inspection camera 1, position measurement scanners 2a, 2b, and 2c for acquiring three-dimensional position information and orientation of the weld inspection camera 1 in a structure 7, receivers 3a and 3b attached to the weld inspection camera 1, and a processing computer 4 for processing the position information and weld images.

[0017] FIG. 1A also shows a worker 5 handling an image analysis camera, a weld 6 to be inspected, and a structure 7 to which the weld belongs.

[0018] The processing computer 4 is equipped with a CPU (Central Processing Unit), a memory, a storage means (storage unit) such as a hard disk, and a network interface.

[0019] 1B shows a functional block diagram of processing computer 4. As shown in FIG. 1B, processing computer 4 receives signals from weld inspection camera 1 and signals from position measurement scanners 2a, 2b, and 2c. Processing computer 4 is composed of coordinate information acquisition unit 41 that acquires the three-dimensional coordinates of weld inspection camera 1, weld coordinate comparison unit 42 that has weld coordinate data 46, image acquisition unit 43 that acquires camera images, image analysis unit 44, and recording unit 45. Processing is automatically performed by an algorithm inside the computer, and the processed data can be saved in recording unit 45.

[0020] The multiple position measuring scanners 2a, 2b, and 2c installed on the structure 7 can, by emitting radio waves or infrared rays, grasp the distance (Dn) between each of them and the multiple receivers 3a and 3b attached to the weld inspection camera 1. The position measuring scanners 2a, 2b, and 2c and the receivers 3a and 3b are collectively referred to as the position measuring unit that measures the position of the camera.

[0021] The coordinate information acquisition unit 41 receives signals from the position measurement scanners 2a, 2b, 2c, and calculates and acquires three-dimensional position information 80 (Xc, Yc, Zc) of each of the receivers 3a, 3b in the structure 7. Then, by calculating a vector (Uc, Vc, Wc) from the obtained plurality of pieces of three-dimensional position information 80 (Xc, Yc, Zc), the shooting direction 81 of the weld inspection camera 1 in the structure 7 can be obtained.

[0022] Here, it has been explained that the position measuring scanners 2a, 2b, 2c calculate the distances (Dn) to the multiple receivers 3a, 3b, but the position measuring scanners 2a, 2b, 2c may also send a time signal indicating the time it takes for radio waves and the like reflected from the multiple receivers 3a, 3b to return to the processing computer 4, so that the coordinate information acquisition unit 41 can also calculate the distances (Dn) between the multiple receivers 3a, 3b and the position measuring scanners 2a, 2b, 2c, and calculate and acquire three-dimensional position information 80 (Xc, Yc, Zc) of each of the receivers 3a, 3b in the structure 7.

[0023] Weld coordinate comparison unit 42 is provided with weld coordinate data 46, and by comparing the position of weld inspection camera 1 in structure 7 obtained from the above-mentioned three-dimensional position information 80 and the camera's shooting direction 81 with point cloud 82 based on weld coordinate data 46, the information is converted into pixel coordinates 85 of image 84 displayed on screen 83 within weld inspection camera 1. Using the three-dimensional point cloud 82 and pixel coordinates 85 corresponding to each weld, weld point cloud 86 and weld inspection specification 87 linked to weld point cloud 86 are associated with each other.

[0024] Image acquisition unit 43 acquires image 84 from weld inspection camera 1. Then, image acquisition unit 43 stores a weld point cloud 86 corresponding to pixel coordinates 85 in image 84, a data group of weld inspection specifications 87, and training data for image analysis corresponding to weld inspection specifications 87.

[0025] The image analysis unit 44 uses image analysis techniques such as machine learning and deep learning to determine whether the weld in the image 84 in the weld inspection camera 1 satisfies the weld inspection specifications 87.

[0026] The recording unit 45 stores, as a record with one row in a table, the three-dimensional point cloud 82 corresponding to each weld in the weld coordinate matching unit 42 using the image 84 as the primary key, the weld point cloud 86 in pixel coordinates 85, the weld inspection specification 87, and the data on the pass / fail judgment of the weld based on the image analysis of the weld in the image analysis unit 44.

[0027] The processing computer 4 can be configured to display data related to the record 45. An example display may show a 3D point cloud 82 of the weld in the structure 7 in the image 84, a weld point cloud 86 in pixel coordinates 85, weld inspection specifications 87, pass / fail results, and inspection date and time. Other data may also be displayed as needed.

[0028] FIG. 2 is a flowchart showing the operation of the processing computer 4. The actual operation of the weld inspection and the data processing flow will be explained below. To explain FIG. 2, we will first explain FIG. 6A and FIG. 6B. FIG. 6A shows a conceptual diagram of the inside of the structure 7 where the actual appearance inspection is performed. FIG. 6B is a conceptual diagram of the inside of the structure 7 when data stored in CAD is used.

[0029] In Figures 6A and 6B, 80 represents the three-dimensional position information of the camera, 81 represents the camera's shooting direction, 82 represents the three-dimensional point cloud of the weld, 83 represents the screen, 84 represents the image on the screen, 85 represents the pixel coordinates of the weld in the screen coordinate system, 86 represents the point cloud of the weld in CAD, and 87 represents the weld inspection specifications.

[0030] The world coordinate system is a coordinate system of three-dimensional position coordinates within the structure 7, and the screen coordinate system is a coordinate system of two-dimensional pixel coordinates. The CAD coordinate system is a coordinate system of position coordinates stored in CAD. The camera coordinate system is a coordinate system in which the z direction is the shooting direction 81 and the x and y directions are the width and length directions of the shot image.

[0031] In step S1, the worker 5 points the weld inspection camera 1 toward the weld 6 of the structure 7, and the image acquisition unit 43 acquires the image 84.

[0032] In step S2, the coordinate information acquisition unit 41 acquires three-dimensional position information 80 (Xc, Yc, Zc) of the weld inspection camera 1 from the positional relationship between the multiple position measuring scanners 2a, 2b, 2c and the single receiver 3a or receiver 3b. The three-dimensional position information of the weld inspection camera 1 may be the three-dimensional position information of either the receiver 3a or 3b, or may be a calculated value obtained by averaging the position information.

[0033] In step S3, the coordinate information acquisition unit 41 acquires, as a vector, the shooting direction 81 of the weld inspection camera 1. Specifically, the vector (Uc, Vc, Wc) created based on three-dimensional receiver position information 80 (Xc, Yc, Zc) calculated from the positional relationships between the multiple position measuring scanners 2a, 2b, 2c and the multiple receivers 3a, 3b is set as the shooting direction 81 of the weld inspection camera 1.

[0034] In step S4, the weld coordinate matching unit 42 plots the values of the three-dimensional position information 80 (Xc, Yc, Zc) and shooting direction 81 (Uc, Vc, Wc) of the weld inspection camera 1 within the three-dimensional coordinate system of the three-dimensional point cloud 82 corresponding to each weld within the structure 7, and uses the camera's angle of view and the values of the width pixel size and vertical pixel size of the captured image to perform matrix calculations from the two-dimensional image pixel coordinates 85 to create a three-dimensional coordinate weld point cloud 86, and outputs it to the recording unit 45.

[0035] In step S5, the image acquisition unit 43 calls from the recording unit 45 a data group of the weld inspection specifications 87 associated with the weld point cloud 86 corresponding to the pixel coordinates 85, which was created in step S4.

[0036] In step S6, the image analysis unit 44 compares the training data for image analysis corresponding to the weld inspection specification 87 with the weld at pixel coordinate 85 in the image 84.

[0037] In step S7, the image analysis unit 44 compares the image analysis training data with the welded part and determines that the welded part has a certain degree of matching similarity, making it a pass, or a fail if it does not, and outputs the determination result and writes it to the recording unit 45.

[0038] An example of the welding inspection operation and processing flow for a structure consisting of the above seven steps has been shown.

[0039] The processor in the processing computer 4 can be configured to call up a program recorded in a recording unit or the like and execute each step from step S1 to step S7.

[0040] Following the flow from step S1 to step S7, 11 welds in the box structure were inspected. The 3D scanner took 22 minutes to measure, and an additional 3 minutes was required for point cloud matching between the 3D scan data obtained above and the 3D-CAD data, so the total time required to determine the pass / fail status of the welds was 25 minutes.

[0041] Figure 3 shows the 3D coordinates of the weld point cloud in the structure. Figure 4 shows the pixel coordinates of the weld in the image corresponding to the camera position and shooting direction. Figure 5 shows the image of the weld in the camera image.

[0042] In this embodiment, in step S4, the 3D point cloud of the weld shown in Figure 3 was converted into pixel coordinates within one field of view image of the camera shown in Figure 4, and the weld shown in the camera image was matched to the actual weld, and inspection was performed.

[0043] Next, using the YOLO program shown in the reference, the weld image in Figure 5 was compared with the training data corresponding to the weld specifications to determine whether the weld was pass or fail. As a result, when the field of view was set to 25, the same as the number of welds, the measurement time was 4 minutes, achieving a speed increase of more than 10 times. The reference here is "Redmon, Joseph & Divvala, Santosh & Girshick, Ross & Farhadi, Ali. (2016). You Only Look Once: Unified, Real-Time Object Detection. 779-788. 10.1109 / CVPR.2016.91."

[0044] According to this embodiment, it is possible to perform high-speed visual inspection of welds within a structure using image analysis with a camera, without using a stationary 3D scanner or a robot and 2D laser displacement meter.

[0045] In this example, the weld was inspected using the weld inspection camera 1. However, it is also effective to use a 3D shape measuring device, such as LIDAR, instead of the weld inspection camera 1 to determine the position of the weld to be inspected in the structure 7 and check whether the weld inspection specifications are met based on the 3D shape.

[0046] In this embodiment, the values of the three-dimensional position information 80 (Xc, Yc, Zc) and the imaging direction 81 (Uc, Vc, Wc) held by the worker 5 are obtained from the positional relationship between the position measurement scanners 2a, 2b, 2c and the receivers 3a, 3b. However, it is also effective to use a beacon or a stereo camera as the position measurement unit to obtain the three-dimensional position information and the imaging direction.

[0047] As described above, in the visual inspection work of the welded portions of the structure, the visual inspection of each welded portion of the structure can be carried out quickly and reliably. [Explanation of symbols]

[0048] 1...weld inspection camera, 2a, 2b, 2c...position measurement scanner, 3a, 3b...receiver, 4...processing computer, 5...worker, 6...weld to be inspected, 7...structure, 41...3D coordinate information acquisition device, 42...weld coordinate matching unit, 43...video acquisition unit, 44...image analysis unit, 45...recording unit, 46...weld coordinate data, 80...3D position information of camera, 81...camera shooting direction, 82...weld point cloud in camera coordinate system, 83...2D screen within camera, 84...image captured by camera, 85...pixel coordinates of weld on 2D screen within camera, 86...weld point cloud corresponding to weld on 2D screen within camera, 87...weld inspection specifications

Claims

1. an imaging unit disposed within the structure; a position measurement unit that measures the position of the imaging unit; a coordinate information acquisition unit that acquires the position of the imaging unit within the structure and the imaging direction; an image acquisition unit that acquires an image from the imaging unit; a weld coordinate collating unit that creates a weld point cloud within the structure based on the position of the photographing unit, the photographing direction, and pixel coordinates of the image; An appearance inspection system having an image analysis unit that determines whether a weld is acceptable or not based on the image from the photographing unit and weld inspection specification information corresponding to the weld point cloud.

2. 2. The visual inspection system according to claim 1, The position measurement unit An appearance inspection system comprising a plurality of position measurement scanners arranged on the structure and a receiver arranged in the photographing unit.

3. 2. The visual inspection system according to claim 1, The visual inspection system, wherein the position measurement unit is a beacon or a stereo camera.

4. 2. The visual inspection system according to claim 1, The photographing unit is a camera for inspecting welded parts.

5. 2. The visual inspection system according to claim 1, The photographing unit is a visual inspection system that is a 3D measuring device.

6. 2. The visual inspection system according to claim 1, The image analysis unit An appearance inspection system that compares the image from the photographing unit with training data for image analysis corresponding to the weld inspection specification information, and determines whether the image passes or fails based on similar trends.

7. A visual inspection method for visually inspecting a welded portion in a structure using an image from an imaging unit, comprising: creating a weld point cloud within the structure from the position of the photographing part, the photographing direction, and pixel coordinates of the image; An appearance inspection method for determining whether a weld is acceptable or not based on the image from the photographing unit and weld inspection specification information corresponding to the weld point cloud.

8. 8. The visual inspection method according to claim 7, The visual inspection method uses information from a position measurement unit disposed on the structure to determine the position and direction of the photographing unit.

Images