Method for inspecting workpiece, method for generating trained model, program, inspection device, trained model generation device, and trained model

A computer-based inspection system analyzes image data to assess brazed joint quality non-destructively, addressing the limitations of conventional methods by accurately identifying defects in the base material and fillet formation.

JP2024165371APending Publication Date: 2024-11-28MITSUBISHI ELECTRIC CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023081519
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional brazing inspection methods require destructive testing and manual visual inspection, which are time-consuming and costly, and existing non-destructive methods cannot effectively assess the state of the base material and fillet formation in soldered joints.

Method used

A non-destructive inspection method using a computer-based system that analyzes image data from brazed joints to determine the condition of the base material and fillet formation by comparing brightness levels and employing a trained model to infer brazing quality.

Benefits of technology

Enables efficient, non-destructive inspection of brazed joints to identify defects such as cracks and improper fillet formation without damaging the workpiece, improving inspection accuracy and reducing costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165371000001_ABST
    Figure 2024165371000001_ABST
Patent Text Reader

Abstract

To provide a technique that enables operators and other users to inspect the conditions of the base material and fillet without performing special processing on the base material and the like.SOLUTION: The present invention provides a method for a computer to inspect a workpiece, including the steps of: a step (S910) for accepting input of image data acquired from capturing a brazed workpiece; a step for comparing the inspection target area in the workpiece with a predetermined brightness, based on the image data; a step (S955) for determining that the workpiece is not correctly brazed, based on the extraction of an area darker than the predetermined brightness from the inspection target area in the workpiece; and a step for outputting the result of the determination.SELECTED DRAWING: Figure 9
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to techniques for inspecting metal-to-metal joints, and more particularly, to techniques for inspecting the appearance of brazed joints. [Background technology]

[0002] Brazing is known as a method for joining a first metal pipe and a second metal pipe. Brazing is a method in which a filler metal with a lower melting point than the base metal is melted and poured into the joint between the first and second pipes.

[0003] Conventionally, brazing inspection required an operator to cut and visually inspect an inspection sample manufactured in the same process as the product to be shipped. However, such destructive inspection requires the manufacture of a greater number of products than the number of products to be shipped. In addition, visual inspection by an operator is essential, which requires extra costs and time. Therefore, there is a demand for an inspection method or device that can automatically inspect the brazing condition non-destructively.

[0004] Regarding a non-destructive inspection technique, for example, Japanese Patent Laid-Open Publication No. 2008-111457 (Patent Document 1) discloses an inspection device for inspecting a joint structure characterized in that "a second pipe is inserted into a joint formed in a first pipe by expanding the diameter near the end so that the peripheral edge of the end abuts against the inner peripheral surface of the joint, and a molten joining material is poured into a gap formed between the inner peripheral surface of the joint and the outer peripheral surface of the second pipe to join the first pipe and the second pipe, and an inspection hole into which the molten joining material flows and is filled is formed in the joint by communicating the outer peripheral surface of the first pipe with the gap when a sufficient amount of the joining material has been poured to join the first pipe and the second pipe" and "the inspection device is characterized in that it is equipped with an imaging unit that images the inspection hole or the vicinity of the inspection hole to generate an image, and a discrimination unit that detects that the inspection hole has reached the filled state based on the image and discriminates that the joining is completed" (claims 1 and 3).

[0005] According to the technology disclosed in Patent Document 1, "when a sufficient amount of joining material has been poured in to join the first and second tubes, an inspection hole is provided at the joint, which connects the gap to the outer surface of the first tube into which the molten joining material flows to fill the gap. When the inspection hole is filled with brazing material, it can be determined that a sufficient amount of joining material has been poured in to join the first and second tubes, so that the brazing condition can be easily inspected in a non-destructive manner" (paragraph 0009). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2008-111457 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the inspection device disclosed in Patent Document 1 has a problem in that although it can check the state of the filler metal flowing into the joint, it cannot inspect the state of the base material itself, such as whether or not cracks have occurred in the base material itself. In addition, the inspection device has a problem in that it cannot inspect whether the "fillet" protruding from the joint (hereinafter referred to as "fillet") is properly formed.

[0008] The present disclosure has been made in view of the above-mentioned background. It is an object of one aspect to provide a technique for inspecting the state of a base material and a fillet. [Means for solving the problem]

[0009] According to one embodiment, a method for inspecting a workpiece by a computer is provided, the method including the steps of receiving input of image data obtained by photographing a brazed workpiece, comparing an inspection target area in the workpiece with a predetermined brightness based on the image data, determining that the workpiece is not normally brazed based on a region that is darker than the predetermined brightness being extracted from the inspection target area in the workpiece, and outputting a result of the determination.

[0010] According to one embodiment, the condition of the base material and the fillets can be inspected.

[0011] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 shows an overview of a configuration of an inspection system 10 according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a hardware configuration of a computer 200 implementing an image processing device 100 according to a first embodiment. [Diagram 3] 13 is a diagram showing details of an optical system for photographing a joint 126 of a workpiece 20. FIG. [Figure 4] 13 is a diagram showing the optical system when a fillet 127 protruding from a joint 126 is photographed. [Diagram 5] 5 is a diagram showing an example of a screen 500 displayed on a monitor 8 of a computer 200 that realizes an image processing device 100. FIG. [Figure 6] 13 is a diagram showing an optical system when a proper fillet is not formed at a joint 126 of a workpiece 20. FIG. [Figure 7] FIG. 13 is a diagram showing an example of an image displayed on monitor 8 when an appropriate fillet is not formed among images captured by camera 113. [Figure 8]1 is a diagram showing an image extracted by binarizing image data obtained by photographing; [Figure 9] 1 is a flowchart showing a part of the processing executed by a CPU 1 of a computer 200 that realizes an image processing device 100. [Figure 10] FIG. 1 is a block diagram showing the configuration of an inspection system 1000. [Figure 11] 13 is a diagram showing a state in which additional information is associated with an image obtained by photographing a workpiece 20 on which brazing has been performed. FIG. [Figure 12] FIG. 12 illustrates an example of a neural network 1200 used by the learning device 1010. [Figure 13] 10 is a flowchart showing a part of the processing executed by a CPU 1 functioning as a learning device 1010. [Figure 14] 11 is a flowchart showing a part of the processing executed by a CPU 1 of a computer 200 functioning as an inference device 1030. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0014] Embodiment 1 <Inspection system configuration> The configuration of the inspection system 10 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an outline of the configuration of the inspection system 10 according to the first embodiment. In one aspect, the inspection system 10 inspects the appearance of a brazed joint or a base material of the pipe. The appearance may include the brazed joint or the base material of the pipe.

[0015] The inspection system 10 includes an image processing device 100. The image processing device 100 includes an imaging control unit 101 and a discrimination unit 102. The inspection system 10 is used by a worker performing brazing (hereinafter also referred to as an "operator").

[0016] The photography control unit 101 is electrically connected to the camera 113, the bar lights 114 and 115, and the sensor 117. The camera 113 photographs the workpiece 20. The workpiece 20 is, for example, a tubular material in which a pipe 121 and a pipe 122 are brazed together at a joint 126. In a certain aspect, each of the workpieces 20 is sequentially transported on a predetermined transport path. The position of the workpiece 20 on the transport path may be positioned at a predetermined fixed location or may vary within the transport path.

[0017] According to this embodiment, the workpiece 20 is transported so that at least the joint 126 can be photographed by the camera 113. For example, in a certain aspect, when the joint 126 is formed on the entire outer circumference of the tubes 121 and 122, the workpiece 20 is rotated in the axial direction by an actuator (not shown) so that the outer circumference can be photographed by the camera 113. In another aspect, when the joint 126 is formed only on a part of the tube material, the posture of the tubes 121 and 122 is positioned so that the joint 126 is within the photographing range of the camera 113. In this way, the joint 126 is directly photographed by the camera 113, so that an abnormality such as a depression defect (hereinafter sometimes referred to as a "void") or a base material crack defect can also be detected. When the brazing joint is properly formed, a part of the joint 126 forms a fillet 127. The fillet 127 is a part of the brazing material protruding from the fitting part of the tubes 121 and 122.

[0018] The photography control unit 101 accepts input of image data acquired by photography by the camera 113. The photography control unit 101 sends signals to the bar lighting 114 and the bar lighting 115, respectively, and controls light emission by each of the bar lighting 114, 115. In one aspect, the bar lighting 114, 115 includes a light source capable of adjusting hue, saturation, and brightness. The photography control unit 101 controls the light source to control the hue, saturation, and brightness of the light emitted from the bar lighting 114, 115, respectively.

[0019] The photography control unit 101 receives input of a signal output by the sensor 117. The sensor 117 detects a workpiece as a subject of photography by the camera 113, and outputs a signal according to the detection result to the photography control unit 101. For example, when the sensor 117 detects that the workpiece 20 is transported to a predetermined photography range, it outputs a signal according to the detection result. The signal is input to the photography control unit 101. When the photography control unit 101 receives input of the signal from the sensor 117, it executes photography by the camera 113.

[0020] Specifically, when the photographing control unit 101 receives a signal from the sensor 117, it generates a photographing trigger. The photographing control unit 101 transmits an imaging trigger to at least the camera 113. When the bar lights 114 and 115 are constantly on, the imaging trigger is transmitted to at least the camera 113. When the bar lights 114 and 115 are not on, the imaging trigger is transmitted to each of the bar lights 114 and 115 and the camera 113. In this case, the photographing control unit 101 transmits an imaging trigger to the bar lights 114 and 115 to turn on the bar lights 114 and 115. After that, after a predetermined time has elapsed, the photographing control unit 101 transmits an imaging trigger to the camera 113.

[0021] When the camera 113 detects receipt of the photographing trigger, it photographs the workpiece 20. Image data generated by photographing is input to the photographing control unit 101. The photographing control unit 101 transmits the image data received from the camera 113 to the discrimination unit 102.

[0022] Although the above example illustrates that the imaging trigger is generated based on the output from the sensor 117, the method of generating the imaging trigger is not limited to this.

[0023] The discrimination unit 102 executes image processing on the image data. Using the results of the image processing, the discrimination unit 102 discriminates whether the workpiece 20 is a non-defective product that has been properly brazed. The discrimination result is output to a monitor or other output device. By checking the discrimination result, an operator can discriminate whether the brazing has been performed properly without destroying the workpiece 20.

[0024] <Hardware configuration of image processing device> A hardware configuration of image processing device 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a hardware configuration of a computer 200 that realizes image processing device 100 according to the first embodiment.

[0025] The main components of the computer 200 include a CPU (Central Processing Unit) 1 that executes a program, a mouse 2 and a keyboard 3 that receive instructions input by the user of the computer 200, a RAM 4 that volatilely stores data generated by the execution of the program by the CPU 1 or data input via the mouse 2 or keyboard 3, a hard disk 5 that nonvolatilely stores data, an optical disk drive 6, a communication interface (I / F) 7, and a monitor 8. The components are mutually connected by a data bus. A CD-ROM (Compact Disc - Read Only Memory) 9 or other optical disks are loaded into the optical disk drive 6.

[0026] The processing in the computer 200 is realized by each piece of hardware and software executed by the CPU 1. Such software may be pre-stored in the hard disk 5. The software may also be stored in a CD-ROM 9 or other recording medium and distributed as a computer program. Alternatively, the software may be provided as a downloadable application program by an information provider connected to the Internet. Such software is read from the recording medium by the optical disk drive 6 or other reading device, or downloaded via the communication interface 7, and then temporarily stored in the hard disk 5. The software is read from the hard disk 5 by the CPU 1 and stored in the RAM 4 in the form of an executable program. The CPU 1 executes the program.

[0027] Each component constituting the computer 200 shown in Fig. 2 is a general one. Therefore, one of the essential parts of the technical idea according to the present disclosure can be said to be software stored in the RAM 4, the hard disk 5, the CD-ROM 9 or other recording medium, or software that can be downloaded via a network. The recording medium may include a non-transitory computer-readable data recording medium. The operation of each piece of hardware of the computer 200 is well known, so a detailed description will not be repeated.

[0028] The recording medium is not limited to a CD-ROM, a FD (Flexible Disk), or a hard disk, but may also be a medium that carries a program in a fixed manner, such as an SSD (Solid State Drive), a magnetic tape, an optical disk (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), an IC (Integrated Circuit) card (including a memory card), an optical card, a mask ROM, an EPROM (Electronically Programmable Read-Only Memory), an EEPROM (Electronically Erasable Programmable Read-Only Memory), or a semiconductor memory such as a flash ROM.

[0029] The program referred to here includes not only a program that can be directly executed by a CPU, but also a source program format program, a compressed program, an encrypted program, and the like.

[0030] Moreover, instead of the CPU 1, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these may be used.

[0031] <Optical system details> The optical system according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing the details of the optical system for photographing the bonded portion 126 of the workpiece 20.

[0032] In one aspect, the camera 113 and the joint 126 are positioned so that the angle between the axis of the workpiece 20 and the optical axis of the camera 113 (i.e., the angle between the outer surface of the workpiece 20 and the optical axis) is θ degrees.

[0033] The bar light 114 is disposed at an angle such that the optical axis of the camera 113 and the optical axis of the bar light 114 form an angle of φ°.

[0034] The bar illuminator 115 is disposed so that the angle between the joint 126 and the optical axis of the bar illuminator 115 (that is, the angle between the outer circumferential surface of the workpiece 20 and the optical axis) is 90 degrees.

[0035] By arranging the camera 113 and the bar lights 114, 115 in this manner, if a proper fillet is formed, the fillet portion is photographed brightly. On the other hand, if a proper fillet is not formed, the fillet portion is photographed darkly. This brightness makes it possible to inspect whether or not the joint 126 has been properly brazed. Details of such inspection will be described later.

[0036] Note that the above angles (θ°, φ°, 90 degrees) are merely examples and are not limited to the above examples. Each angle can be changed depending on the size or shape of the target workpiece 20. In addition, the number of lighting bars to be installed can also be changed depending on the size and shape of the target workpiece 20.

[0037] <When a fillet is formed> A case where an appropriate fillet 127 is formed at joint 126 of workpiece 20 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an optical system when fillet 127 protruding from joint 126 is photographed.

[0038] In general, shiny surfaces such as metals have strong specular reflection characteristics. Specular reflection refers to the reflection of light from a light source at the same angle as the light that was emitted from the light source. In contrast, diffuse light refers to the light that is reflected in all directions by diffusing the light emitted from the light source.

[0039] According to this embodiment, light 410 emitted from bar lighting 114 and light 420 emitted from bar lighting 115 reach surface 128 of fillet 127 formed of wax. When light 410, 420 is specularly reflected, reflected light 411, 421 reaches camera 113. Therefore, when fillet 127 is appropriately formed as described above, the joint 126, particularly fillet 127, is captured by camera 113 as a bright image.

[0040] <Image> 5 to 8, a description will be given of images acquired by photographing with camera 113. Fig. 5 is a diagram showing an example of a screen 500 displayed on monitor 8 of computer 200 which realizes image processing device 100.

[0041] Screen 500 displays an image of workpiece 20 on which a fillet has been appropriately formed. The image includes image 501 of tube 121, image 502 of tube 122, and image 561 corresponding to fillet 127. When appropriate fillet 127 is formed at joint 126 of tube 121 and tube 122, an image of the vicinity of the joint is captured brightly as in image 561. Therefore, discrimination unit 102 can easily discriminate whether or not fillet 127 that should be formed on workpiece 20 has been appropriately formed.

[0042] 5, about one-third (120 degrees) of the circumference of the workpiece 20 is photographed due to the distance between the camera 113 and the workpiece 20. Therefore, when brazing is performed on the entire circumference, it is possible to photograph the entire circumference by rotating the workpiece 20 around its axis according to the photographing range.

[0043] If the fillet is not formed A case where a fillet is not formed will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an optical system when a proper fillet is not formed at joint 126 of workpiece 20.

[0044] In another aspect, if joint 126 of workpiece 20 is not normally brazed, a fillet is not formed at joint 126. Therefore, the lights irradiated from bar lighting 114 and bar lighting 115 to joint 126 are not specularly reflected by joint 126, and the reflected lights do not reach camera 113. As a result, the portion where a fillet should be formed is captured as a dark image.

[0045] FIG. 7 is a diagram showing an example of an image displayed on monitor 8 when an appropriate fillet is not formed among images captured by camera 113 (FIG. 6).

[0046] 7A shows an image 700 obtained by photographing a defective product according to a certain aspect. More specifically, image 700 represents a case where a fillet has not been formed at joint 126. Image 700 includes image 701 of tube 121, image 702 of tube 122, and image 706 of joint 126.

[0047] If brazing is not performed properly on joint 126 of workpiece 20, a fillet will not be formed at joint 126 between tube 121 and tube 122. Therefore, image 706 of joint 126 appears dark.

[0048] 7B shows an image 710 obtained by photographing a defective product from another perspective. More specifically, image 710 shows a case where a fillet has been formed but a void (defective depression) exists. Image 710 includes image 701 of tube 121, image 702 of tube 122, and image 716 of joint 126. Image 716 includes image 708 corresponding to the void.

[0049] In another aspect, even if a fillet is formed in the joint 126, a void may be formed in a part of the brazed region. In this case, the light irradiated from each of the bar lights 114 and 115 is not reflected specularly in the region where the void occurs. Therefore, the reflected light does not reach the camera 113. As a result, the region where the void is formed is photographed as an image darker than the surrounding regions, as in image 708 in image 710. Therefore, the discrimination unit 102 can determine whether the brazing has been performed normally on the joint 126 by detecting an area darker than an area having a specified brightness from the image 710.

[0050] In the image processing device 100 according to the present embodiment, the discrimination unit 102 detects a dark area of ​​the joint 126 by image processing of the image (see Figs. 5 and 7) obtained by the imaging control unit 101. The discrimination unit 102 compares the brightness of the detected area with a threshold value set as the brightness for a non-defective product, thereby determining whether the brazed workpiece is a non-defective product or a defective product.

[0051] As described above, according to this embodiment, the brazed joint 126 is photographed, and the image obtained by photographing is subjected to image processing. Using the image data after the image processing, the discrimination unit 102 of the image processing device 100 can inspect whether the brazing has been performed normally or not, without the need to drill an inspection hole in the base material.

[0052] As shown in FIG. 5, the field of view of the image obtained by the camera 113 cannot cover the entire tubular workpiece 20, and may be about 120 degrees around the workpiece 20. Therefore, when performing a full-circumference inspection of the tubular workpiece 20, the remaining 240 degrees are also captured by changing the relative position between the optical system and the workpiece 20. For example, three optical systems having the same configuration may be arranged at intervals of 120 degrees. In this case, each optical system captures the workpiece 20, eliminating blind spots that may occur when a single optical system captures the workpiece 20.

[0053] In another aspect, the workpiece 20 transported to the photographing area may be rotated about an axis. For example, after one optical system photographs the workpiece 20, the workpiece 20 is rotated about 120 degrees. Then, the one optical system photographs the workpiece 20 again. Then, the workpiece 20 is further rotated about 120 degrees, and the one optical system photographs the workpiece 20. The discrimination unit 102 can determine whether the brazing has been performed appropriately by photographing the entire circumference of the workpiece 20.

[0054] In the above example, the shooting range is one third of the entire circumference of the workpiece 20 (rotation angle is 120 degrees), but the rotation angle is not limited to 120 degrees. Depending on the distance between the camera 113 and the workpiece 20, more shooting may be performed at a greater number of rotations (smaller rotation angle).

[0055] Image after binarization 8 is a diagram showing an image extracted by binarizing the image data obtained by photographing. The image processing device 100 according to the present embodiment performs binarization processing on the acquired image data in order to determine whether the workpiece 20 is a non-defective product. The discrimination unit 102 uses the image after the binarization processing to determine whether the workpiece 20 is a non-defective product.

[0056] In one aspect, if the brazing at the joint 126 is not performed properly, an image 806 corresponding to the joint 126 is extracted as a dark region, as shown in Fig. 8(A). In another aspect, if the brazing is performed properly but a depression (void) occurs (see Fig. 7(B)), an image 808 corresponding to the void is extracted as a dark region. In either case, the discrimination unit 102 discriminates that the workpiece 20 is defective.

[0057] <Control Structure> A control structure of image processing device 100 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing a part of the process executed by CPU 1 of computer 200 which realizes image processing device 100.

[0058] In step S905, the CPU 1 initializes each variable to be used in the subsequent processes.

[0059] In step S910, the CPU 1 acquires image data from the imaging control unit 101 and captures the image. Through this process, an image to be used for determining the brazing joint of the workpiece 20 is acquired.

[0060] In step S915, the CPU 1 executes a positioning process. The positioning process includes steps S920, S925, and S930. In general, it is difficult to arrange the workpieces 20 on the conveyor so that the reproducibility after shooting is high. This is because it is expected that if the workpieces 20 are arranged in such a way, the productivity will decrease due to the reduction in the number of workpieces 20 to be transported, or the cost of the conveyor will increase in order to realize such an arrangement. Therefore, in an image obtained by photographing the workpieces 20 transported without considering the reproducibility, the joints 126 are not necessarily located at a fixed location in the image, and a "position shift" of the joints 126 may occur. Therefore, it is desirable to correct each image obtained by photographing so that the joints 126 to be inspected are captured at the fixed location.

[0061] When the workpiece 20 is transported without being fixed, in order for the CPU 1 to use the image to determine whether it is a good product or a defective product, a process (positioning process) is required to detect the position on the image of the joint 126 to be inspected and correct the position to a position predetermined for photographing according to the detection result. In another aspect, when the workpiece 20 is photographed while being fixed by a jig or the like, the workpiece 20 appears at a fixed position in the image obtained by photographing, and such a positioning process is not required.

[0062] Therefore, in this embodiment, the positioning process of the inspection object is performed by focusing on the color components of the tube and extracting a specific color. In general, color extraction using HSV, which expresses color using three elements of hue, saturation, and value, is known to be more resistant to brightness fluctuations and easier to set color extraction parameters than color extraction using RGB. For example, if the base material of the tube to be brazed is a copper tube, the copper tube part in the image will show a red hue, so color extraction using HSV is effective.

[0063] In step S920, the CPU 1 executes HSV conversion as a pre-processing for color extraction.

[0064] In step S925, the CPU 1 executes a color extraction process to extract only the hue of a specific color (for example, red in the case of a copper pipe) in the image, thereby extracting only the region of the image of the workpiece 20 to be inspected.

[0065] In step S930, the CPU 1 calculates the center of gravity position. More specifically, the CPU 1 determines the inspection range for the area extracted in step S925. As an example, the CPU 1 obtains the center of gravity position of the extracted area, and cuts out only the inspection range from the captured image based on the coordinate value of the center of gravity position. When the inspection range is set in this manner, factors such as the inability to obtain correct inspection results due to the influence of parts other than the workpiece 20 on the image can be suppressed.

[0066] In step S935, the CPU 1 executes a known binarization process. For example, the CPU 1 generates a binarized image from the image for determination by using a preset binarization threshold value.

[0067] In step S940, the CPU 1 executes a labeling process on the binarized image. In this embodiment, the labeling process refers to attaching a label to each connected pixel. The labeling process makes it possible to classify a plurality of regions on an image into groups. In this embodiment, the regions that are divided into groups by the labeling process are dark regions. Note that the labeling process is a well-known technique, and therefore detailed description will not be repeated.

[0068] In step S945, CPU 1 determines whether the area value of the regions grouped in the labeling process is equal to or less than a preset threshold value. If the area value is equal to or less than the threshold value (YES in step S945), CPU 1 switches control to step S950. If not (NO in step S945), CPU 1 switches control to step S955.

[0069] For example, in a certain aspect, when an appropriate fillet is formed at the joint 126 of the workpiece 20 (YES in step S945), the joint 126 is photographed as a bright area. In this case, the CPU 1 does not detect a dark area, or the area value of the detected area is close to 0.

[0070] On the other hand, in another aspect, when an appropriate fillet is not formed on the workpiece 20 (NO in step S945), the joint 126 is photographed as a dark area. In this case, the CPU 1 obtains a binarized image in which only the joint 126 is detected as an image 806 of a dark area, as in image 800 (FIG. 8(A)). Alternatively, although brazing has been performed on the workpiece 20, voids have occurred, and a binarized image in which only an image 808 corresponding to the void is detected as a dark area, as in image 810 (FIG. 8(B)), is obtained. In this way, the CPU 1 calculates the area value of the dark area, and can determine whether the workpiece 20 that has been brazed is a non-defective product using a preset threshold value as a discrimination criterion.

[0071] In step S950, the CPU 1 judges that the work 20 is a non-defective product (OK judgment). The CPU 1 outputs the result of the judgment. The output destination is the monitor 8, another information display device connected to the communication IF 7, etc.

[0072] In step S955, the CPU 1 judges that the workpiece 20 is defective (NG judgment).The CPU 1 outputs the result of the judgment.

[0073] In this manner, the image processing device 100 according to this embodiment can extract dark areas from the image for judgment, thereby determining whether the workpiece 20 that is brazed and joined is good or bad, without destroying the workpiece.

[0074] The image processing device 100 according to the present embodiment calculates the area of ​​each region formed by a plurality of pixels to which the same label is assigned, and executes a discrimination process based on the area value. The discrimination mode is not limited to the above mode. For example, an image processing device according to another aspect may determine whether the workpiece 20 is a good product or a defective product using different feature amounts such as the perimeter, the length of the main axis, and the length of the secondary axis according to the inspection standard.

[0075] Embodiment 2 A second embodiment will be described with reference to Figures 10 to 14. An inspection system 1000 according to the second embodiment differs from the inspection system 10 according to the above-described embodiment in that it has a learning function and an inference function.

[0076] Fig. 10 is a block diagram showing the configuration of an inspection system 1000. As shown in Fig. 10, the inspection system 1000 includes a learning device 1010, a learned model storage unit 1020, an inference device 1030, and a display unit 1040 in addition to the configuration of the inspection system 10. The learning device 1010 includes a data acquisition unit 1011 and a model generation unit 1012. The inference device 1030 includes a data acquisition unit 1031 and an inference unit 1032. The inference device 1030 is connected to the display unit 1040.

[0077] In the learning device 1010, the data acquisition unit 1011 acquires an image that the operator judges to be a good product (hereinafter referred to as a "good product image") and an image that the operator judges to be a defective product (hereinafter referred to as a "defective product image") as post-brazing images (first input). The post-brazing images are also called learning images, and images acquired by the photography control unit 101 of the image processing device 100 are used. Furthermore, the data acquisition unit 1011 acquires correct answer information (second input) regarding the performance of the brazing. The correct answer information is input to the learning device 1010 by, for example, a user of the inspection system 1000 (a brazing joint worker or other operator). The first input and the second input are provided to the model generation unit 1012.

[0078] The model generation unit 1012 learns the brazing performance and the image at that time based on "learning data" consisting of a combination of a learning image (first input) and correct answer information (second input). The model generation unit 1012 generates a trained model that infers the state of the workpiece 20 when brazing is performed appropriately from the post-brazing image and the correct answer information.

[0079] In this embodiment, the learning data is data in which the post-brazing image and the workmanship information are associated with each other. In this data association, information on the brazing state, such as "good product", may be associated with a normal good product image. In other aspects, additional information may be associated with an item classified as a good product.

[0080] Therefore, the association of additional information will be described with reference to Fig. 11. Fig. 11 is a diagram showing a state in which additional information is associated with an image obtained by photographing a workpiece 20 on which brazing has been performed.

[0081] 11(A) shows an image 1100 of the workpiece 20, which is classified as a non-defective product but in which an excessive amount of brazing material has been supplied to join the tubes 121 and 122. The image 1100 may be displayed on a display device such as the monitor 8 or the display unit 1040. The image 1100 includes an image 1110. The image 1110 corresponds to the brazing material that has been supplied to the workpiece 20.

[0082] As is clear from image 1110, the brazing filler metal has been excessively added to workpiece 20 (more than the amount necessary to maintain the strength of joint 126). Thus, while the image of workpiece 20 is classified as a "good product image," information such as "excessive brazing filler metal added" is associated with the good product image because an excessive amount of brazing filler metal has been added. Although excessive brazing filler metal is classified as a good product from the viewpoint of "brazing joint," it may be a state that should be avoided in terms of the stability of brazing quality.

[0083] 11(B) shows an image 1120 (defective product image) of the workpiece 20 that is classified as a defective product because the fillet was not properly formed. The image 1120 includes an image 1130. The image 1130 corresponds to a crack (hereinafter also referred to as a "base material crack") that has occurred in the base material (pipe 121).

[0084] The performance information associated with the image 1120 and other defective product images is, for example, “base material crack.” If a crack or the like occurs in the base material (the tube 121 or the tube 122) of the workpiece 20, even if the fillet is properly formed, the workpiece should be classified as defective in terms of brazing quality.

[0085] Therefore, it is desirable to feed back such various pieces of brazing performance information to the worker (the person performing the brazing joint). This feedback can notify the worker of the occurrence of such a condition in a timely manner. The worker can adjust the operating conditions of the brazing joint (for example, the positioning of the brazing mechanism (not shown) and the joint, the position at which the base material is gripped, etc.), which can result in stabilization of the brazing quality.

[0086] In the present embodiment, the case where the brazing performance information is fed back to the worker is exemplified, but the recipient of the feedback is not limited to the worker. For example, the brazing performance information may be fed back to an automatic brazing device or the like. The automatic brazing device may autonomously adjust the brazing operating conditions according to the content of the feedback.

[0087] In the present embodiment, the excessive supply of brazing material and the base metal cracking are exemplified as examples of the brazing performance information, but the performance information is not limited to these. The brazing performance information learned as the learning data may include various abnormalities that affect the quality of the brazing joint in addition to the excessive supply of brazing material and the base metal cracking.

[0088] The learning device 1010 is used to learn the performance state of the brazing after the brazing joint process is performed. The learning device 1010 may be provided outside the inspection system 1000. For example, in another aspect, the learning device 1010 may be connected to the image processing device 100 via a network such as the Internet or an intranet. In yet another aspect, the learning device 1010 may be built into the image processing device 100. Furthermore, the learning device 1010 may exist on a cloud server.

[0089] The model generation unit 1012 may use, as a learning algorithm, a known algorithm such as supervised learning, unsupervised learning, or reinforcement learning.

[0090] A case where a neural network is used as a learning algorithm will now be described with reference to Fig. 12. Fig. 12 is a diagram illustrating a neural network 1200 used by the learning device 1010. The neural network 1200 is composed of input layers X1, X2, and X3 consisting of a plurality of neurons, intermediate layers (hidden layers) Y2 and Y2 consisting of a plurality of neurons, and output layers Z1, Z2, and Z3 consisting of a plurality of neurons. The intermediate layer may be one layer or two or more layers.

[0091] In one aspect, the model generation unit 1012 learns brazing performance information by so-called supervised learning according to a neural network model. Supervised learning refers to a method in which a set of data, that is, an input and a result (label), is provided to the learning device 1010, and the model generation unit 1012 learns the features inherent in the set of data and infers the result from the input.

[0092] In the neural network 1200 illustrated in FIG. 12, when multiple inputs are input to the input layers X1, X2, and X3, the input values ​​are multiplied by weights W1 (w11, w12, w13, w14, w15, and w16). The results are input to the intermediate layers Y1 and Y2. The intermediate layers perform calculations on the input values. The results of the calculations are multiplied by weights W2 (w21, w22, w23, w24, w25, and w26). The results are input to the output layers Z1, Z2, and Z3. The output layers Z1, Z2, and Z3 perform calculations on the input values ​​and output the results of the calculations. The output results vary depending on the values ​​of weights W1 and W2.

[0093] In this embodiment, the neural network 1200 learns information about the brazing state by so-called supervised learning in accordance with learning data created based on a combination of the “post-brazing image” and the “brazing performance information” acquired by the data acquisition unit 1011.

[0094] That is, the neural network 1200 learns by inputting post-brazing images to input layers X1, X2, and X3 and adjusting weights W1 and W2 so that the results output from output layers Z1, Z2, and Z3 approach the brazing performance information.

[0095] The model generation unit 1012 generates a trained model by executing the above-mentioned learning, and outputs the trained model to the trained model storage unit 1020. The trained model storage unit 1020 stores the trained model output from the model generation unit 1012. The trained model storage unit 1020 is realized by a hard disk 5, a RAM 4, or other storage device.

[0096] Next, the learning process by the learning device 1010 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing a part of the process executed by the CPU 1 functioning as the learning device 1010.

[0097] In step S1310, the CPU 1, as the data acquisition unit 1011, acquires the post-brazing image and the brazing performance information from the image processing device 100. The timing of acquisition is not particularly limited. It is sufficient that the CPU 1 can acquire at least the post-brazing image and the brazing performance information in a state in which they are associated with each other. In one aspect, the CPU 1 can acquire the post-brazing image and the brazing performance information simultaneously. In another aspect, the CPU 1 can acquire the post-brazing image and the brazing performance information at different times.

[0098] In step S1320, the CPU 1, as the model generation unit 1012, learns information about the brazing state by so-called supervised learning according to learning data created based on a combination of the post-brazing image and brazing performance information, and generates a learned model.

[0099] In step S1330, the CPU 1, functioning as the model generation unit 1012, stores the generated trained model in the trained model storage unit 1020. In this manner, the learning device 1010 generates a trained model.

[0100] In the present embodiment, the model generating unit 1012 uses supervised learning as a learning algorithm, but the learning algorithm used is not limited to supervised learning. The model generating unit 1012 may use other learning algorithms such as reinforcement learning, unsupervised learning, or semi-supervised learning.

[0101] In another aspect, the model generating unit 1012 may learn information about the brazing state according to learning data created for a plurality of inspection systems. For example, the model generating unit 1012 may acquire learning data from a plurality of inspection systems used in the same area, or may learn information about the brazing state by using learning data collected from a plurality of inspection systems operating independently in different areas.

[0102] In addition, the inspection system that collects the learning data may be added to the collection target during the process, or may be removed from the collection target.

[0103] Furthermore, the learning device 1010 that has learned information about the brazing state for a certain inspection system may be applied to another inspection system, and the brazing state information may be re-learned for the other inspection system to update the learned model.

[0104] In another aspect, the model generating unit 1012 may use deep learning, which learns to extract features themselves, as a learning algorithm. Furthermore, the model generating unit 1012 may execute machine learning according to other known methods, such as genetic programming, functional logic programming, and support vector machines.

[0105] The inference device 1030 in this embodiment will be described with reference to Fig. 10 again. In the inference device 1030, a data acquisition unit 1031 takes in image data acquired by an imaging control unit 101 that drives a camera 113, and acquires an image for inference.

[0106] The inference unit 1032 accesses the trained model stored in the trained model storage unit 1020, inputs the acquired image for inference to the trained model, and outputs information on the brazing state. That is, the after-brazing image acquired by the data acquisition unit 1031 is input to the trained model, and the inference device 1030 outputs information on the brazing state inferred from the after-brazing image.

[0107] In this embodiment, the inference device 1030 outputs information on the brazing state using a trained model trained by the model generation unit 1012. However, the trained model used may be a trained model created by another learning device, and the inference device 1030 may output information on the brazing state based on this trained model.

[0108] <Inference processing> 14, the inference processing by inference device 1030 will be described. FIG. 14 is a flowchart showing a part of the processing executed by CPU 1 of computer 200 functioning as inference device 1030.

[0109] In step S1410, the CPU 1 initializes each variable to be used in the subsequent processes.

[0110] In step S1415, the CPU 1, functioning as the data acquisition unit 1031, acquires image data and acquires an image for inference from the image data.

[0111] In step S1420, the CPU 1, functioning as the inference unit 1032, inputs the image for inference to the trained model stored in the trained model storage unit 1020.

[0112] In step S1425, CPU 1 determines whether the inference image is OK. This determination may be made based on the results of steps S950 and S955, for example. In other aspects, this determination may be made based on brazing information output from the trained model. If CPU 1 determines that the inference image is OK (YES in step S1425), it switches control to step S1430. If not, that is, if the inference image is NG (NO in step S1425), CPU 1 switches control to step S1450.

[0113] In step S1430, CPU 1 determines whether or not there is an abnormality in the inference image. This determination is made, for example, based on brazing information output from the trained model. An abnormality when the workpiece 20 is a non-defective product is, for example, "excessive brazing material input." If CPU 1 determines that there is an abnormality (YES in step S1430), it switches control to step S1435. If not (NO in step S1430), CPU 1 switches control to step S1440.

[0114] In step S1435, the CPU 1 outputs a result indicating that the workpiece 20 is a non-defective product and an alert. The alert may be, for example, dripping.

[0115] In step S1440, the CPU 1 outputs a result indicating that the workpiece 20 is a non-defective product.

[0116] In step S1450, the CPU 1 identifies the cause of the inference image being determined to be NG. This identification is performed based on the brazing information output from the trained model. The cause includes, for example, anomalies that may occur in brazing joints, such as voids and base material cracks. The identification is performed based on the relationship between the size of a dark area included in the image and a threshold value. When the CPU 1 identifies that the cause is a void, the CPU 1 switches control to step S1455. On the other hand, when the CPU 1 identifies that the cause is a base material crack, the CPU 1 switches control to step S1460.

[0117] In step S1455, the CPU 1 determines that the inference image is NG because it contains a void.

[0118] In step S1460, the CPU 1 determines that the inference image is NG because it includes a base material crack.

[0119] In step S1470, the CPU 1 outputs the determination result.

[0120] As described above, according to this embodiment, the inspection system 1000 has a learning function and an inference function, so that the efficiency of judgment in the inspection of the workpiece 20 can be improved.

[0121] According to the embodiment described above in detail, the condition of the base material and the fillet can be inspected with a minimum necessary configuration without requiring special processing of the base material, etc. Also, an inspection device that contributes to quality improvement can be provided.

[0122] Some of the technical features disclosed in this specification can be summarized, for example, as follows.

[0123] [Example 1] According to one aspect, a method is provided for a computer to inspect a workpiece 20. This method includes a step in which a CPU 1 receives input of image data obtained by photographing the brazed workpiece 20, a step in which the CPU 1 compares an inspection target area in the workpiece 20 with a predetermined brightness based on the image data, a step in which the CPU 1 determines that the workpiece 20 is not normally brazed based on the extraction of an area darker than the predetermined brightness from the inspection target area in the workpiece 20, and a step of outputting a result of the determination.

[0124] [Example 2] In addition to the configuration shown in Example 1, a method according to a certain aspect further includes a step of causing the CPU 1 to irradiate light from a light source onto the workpiece 20, and a step of causing the CPU 1 to cause an imaging device to capture an image of the workpiece 20 irradiated with the light. The step of receiving an input includes a step of receiving image data from the imaging device.

[0125] [Example 3] In a method according to a certain aspect, in addition to the configuration shown in Example 2, the axis connecting the workpiece 20 and the imaging device is shifted from the axis connecting the light source and the workpiece 20.

[0126] [Example 4] A method according to a certain aspect includes, in addition to the configuration shown in any one of Examples 1 to 3, a step in which the CPU 1 extracts an area corresponding to the workpiece 20 from the input image data based on a color component designated in advance. The comparing step includes a step of comparing the inspection target area and a predetermined brightness in the extracted area.

[0127] [Example 5] According to another aspect, a method for generating a trained model for inferring a state of a brazed joint is provided. This method includes a step of receiving, by a CPU 1, an input of image data obtained by photographing a brazed workpiece 20, a step of receiving, by the CPU 1, an input of training data including performance information representing a state of the brazed joint of the workpiece 20 for the image data and the image data obtained by photographing the brazed workpiece 20, and a step of generating, by the CPU 1, a trained model for inferring a state of the brazed joint of the workpiece 20 from the image data of the workpiece 20 on which brazing has been performed, using the training data.

[0128] [Example 6] In a method according to one aspect, in addition to the configuration shown in Example 5, the performance information includes information indicating that a fillet has been formed at the joint to meet a predetermined criterion, and information indicating that a fillet has not been formed at the joint to meet a predetermined criterion.

[0129] [Example 7] In a method according to one aspect, in addition to the configuration shown in Example 6, the performance information further includes information indicating that the fillet has been formed at the joint to meet predetermined criteria, but that the base material of the workpiece 20 has been damaged.

[0130] [Example 8] According to another aspect, a program (computer software) is provided, which causes a computer to implement the method according to any one of Examples 1 to 7.

[0131] [Example 9] According to another aspect, an inspection device is provided, the inspection device including a memory storing a program for causing a computer to realize the method according to any one of Examples 1 to 4, and a processor for executing the program.

[0132] [Example 10] According to another aspect, a trained model generation device is provided. The trained model generation device is realized by a computer 200. The trained model generation device includes a memory that stores a program that causes a computer to realize the method described in any one of Examples 5 to 7, and a processor that executes the program.

[0133] [Example 11] According to another aspect, a trained model is provided that causes a computer 200 to judge the quality of a brazing joint. The trained model may be stored in a storage device of the computer. The trained model may be stored in a CD-ROM 9 or other recording medium and distributed as a computer program. Alternatively, the trained model may be provided as a trained model that can be downloaded by an information provider connected to the Internet. Furthermore, the trained model may be stored in a storage device of a server computer that provides a cloud service as a so-called SaaS (Software as a service). The trained model receives an input of inference image data obtained by photographing the brazed workpiece 20. The trained model infers the state of the brazing joint of the workpiece 20 from the inference image data of the workpiece 20 on which the brazing joint has been performed, based on the training image data obtained by photographing the brazed workpiece 20 and the performance information representing the state of the brazing joint of the workpiece 20.

[0134] [Example 12] In another aspect, in addition to the configuration shown in Example 11, the trained model includes performance information indicating that a fillet is formed at the joint to meet a predetermined criterion and information indicating that a fillet is not formed at the joint to meet a predetermined criterion.

[0135] [Example 13] In accordance with another aspect, the trained model includes, in addition to the configuration shown in Example 12, the performance information further includes information indicating that a fillet has been formed at the joint to meet predetermined criteria, but that the base material of the workpiece 20 has been damaged.

[0136] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0137] 1 CPU, 2 mouse, 3 keyboard, 4 RAM, 5 hard disk, 6 optical disk drive, 7 communication interface, 8 monitor, 9 CD-ROM, 10,1000 inspection system, 20 workpiece, 100 image processing device, 101 photography control unit, 102 discrimination unit, 113 camera, 114,115 bar lighting, 117 sensor, 121,122 tube, 126 joint, 127 fillet, 128 surface, 200 computer, 410,420 light, 411,421 reflected light, 500 screen, 501,502,561,700,701,702,706,708,710,716,800,806,808,810,1100,1110,1120,1130 image, 1010 Learning device, 1011, 1031 data acquisition unit, 1012 model generation unit, 1020 trained model memory unit, 1030 inference device, 1032 inference unit, 1040 display unit, 1200 neural network.

Claims

1. 1. A method for computer-implemented inspection of a workpiece, comprising: Receiving input of image data obtained by photographing the brazed workpiece; A step of comparing an inspection target area in the workpiece with a predetermined brightness based on the image data; determining that the workpiece is not normally brazed based on the fact that an area darker than the predetermined brightness is extracted from the inspection target area of ​​the workpiece; and outputting a result of the determination.

2. A step of irradiating light from a light source onto the workpiece; The method further includes a step of photographing the workpiece irradiated with the light by an imaging device, The method of claim 1 , wherein the step of receiving the input comprises the step of receiving the image data from the imaging device.

3. The method of claim 2 , wherein an axis connecting the workpiece and the imaging device is offset from an axis connecting the light source and the workpiece.

4. Further comprising a step of extracting an area corresponding to the work from the input image data based on a pre-specified color component; The method of claim 1 , wherein the comparing step includes the step of comparing the area of ​​interest to the predetermined brightness in an extracted region.

5. 1. A method for generating a trained model for inferring a state of a brazed joint, comprising: Receiving input of image data obtained by photographing the brazed workpiece; A step of receiving input of learning data including performance information representing a state of the brazing joint of the workpiece with respect to the image data and image data obtained by photographing the brazed joint of the workpiece; and using the learning data to generate a trained model that infers the state of the brazing joint of a workpiece from image data of the workpiece on which brazing joint has been performed.

6. The performance information is Information indicating that a fillet has been formed at the joint to meet a predetermined criterion; and and information indicating that a fillet has not been formed at the joint to meet said predetermined criteria.

7. The method of claim 6 , wherein the performance information further comprises information indicating that a fillet has been formed at the joint to meet the predetermined criteria, but that a base material of the workpiece has been damaged.

8. A program for causing a computer to implement the method according to any one of claims 1 to 7.

9. A memory storing a program for causing a computer to implement the method according to any one of claims 1 to 4; and a processor for executing the program.

10. A memory storing a program for causing a computer to implement the method according to any one of claims 5 to 7; A trained model generation device comprising: a processor that executes the program.

11. A trained model that allows a computer to infer a state of a brazed joint, the trained model comprising: The image data for inference obtained by photographing the brazed workpiece is input, A trained model for operating a computer to infer the state of a brazed joint of a workpiece from the inference image data of the brazed joint, based on training image data obtained by photographing the brazed joint and performance information representing the state of the brazed joint of the workpiece.

12. The performance information is Information indicating that a fillet has been formed at the joint to meet a predetermined criterion; and and information indicating that a fillet has not been formed at the joint to meet the predetermined criteria.

13. The trained model of claim 12, wherein the performance information further includes information indicating that a fillet is formed at the joint to meet the predetermined criteria, but that a base material of the workpiece is damaged.

Citation Information

Patent Citations

  • Joining structure, inspection device, and inspection method

    JP2008111457A