Analysis method, analysis apparatus, analysis system, program, storage medium, and head-mounted display

Abstract

The present invention provides an analysis method, analysis apparatus, analysis system, program, storage medium, and head-mounted display capable of improving the accuracy of image-based brazing analysis. [Solution] The analysis method according to the embodiment analyzes the brazing work of a first member and a second member. In the analysis method, a computer acquires a first image in a first wavelength band in which the brazing work is captured. The computer acquires a second image in a second wavelength band in which the brazing work is captured and which has a wavelength longer than the wavelength of the first wavelength band. The computer determines a first period in which disturbances to the analysis occur based on a plurality of pixel values, including the pixel values ​​of the first image and the pixel values ​​of one or more first images acquired in the past. The computer determines whether the timing in which the second image was taken is included in the first period. If the timing is not included in the first period, the computer uses the second image to determine whether the heating state of the first member and the heating state of the second member are sufficient.

Description

Technical Field

[0001] Embodiments of the present invention relate to an analysis method, an analysis apparatus, an analysis system, a program, a storage medium, and a head-mounted display.

Background Art

[0002] When manufacturing an article, a brazing operation may be performed. In order to improve the quality of the brazed product, attempts have been made to analyze the brazing operation from an image. There is a need for a technology that can improve the analysis accuracy of the brazing operation using an image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the embodiments of the present invention is to provide an analysis method, an analysis apparatus, an analysis system, a program, a storage medium, and a head-mounted display that can improve the analysis accuracy of the brazing operation using an image.

Means for Solving the Problems

[0005] The analysis method according to the embodiment analyzes the brazing work of a first member and a second member. In the analysis method, a computer acquires a first image in a first wavelength band in which the brazing work is captured. The computer acquires a second image in a second wavelength band, which is longer than the wavelength of the first wavelength band, in which the brazing work is captured. The computer determines a first period in which disturbances to the analysis occur, based on a plurality of pixel values, including the pixel values ​​of the first image and one or more pixel values ​​of the first image acquired in the past. The computer determines whether the timing in which the second image was taken is included in the first period. If the timing is not included in the first period, the computer performs a first determination using the second image to determine whether the heating state of the first member and the heating state of the second member are sufficient. [Brief explanation of the drawing]

[0006] [Figure 1] Figure 1 is a schematic diagram showing the configuration of the analysis system according to the embodiment. [Figure 2] Figure 2 is a flowchart showing the analysis method according to the embodiment. [Figure 3] Figure 3 is an image showing the brazing process. [Figure 4] Figure 4(a) is an example of the first image in the first wavelength band. Figure 4(b) is an example of the second image in the second wavelength band. [Figure 5] Figure 5 is a graph showing an example of the first waveform. [Figure 6] Figure 6 is a graph showing another example of the first waveform. [Figure 7] Figures 7(a) and 7(b) are images showing the brazing process. [Figure 8] Figure 8 is a flowchart showing a specific example of the first determination. [Figure 9] Figures 9(a) to 9(c) are images illustrating specific examples of the first judgment. [Figure 10] Figure 10 is a graph showing the pixel values ​​on line XX in Figure 9(c). [Figure 11] Figure 11 is a flowchart showing a specific example of the second judgment. [Figure 12] Figure 12 is a graph showing the pixel values ​​on the XX line in Figure 9(c). [Figure 13] Figure 13 is a graph showing pixel values ​​in another image. [Figure 14] Figure 14 is a graph showing an example of the relationship between wavelength and pixel value. [Figure 15] Figure 15 is a graph showing an example of how pixel values ​​change with respect to wavelength. [Figure 16] Figure 16 is a schematic diagram showing the configuration of the imaging device. [Figure 17] Figure 17 is a schematic diagram illustrating the configuration of a computer that performs the analysis method according to the embodiment. [Figure 18] Figure 18 is a perspective view showing an example of a head-mounted display. [Figure 19] Figure 19 is a schematic diagram showing an example of a display using a head-mounted display. [Figure 20] Figure 20 is a schematic diagram showing an example of a display using a head-mounted display. [Figure 21] Figure 21 is a schematic diagram showing an example of a display using a head-mounted display. [Modes for carrying out the invention]

[0007] The embodiments of the present invention will be described below with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes of the parts, etc., are not necessarily the same as those of reality. Furthermore, even when representing the same part, the dimensions and ratios may be represented differently in the drawings. In this specification and each drawing, elements similar to those already described are denoted by the same reference numerals, and detailed explanations are omitted as appropriate.

[0008] Embodiments of the present invention are applied to brazing operations. In a brazing operation, two members are joined by a brazing material. First, the two members are brought close to each other, and the joint portions of the members are sufficiently heated by a flame. When each joint portion is sufficiently heated, the brazing material is applied to the joint portions. The brazing material melts and adheres to each joint portion. When a sufficient amount of the brazing material has adhered to each joint portion, the heating is stopped and each member is cooled. By cooling, the brazing material solidifies and the two members are joined.

[0009] The members (base materials) to be joined are made of metal or ceramics. For example, when the base material is made of metal, the base material includes copper or steel. When the base material is made of ceramics, the base material includes aluminum nitride, aluminum oxide, zirconia, or the like. In joining these base materials, a metal having a melting point lower than that of the base material, such as phosphor bronze or aluminum, is used as the brazing material.

[0010] FIG. 1 is a schematic diagram showing the configuration of an analysis system according to an embodiment. As shown in FIG. 1, an analysis system 1 according to an embodiment includes a photographing device 10, a computer 20, and a display device 30.

[0011] In the example shown in FIG. 1, an operator W is brazing a first member 41 and a second member 42. The operator W holds a burner 43 in the right hand and a linear brazing material 44 in the left hand.

[0012] The photographing device 10 photographs the joint portion of the first member 41 and the second member 42 and acquires an image. At this time, the photographing device 10 acquires images in two different wavelength bands (a first wavelength band and a second wavelength band). The wavelength of the second wavelength band is longer than the wavelength of the first wavelength band. The first wavelength band and the second wavelength band do not overlap. The photographing device 10 acquires a first image in the first wavelength band and a second image in the second wavelength band photographed at the same timing. The photographing range of the first image and the photographing range of the second image are the same. The photographing device 10 repeatedly acquires the first image and the second image during the brazing operation.

[0013] Computer 20 includes processing circuits that perform various processes, memory for storing programs, etc. Computer 20 may be a general-purpose personal computer (PC). The processing circuits acquire the first image and the second image acquired by the imaging device 10. Computer 20 may receive the images directly from the imaging device 10. Alternatively, the imaging device 10 may store the images in a storage device or a network server, and computer 20 may retrieve the images from that storage device.

[0014] Computer 20 functions as an analytical device that performs analysis related to the brazing operation. Computer 20 analyzes the brazing operation using the acquired images. Display device 30 displays the analysis results from computer 20 to the worker W.

[0015] Figure 2 is a flowchart illustrating the analysis method according to this embodiment. Figure 3 is an image showing the brazing process. In the analysis method AM shown in Figure 2, first, worker W heats the first member 41 and the second member 42 using a burner 43 (step S0). Figure 3 shows the view from worker W's perspective. In the example in Figure 3, the first member 41 and the second member 42 are heated by the flame F1 of the burner 43. For example, the first member 41 and the second member 42 are made of copper and are brown when not heated. The temperature of the flame F1 is about 1400-1800°C near the burner 43, and the flame F1 is blue.

[0016] The imaging device 10 photographs the brazing process and saves a first image in the first wavelength band and a second image in the second wavelength band (step S1). The computer 20 acquires the first and second images (step S2).

[0017] Figure 4(a) is an example of the first image in the first wavelength band. Figure 4(b) is an example of the second image in the second wavelength band. Figures 4(a) and 4(b) are the first and second images, respectively, taken at the timing shown in Figure 3. In this example, the first wavelength band is 530 ± 40 nm. The first image, IMG1, shown in Figure 4(a), is an optical image based on light with a wavelength in the range of 530 ± 40 nm. The second wavelength band is 630 ± 60 nm. The second image, IMG2, shown in Figure 4(b), is an optical image based on light with a wavelength in the range of 630 ± 60 nm.

[0018] Computer 20 obtains pixel values ​​from the first image. Computer 20 also refers to the pixel values ​​of multiple first images obtained in the past. Computer 20 generates a first waveform using the pixel values ​​of the most recent first image and the pixel values ​​of multiple past first images (step S3).

[0019] Figure 5 is a graph showing an example of the first waveform. Figure 6 is a graph showing another example of the first waveform. The computer 20 generates a first waveform, for example, as shown in Figure 5. In Figure 5, the horizontal axis represents time T, and the vertical axis represents the pixel value (luminance) V. The imaging device 10 repeatedly takes images during the brazing process. By repeatedly taking images, multiple first images and multiple second images are obtained. As a result, a first waveform is generated using the pixel values ​​of the first images acquired up to that point.

[0020] The computer 20 determines from the first waveform the first period during which disturbances to the analysis occur (step S4). For example, as shown in Figure 5, the computer 20 compares the pixel value V with a preset threshold th1. The period during which the pixel value V is greater than or equal to the threshold th1 is determined to be the first period P1. If there are multiple periods during which the pixel value V is greater than or equal to the threshold th1, and the intervals between these periods are shorter than a predetermined time length, the computer 20 may determine that the multiple periods and the periods between them together constitute the first period P1.

[0021] The computer 20 may also differentiate the pixel value V with respect to time T, as shown in Figure 6, to generate a first waveform representing the change in the pixel value dV / dT with respect to time T. The computer 20 compares the absolute value of the change in dV / dT with a preset threshold th2. The period during which the change in dV / dT is greater than or equal to the threshold th2 is determined to be the first period P1. If there are multiple periods during which the change in dV / dT is greater than or equal to the threshold th2, and the intervals between these periods are shorter than a predetermined time length, the computer 20 may determine that the multiple periods and the periods between them together constitute the first period P1.

[0022] Figures 7(a) and 7(b) are images showing the brazing process. Disturbances to the analysis of brazing operations include the generation of flames from the heated base material and the generation of flames from scattered base material components. Figure 7(a) shows an image when no flames are generated from the base material. The first member 41 and the second member 42 are heated by the flame F1 from the burner 43, causing the first member 41 and the second member 42 to glow red and become bright. In Figure 7(b), flames F2 are generated from the first member 41 and the second member 42. Flames F2 are brighter than flames F1, the first member 41, and the second member 42. Therefore, flames F2 can be a disturbance that hinders analysis.

[0023] The computer 20 determines whether the timing of the capture of the second image falls within the first period P1 (step S5). If the capture timing does not fall within the first period P1, the computer 20 performs a first determination (step S6). In the first determination, the second image is used to determine whether the heating state of the first member 41 is sufficient, and whether the heating state of the second member 42 is sufficient.

[0024] Metallic components change color when their temperature rises due to heating. For example, if the first component 41 and the second component 42 are made of copper, they will turn red when heated to about 700-800°C. The brightness of the red color depends on the temperature. That is, the higher the temperature of the first component 41 and the second component 42, the larger the pixel values ​​of the first component 41 and the second component 42 that appear in the second image. The computer 20 can perform a first determination using the pixel values ​​of the first component 41 and the second component 42 in the second image.

[0025] For example, the imaging device 10 is positioned so that only the joint area is visible in the second image. In this case, the computer 20 calculates the average of the total pixel values ​​in the second image. If the average value is above a preset threshold, the computer 20 determines that the heating state of the first member 41 is sufficient, and also determines that the heating state of the second member 42 is sufficient.

[0026] A separate imaging device 10 may be provided to photograph only the joint of the first member 41, and another imaging device 10 may be provided to photograph only the joint of the second member 42. In this case, the computer 20 calculates the average of the total pixel values ​​of the second image captured by one imaging device 10 as the pixel value of the first member 41. The computer 20 calculates the average of the total pixel values ​​of the second image captured by another imaging device 10 as the pixel value of the second member 42. The computer 20 determines that the first member 41 is sufficiently heated if the pixel value of the first member 41 is above a preset threshold. The computer 20 determines that the second member 42 is sufficiently heated if the pixel value of the second member 42 is above a preset threshold.

[0027] If an object or space other than the first member 41 and the second member 42 is captured in a second image, the computer 20 may extract the first member 41 and the second member 42 from the second image. For example, the regions of the first member 41 and the second member 42 are predefined in the second image. The computer 20 cuts out the region set as the first member 41 and the region set as the second member 42 from the second image.

[0028] Alternatively, the computer 20 may extract the edges of the first member 41 and the edges of the second member 42 from the second image. The computer 20 determines that the region enclosed by the edges of the first member 41 is the first member 41. The computer 20 determines that the region enclosed by the edges of the second member 42 is the second member 42. The computer 20 calculates the average of the pixel values ​​of at least some of the region of the first member 41. If the average value is greater than or equal to a preset threshold, the computer 20 determines that the first member 41 is sufficiently heated. The computer 20 calculates the average of the pixel values ​​of at least some of the region of the second member 42. If the average value is greater than or equal to a preset threshold, the computer 20 determines that the second member 42 is sufficiently heated.

[0029] Edge detection using image processing can be used for edge extraction. Alternatively, an image processing model for edge extraction may be provided. The image processing model is pre-trained to output the edges of the first member 41 and the second member 42 in response to an input image containing the first member 41 and the second member 42. Preferably, the image processing model includes a convolutional neural network.

[0030] The computer 20 may perform a second determination (step S7). In the second determination, the uniformity between the heating state of the first member 41 and the heating state of the second member 42 is determined. As described above, the pixel values ​​of the first member 41 and the second member 42 depend on the temperature of the first member 41 and the temperature of the second member 42, respectively. Therefore, if the difference between the pixel values ​​of the first member 41 and the second member 42 is large in the second image, it indicates that the first member 41 and the second member 42 are not heated uniformly. The computer 20 determines that the first member 41 and the second member 42 are heated uniformly if the difference between the pixel values ​​of the first member 41 and the second member 42 in the second image is less than a preset threshold.

[0031] For example, as described above, a second image is obtained showing only the joint area of ​​the first member 41, and a second image is obtained showing only the joint area of ​​the second member 42. Alternatively, the regions of the first member 41 and the second member 42 are predefined in the second image. The edges of the first member 41 and the edges of the second member 42 may be extracted from the second image. By either method, the pixel values ​​of the first member 41 and the pixel values ​​of the second member 42 are obtained, and the difference between the pixel values ​​of the first member 41 and the pixel values ​​of the second member 42 is calculated.

[0032] If it is determined in step S5 that the shooting timing is included in the first period P1, the computer 20 does not perform the first and second determinations. Alternatively, the computer 20 may perform the first and second determinations, but treats the results of those determinations as invalid.

[0033] The computer 20 then determines whether the brazing process is complete (step S8). If the brazing process is not complete, the process returns to step S1. This causes the brazing process, which is currently heating, to be filmed again.

[0034] Figure 8 is a flowchart showing a specific example of the first determination. Referring to Figure 8, an example of the specific processing for the first determination will be explained. First, the computer 20 extracts the edges of the first member 41 and the second member 42 from the image (step S6a). Either the first image or the second image may be used for edge extraction. Based on the edge extraction results, the computer 20 determines the first member 41 and the second member 42 that are depicted in the image (step S6b). For example, a function such as findContours is used to extract the region surrounded by the edges. Based on the positional relationship and area size relationship of the extracted multiple regions, one of the multiple regions is determined to be the first member 41, and another of the multiple regions is determined to be the second member 42. In the example shown in Figure 7, the largest region at the top of the image is determined to be the first member 41. The largest region at the bottom of the image is determined to be the second member 42.

[0035] The computer 20 sets a first determination area in the region of the image that is determined to be the first member 41. The computer 20 sets a second determination area in the region of the image that is determined to be the second member 42 (step S6c). The first determination area and the second determination area are areas used for the first determination. For example, the vicinity of the center of the joint of the first member 41 is set as the first determination area. The vicinity of the center of the joint of the second member 42 is set as the second determination area.

[0036] Computer 20 calculates the average of the pixel values ​​in the first determination area and obtains the first average value. Computer 20 calculates the average of the pixel values ​​in the second determination area and obtains the second average value (step S6d). The average may be a simple average or a weighted average.

[0037] The computer 20 compares the first average value with a preset first threshold (step S6e). If the first average value is greater than or equal to the first threshold, the computer 20 determines that the heating state of the first component 41 is sufficient. If the first average value is less than the first threshold, the computer 20 determines that the heating state of the first component 41 is insufficient.

[0038] The computer 20 compares the second average value with a preset second threshold (step S6f). The second threshold may be the same as the first threshold, or it may be different from the first threshold. If the second average value is greater than or equal to the second threshold, the computer 20 determines that the heating state of the second component 42 is sufficient. If the second average value is less than the second threshold, the computer 20 determines that the heating state of the second component 42 is insufficient.

[0039] Figures 9(a) to 9(c) are images illustrating specific examples of the first judgment. The imaging device 10 acquires an image IMG3, for example, as shown in Figure 9(a). The computer 20 extracts the edge E1 of the first member 41 and the edge E2 of the second member 42 from the image IMG3, as shown in Figure 9(b). The computer 20 determines that the area enclosed by edge E1 is the first member 41. The computer 20 determines that the area enclosed by edge E2 is the second member 42.

[0040] As shown in Figure 9(c), the computer 20 sets a first determination region R1 in the first member 41. In the illustrated example, the first member 41 and the second member 42 are pipes extending in the first direction D1. The first determination region R1 is located on the side of the second member 42 in the first member 41. Also, the first determination region R1 is away from edge E1 in the second direction D2, which is perpendicular to the first direction D1, and is located in the central part of the first member 41. For example, the computer 20 divides the first member 41 into three equal parts in the second direction D2. The computer 20 sets the first determination region R1 in the central region of the three divided areas.

[0041] The computer 20 sets a second determination region R2 in the second member 42. The second determination region R2 is located on the side of the first member 41 in the second member 42. The second determination region R2 is away from edge E2 in the second direction D2 and is located in the central part of the second member 42. For example, the second determination region R2 is set in the central region of the region of the second member 42 that is divided into three equal parts in the second direction D2.

[0042] Figure 10 is a graph showing the pixel values ​​on line XX in Figure 9(c). In Figure 10, the horizontal axis represents the position P in the first direction D1. The vertical axis represents the pixel value (luminance) V. Computer 20 calculates the first average value av1 in the region determined to be the first member 41. It also calculates the second average value av2 in the region determined to be the second member 42.

[0043] The computer 20 compares the first mean value av1 and the second mean value av2 with the threshold th3. The threshold th3 is an example of the first and second thresholds. In this example, the first and second thresholds are the same. The first mean value av1 and the second mean value av2 are greater than the threshold th3. Therefore, the computer 20 determines that the first member 41 is sufficiently heated and the second member 42 is sufficiently heated.

[0044] Figure 11 is a flowchart showing a specific example of the second judgment. Referring to Figure 11, an example of the specific processing for the second determination will be explained. First, the computer 20 extracts the edges of the first member 41 and the second member 42 from the image (step S7a). The computer 20 determines whether the first member 41 and the second member 42 are the same based on the edge extraction results (step S7b). The computer 20 sets the first determination area and the second determination area (step S7c). The computer 20 calculates the first average value and the second average value (step S7d). Steps S7a to S7d can be performed in the same manner as steps S6a to S6d. Steps S7a to S7d may be omitted by using the results of steps S6a to S6d in the second determination.

[0045] The computer 20 calculates the difference between the first average value and the second average value (step S7e). The computer 20 compares the difference with a preset third threshold (step S7f). If the difference is greater than or equal to the third threshold, the computer 20 determines that the heating state of the first member 41 and the heating state of the second member 42 are not uniform. If the difference is less than the third threshold, the computer 20 determines that the heating state of the first member 41 and the heating state of the second member 42 are uniform.

[0046] Instead of the difference, the ratio of the first mean value to the second mean value may be used. For example, if the ratio of the second mean value to the first mean value is within a predetermined range, the computer 20 determines that the heating state of the first component 41 and the heating state of the second component 42 are uniform.

[0047] Figure 12 is a graph showing the pixel values ​​on the XX line in Figure 9(c). The graph in Figure 12 is the same as the graph in Figure 10. As shown in Figure 12, the computer 20 calculates the difference df between the first mean value av1 and the second mean value av2. The computer 20 compares the difference df with a preset threshold th4 (an example of a third threshold). In the illustrated example, the difference df is greater than the threshold th4. Therefore, the computer 20 determines that the heating state of the first member 41 and the heating state of the second member 42 are not uniform.

[0048] Figure 13 is a graph showing pixel values ​​in another image. In the example shown in Figure 13, both the first mean value av1 and the second mean value av2 are greater than the threshold th3. Therefore, the computer 20 determines that the heating state of the first member 41 is sufficient and the heating state of the second member 42 is sufficient. Also, the difference df between the first mean value av1 and the second mean value av2 is smaller than the threshold th4. Therefore, the computer 20 determines that the heating state of the first member 41 and the heating state of the second member 42 are uniform.

[0049] The advantages of the embodiment will be explained. When manufacturing goods, workers sometimes perform brazing. When workers perform brazing, the quality of the brazing depends on the worker's experience and skill. For example, it is desirable that the brazing material be supplied after the base material has been sufficiently heated. If the brazing material is supplied before the base material is sufficiently heated, the brazing material will not melt, and proper brazing cannot be performed. Also, it is necessary to continue heating the brazing material until it melts on the base material. In this case, the base material surrounding the brazing material may be unnecessarily heated, potentially causing the base material to deteriorate.

[0050] One method to support proper brazing is to attach multiple sensors to the worker, the burner, and their surroundings. In this method, the brazing process is analyzed based on the sensor detection results. However, this method requires time-consuming sensor installation, and the sensors may potentially interfere with the work.

[0051] To eliminate the need for sensors, there is also an analysis method that uses cameras. By photographing the brazing process with a camera and analyzing the images, proper brazing techniques can be supported. This method eliminates the need to attach multiple sensors, making it a simple way to analyze brazing processes.

[0052] On the other hand, when using a camera, disturbances to the analysis become a challenge. Disturbances arise when components on the heated surface of the base material, or components scattered from the base material, burn and exhibit a flame test. The flame, whose color has changed due to the flame test, is much brighter than a burner flame or a heated base material. Therefore, when a flame test occurs, the image changes significantly. The occurrence of disturbances makes analysis using a camera difficult. Hereafter, the flame that produces a flame test on the base material will also be referred to as the "flame generated from the base material."

[0053] In the analysis method according to this embodiment, two types of images are used to analyze the brazing process: a first image in the first wavelength band and a second image in the second wavelength band. The first and second images are optical images based on light in different wavelength bands. The first wavelength band is set to match the color of the flame generated from the base material. The first image is acquired to investigate the occurrence of disturbances during the analysis. The second wavelength band is set to match the color of the heated base material. The second image is acquired for the analysis of the brazing process.

[0054] Once the first and second images are acquired, the first period in which disturbances to the analysis occur is determined based on the pixel values ​​of the most recent first image and the pixel values ​​of multiple first images acquired in the past. For example, as shown in Figure 5, the first period includes a period in which the pixel values ​​are greater than a preset threshold. Alternatively, as shown in Figure 6, the first period includes a period in which the change in pixel values ​​over time is greater than a preset threshold.

[0055] Once the first period in which the disturbance occurs is identified, it is determined whether the timing of the capture of the second image falls within that first period. If the capture timing does not fall within the first period, the first determination is performed. In the first determination, the second image is used to determine whether the heating state of the first component and the heating state of the second component are sufficient.

[0056] According to this embodiment, the brazing process is analyzed using a second image taken at a time when no disturbances are occurring. This improves the accuracy of determining the heating state of the first member and the heating state of the second member.

[0057] In the analysis method according to the embodiment, a second determination may be performed in addition to the first determination. Even if the first member and the second member are sufficiently heated, if the temperature difference between the first member and the second member is large, the brazing material will not be properly supplied to the joint between the first member and the second member, and the joint strength will decrease. Also, the way the brazing material melts will change between the first member and the second member, and the first member and the second member will not be joined uniformly, and the joint strength will decrease. In the second determination, the uniformity between the heating state of the first member and the heating state of the second member is determined. If the second determination determines that the heating state between the first member and the heating state of the second member is uniform, the joint strength can be improved by supplying brazing material to the joint.

[0058] Generally, the flame of a gas burner is blue. When a base material is heated with a burner, the flame that can be generated from the base material is blue to yellow. For this reason, the first wavelength band can be selected from the range of 420 nm to less than 620 nm. By selecting the first wavelength band from the range of 420 nm to less than 620 nm, the first period in which disturbances occur can be determined with greater accuracy.

[0059] The base material is made of metal. Generally, when metal is heated to around the melting point of the brazing material, it appears orange to red. For this reason, the second wavelength band can be selected from within the range of 650 nm to less than 950 nm. By selecting the second wavelength band from within the range of 650 nm to less than 950 nm, the flame becomes less likely to appear in the image, thereby improving the accuracy of the first judgment.

[0060] To improve the accuracy of the analysis, it is preferable that the ranges of the first and second wavelength bands are narrow. For example, the first wavelength band is set narrowly to correspond to the color of the flame generated from the base material. This allows for more accurate determination of the first period based solely on the flame generated from the base material. The second wavelength band is set narrowly to correspond to the color of the base material when heated to near the melting point of the brazing material. This allows for improved accuracy of the first determination based solely on the temperature of the heated base material.

[0061] For example, the difference between the upper and lower limits of the first wavelength band is less than 20 nm, and the difference between the upper and lower limits of the second wavelength band is less than 20 nm. These differences are preferably less than 15 nm, and more preferably less than 10 nm.

[0062] Figure 14 is a graph showing an example of the relationship between wavelength and pixel value. Figure 15 is a graph showing an example of the change in pixel value with respect to wavelength. Referring to Figures 14 and 15, an example of a method for selecting the first and second wavelength bands will be explained. First, the first member 41 and the second member 42 are heated while being photographed. The relationship between wavelength and pixel value (spectral data) is obtained by the photography. Spectral data can be obtained using a spectroscopic camera (hyperspectral camera, multispectral camera, etc.).

[0063] Spectral data is obtained when a flame is generated from the base material. Figure 14 shows an example of spectral data when a flame is generated from the base material. In Figure 14, the horizontal axis represents wavelength λ. The vertical axis represents the sum of the pixel values ​​Vs of light at each wavelength contained in the image. In Figure 14, peaks p1 to p9 are generated. For example, the wavelength band in which any of the peaks p1 to p9 are generated can be used as the first wavelength band.

[0064] Alternatively, the spectral data may be differentiated with respect to wavelength. Figure 15 shows the result of differentiating the pixel values ​​of the spectral data from Figure 14 with respect to wavelength. In Figure 15, peaks p11 to p19 of dVs / dλ are observed. For example, the wavelength band in which any of the peaks p11 to p19 occur can be used as the first wavelength band.

[0065] The second wavelength band is selected from within the range of the base material's color when heated. At this time, since peaks p1-p9 and p11-p19 are wavelength bands prone to disturbances, the second wavelength band is selected while avoiding the wavelength bands where peaks p1-p9 and p11-p19 occur. Even when peaks p1-p9 and p11-p19 are avoided, disturbances still affect pixel values ​​in other wavelength bands. Therefore, to improve the analysis accuracy of the brazing process, it is desirable to perform the first judgment while avoiding the first period during which disturbances occur.

[0066] For example, the base material is made of copper, and the brazing material is made of phosphorus copper. The melting point (solidus) of phosphorus copper brazing material (BCuP-2) is 710°C. In this case, the base material is heated to approximately 700°C. The first wavelength band is set to 544 nm ± 10 nm. The second wavelength band is set to 780 nm ± 10 nm.

[0067] As another example, the base material is made of steel and the brazing material is made of silver. The melting point (solidus) of silver brazing material (BAg-24) is 660°C. In this case, the base material is heated to approximately 600°C. The first wavelength band is set to 460 nm ± 10 nm. The second wavelength band is set to 600 nm ± 10 nm.

[0068] Figure 16 is a schematic diagram showing the configuration of the imaging device. In this embodiment, it is sufficient to acquire only the light contained in the first wavelength band and the light contained in the second wavelength band. For example, as shown in Figure 16, the imaging device 10 includes an optical system 11, a beam splitter 12, a first filter 13, a first image sensor 14, a second filter 15, and a second image sensor 16.

[0069] The optical system 11 includes one or more lenses. Light L is incident on the optical system 11. The light that has passed through the optical system 11 is incident on the beam splitter 12. The beam splitter 12 splits the light L into light L1 and light L2. Light L1 is incident on the first filter 13. The first filter 13 is a filter that transmits only light in the first wavelength band. The light L1 that has passed through the first filter 13 is incident on the first image sensor 14. The first image sensor 14 converts the light L1 in the first wavelength band into an electrical signal and generates a first image.

[0070] Light L2 enters the second filter 15. The second filter 15 is a filter that transmits only light in the second wavelength band. The light L2 that has passed through the second filter 15 enters the second image sensor 16. The second image sensor 16 converts the light L2 in the second wavelength band into an electrical signal and generates a second image.

[0071] Light in the first wavelength band and light in the second wavelength band can also be acquired, for example, by a spectroscopic camera. However, spectroscopic cameras are larger than ordinary cameras. When a camera is attached to a worker, the spectroscopic camera may interfere with the work. With the imaging device 10 configured as shown in Figure 16, it is possible to acquire a first image based on light in the first wavelength band and a second image based on light in the second wavelength band while suppressing the increase in size of the imaging device 10. The imaging device 10 shown in Figure 16 is suitable for brazing work.

[0072] Figure 17 is a schematic diagram illustrating the configuration of a computer that performs the analysis method according to the embodiment. The computer 90 includes a CPU 91, ROM 92, RAM 93, storage device 94, input interface 95, output interface 96, and communication interface 97.

[0073] ROM92 stores programs that control the operation of computer 90. ROM92 contains the programs necessary for computer 90 to perform each of the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are loaded.

[0074] The CPU 91 includes processing circuits. The CPU 91 uses the RAM 93 as work memory and executes programs stored in at least one of the ROM 92 or the storage device 94. During program execution, the CPU 91 controls each component via the system bus 98 and performs various processes.

[0075] The memory device 94 stores data necessary for program execution and data obtained through program execution.

[0076] The input interface (I / F) 95 connects the computer 90 to the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0077] The output interface (I / F) 96 can connect the computer 90 to the output device 96a. The output I / F 96 is a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI®). The CPU 91 can transmit data to the output device 96a via the output I / F 96 and display an image on the output device 96a.

[0078] The communication interface (I / F) 97 allows the computer 90 to connect with a server 97a located outside the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97.

[0079] The storage device 94 includes one or more selected from Hard Disk Drives (HDDs) and Solid State Drives (SSDs). The input device 95a includes one or more selected from a mouse, keyboard, microphone (voice input), and touchpad. The output device 96a includes one or more selected from a monitor, projector, printer, and speaker. Devices that have the functions of both input device 95a and output device 96a, such as a touch panel, may also be used.

[0080] The analysis process may be performed by one computer 90 or by multiple computers 90.

[0081] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.

[0082] For example, data on a recording medium is read by a computer (or embedded system). The recording format (storage format) on the recording medium is arbitrary. For example, a computer reads a program from the recording medium and causes the CPU to execute instructions based on this program. The acquisition (or reading) of the program by the computer may be done via a network.

[0083] Figure 18 is a perspective view showing an example of a head-mounted display. The computer that performs the analysis method may be integrated into a head-mounted display (HMD). For example, as shown in Figure 18, the HMD 100 includes a frame 101, lenses 111 and 112, projection devices 121 and 122, an image camera 131, a depth camera 132, a light source 133, an eye-tracking camera 134, a sensor 140, a microphone 141, a processing unit 150, a battery 160, and a storage device 170.

[0084] In the illustrated example, the HMD100 is a dual-lens head-mounted display. Two lenses, 111 and 112, are fitted into the frame 101. Projection devices 121 and 122 project information onto lenses 111 and 112, respectively.

[0085] Projection devices 121 and 122 display information about the brazing operation, the results of various judgments, etc., on lenses 111 and 112. Alternatively, only one of the projection devices 121 or 122 may be provided, and information may be displayed on only one of the lenses 111 or 112.

[0086] Lenses 111 and 112 are light-transmitting. The worker can see the real world through lenses 111 and 112. The worker can also see the information projected onto lenses 111 and 112 by projection devices 121 and 122. The information is superimposed onto the real world by projection from projection devices 121 and 122.

[0087] Image camera 131 detects visible light and obtains a two-dimensional image. Image camera 131 may be a spectrophotometer, but preferably has the configuration shown in Figure 16. Depth camera 132 irradiates infrared light and obtains a depth image based on the reflected infrared light. Light source 133 irradiates light (e.g., infrared light) toward the wearer's eyeballs. Eye tracking camera 134 detects the light reflected by the wearer's eyeballs. Sensor 140 is a 6-axis detection sensor capable of detecting angular velocity in 3 axes and acceleration in 3 axes. Microphone 141 accepts voice input.

[0088] The processing unit 150 controls each element of the HMD 100. For example, the processing unit 150 controls the projection devices 121 and 122 to display information on lenses 111 and 112. Hereafter, the process by which the processing unit 150 displays information on lenses 111 and 112 using projection devices 121 and 122 will also be simply referred to as "the processing unit displays information." In addition, the processing unit 150 detects movement of the field of view based on the detection results from the sensor 140. The processing unit 150 changes the display by projection devices 121 and 122 according to the movement of the field of view.

[0089] The processing unit 150 uses the image acquired by the image camera 131 to perform steps S2 to S8 of the analysis method shown in Figure 2. The processing unit 150 may also display the processing results obtained in the analysis method. In addition, the processing unit 150 may recognize the surface shape of an object from the image acquired by the depth camera 132. The processing unit 150 may also calculate the viewpoint and line of sight of the worker's eyes from the detection results obtained by the eye-tracking camera 134.

[0090] The battery 160 supplies the power necessary for operation to each element of the HMD 100. The storage device 170 stores data necessary for processing by the processing device 150, data obtained from processing by the processing device 150, etc. The storage device 170 may be located outside the HMD 100 and may communicate with the processing device 150.

[0091] The display device is not limited to the illustrated example; it may also be a single-lens head-mounted display. The display device may also be a glasses-type device as shown in the illustration, or a helmet-type device.

[0092] The HMD100 may be equipped with a display instead of lenses 111, 112, projection device 121, and projection device 122. In that case, the display will show video captured by the image camera 131. The wearer will be able to understand the surrounding situation through the video. In addition, information on the brazing work and the results of various judgments will be displayed on the display.

[0093] Figures 19 to 21 are schematic diagrams showing examples of displays using head-mounted displays. For example, as shown in Figure 19, worker W views the first member 41 and the second member 42 through transmissive lenses 111 and 112. When the brazing operation begins, the image camera 131 takes pictures of the first member 41 and the second member 42. The processing device 150 uses the acquired images to perform steps S2 to S8 of the analysis method AM shown in Figure 2.

[0094] For example, the first determination may determine that the first member 41 and the second member 42 are sufficiently heated, and the second determination may determine that the first member 41 and the second member 42 are uniformly heated. In that case, the processing device 150 displays a message M1 indicating the determination result, as shown in Figure 19. Message M1 may also be output by voice.

[0095] As another example, the first determination may be that the heating state of the first component 41 is insufficient. In that case, the processing device 150 displays a message M2 indicating the determination result, as shown in Figure 20. Message M2 may also include a work instruction to the operator prompting them to heat the first component 41. Message M2 may also be output by voice.

[0096] As yet another example, the first determination determines that the heating state of the first member 41 and the second member 42 is sufficient, and the second determination determines that the heating state of the first member 41 and the second member 42 is uneven. In that case, the processing device 150 displays a message M3 indicating the determination result, as shown in Figure 21. Message M3 may include a work instruction to the operator prompting them to heat one of the members more evenly in order to make the heating of the first member 41 and the second member 42 more uniform. Message M3 may also be output by voice.

[0097] In the display example shown in Figure 21, the specific area of ​​the component to be heated may be displayed, as in Figure 9(c). For example, if the processing device 150 determines that the heating state of the first component 41 and the heating state of the second component 42 are not uniform, it displays a determination area set for the component to be heated. This allows the operator to specifically understand which part of the component should be heated.

[0098] With respect to the HMD according to the embodiment, at least part of the processing in steps S2 to S8 shown in Figure 2 may be performed by an external computer. For example, the HMD is connected to a computer prepared separately from the HMD by wireless communication. The HMD transmits the acquired image or the result of processing using the image to the external computer. The computer performs processing using the received data and transmits the processing result to the HMD.

[0099] As a specific example, when the HMD acquires the first and second images, it transmits these images to an external computer. The computer collects the first and second images and executes steps S3 to S7. The computer then transmits each of the judgment results obtained in steps S3 to S7 to the HMD.

[0100] In such cases, an external computer can also be considered part of the HMD100 according to the embodiment. By having at least a portion of the data processing performed by an external computer, the processing unit 150 can be made smaller and lighter, improving the usability of the HMD100.

[0101] Embodiments of the present invention include the following features. (Feature 1) An analytical method for analyzing the brazing work between a first member and a second member, Computers A first image in the first wavelength band showing the brazing operation is acquired. The brazing operation is captured, and a second image is obtained in a second wavelength band that is longer than the wavelength of the first wavelength band. Based on a plurality of pixel values ​​including the pixel values ​​of the first image and one or more pixel values ​​of the first image acquired in the past, a first period in which disturbances to the analysis occur is determined. Determine whether the timing at which the second image was taken falls within the first period. If the aforementioned timing is not included in the first period, a first determination is performed using the second image to determine whether the heating state of the first member is sufficient and whether the heating state of the second member is sufficient. Analysis method. (Feature 2) The analysis method according to Feature 1, wherein the computer does not perform the first determination or invalidates the result of the first determination if the timing is included in the first period. (Feature 3) The analysis method according to feature 1 or 2, wherein the computer further performs a second determination using the second image to determine the uniformity between the heating state of the first member and the heating state of the second member when the timing is not included in the first period. (Feature 4) The first wavelength band is selected from the range of 420 nm to less than 620 nm. The second wavelength band is selected from within the range of 650 nm to less than 950 nm, as described in any one of features 1 to 3 of the analysis method. (Feature 5) The difference between the upper and lower limits of the first wavelength band is less than 20 nm. The difference between the upper and lower limits of the second wavelength band is less than 20 nm, as described in Feature 4 of the analysis method. (Feature 6) The aforementioned computer, A first waveform is obtained that shows the relationship between time and pixel value or the relationship between time and the change in pixel value. In the first waveform, the period during which the pixel value is greater than a threshold or the change in the pixel value is greater than a threshold is determined to be the first period. The analysis method described in one of the features 1-5. (Feature 7) The aforementioned computer, From the first image or the second image, the edges of the first member and the edges of the second member are extracted. In the first image, the first period is determined using the average value of the pixel values ​​in a part of the first member and the average value of the pixel values ​​in a part of the second member. The analysis method described in one of the features 1-5. (Feature 8) The aforementioned computer, From the first image or the second image, the edges of the first member and the edges of the second member are extracted. In the second image, the first determination is performed using the average value of the pixel values ​​in a part of the first member and the average value of the pixel values ​​in a part of the second member. The analysis method described in one of the features 1-7. (Feature 9) Equipped with a computer including processing circuits, An analytical apparatus that performs the analysis method described in any one of features 1 to 8 of the aforementioned processing circuit. (Feature 10) The analytical apparatus described in Feature 9, A camera that acquires the first image and the second image, An analytical system equipped with the following features. (Feature 11) The aforementioned imaging device is A first filter that selectively transmits light included in the first wavelength band, A first image sensor that receives light transmitted through the first filter, A second filter that selectively transmits light included in the second wavelength band, A second image sensor that receives light transmitted through the second filter, The analysis system described in Feature 10, including the above. (Feature 12) A program that causes a computer to perform an analysis method described in one of the features 1 to 8. (Feature 13) A storage medium containing the program described in Feature 12. (Feature 14) A photographic device that repeatedly acquires a first image in the first wavelength band and a second image in the second wavelength band which has a longer wavelength than the first wavelength band by photographing the brazing process of the first and second members, A display device that shows information to the wearer, A processing circuit is provided, The aforementioned processing circuit is Based on the pixel values ​​of multiple first images, the disturbance time period is determined. From among the multiple second images, extract the second images acquired during time periods other than the disturbance period. Using the extracted second image, a first determination is performed to determine whether the heating state of the first member is sufficient and whether the heating state of the second member is sufficient. The display device is a head-mounted display that displays an alert when the first member is not sufficiently heated or when the second member is not sufficiently heated. (Feature 15) The aforementioned display device is If the heating state of the first member is insufficient, the area of ​​the first member that is insufficiently heated will be indicated. If the heating state of the second member is insufficient, the portion of the second member that is insufficiently heated will be indicated. Head-mounted display as described in Feature 14. (Feature 16) The head-mounted display according to feature 14 or 15, wherein the processing circuit further performs a second determination using the extracted second image to determine the uniformity between the heating state of the first member and the heating state of the second member. (Feature 17) The head-mounted display according to feature 16, wherein the display device indicates a member or area to be heated when the heating state of the first member and the heating state of the second member are not uniform. (Feature 18) The processing circuit extracts the edges of the first member and the edges of the second member from a plurality of first images or a plurality of second images. The head-mounted display according to any one of features 14 to 17, wherein the display device displays the edge of the first member and the edge of the second member. (Feature 19) The aforementioned imaging device is A first filter that selectively transmits light included in the first wavelength band, A first image sensor that receives light transmitted through the first filter, A second filter that selectively transmits light included in the second wavelength band, A second image sensor that receives light transmitted through the second filter, A head-mounted display as described in any one of features 14-18, including the above.

[0102] In this specification, "or" indicates that "at least one" of the items listed in the text may be adopted.

[0103] According to the embodiments described above, an analysis method, analysis apparatus, analysis system, program, storage medium, and head-mounted display are provided that can improve the accuracy of image-based brazing analysis.

[0104] Although several embodiments of the present invention have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other. [Explanation of Symbols]

[0105] 1: Analysis system, 10: Imaging device, 11: Optical system, 12: Beam splitter, 13: First filter, 14: First image sensor, 15: Second filter, 16: Second image sensor, 20: Computer, 30: Display device, 41: First component, 42: Second component, 43: Burner, 44: Brazing material, 90: Computer, 91: CPU, 92: ROM, 93: RAM, 94: Storage device, 95: Input interface, 95a: Input device, 96: Output interface, 96a: Output device, 97: Communication interface, 97a: Server, 98: System bus, 100: HMD, 101: Frame, 111,112: Lens, 121,122: Projection device, 131: Image camera, 132: Depth camera, 133: Light source, 134: Eye-tracking camera, 140: Sensor, 141: Microphone, 150: Processing unit, 160: Battery, 170: Memory device, AM: Analysis method, D1: First direction, D2: Second direction, E1, E2: Edge, F1, F2: Flame, IMG1: First image, IMG2: Second image, IMG3: Image, L, L1, L2: Light, M1~M3: Message, P1: First period, R1: First judgment area, R2: Second judgment area, V: Pixel value, W: Worker, av1: First mean value, av2: Second mean value, p1~p9, p11~p19: Peak, th1~th4: Threshold

Claims

1. An analytical method for analyzing the brazing work between a first member and a second member, Computers A first image in the first wavelength band showing the brazing operation is acquired. The brazing operation is captured, and a second image is obtained in a second wavelength band that is longer than the wavelength of the first wavelength band. Based on a plurality of pixel values ​​including the pixel values ​​of the first image and one or more pixel values ​​of the first image acquired in the past, a first period in which disturbances to the analysis occur is determined. Determine whether the timing at which the second image was taken falls within the first period. If the aforementioned timing is not included in the first period, a first determination is performed using the second image to determine whether the heating state of the first member is sufficient and whether the heating state of the second member is sufficient. Analysis method.

2. The analysis method according to claim 1, wherein the computer does not perform the first determination or invalidates the result of the first determination if the timing is included in the first period.

3. The analysis method according to claim 1, wherein the computer further performs a second determination using the second image to determine the uniformity between the heating state of the first member and the heating state of the second member when the timing is not included in the first period.

4. The first wavelength band is selected from the range of 420 nm or more and less than 620 nm. The analytical method according to claim 1, wherein the second wavelength band is selected from within the range of 650 nm or more and less than 950 nm.

5. The difference between the upper and lower limits of the first wavelength band is less than 20 nm. The analytical method according to claim 4, wherein the difference between the upper and lower limits of the second wavelength band is less than 20 nm.

6. The aforementioned computer, A first waveform is obtained that shows the relationship between time and pixel value or the relationship between time and the change in pixel value. In the first waveform, the period during which the pixel value is greater than a threshold or the change in the pixel value is greater than a threshold is determined to be the first period. The analytical method according to claim 1.

7. The aforementioned computer, From the first image or the second image, the edges of the first member and the edges of the second member are extracted. In the first image, the first period is determined using the average value of the pixel values ​​in a part of the first member and the average value of the pixel values ​​in a part of the second member. The analytical method according to claim 1.

8. The aforementioned computer, From the first image or the second image, the edges of the first member and the edges of the second member are extracted. In the second image, the first determination is performed using the average value of the pixel values ​​in a part of the first member and the average value of the pixel values ​​in a part of the second member. The analytical method according to claim 1.

9. Equipped with a computer including processing circuits, An analytical apparatus wherein the processing circuit performs the analytical method according to any one of claims 1 to 8.

10. The analytical apparatus according to claim 9, A camera that acquires the first image and the second image, An analytical system equipped with the following features.

11. The aforementioned imaging device is A first filter that selectively transmits light included in the first wavelength band, A first image sensor that receives light transmitted through the first filter, A second filter that selectively transmits light included in the second wavelength band, A second image sensor that receives light transmitted through the second filter, The analysis system according to claim 10, including the following:

12. A program that causes a computer to perform the analysis method described in any one of claims 1 to 8.

13. A storage medium storing the program described in claim 12.

14. A photographic device that repeatedly acquires a first image in the first wavelength band and a second image in the second wavelength band which has a longer wavelength than the first wavelength band by photographing the brazing process of the first and second members, A display device that shows information to the wearer, A processing circuit is provided, The aforementioned processing circuit is Based on the pixel values ​​of multiple first images, the disturbance time period is determined. From among the multiple second images, extract the second images acquired during time periods other than the disturbance period. Using the extracted second image, a first determination is performed to determine whether the heating state of the first member is sufficient and whether the heating state of the second member is sufficient. The display device is a head-mounted display that displays an alert when the first member is not sufficiently heated or when the second member is not sufficiently heated.

15. The aforementioned display device is If the heating state of the first member is insufficient, the area of ​​the first member that is insufficiently heated will be indicated. If the heating state of the second member is insufficient, the portion of the second member that is insufficiently heated will be indicated. The head-mounted display according to claim 14.

16. The head-mounted display according to claim 14, wherein the processing circuit further performs a second determination using the extracted second image to determine the uniformity between the heating state of the first member and the heating state of the second member.

17. The head-mounted display according to claim 16, wherein the display device indicates a member or location to be heated when the heating state of the first member and the heating state of the second member are not uniform.

18. The processing circuit extracts the edges of the first member and the edges of the second member from a plurality of first images or a plurality of second images. The head-mounted display according to any one of claims 14, wherein the display device displays the edge of the first member and the edge of the second member.

19. The aforementioned imaging device is A first filter that selectively transmits light included in the first wavelength band, A first image sensor that receives light transmitted through the first filter, A second filter that selectively transmits light included in the second wavelength band, A second image sensor that receives light transmitted through the second filter, A head-mounted display according to any one of claims 14 to 18, including the following:

Images