Display control device, display system, welding system, welding method, display control method, and storage medium

Abstract

Provided is a display control device, display system, welding system, welding method, display control method, and storage medium that can improve the quality of a welding operation. A display control device according to an embodiment extracts a first feature from a first image obtained by photographing a welding operation. The display control device causes a display device to display the first feature and guidance related to the welding operation.

Description

Technical Field

[0001] Embodiments of the present invention relate to a display control device, a display system, a welding system, a welding method, a display control method, and a storage medium. Background Technology

[0002] Welding operations are sometimes performed manually. The quality of manually performed operations depends on the operator's experience and knowledge. The aim is to find a technique that can improve the quality of welding operations even for operators lacking experience and knowledge.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-006419 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] The problem to be solved by the present invention is to provide a display control device, a display system, a welding system, a welding method, a display control method, and a storage medium that can improve the quality of welding operations.

[0008] Solution for solving the problem

[0009] The display control device involved in the embodiment extracts a first feature from a first image obtained by capturing a welding operation. The display control device causes a display device to display the first feature and guidance related to the welding operation. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating the structure of the display system involved in the implementation method.

[0011] Figure 2 This is a flowchart illustrating the display control method involved in the implementation method.

[0012] Figure 3 (a) and Figure 3 (b) is a schematic diagram illustrating a specific example of the processing of the display control device involved in the embodiment.

[0013] Figure 4 (a) and Figure 4 (b) is a schematic diagram showing a display example of the display control device according to the embodiment.

[0014] Figure 5 This is a schematic diagram illustrating a display example of a display control device according to the embodiment.

[0015] Figure 6(a) and Figure 6 (b) is a graph representing the learning data.

[0016] Figure 7 (a)~ Figure 7 (c) is a graph representing the learning data.

[0017] Figure 8 (a)~ Figure 8 (c) is an image representing a three-dimensional model of the various elements related to the welding operation. Figure 8 (d) is an image showing a three-dimensional model of the welding operation.

[0018] Figure 9 (a) and Figure 9 (b) is a graph representing the learning data.

[0019] Figure 10 (a)~ Figure 10 (c) is a graph representing the learning data.

[0020] Figure 11 This is a schematic diagram illustrating another display example of the display control device according to the embodiment.

[0021] Figure 12 This is a schematic diagram illustrating a portion of the structure of another display system involved in the implementation.

[0022] Figure 13 It is a schematic diagram representing the hardware structure. Detailed Implementation

[0023] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings.

[0024] The accompanying drawings are schematic or conceptual, and the thickness and width relationships of the parts, as well as the proportions between the parts, may not be the same as in reality. Furthermore, even when representing the same part, the dimensions or proportions may differ depending on the accompanying drawing.

[0025] In this application specification and figures, elements that are the same as those already described are labeled with the same reference numerals and detailed descriptions are omitted where appropriate.

[0026] Figure 1 This is a schematic diagram illustrating the structure of the display system involved in the implementation method.

[0027] like Figure 1 As shown, the display system 1 includes a display control device 10, a storage device 15, a photographic device 21 (first photographic device), a display device 30, and a lighting device 40.

[0028] The camera device 21 captures images of worker W performing welding operations. The camera device 21 can acquire moving images and extract still images from them. The camera device 21 is, for example, a camera including a CMOS image sensor or a CCD image sensor.

[0029] The display control device 10 acquires the captured images. Additionally, the display control device 10 refers to a database related to the welding operation stored in the storage device 15. Based on the image and database data, the display control device 10 outputs guidance related to the welding operation to the display device 30.

[0030] The display device 30 displays guidance based on the output signal from the display control device 10. In the illustrated example, the worker W wears a welding mask H. A camera device 21 and the display device 30 are embedded in the mask H.

[0031] Illumination 40 illuminates the component M to be welded. Illumination 40 includes a light-emitting diode or a lamp. Illumination 40 may also emit a laser. For example, illumination 40 illuminates invisible light. The light illuminating from illumination 40 may also include components other than invisible light (visible light). Infrared light is preferred as invisible light.

[0032] For example, the imaging device 21 receives light transmitted through the optical filter 25. The optical filter 25 can be provided separately from the imaging device 21 or embedded in the imaging device 21. The optical filter 25 selectively transmits light of a specific wavelength. The imaging device 21 generates an image based on the light transmitted through the optical filter 25. By using the optical filter 25, overexposure of the image caused by the emission of the electric arc can be suppressed. As the optical filter 25, a bandpass filter can be used. Alternatively, another optical filter 25 can be further provided in the optical path between the illumination 40 and the component M.

[0033] The wavelength band of the light irradiated from illumination 40 is set according to the wavelength band of the light from transmission optical filter 25. For example, in the case where infrared light is selectively transmitted through optical filter 25, illumination 40 irradiates light containing infrared light. In this case, to reduce power consumption, it is preferable that the intensity of the infrared light component in the light irradiated from illumination 40 is greater than the intensity of other components. By suppressing the intensity of the light components clipped by optical filter 25, the power consumption of illumination 40 can be reduced.

[0034] Welding apparatus 50 is used in welding operations. In welding operations, two or more components are welded together to form a joint. Welding operations performed using welding apparatus 50 include arc welding, laser welding, and brazing (hard brazing). The specific structure of welding apparatus 50 can be appropriately modified according to the welding operation being performed.

[0035] Here, an example of performing arc welding using welding apparatus 50 is described. Specifically, the arc welding is Tungsten Inert Gas (TIG) welding, Metal Inert Gas (MIG) welding, Metal Active Gas (MAG) welding, or carbon dioxide arc welding, etc. In the apparatus shown, MAG welding or MIG welding is performed.

[0036] The welding apparatus 50 includes a welding wire supply unit 51, a welding wire 52, a welding wire coil 53, a welding torch 54, a tube 55, a gas supply unit 56, a gas source 57, and a control device 58.

[0037] The welding wire supply unit 51 has a welding wire coil 53 formed by winding welding wire 52 (consumable electrode). The welding wire supply unit 51 supplies the welding wire 52 of the welding wire coil 53 to the welding gun 54 through the tube 55.

[0038] The gas supply unit 56 is connected to the gas source 57 and supplies the shielding gas stored in the gas source 57 to the welding torch 54. In the illustrated example, the shielding gas is supplied to the welding torch 54 via the welding wire supply unit 51 and the tube 55. As the shielding gas, an inert gas (e.g., argon) can be used. In the case of performing MAG welding, the shielding gas may contain reactive gases such as carbon dioxide in addition to inert gases.

[0039] Welding wire 52 and shielding gas are supplied to welding torch 54. At the tip of welding torch 54, welding wire 52 protrudes toward part M. In addition, welding torch 54 sprays the supplied shielding gas around welding wire 52.

[0040] The control device 58 controls various elements of the welding apparatus 50. Specifically, the control device 58 controls the wire supply speed of the wire supply section 51, the flow rate of the shielding gas in the gas supply section 56, the potential at the tip of the wire 52, the potential of component M, the current flowing through the tip of the wire 52, and the waveform of the current. For example, the control device 58 controls the potentials of component M and wire 52 in such a way that the wire 52 is the positive electrode (+) and component M is the negative electrode (-).

[0041] The control device 58 sets each setting condition to a pre-registered value. The pre-registered setting conditions include the wire supply speed, shielding gas flow rate, voltage and current values ​​during welding, and current waveform.

[0042] Operator W holds the welding torch 54 and brings the tip of the welding wire 52 close to the component M. An electric arc is generated by the potential difference between the component M and the welding wire 52. Operator W moves the welding torch 54 along the specified welding line to weld the object.

[0043] Figure 2This is a flowchart illustrating the display control method involved in the implementation method. Figure 3 (a) and Figure 3 (b) is a schematic diagram illustrating a specific example of the processing of the display control device involved in the embodiment. Figure 4 (a) and Figure 4 (b) is a schematic diagram showing a display example of the display control device according to the embodiment.

[0044] The display control device 10 acquires an image of the welding operation (first image) captured by the photography device 21 (step S1). Figure 3 (a) is an example of an image. Figure 3 In the image IMG shown in (a), components M1 and M2 are being welded. The welding torch 54 is moving in the direction of travel D1. Near the tip of the welding wire 52, due to the heat of the arc, the welding wire 52, component M1, and component M2 melt, forming a molten pool WP. On the opposite side of the molten pool WP, in the opposite direction D2, a weld bead B is formed. The opposite direction D2 is opposite to the direction of travel D1.

[0045] The display control device 10 extracts a first feature from the image (step S2). The first feature indicates the shape or position of an element related to the welding operation. For example, the display control device 10 extracts the first feature through image processing. As image processing, edge detection can be used. Preprocessing such as noise reduction, smoothing, and edge emphasis can also be appropriately performed before edge detection. Alternatively, the display control device 10 can input the image into a learned model. The display control device 10 acquires the first feature output from the model. For example, the model contains a neural network that is pre-learned to output features in response to the input image. Alternatively, image processing can be combined with the model in the acquisition of the first feature. For example, the image obtained by edge detection is input into the model. In this case, the model is pre-learned to output features based on the image containing edges.

[0046] As an example, from Figure 3 Extracted from the image IMG shown in (a) Figure 3 The first feature is shown in (b). The first feature includes features F1a to F1c. Feature F1a shows the outer edge of the welding torch 54. Feature F1b shows the welding wire 52. Feature F1c shows the bevel between components M1 and M2. Features F1a to F1c show the shape and position of the welding wire 52, the welding torch 54, and the bevel. In addition, the outer edge of the molten pool WP, the outer edge of the weld bead B, etc., can also be extracted as features.

[0047] The display control device 10 calculates the first welding conditions based on the first feature (step S3). The first welding conditions, for the welding operation being performed, include the welding torch angle and the bevel position. For example... Figure 3As shown in (a), the welding torch angle is the angle θ of the welding torch 54 relative to the bevel (weld line). The bevel is the gap between parts M1 and M2. The bevel position is the position of the bevel in the longitudinal direction D3. The longitudinal direction D3 is perpendicular to the travel direction D1 and the opposite direction D2, and parallel to the stacking direction of the parts. Alternatively, the bevel position can also be calculated as the position of the bevel on a plane perpendicular to the travel direction D1 and the opposite direction D2.

[0048] The display control device 10 refers to the database (step S4). The database stores data used to guide the generation of welding operations. For example, the database contains a large amount of data accumulated through repeated welding simulations.

[0049] The database stores multiple welding conditions and evaluations for each condition. For example, welding conditions include the torch angle and bevel position. Evaluations are set based on the possible outcomes for each welding condition (the combination of torch angle and bevel position). Welding conditions with a high probability of undesirable results are given a negative evaluation. Welding conditions with a high probability of desirable results are given a positive evaluation. Undesirable results include, for example, the formation of defects or incomplete joints. An example of registering two evaluations is given here, but it is also possible to register three or more evaluations.

[0050] The data in the database is generated using simulation. In the simulation, various welding conditions are set, and the welding conditions are evaluated based on the simulation results. In the simulation, the analysis of the arc-based heat source is combined with the thermal fluid analysis of the molten pool to model the welding phenomenon. Using the model, numerical analysis is performed on the shape of the arc-based heat source, the shape of the molten pool, the situation inside the molten pool, the weld bead shape, etc. From the perspective of welding conditions and the accuracy of evaluation, the simulation using thermal fluid analysis is effective. This simulation is recorded in "Satoshi Sasamune and Masashi Mori, 'Simulation Development of TIG Arc Welding Phenomena', IHI Technical Report, 57(1), 2017", etc.

[0051] Alternatively, the database data can be generated using images of repeated welding operations. For example, multiple operators perform the same welding operation. A photographic device acquires an image of the welding operation. A second feature is extracted from the image. The second feature contains features of the same kind as the first feature. As described above, when the outer edge of the welding torch 54, welding wire 52, bevel, etc., are extracted as the first feature, the second feature also includes these features. Welding conditions are calculated based on the second feature. Furthermore, the welding conditions are evaluated by a user. The user may be an operator or the administrator of the display system 1, etc. The obtained welding conditions and evaluation are registered in the database. These processes can be performed by the display control device 10 or by other processing devices.

[0052] In addition to registering the welding torch angle and bevel position, welding conditions can also include the width of the molten pool WP and the moving speed of the welding torch 54. Features extracted from the image can be appropriately modified based on the welding conditions registered in the database. For example, if the welding conditions include the width of the molten pool WP, features showing the outer edge of the molten pool can be extracted from the image.

[0053] The display control device 10 refers to the evaluation for the first welding condition in the database (step S5). The display control device 10 generates guidance for the operator W based on the first welding condition and its evaluation (step S6). The display control device 10 causes the display device 30 to display the generated guidance (step S7).

[0054] If the evaluation of the first welding condition is positive, the display control device 10 generates guidance to perform welding as is. If the evaluation of the first welding condition is not positive, the display control device 10 generates guidance indicating a more appropriate second welding condition. For example, a welding condition that is most similar to the first welding condition and has a higher evaluation than the first welding condition is extracted from the database and used as the second welding condition.

[0055] Figure 4 (a) represents a case where the evaluation of the first welding condition is positive. Figure 4 (b) represents a case where the evaluation of the first welding condition is not affirmative.

[0056] exist Figure 4 In example (a), as a guide, the message Me1 is shown, prompting the soldering to proceed as is. Figure 4 In example (b), as a guide, the message Me2 that prompts the modification of the welding conditions and the second feature corresponding to the second welding condition are shown. Additionally, in Figure 4 (a) and Figure 4 In (b), the X-direction of the coordinate system displayed in the guide shows a direction parallel to the travel direction D1 and the opposite direction D2. The Y-direction shows a direction parallel to the longitudinal direction D3.

[0057] The second feature includes features F2a to F2c. Feature F2a shows the outer edge of the welding torch 54. Feature F2b shows the welding wire 52. Feature F2c shows the bevel. That is, the second feature includes features corresponding to the first feature. The second feature is displayed in a different manner than the first feature. For example, the color, thickness, or line type of the second feature is different from that of the first feature. Figure 4In example (b), the first feature is shown as a dashed line and the second feature as a solid line. With guidance, the operator W can easily grasp that the welding torch 54 can be moved in a manner that positions the welding torch on the second feature.

[0058] Explain the advantages of the implementation method.

[0059] The display control device 10 causes the display device 30 to display a first feature extracted from the image and guidance related to the welding operation. By displaying the first feature, the operator W can easily grasp the key points of the current welding operation. By referring to the guidance, the operator W can easily determine whether their welding operation is appropriate. In addition, by displaying the guidance together with the first feature, the operator W can specifically understand how to improve their welding operation. Even operators lacking experience or knowledge can improve the quality of manually performed welding operations by referring to the displayed first feature and guidance.

[0060] Display control device 10, such as Figure 4 As shown in (b), in addition to displaying the first feature extracted from the first image, a second feature is also displayed. The second feature corresponds to a more appropriate second welding condition. By displaying the second feature, the operator W can more easily grasp how to improve the welding operation. As a result, the quality of the welding operation can be further improved. The convenience of the display system 1 can be improved.

[0061] The display control device 10 can also generate historical data (log data) related to the welding operation. For example, the display control device 10 records more than one selected from setting conditions, welding conditions, and evaluations by time. The welding conditions are calculated based on images of the welding operation. The evaluations are obtained from a database based on the welding conditions.

[0062] The display control device 10 outputs historical data after the welding operation is completed. For example, the display control device 10 sends historical data to an external server via File Transfer Protocol (FTP). The display control device 10 can also perform database communication and use Open Database Connectivity (ODBC) to insert data into an external database server. The display control device 10 can also output historical data in a specified file format such as Comma Separated Value (CSV) and write it to a recording medium such as flash memory. The display control device 10 can also extract only historical data when the evaluation is not positive.

[0063] Welding defects are sometimes difficult to detect visually. Operators or managers can assess weld quality by referring to historical data, even for defects not visible to the naked eye. Alternatively, historical data can be used to determine the quality of the final weld. By using historical data, even defects invisible to the naked eye can be reflected in the overall quality of the weld.

[0064] The display control device 10 can automatically determine the quality of the joint based on historical data. For example, three evaluation criteria—"Good," "Acceptable," and "No"—are set for various welding conditions. The quality of the joint is determined based on the ratio of the time spent performing the welding condition with an "Acceptable" evaluation to the total welding operation time. A larger ratio results in a smaller ratio for the welding condition with a "Good" evaluation. The higher the ratio, the lower the quality of the joint is determined. Furthermore, even if a welding condition with a "No" evaluation is slightly performed, a low quality is determined for that joint. For example, to avoid performing a "No" welding condition, the display control device 10 may display guidance to perform a "Good" welding condition even when a "Acceptable" welding condition has been performed.

[0065] The display device 30 may be a PC monitor or a projector, but from a convenience point of view, a head-mounted display (HMD) is preferred. The display control device 10 uses mixed reality (MR) technology to display the first feature and guidance overlapping in the real scene.

[0066] Figure 5 This is a schematic diagram illustrating a display example of a display control device according to the embodiment.

[0067] exist Figure 5 In this example, a display device 30 is embedded in the glasses 31 of the HMD. Features F1a-F1c (first features), features F2a-F2c (second features), and message Me2 are displayed on the glasses 31. By using the HMD, the operator W can confirm the first features and receive guidance without moving their line of sight during welding operations. This improves the convenience of the display system 1.

[0068] The construction of the glasses 31, which includes the display device 30, is arbitrary. The glasses 31 can be, for example... Figure 5 The image shown is a single-lens type, but it can also be a dual-lens type. The specific structure of the HMD can be modified appropriately according to the welding operation. The structure of the HMD can be any type, such as goggle type, eyeglass type, windproof type, etc. The HMD can also be... Figure 1 As shown, it is integrated with the mask H. As an HMD, it can also be used with smart glasses.

[0069] In the extraction of the first feature, from an accuracy perspective, a model is preferred. The model is pre-learned through supervised learning. Multiple learning data sets are used in the learning process. Each learning data set contains a learning image and answer data illustrating the features of that learning image. The model learns by outputting answer data in response to the input of the learning image.

[0070] Figure 6 of (a), Figure 6 (b) and Figure 7 (a)~ Figure 7 (c) is a graph representing the learning data.

[0071] Figure 6 (a) is the learning image TI1 input to the model. Figure 6 (b) indicates Figure 6 The learning image TI1 in (a) is characterized by feature F3. Feature F3 comprises features F3a and F3b. Feature F3a shows the outer edge of the molten pool WP. Feature F3b shows the welding wire 52. The model is in response to... Figure 6 The learning image TI1 shown in (a) is used as the input to output. Figure 6 The features shown in (b) are learned in the F3 manner. Furthermore, the taught features are set according to the features desired for the model's output. For example... Figure 3 As shown in (b), when the model outputs features such as the outer edge and bevel of the welding torch 54, the answer data showing these features is used.

[0072] To maximize model accuracy, it is preferable to prepare a large amount of training data. However, preparing training data takes time. To shorten the preparation time, processing training images to generate other training images is an effective method.

[0073] Figure 7 (a)~ Figure 7 (c) is based on Figure 6 Other learning images TI1a to TI1c are generated from the learning image shown in (a). By making... Figure 6 The overall brightness of the learning image TI1 shown in (a) increases, generating... Figure 7 The learning image TI1a is shown in (a). By reducing the overall brightness of the learning image TI1a, a generation is achieved. Figure 7 The learning image TI1b is shown in (b). By adding defect D to the learning image TI1, a... Figure 7 The learning image TI1c is shown in (c). As a target Figure 7 (a)~ Figure 7 The answer data of the learning images TI1a to TI1c of (c) can be used Figure 6Feature F3 of (b).

[0074] By utilizing image processing to generate learning images, it is no longer necessary to photograph actual welding operations to create them. Furthermore, the learning images generated through image processing can share answer data with the original learning images. Therefore, it is possible to increase the amount of learning data while suppressing the increase in the time required to prepare the learning data.

[0075] Figure 8 (a)~ Figure 8 (c) is an image representing a three-dimensional model of the various elements related to the welding operation. Figure 8 (d) is an image showing a three-dimensional model of the welding operation.

[0076] Three-dimensional (3D) models can be used to generate learning data. For example, Figure 8 (a)~ Figure 8 Prepare 3D models Mo1 to Mo3 as shown in (c). 3D model Mo1 shows the component M to be welded. 3D model Mo2 shows the welding wire 52 and the welding torch 54. 3D model Mo3 shows the molten pool WP. The 3D models can be generated, for example, through computer-aided design (CAD). Alternatively, models can be generated by scanning a physical object using a 3D scanner, etc. Figure 8 (a)~ Figure 8 To generate 3D models of the elements shown in (c). Figure 8 The 3D model of the welding operation shown in (d) is Mo4.

[0077] use Figure 8 The 3D model Mo4 shown in (d) is used to perform optical simulation and physical rendering. In the optical simulation, ray tracing is performed from the viewpoint (photographic device 21) and from the illumination 40 to reproduce the behavior of light in the space where the 3D model is configured. Through simulation, physical values ​​(illuminance, brightness, luminance, or chromaticity) at various points of the 3D model are obtained. By rendering the simulation results, an image reproducing the welding operation is obtained.

[0078] Figure 9 of (a), Figure 9 (b) and Figure 10 (a)~ Figure 10 (c) represents an image of the learning data produced through optical simulation.

[0079] Figure 9 (a) is using Figure 8 The learning image TI2 generated from the 3D model Mo4 shown in (d) is shown. Figure 9 (b) is Figure 9The features of the learning image in (a). Feature F4 includes features F4a and F4b. Feature F4a shows the outer edge of the molten pool WP. Feature F4b shows the welding wire 52. The model is adapted to... Figure 9 The learning image TI2 shown in (a) is used as the input to the output. Figure 9 The features shown in (b) are learned in the manner of F4.

[0080] When using optical simulation, variations in the learned image can be easily generated. For example, different learned images can be generated by changing the distance between the welding torch 54 and the viewpoint, or the orientation of the line of sight relative to the welding torch 54. As a specific example, by making the viewpoint farther than the viewpoint of the learned image TI2, variations can be generated. Figure 10 The learning image TI2a is shown in (a). The angle of the viewpoint relative to the state change shown in the learning image TI2a is used to generate... Figure 10 The learning image TI2b is shown in (b). A generation is achieved by making the viewpoint closer to the viewpoint of the learning image TI2. Figure 10 The learning image TI2c is shown in (c).

[0081] Furthermore, the features correspond to the shapes of the elements contained in the 3D model. Therefore, features can be generated using the 3D model. For example, it is possible to use... Figure 8 The 3D model Mo3 shown in (c) is used to generate Figure 9 Feature F4a is shown in (b). It is possible to use Figure 8 The 3D model Mo2 shown in (b) is used to generate feature F4b. As a specific example, the colors of each element are made distinct during rendering. Edge detection is performed on the rendered image. Boundaries between colors are detected as edges. The parameters for edge detection are set based on the colors of the elements adjacent to the boundary to be detected as an edge. According to this method, there is no need for the user to prepare answer data, thus reducing the time required to prepare learning data. Furthermore, image processing such as defect addition can be applied to the rendered image.

[0082] The learning of the aforementioned model and the generation of learning data are performed by the learning device. Either the display control device 10 or other processing devices can be used as the learning device.

[0083] The above describes examples of performing MAG welding or MIG welding. The invention described in the embodiments can also be applied to other welding operations.

[0084] Figure 11 This is a schematic diagram illustrating another display example of the display control device according to the embodiment.

[0085] exist Figure 11In the example, brazing is performed as a welding operation. The filler metal Br is heated and melted by the burner Bu. Alternatively, the filler metal Br is heated and melted by contact with components that have already been heated by the burner Bu. The molten filler metal Br solidifies, thereby joining components M3 and M4. The burner Bu is part of the welding apparatus.

[0086] The photographic device 21 captures the brazing process and acquires an image. The display control device 10 extracts a first feature from the image. The first feature includes features F11a to F11c. Feature F11a shows the outer edge of the burner Bu. Feature F11b shows the outer edge of the component M4. Feature F11c shows the brazing filler metal Br.

[0087] The display control device 10 calculates the first welding conditions based on the extracted first feature. For example, the first welding conditions include the angle between the solder Br and the burner Bu, the relative position of the burner Bu with respect to the component M4, etc. The display control device 10 refers to a database and displays the second feature showing the outer edge of the burner Bu and the message Me3.

[0088] like Figure 11 As shown, the guidance can also include a message Me4 indicating the steps of the welding operation. The instructions indicate the steps of the welding operation. For example, to perform ideal brazing, the steps are displayed when the temperature of component M3 or component M4 reaches an appropriate value. This reduces damage to component M3 or component M4 due to overheating. The brazing filler metal Br flows appropriately between component M3 and component M4, achieving ideal brazing. By displaying instructions at the appropriate time, even inexperienced or unskilled operators can perform welding operations smoothly. This improves the convenience of the display system 1.

[0089] Figure 12 This is a schematic diagram illustrating a portion of the structure of another display system involved in the implementation.

[0090] like Figure 12 As shown, the display system 1 may also include a camera device 22 (second camera device), a camera device 23 (third camera device), and a camera device 24 (fourth camera device). For example, camera devices 21 to 24 are mounted on a mask H worn by the worker W.

[0091] The imaging device 22 is a depth camera that measures the depth of the subject. Depth, in other words, is the distance between the subject and the camera. The display control device 10 adjusts the focus of the imaging device 21 based on the measurement results of the imaging device 22. Alternatively, a depth sensor can be embedded in an image sensor, using a single imaging device that combines the functions of both imaging devices 21 and 22.

[0092] Camera 23 is a camera that captures the eyes of worker W. Camera 24 is a depth camera that measures the depth of worker W's eyes. Display control device 10 tracks worker W's gaze based on data obtained from camera 23 and 24. Alternatively, a depth sensor can be embedded in an image sensor, using a single camera device that incorporates the functions of both camera 23 and 24.

[0093] The display control device 10 calculates the eye position, gaze direction, etc., based on the photographic results from the imaging device 23 and the measurement results from the imaging device 24. Based on these calculations, the display control device 10 calculates the viewpoint in the display device 30. That is, the display control device 10, imaging device 23, and imaging device 24 function as eye trackers. The display control device 10 adjusts the guided display position in the display device 30 according to the calculated viewpoint. Thus, for example, the second feature can be displayed in a more appropriate position. As a result, the quality of the welding operation can be further improved.

[0094] Figure 13 It is a schematic diagram representing the hardware structure.

[0095] The aforementioned display control device 10, control device 58, learning device, and processing device are implemented using either general-purpose or dedicated computers. The functions of the display control device 10, control device 58, learning device, and processing device can be implemented either through the collaboration of multiple computers or through a single computer. For example, the display control device 10, control device 58, learning device, and processing device respectively include... Figure 13 The hardware structure shown.

[0096] Figure 13 The computer 90 shown includes a CPU 91, ROM 92, RAM 93, storage device 94, input interface 95, output interface 96, and communication interface 97.

[0097] ROM 92 stores the programs that control the computer's operations. ROM 92 contains the programs required for the computer to perform the aforementioned processes. RAM 93 functions as a storage area for expanding the programs stored in ROM 92.

[0098] CPU 91 includes processing circuitry. CPU 91 uses RAM 93 as working memory to execute programs stored in at least one of ROM 92 or storage device 94. During program execution, CPU 91 controls various structures via system bus 98 to perform various processes.

[0099] Storage device 94 stores the data required for program execution and the data obtained through program execution. Storage device 94 functions as storage device 35.

[0100] The input interface (I / F) 95 connects the computer 90 to the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0101] Output interface (I / F) 96 connects computer 90 to display device 96a. Output I / F 96 may be, for example, a Digital Visual Interface (DVI) or High-Definition Multimedia Interface (HDMI). CPU 91 can send data to display device 96a via output I / F 96, causing display device 96a to display images.

[0102] Communication interface (I / F) 97 connects server 97a external to computer 90 to computer 90. Communication I / F 97 is, for example, a network card such as a LAN card. CPU 91 can read various data from server 97a via communication I / F 97. For example, photographic devices 21-24 save the acquired data to server 97a.

[0103] Storage device 94 includes one or more selected from Hard Disk Drive (HDD) and Solid State Drive (SSD). Storage device 94 may also be used as storage device 15. Input device 95a includes one or more selected from mouse, keyboard, microphone (voice input), and touchpad. Display device 96a includes one or more selected from monitor and projector. Display device 96a may also be used as display device 30.

[0104] The processing of the various data described above can also be recorded as programs that can be executed by a computer on a disk (floppy disk and hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), semiconductor memory, or other non-transitory computer-readable storage medium.

[0105] For example, information recorded on a recording medium can be read by a computer (or embedded system). The recording format (storage format) on the recording medium is arbitrary. For example, a computer reads a program from the recording medium and executes the instructions described in the program in the CPU based on that program. In a computer, program retrieval (or reading) can also be performed via a network.

[0106] According to the embodiments described above, a display control device and a display system capable of displaying guidance related to welding operations are provided. By using the display control device or display system, the quality of welding operations can be improved. Similarly, by using a welding system equipped with a display system and a welding apparatus, the quality of welding operations can be improved. Furthermore, by using a welding method utilizing a display control device, a display control method that displays guidance, or a program that executes the display control method on a computer, the quality of welding operations can also be improved.

[0107] The above examples illustrate several embodiments of the present invention, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, modifications, etc., can be made without departing from the spirit of the invention. These embodiments or variations thereof are included within the scope or spirit of the invention, and are included within the scope of the invention as described in the claims and its equivalents. Furthermore, the foregoing embodiments can be combined with each other for implementation.

Claims

1. A display control device, wherein, A first feature is extracted from a first image obtained from photographing the welding operation. This first feature shows the outer edge of the welding torch, the welding wire, and the bevel. The first welding condition is calculated based on the first feature, and the first welding condition includes the angle of the welding torch relative to the bevel and the position of the bevel. Referencing the evaluation for the first welding condition in the database, If the evaluation is positive, a first guide is generated containing a first message prompting the welding to proceed as is. The display device displays the first feature and the first guide. If the evaluation is not affirmative, a second guide is generated, which includes a second message prompting a correction of the welding conditions and a second feature corresponding to the second welding conditions, which are welding conditions with a higher evaluation than the first welding conditions. The second feature indicates the outer edge of the welding torch, the welding wire, and the bevel corresponding to the second welding conditions. The display device displays the first feature and the second guide.

2. The display control device according to claim 1, wherein, Generate historical data related to the welding operation. The historical data includes one or more selected from the first welding condition and the evaluation for the first welding condition.

3. The display control device according to claim 1 or 2, wherein, The first feature is extracted using at least one of image processing and feature extraction models.

4. The display control device according to claim 3, wherein, The model was learned using multiple training images. The multiple learning images were generated through optical simulation using a three-dimensional model illustrating the welding operation.

5. A display system comprising: The display control device according to any one of claims 1 to 4; A first photographic device, which captures the first image; and The display device.

6. The display system according to claim 5, wherein, It also has an optical filter. The first photographic device generates the first image based on light transmitted through the optical filter.

7. A welding system comprising: The display system according to claim 5 or 6; and The welding apparatus used in the welding operation.

8. A welding method, wherein, The welding operation is performed while referring to the first guide or the second guide of the display control device according to any one of claims 1 to 4.

9. A display control method that causes a processing device to perform the following steps: A first feature is extracted from a first image obtained from photographing the welding operation. This first feature shows the outer edge of the welding torch, the welding wire, and the bevel. The first welding condition is calculated based on the first feature, and the first welding condition includes the angle of the welding torch relative to the bevel and the position of the bevel. Referencing the evaluation for the first welding condition in the database, If the evaluation is positive, a first guide is generated containing a first message prompting the welding to proceed as is. The display device displays the first feature and the first guide. If the evaluation is not affirmative, a second guide is generated, which includes a second message prompting a correction of the welding conditions and a second feature corresponding to the second welding conditions, which are welding conditions with a higher evaluation than the first welding conditions. The second feature indicates the outer edge of the welding torch, the welding wire, and the bevel corresponding to the second welding conditions. The display device displays the first feature and the second guide.

10. A storage medium storing a program that causes the processing device to perform the display control method according to claim 9.

Images