Training data generation apparatus, training data generation method, and program

Abstract

To reduce workload of a user.SOLUTION: A training data generation apparatus includes: an input control unit which receives inputs of one or more weld images, and feature point information including position information of feature points in each of the weld images and evaluation values indicating the reliability of the position information; a display control unit which displays, on a display unit, the weld images presenting the feature points so as to be edited, the feature point information corresponding to the weld images, and an image file list which is a list of file names and storage locations of the one or more input weld images, all together; an editing unit which receives, on the weld image displayed on the display unit, editing on the feature points from a user, to modify the position information of the feature points; and a generation unit which generates training data including the weld images and true values of the position information of the feature points, the true values of the position information of the feature points being the modified position information of the feature points.SELECTED DRAWING: Figure 2B

Description

[Technical field]

[0001] An embodiment of the present invention relates to a teacher data generating device, a teacher data generating method, and a program. [Background technology]

[0002] There is known a technique for detecting feature points from a welding image, which is an image of a welding target, and controlling welding using the positions of the detected feature points. In detecting feature points from an image, for example, a detection model is used that is trained using training data including a learning image and true values ​​(i.e., ground truth data) of the positions of feature points in the image.

[0003] One possible learning method that reduces the load required for acquiring training data is to use images obtained during operation of a welding system as training data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-182966 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above learning method, for example, when the target of application changes, the accuracy of feature point extraction may decrease, and a huge amount of training data is required to create a new extraction model. Also, for example, specialized programming language skills are required at the technology introduction site to create the extraction model.

[0006] The present embodiment aims to provide a teacher data generation device, a teacher data generation method, and a program that can reduce the workload of a user. [Means for solving the problem]

[0007] A teacher data generation device of an embodiment is a teacher data generation device that generates teacher data used to learn a detection model that inputs a welding image of a welding target and outputs position information of a plurality of feature points in the welding image, and includes an input control unit that accepts input of one or more of the welding images and feature point information including position information of the feature points for each of the welding images and an evaluation value indicating reliability of the position information, a display control unit that displays, in parallel on a display unit, the welding image in which the feature points are indicated in an editable manner, the feature point information corresponding to the welding image, and an image file list that is a list of storage locations and file names of the input one or more welding images, an editing unit that accepts editing of the feature points from a user on the welding image displayed on the display unit and changes the position information of the feature points, and a generation unit that sets the changed position information of the feature points as true values ​​of the position information of the feature points, and generates teacher data including the welding image and the true values ​​of the position information of the feature points. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of a welding system including an information processing device according to an embodiment. [Figure 2A] FIG. 2A is a block diagram showing an example of the configuration of an information processing device. [Figure 2B] FIG. 2B is a block diagram illustrating an example of a functional configuration of the teacher data generation device according to the embodiment. [Diagram 3] FIG. 3 is a flowchart illustrating an example of a learning process in the embodiment. [Figure 4] FIG. 4 is a diagram for explaining an example of a feature point in the embodiment. [Diagram 5] FIG. 5 is a diagram for explaining an example of a feature point in the embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of a feature point in the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of the information collecting process according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of a data structure of an image information file according to the embodiment. [Figure 9] FIG. 9 is a flowchart illustrating an example of a procedure of the teacher data generation process according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a template of an image information file according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a feature point editing screen according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a procedure of the feature point editing process according to the embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 14] FIG. 14 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 15] FIG. 15 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 16] FIG. 16 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a feature point editing screen during feature point editing in the embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of an image file list according to the embodiment. [Figure 20] FIG. 20 is a diagram illustrating an example of a learning model generation screen according to the embodiment. [Figure 21] FIG. 21 is a diagram illustrating an example of a procedure of a learning process of a detection model using training data according to an embodiment. [Figure 22] FIG. 22 is a flowchart illustrating an example of a procedure of a welding control process in the embodiment. [Diagram 23] FIG. 23 is a flowchart illustrating an example of a procedure for an editing process according to the first modification. [Figure 24] FIG. 24 is a flowchart illustrating an example of a procedure for a teacher data generation process according to the second modification. [Diagram 25] FIG. 25 is a flowchart illustrating an example of a procedure for a learning process of a detection model in the second modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Preferred embodiments of a teacher data generating device, a teacher data generating method, and a program will be described in detail below with reference to the accompanying drawings.

[0010] (Embodiment) 1 is a schematic diagram illustrating a configuration of a welding system including an information processing device according to an embodiment. The welding system 1 includes an information processing device 10, a welding device 20, a storage device 30, a PoE (Power over Ethernet) hub 40, and a PLC (Programmable Logic Controller) 50.

[0011] The welding device 20 welds two or more members together to form an integrated body. The welding device 20 performs, for example, arc welding or laser welding. Specific examples of arc welding include Tungsten Inert Gas (TIG) welding, Metal Inert Gas (MIG) welding, Metal Active Gas (MAG) welding, and carbon dioxide gas arc welding. Here, an example in which the welding device 20 performs TIG welding will be mainly described.

[0012] Welding device 20 includes, for example, head 21, arm 22, power supply unit 26a, gas supply unit 26b, wire 23, imaging unit 24, lighting unit 25, and welding control unit 26. Welding control unit 26 includes power supply unit 26a, gas supply unit 26b, and control unit 26c.

[0013] A tungsten electrode 21a is provided on the head 21. The tip of the electrode 21a is exposed from the head 21. For example, the head 21 is attached to a multi-joint arm 22 including a plurality of links. Alternatively, the head 21 may be provided on a welding torch held by a worker.

[0014] The power supply unit 26a is electrically connected to the electrode 21a and the welding object S. A voltage is applied between the electrode 21a and the welding object S by the power supply unit 26a, and an arc discharge occurs. One of the electrode 21a and the welding object S may be set to a common potential (e.g., ground potential), and the power supply unit 26a may control only the potential of the other of the electrode 21a and the welding object S.

[0015] The gas supply unit 26b is connected to the head 21. The gas supply unit 26b supplies an inert gas to the head 21. Alternatively, the gas supply unit 26b may supply a mixed gas of an inert gas and an active gas. The gas supplied to the head 21 is sprayed toward the welding object S from the tip of the head 21 where the electrode 21a is exposed.

[0016] The tip of wire 23 is placed in the space where arc discharge is occurring. The tip of wire 2325 melts due to arc discharge and drips onto the welding object S. The molten wire 23 solidifies, thereby welding the welding object S. Wire 23 is fixed to arm 22, for example, and is automatically fed as melting progresses.

[0017] The imaging unit 24 captures an image of a location where welding is being performed during welding. The imaging unit 24 captures an image of the welding location to obtain a still image. Alternatively, the imaging unit 24 may capture a moving image. The imaging unit 24 captures a still image by cutting out a part of the moving image. The imaging unit 24 is, for example, a camera including a CCD image sensor or a CMOS image sensor.

[0018] The illumination unit 25 illuminates the welding area during welding so that a clearer image can be obtained by the imaging unit 24. If an image that can be used in subsequent processing can be obtained without illuminating the welding area, the illumination unit 25 does not need to be provided.

[0019] The control unit 26c controls the operation of each of the above-mentioned components of the welding device 20. For example, the control unit 26c generates an arc discharge while driving the arm 22, and welds the welding object S in a predetermined direction. The control unit 26c may also control the settings of the imaging unit 24, the lighting unit 25, and the like.

[0020] The PoE hub 40 is a hub that connects the imaging unit 24, the information processing device 10, and the PLC 50. The PLC 50 is connected to the welding control unit 26 (or the control unit 26c) of the welding device 20.

[0021] The image captured by the imaging unit 24 is transmitted to the information processing device 10, for example, via the PoE hub 40. The information processing device 10 receives the image captured by the imaging unit 24 and stores it in the storage device 30 connected to the information processing device 10. For example, the information processing device 10 stores the captured image in the storage device 30 in association with the welding conditions at the time of capturing the image and the capturing conditions at the time of capturing the image.

[0022] The welding conditions include, for example, an applied voltage, a gas flow rate, a current value, a wire feed speed, or a welding speed. The photographing conditions include, for example, settings of the imaging unit 24, such as an exposure time, an aperture, or a sensitivity (ISO). The photographing conditions may include settings of the illumination unit 25. For example, when a pulse current is supplied to the illumination unit 25, the photographing conditions further include a pulse width, a pulse frequency, a duty ratio, or a peak value. Here, when multiple items are listed connected by "or", this means that all of the items may be included, or only some of the items may be included.

[0023] Fig. 2A is a block diagram showing an example of the configuration of the information processing device 10. As shown in Fig. 2A, the information processing device 10 includes an acquisition unit 101, a teacher data generation device 102, a learning unit 103, an output control unit 104, an input unit 105, a display unit 106, and a storage unit 121.

[0024] The input unit 105 is an input device such as a keyboard, a mouse, a touch panel, etc. The display unit 106 is a display device such as a display device.

[0025] The acquiring unit 101 acquires various information used in the information processing device 10. For example, the acquiring unit 101 acquires one or more welding images (which may be still images or moving images) of an object to be welded captured by the imaging unit 2426 of the welding device 20. The acquiring unit 101 also acquires designation of ground truth data that indicates true values ​​of positions of multiple feature points in the welding image.

[0026] The ground truth data is used together with the welding image to machine-train a detection model that detects the positions of specific feature points in an image, which is one of the image recognition techniques (image recognition tasks). That is, one or more pieces of training data including a welding image and true values ​​of the positions of multiple feature points in the welding image are used to train the detection model.

[0027] Therefore, for example, the user specifies the positions of feature points in the image, the number of which is the number of feature points to be detected. The positions of the feature points may be specified by any method, and may include, for example, a method of directly inputting coordinate values ​​on the image, and a method of specifying the positions in the image with a mouse or the like using a graphical user interface separately provided. The acquisition unit 101 acquires ground truth data representing the positions of the feature points specified in this manner.

[0028] The acquisition unit 101 acquires welding images captured by the welding device 20 from the storage device 30. The acquisition unit 101 also extracts feature points of the acquired welding images, and obtains position information (i.e., position coordinates) of the feature points and a score indicating the reliability of the position information. The acquisition unit 101 associates the storage location and file name of the welding image, the position information of the feature points of the welding image, and the score of the position information, and stores them in the storage unit 121 as an image information file 1211. The position information of the feature points of the welding image and the score of the position information are referred to as feature point information.

[0029] That is, the image information file 1211 is data in which the storage location (also referred to as address) of the image file of the welding image is associated with the feature point information of the welding image (i.e., position information and score) for each image file (see FIG. 8). The teacher data generated by the teacher data generating device 102 is the storage location and file name of the image file of the welding image and the true value of the position information, and is saved in the format of the image information file 1211. Details of the image information file 1211 will be described later. The storage location and file name of the image file of the welding image may be referred to as image file information.

[0030] The teacher data generating device 102 edits the welding image and position information of the feature points obtained by the acquiring unit 101, or generates one or more pieces of teacher data from the welding image obtained by the acquiring unit 101. Here, the teacher data is data having a welding image and feature point information including true values ​​(correct answer data) of position information of the feature points in the welding image.

[0031] The learning unit 103 learns a detection model using the teacher data generated by the teacher data generating device 102. The detection model is a model that detects the position of a specific feature point in an input image, which is one of image recognition technologies. The detection model may be a model of any structure, and may be, for example, a model using DarkPose, which is one of the methods using a convolutional neural network. In addition, the learning method by the learning unit 103 may be any method that can be applied to the detection model to be adopted. The learning unit 103 may be provided as a device separate from the information processing device 10. The learning unit 103 is an example of a learning device.

[0032] The output control unit 104 controls the output of various information used in the information processing device 10. For example, the output control unit 104 outputs information (parameters, etc.) related to the learned detection model to the welding device 20. This allows the welding device 20 to use the learned detection model for controlling welding. For example, the control unit 26c inputs an image (welding image) captured by the imaging unit 24 to the detection model, and controls welding using the positions of multiple feature points output by the detection model. In addition, the output control unit 104 performs display control on the display unit 106.

[0033] The storage unit 121 stores various information used in various processes of the information processing device 10. For example, the storage unit 121 stores the image acquired by the acquisition unit 101 and the designation of the correct answer data. In this embodiment, the storage unit 121 stores an image information file 1211. The information processing device 10 may be configured to use the storage device 30 instead of the storage unit 121.

[0034] The memory unit 121 and the storage device 30 can be composed of any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disc Drive), an SSD (Solid State Drive), and an optical disc.

[0035] Next, the teacher data generating device 102 will be described in detail. FIG. 2B is a block diagram showing an example of a functional configuration of the teacher data generating device 102 according to the embodiment. 2B, the teacher data generation device 102 mainly includes an input control unit 1021, an editing unit 1022, a display control unit 1023, and a generation unit 1024. The teacher data generation device 102 is executed as a teacher data generation application (program) by the CPU of the information processing device 10.

[0036] The input control unit 1021 accepts input of one or more welding images and feature point information including position information of feature points for each welding image and a score of the position information. In this embodiment, the input control unit 1021 accepts input of image file information that associates the welding image with the feature point information.

[0037] The display control unit 1023 displays, in parallel on the display unit 106, a welding image with editable feature points, feature point information corresponding to the welding image, and an image file list which is a list of storage locations (sometimes also referred to as addresses) and file names of one or more welding images input by the input control unit 1021.

[0038] The editing unit 1022 accepts editing from the user for changing the position information of the feature points to true values ​​on the welding image displayed on the display unit 106, and changes the position information of the feature points to true values.

[0039] The generating unit 1024 sets the position information of the feature point changed by the editing unit 1022 as the true value of the position information of the feature point, and generates teacher data including the welding image and the true value of the position information of the feature point. In this embodiment, the generating unit 1024 generates, as the teacher data, an image information file in which the welding image, the true value of the position information of the feature point, and a score are associated with each other.

[0040] Next, the detection model learning process performed by the information processing device 10 will be described. FIG. 3 is a flowchart showing an example of the learning process in this embodiment.

[0041] The acquiring unit 101 collects various information (S101). Specifically, the acquiring unit 101 acquires one or more welding images captured by the welding device 20 of the welding system 1 from the welding device 20. The acquiring unit 101 then extracts feature points of each welding image, generates an image information file 1211 in which image file information (storage location and file name of the image file) of the welding image is associated with the feature point information, and stores the image information file 1211 in the storage unit 121.

[0042] Next, the teacher data generation device 102 generates teacher data using the image information file 1211 stored in the storage unit 121 (S102). The teacher data is also stored in the storage unit 121 in the format of the image information file 1211.

[0043] Next, the learning unit 103 uses the teacher data to learn a detection model (S103), and ends the learning process.

[0044] The details of each of the above processes will be further described below. First, an example of the feature points used in welding system 1 will be described. Figs. 4 to 6 are diagrams for explaining examples of the feature points.

[0045] Fig. 4 shows an example of a welding image in which objects 501, 502, 503, and 504 corresponding to the above four objects (OBJ1) to (OBJ4) are superimposed. Fig. 5 shows an example of a welding image in which feature points 601 and 602 corresponding to object 501 (electrode tip) and object 502 (wire tip) are detected from among these objects. Fig. 6 shows an example of a welding image in which feature points 701a, 701b, 702a, and 702b corresponding to object 503 (groove wall surface) and feature points 711a, 711b, 712a, and 722b corresponding to object 504 (molten pool contour) are detected from among these objects.

[0046] In this way, an object for which feature points are to be detected may be represented by one feature point or multiple feature points. For example, as shown in Fig. 5, the electrode tip and the wire tip are each represented by one feature point. Also, as shown in Fig. 6, the two groove walls (left and right groove walls) are each represented by two feature points, and the molten pool contour is represented by four feature points.

[0047] In addition, since strong light is emitted during welding, two welding images as shown in Figures 5 and 6 can be obtained by taking images in two modes with different exposure times. All feature points may be detected from one welding image by taking an image with one type of exposure time. In this embodiment, a weld pool image including an image of a weld pool will be described as an example of a weld image, but an image other than a weld pool image may also be used as the weld image.

[0048] First, the information collection process in S101 of FIG. 3 performed by the welding system 1 and the acquisition unit 101 of the information processing device 10 will be described. FIG. 7 is a flowchart illustrating an example of a procedure of the information collecting process according to the embodiment.

[0049] The control unit 26c of the welding control unit 26 turns on the imaging function and captures a molten pool image by the imaging device 24 (S701). Next, the control unit 26c turns on the storage function and stores the molten pool image captured by the imaging device 24 in the storage device 30 (S702).

[0050] Next, the acquisition unit 101 of the information processing device 10 turns on the feature point extraction function, acquires the molten pool image from the storage device 30, performs image processing on the molten pool image, and extracts position information and scores of feature points in the molten pool image (S703).

[0051] Specifically, the acquisition unit 101 extracts feature points from the molten pool image using an extraction model (not shown), which is a trained model prepared in advance. The acquisition unit 101 also obtains an index of the reliability of the position information of the feature points, i.e., a score that is an index of how confident the extraction model is in extracting the position information, based on the similarity between the teacher data that the detection model has learned so far and the inference data (i.e., the position information of the extracted feature points). For example, if the teacher data that the detection model has learned so far and the inference data are similar, the score will be high, and if the detection model infers data that it has not learned, the score will be low.

[0052] Here, the processes of S701 and S702 are repeatedly executed until each function is stopped by the control unit 26c, and the process of S703 is repeatedly executed until each function is stopped by the acquisition unit 101. The capturing of the molten pool image by S701 and the saving of the molten pool image by S702 are executed at a processing speed of 20 fps, for example. The processes of S701, S702, and S703 may be configured to be executed in parallel. In this case, the order of the processes of S702 and S703 does not matter.

[0053] Next, when the acquisition of the molten pool image and the extraction of the characteristic points are completed for a predetermined amount, the acquisition unit 101 stops the function of extracting the characteristic points (S704).Then, the acquisition unit 101 registers and saves the image file information (i.e., the storage location and file name) of the acquired molten pool image, which is the output result while the characteristic point extraction was functioning, and the extracted characteristic point position information and score in the image information file 1211 (S705).

[0054] 8 is a diagram showing an example of the data structure of the image information file 1211 according to the embodiment. The image information file 1211 is, as an example, a data file in CSV format. However, the format is not limited to this.

[0055] As shown in Fig. 8, the image information file 1211 links the image file with feature point information (i.e., position information and score of the feature point). Here, the image file, as shown in Fig. 8, is image file information, i.e., the storage location (address) of the molten pool image and the file name. In the example of Fig. 8, the electrode and wire are extracted as feature points of the molten pool image, and position information (X coordinate, Y coordinate) and a score are associated and registered for each. Here, since the image information file 1211 links the feature point information and the image file, the storage location of the image file may be different for each image file.

[0056] 7, next, the control unit 26c stops the function of saving the molten pool image (S706). Next, the control unit 26c stops the function of capturing the molten pool image by the imaging device 124 (S707). This ends the information collection process.

[0057] Next, the teacher data generation process by the teacher data generation device 102 in S102 of FIG. 3 will be described. FIG. 9 is a flowchart illustrating an example of a procedure of the teacher data generation process according to the embodiment.

[0058] Here, the teacher data generation device 102 generates teacher data by editing the position information of the feature points of the image information file 1211 generated by the welding system 1. However, this is not limited to this, and teacher data can also be generated by obtaining new position information of the feature points and generating the image information file 1211. In preparation for generating a new image information file 1211 in this way, the teacher data generation device 102 prepares a template of the image information file.

[0059] Fig. 10 is a diagram showing an example of a template of an image information file according to an embodiment. As shown in Fig. 10, the data structure of the template of the image information file is the same as the data structure of the image information file shown in Fig. 8, and no data has been written yet.

[0060] Returning to Fig. 9, the input control unit 1021 of the teacher data generating device 102 accepts input of an image information file 1211 from a user (S901). Next, the display control unit 1023 retrieves the input image information file 1211 from the storage unit 121, references it, reads feature point information registered in the image information file (S902), and identifies the storage location (address) of the image file of the molten pool image registered in the image information file (S903). Then, the display control unit 1023 reads one or more image files of the molten pool image registered in the image information file from the identified storage location (S904).

[0061] Next, the display control unit 1023 links the read feature point information and the image file of the molten pool image according to the image information file 1211 (S905). Then, the display control unit 1023 displays a feature point editing screen of the teacher data generation application on the display unit 106.

[0062] FIG. 11 is a diagram showing an example of a feature point editing screen according to an embodiment. The feature point editing screen has a user interface specialized for editing feature points. As shown in FIG. 11, the feature point editing screen has an image display area 1003, a feature point information area 1001, an image file list area 1002, and a button area 1004. The image display area 1003 is an area for displaying an image such as a molten pool image. The feature point information area 1001 is an area for displaying feature point information. By looking at the feature point information displayed in the feature point information area 1001, the user can know whether the score of the feature point is high or low.

[0063] The image file list area 1002 is an area that displays a list of image file information, i.e., storage locations (addresses) and file names, of one or more molten pool images. The button area 1004 is an area where various buttons are arranged that the user can operate to give instructions.

[0064] The display control unit 1023 displays the feature point information in a feature point information area 1001 of the feature point editing screen shown in Fig. 11 (S906). The display control unit 1023 also displays an image file list of molten pool images in an image file list area 1002 of the feature point editing screen shown in Fig. 11 (S907). When the user designates an image file in the image file list in the image file list area 1002 with a mouse or the like, the display control unit 1023 displays the molten pool image of the designated image file in the image display area 1003, as shown in Fig. 11.

[0065] Next, when the user performs an editing operation on the feature point editing screen to set the position information of the feature point to a true value, the editing unit 1022 accepts the operation and executes an editing process (S908). The details of the editing process will be described later.

[0066] When the editing process is completed, the generation unit 1024 registers the position information of all the edited feature points in the image information file 1211 in association with the molten pool image, and stores the image information file 1211 in the storage unit 121 (S909). This image information file 1211 becomes the training data.

[0067] Next, the editing process in S909 will be described in detail. 12 is a flow chart showing an example of a procedure of feature point editing processing according to an embodiment. When a user selects a desired image file as an edit target from the image file list in the image file list area 1002 of the feature point editing screen, the input control unit 1021 accepts the selection of the image file of the molten pool image to be edited (S1201). Then, the display control unit 1023 displays the molten pool image of the selected image file in the image display area 1003 of the feature point editing screen (S1202). Here, the image file is selected by the user clicking the desired image file with the mouse in the image file list in the image file list area 1002, or by pressing the "Next Image" button (next image) or the "Prev Image" button (previous image) in the button area 1004.

[0068] 13 to 18 are diagrams showing an example of a feature point editing screen during feature point editing in an embodiment. As shown in Fig. 13, when a user selects and instructs the top image file of the image file list in the image file list area 1002, the molten pool image of the selected image file is displayed in the image display area 1003.

[0069] At this time, as described above, the feature point information corresponding to the selected image file is displayed in the feature point information area 1001 of the feature point editing screen. The display control unit 1023 also displays all feature points registered in the image feature point information at the positions indicated by the position information of each of the image feature point information on the molten pool image. In the example of Fig. 13, the points indicated by G0, GM0, G1, G2, GM1, G2, G3, MW0, and MW1 are the feature points.

[0070] When the user moves the position of the feature point by operating the mouse or the like on the molten pool image displayed in the image display area 1003, the input control unit 1021 accepts the editing of the position of the feature point by the mouse operation (S1203). For example, the user can click and grab a desired feature point among the feature points G0, GM0, G1, G2, GM1, G2, G3, MW0, and MW1 on the molten pool image in the image display area 1003 of FIG. 13 with the mouse, and move the position by dragging and dropping the feature point. The editing unit 1022 changes the position information of the feature point to the position to which the user moved the feature point by operating the mouse on the molten pool image. Then, the display control unit 1023 changes the position information of the feature point to the changed position information and displays it in the feature point information area 1001 of the feature point editing screen (S1204).

[0071] When the user completes editing the feature points to be displayed in the selected molten pool image on the feature point editing screen (S1205), the editing unit 1022 links the position information of the edited feature points to the molten pool image (S1206). Here, the editing unit 1022 changes the position information of the feature points but does not change the scores.

[0072] Then, the editing unit 1022 determines whether there are any more image files to be edited (S1207) based on the user's designation of the final image file by mouse operation, etc. If there are any image files to be edited (S1207: Yes), the input control unit 1021 accepts the selection of the next image file from the user (S1208), and the process returns to S1202, and the processes from S1202 to S1206 are repeated.

[0073] For example, the example in Fig. 14 indicates that the user has selected the second image file from the top in the image file list in the image file list area 1002. In this way, as shown in Figs. 15 to 18, image files in the image file list are selected in order by the user, and the editing unit 1022 edits them based on instructions from the user.

[0074] On the other hand, if it is determined in S1207 that there are no more image files to be edited (S1207: No), the editing process ends and the process returns to the caller.

[0075] In this embodiment, the display control unit 1023 has a function of copying and displaying the changed position information of the molten pool image in which the position information of the feature point has been changed, to the position information of the feature point corresponding to the molten pool image to be selected next as the object to be edited in the image file list displayed in the image file list area 1002. Therefore, the feature point of the previously edited position information is displayed on the molten pool image displayed next in the image display area 1003, and the position information of the feature point in the feature point information area 1001 is also changed and displayed.

[0076] Since a large number of molten pool images are captured by the imaging device 24, such as tens of thousands to hundreds of thousands, and are stored, it is very laborious to correct the position information of the feature points of a large number of molten pool images to true values ​​one by one. On the other hand, since the molten pool images are captured continuously by the imaging device 24, the positions of the feature points of the molten pool images are likely to be similar. In some molten pool images, the positions of the feature points may not change. For this reason, the teacher data generation device 102 of this embodiment is provided with a function for copying the positions of such feature points, so that the user does not need to correct the positions of the feature points when editing the position information of the feature points, or can simply fine-tune the positions. For this reason, this embodiment reduces the labor of the user in editing the position information of the feature points.

[0077] The user can turn off the copy function for the position information of this feature point by clicking a CopyOFF button 10041 in the button area 1004 shown in FIG.

[0078] Furthermore, in this embodiment, the display control unit 1023 may be configured to display, in the image file list displayed in the image file list area 1002, image files in which the position information of the feature points has been edited and image files in which the position information of the feature points has not been edited in different display forms. Here, an image file to which the true value of the position information of the feature points has been assigned is referred to as a taught image file, and an image file to which the true value of the position information of the feature points has not yet been assigned is referred to as an untaught image file. The different display forms can be determined arbitrarily. For example, in the image file list, the background color of a taught image file and the background color of an untaught image file can be displayed in different colors.

[0079] Fig. 19 is a diagram showing an example of an image file list according to an embodiment. In the example of Fig. 19, an example is shown in which the background color of the address of a taught image file and the background color of the address of an untaught image file are displayed differently in the image file list. However, the display form is not limited to this.

[0080] Here, the subject of this embodiment is narrow-gap automatic TIG welding, but if the materials (base metal, welding material), welding posture, and groove angle are different, the obtained molten pool image will also change. It is expected that erroneous detection will occur due to differences in the objects being photographed and the influence of disturbances. For this reason, in this embodiment, it is expected that the accuracy rate of the detection model will be improved by identifying molten pool images that are prone to erroneous detection by the detection model, correcting the position information of the feature points, and re-learning.

[0081] That is, to improve the accuracy of the detection model, it is effective to effectively utilize feature points with low scores. For the learning of the detection model, it is more useful to use location information with low reliability that the extraction model of the acquisition unit 101 is not good at, that is, location information of feature points with low scores, as training data for improving the accuracy rate, rather than location information with high scores obtained with high reliability by the extraction model of the acquisition unit 101. For this reason, preferentially selecting, correcting, and learning feature points with location information with low scores leads to an improvement in the accuracy of the detection model.

[0082] For this reason, in this embodiment, the teacher data generating device 102 re-assigns correct position information to the feature points with low scores and adds them to the teacher data. Then, the learning unit 103, which will be described later, re-learns using this teacher data, thereby making it possible to improve the accuracy of the detection model. In the welding system 1, the molten pool image is captured and saved at a processing speed of, for example, 20 fps. The user needs to analyze the image information file 1211 for tens of thousands to hundreds of thousands of molten pool images, pick out feature points with low scores, and edit the position information of the feature points to their true values.

[0083] Therefore, in this embodiment, the image information file 1211 output by the welding system 1 is input to the teacher data generating device 102, and the score is displayed together with the position information of the feature points in the feature point information area 1001 of the feature point editing screen. Therefore, the user can view the scores of the feature points displayed in the feature point information area 1001, select feature points and molten pool images with low scores, and edit the feature point information, so that the user can easily extract image files with low scores from a large number of molten pool images.

[0084] In addition, the storage location and file name of the molten pool image are associated with the feature point information and registered in the image information file 1211, and are also displayed in the image file list area 1002 of the feature point editing screen. This eliminates the need for the user to manage the image files of the molten pool image in the same folder, thereby reducing the user's management work.

[0085] In this embodiment, the training data generating device 102 outputs training data corresponding to the molten pool image, including the changed position information of the feature points as true values ​​in an image information file of the same format. Further, the learning unit 103 inputs this image information file as training data and performs a learning process.

[0086] Next, the details of the learning process of the detection model using the training data in S103 of FIG. 3 will be described. In the learning process, first, the output control unit 104 of the information processing device 10 displays a learning model generation screen on the display unit 106.

[0087] Fig. 20 is a diagram showing an example of a learning model generation screen according to an embodiment. As shown in Fig. 20, the learning model generation screen displays an extraction model selection button 2001, a setting file input field 2002, a loaded image save folder input field 2003, a loaded data file input field 2004, a model file name input field 2005, a model file save destination input field 2006, execution instruction buttons 2007, a loaded number field 2008, a current status display field 2009, and the like.

[0088] In this embodiment, it is assumed that a detection model for extracting electrodes and wires and a detection model for extracting groove walls and molten pool contours are used, and one of the detection models is selected using the selection button 2001 on the learning model generation screen.

[0089] The setting file input field 2002 is a field for inputting a setting file. The setting file is a file in which detailed setting parameters for learning, such as parameters for padding the teacher data and the number of learning rounds, are set.

[0090] The read image save folder input field 2003 is a field for specifying a folder in which an image file of a molten pool image is saved. The read data file input field 2004 is an input field for specifying data in which feature point information (i.e., position information and score of feature points) is registered.

[0091] In this embodiment, as described above, the storage location (address) and file name of the image file of the molten pool image are linked to the feature point information and registered in the image information file 1211. Therefore, the user can load this image information file 1211 and specify it in the data file input field 2004, and the learning unit 103 can identify the storage location of the image file of each molten pool image by referring to the input image information file 1211. Therefore, the user does not need to specify the loaded image storage folder input field 2003.

[0092] A model file save destination input field 2006 is a field for specifying the model name of the detection model to be generated. Each execution instruction button 2007 is a button for instructing by clicking to start learning, suspend / resume learning, or end learning.

[0093] The number of loaded image files field 2008 displays the number of loaded image files, that is, the number of molten pool images. The current status of the learning process is displayed in the current status display field 2009. Examples of the current learning process status include "Waiting" as shown in the example of FIG. 20, as well as "Processing". When processing, the remaining time is also displayed, such as "Processing (remaining time: XXhXXm)". This is to inform the user of the progress, since the time required to generate a detection model varies depending on the number of teacher data and the number of learning rounds, and the operating time of the information processing device 10 may be long.

[0094] In this embodiment, as described above, the user selects the detection model to be extracted using selection button 2001, but the learning model generation screen may be configured to provide a user interface that allows the user to directly input the detection model to be extracted.

[0095] In addition, in this embodiment, detailed parameters during learning are registered in advance in a setting file, and the detailed parameters during learning are set by specifying and reading this setting file in the setting file input field 2002, but this is not limited to this. For example, the learning model generation screen may be configured to provide a user interface for inputting each of the detailed parameters during learning individually.

[0096] FIG. 21 is a diagram illustrating an example of a procedure of a learning process of a detection model using training data according to an embodiment. It is assumed that the user designates the selection button 2001 for the detection model to be extracted and inputs data into each input field on the learning model generation screen shown in FIG. 20, and then clicks the learning start button in the instruction buttons 2007.

[0097] The learning unit 103 first reads (S2101) the selection of the detection model made by the selection button 2001. Next, the learning unit 103 reads the setting file specified in the setting file input field 2002 and reads detailed setting parameters for learning (S2102).

[0098] Next, the learning unit 103 reads the image information file 1211 specified in the read data file input field 2004. Here, the image information file 1211 is specified and read as the teacher data generated by the teacher data generating device 102.

[0099] Then, the learning unit 103 identifies the storage location (address) of the image file from the image information file 1211, reads one or more image files into the storage location, and identifies feature point information linked to the image file in the image information file 1211 (S2103).

[0100] Then, the learning unit 103 starts the learning process (S2104). During the execution of the learning process, the learning unit 103 displays the remaining time of learning on the learning model generation screen (S2105).

[0101] When the learning process is completed (S2106), the learning unit 103 saves the detection model, which is a learned model, in the save destination specified in the model file save destination input field 2006 under the model file name specified in the model file name input field 2005 (S2107). This completes the learning process.

[0102] Next, a welding control process using the detection model by welding device 20 will be described. FIG. 22 is a flowchart illustrating an example of a procedure of a welding control process in the embodiment.

[0103] The imaging unit 24 acquires images of the welding area, for example, in chronological order (S2201). The control unit 26c detects a plurality of feature points from each acquired welding image, for example, using a learned detection model generated by the learning unit 103 of the information processing device 10 (S2202). The control unit 26c controls welding by the welding device 20 based on the detected feature points so as to weld the optimal position (S2203).

[0104] Any method for controlling welding based on feature points may be used. For example, a case where feature points corresponding to the following objects are used will be described as an example. (OBJ1) Tip of electrode 21a (hereinafter, electrode tip) (OBJ2) Tip of wire 23 (hereafter referred to as wire tip) (OBJ3) Groove wall (OBJ4) Molten pool contour

[0105] In this case, the control unit 26c drives the arm 22 by feedback control so that the electrode tip and the wire tip are positioned at the center of the groove wall surface, for example. The control unit 26c also controls the speed at which the electrode tip and the wire tip are moved in a direction along the groove wall surface so that the molten pool contour comes into contact with the groove wall surface.

[0106] In this embodiment, the teacher data generating device 102 includes an input control unit 1021 that receives input of one or more molten pool images, feature point information including position information of feature points for each molten pool image and a score indicating the reliability of the position information, a display control unit 1023 that displays, in parallel on the display unit 106, a molten pool image with editable feature points, feature point information corresponding to the molten pool image, and an image file list that lists the storage locations and file names of the input one or more molten pool images, an editing unit 1022 that receives editing of feature points from a user on the molten pool image displayed on the display unit 106 and changes the position information of the feature points, and a generating unit 1024 that generates teacher data including a molten pool image and the true value of the position information of the feature points by using the changed position information of the feature points as the true value of the position information of the feature points. Therefore, according to this embodiment, it is possible to easily and efficiently generate teacher data used to learn a detection model for use in controlling welding without requiring specialized skills such as programming technology. Therefore, according to this embodiment, it is possible to reduce the workload of a user in generating teacher data.

[0107] In this embodiment, the input control unit 1021 of the teacher data generating device 102 inputs an image information file in which a molten pool image is associated with feature point information, and the generating unit 1024 generates, as teacher data, an image information file in which a molten pool image is associated with feature point information including true values ​​and scores of position information of feature points. Therefore, in this embodiment, image files can be managed in association with feature point information without being stored in a single location, which further reduces the burden on the user of managing image files and feature points.

[0108] In this embodiment, the image information file 1211 is output by the acquisition unit 101 of the welding system 1 and the information processing device 10, which capture an image of a weld target to generate a molten pool image and generate position information and scores of a plurality of feature points in the molten pool image, and is in a format (e.g., CSV format) that can be input to the learning unit 103 that trains a detection model. In this way, in this embodiment, the data handled by the acquisition unit 101, teacher data generation device 102, and learning unit 103 of the welding system 1 and the information processing device 10 is common, so that the efficiency of the teacher data generation process and the learning process can be improved and the workload of the user can be further reduced.

[0109] In addition, in this embodiment, the display control unit 1023 displays the image file list in different display forms for taught image files to which true values ​​of the position information of feature points have been assigned and untaught image files to which no position information of feature points has been assigned. According to this embodiment, the user can easily distinguish between taught image files and untaught image files in the image file list by the difference in their display forms. Therefore, according to this embodiment, the user can easily grasp and select image files to be edited to true values ​​in the image file list in which a list of a large number of image files is displayed, thereby further reducing the workload of the operator.

[0110] In addition, in this embodiment, the display control unit 1023 of the teacher data generating device 102 copies the changed position information of the molten pool image, in which the position information of the feature points has been changed, to the position information of the feature points corresponding to the molten pool image to be edited next in the image file list, and displays it. Therefore, in this embodiment, the position information of the feature points is copied and displayed in consideration of the fact that the positions of the feature points of the molten pool images captured successively are similar, so that the user does not need to correct the positions when editing the position information of the feature points, or only needs to fine-tune the positions. Therefore, according to this embodiment, the editing work of changing the position information of the feature points of a large number of molten pool images, from tens of thousands to hundreds of thousands, to true values ​​by the user is significantly reduced, and the burden of the editing work on the user can be significantly reduced.

[0111] (Variation 1) There are various modifications to the above embodiment. The display control unit 1023 of the teacher data generation device 102 of variant example 1 selects feature point information having a score below a predetermined threshold from the feature point information, and displays the selected feature point information and the molten pool image corresponding to the feature point information on the editing screen as the subject for editing the feature point position information.

[0112] 23 is a flowchart showing an example of the procedure of the editing process of Modification 1. The processes (S901 to S907) from receiving the input of an image information file to displaying the image file list in the image file list area 1002 of the editing screen are performed in the same manner as in the above embodiment.

[0113] Next, the display control unit 1023 selects feature point information having a score below a predetermined threshold from the feature point information (S2301).Then, the display control unit 1023 displays the selected feature point information and the molten pool image corresponding to the feature point information on the editing screen as the editing target for the feature point position information, and the editing process is performed as in the above embodiment (S908).Then, the generation unit 1024 generates teaching data and saves the image information file (S909).

[0114] In order to improve the accuracy of the detection model, the user must select molten pool images with low-score feature point information from a large number of molten pool images, edit the position information to true values, and generate teacher data. However, in this modification, the teacher data generation device 102 selects and displays feature point information with a score below a predetermined threshold, making it easy for the user to edit the selected feature point information with a low score to true values, and the accuracy of the detection model can be easily improved by performing a learning process using the teacher data generated in this way.

[0115] (Variation 2) In the above embodiment, the teacher data generation device 102 inputs an image information file that associates the address of the image file of the molten pool image with feature point information, and edits the position information of the feature points and generates teacher data, but this is not limited to this.

[0116] In addition, in the above embodiment, the learning unit 103 inputs the image information file output by the teacher data generating device 102 and performs learning processing using the teacher data contained in the image information file, but this is not limited to this.

[0117] In the modified example 2, the teacher data generating device 102 performs editing processing and teacher data generation processing by inputting the storage destination (address) of the molten image and the feature point information separately, without using an image information file. Also, the learning unit 103 of the modified example 2 performs learning processing by inputting the storage destination (address) of the molten image and the feature point information separately, without using an image information file.

[0118] Fig. 24 is a flowchart showing an example of a procedure of a teacher data generation process in Modification 2. The input control unit 1021 receives an input of the image information template shown in Fig. 10 from a user (S2401).

[0119] Next, the input control unit 1021 receives an input of a destination (address) for saving the image file from the user (S2402). The input control unit 1021 receives and reads the input of feature point information (S2403). Then, the display control unit 1023 displays the read feature point information in the feature point information area 1001 of the editing screen (S2404).

[0120] Next, the input control unit 1021 reads the image file of the molten pool from the storage destination of the image file input in S2402 (S2405), and the display control unit 1023 displays the image file list in the image file list area 1002 of the editing screen (S2406).

[0121] Then, similarly to the above embodiment, editing processing (S908) and image information file saving processing (S909) are performed.

[0122] FIG. 25 is a flowchart illustrating an example of a procedure for a learning process of a detection model in the second modification. The learning unit 103 performs the processes of selecting a detection model (S2101) and reading setting parameters for learning (S2102) in the same manner as in the above embodiment.

[0123] Next, the user inputs the storage location (address) of the image file in the loaded image storage folder input field 2003 on the learning model generation screen, and the learning unit 103 accepts the input of the storage location of the image file and loads the image file from the storage location (S2301). Here, in this modified example, all image files need to be stored in the storage location specified in S2301.

[0124] Next, the user specifies data of feature point information in the read data file input field 2004, and the learning unit 103 accepts the input and reads the feature point information (S2302). The learning unit 103 then links the image file read in S2301 with the feature point information read in S2302 (S2303). The subsequent processes (S2104 to S2107) are performed in the same manner as in the above embodiment.

[0125] In the above embodiment and modified example, the teacher data generation device 102 is configured to be included in the information processing device 10, but the teacher data generation device 102 and the information processing device 10 may be configured separately.

[0126] The information processing device 10 of the above embodiment and modified examples (and further, the teacher data generating device 102 when the teacher data generating device 102 is provided separately from the information processing device 10) includes a control device such as a CPU, a memory device such as a ROM (Read Only Memory) or RAM, an external memory device such as an HDD, SSD or CD drive device, a display device such as a display device, and input devices such as a keyboard and a mouse, and has a hardware configuration that utilizes a normal computer.

[0127] Each unit of the information processing device 10 according to the above embodiment and modified examples (acquisition unit 101, teacher data generation device 102, learning unit 103, and output control unit 104) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU execute a program, that is, by software. Also, each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0128] The teacher data generation program executed by the teacher data generation device of the above embodiments and variations is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or a Blu-ray (registered trademark) Disc.

[0129] The teacher data generation program executed by the teacher data generation device according to the above embodiment and modification may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The teacher data generation program executed by the teacher data generation device according to the above embodiment and modification may be provided or distributed via a network such as the Internet.

[0130] Furthermore, the teacher data generating program executed by the teacher data generating device according to the above embodiment and modified example may be configured to be provided by being pre-installed in a ROM or the like.

[0131] The teacher data generation program executed in the teacher data generation device of the above-mentioned embodiment and variant examples has a modular structure including each of the above-mentioned units (input control unit 1021, editing unit 1022, display control unit 1023, generation unit 1024), and in terms of actual hardware, a CPU (processor) reads out the teacher data generation program from the above-mentioned storage medium and executes it, thereby loading each of the above-mentioned units onto a main memory device, and the input control unit 1021, editing unit 1022, display control unit 1023, and generation unit 1024 are generated on the main memory device.

[0132] In addition, each of the above units of the teacher data generation device 102 may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0133] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0134] 10. Information processing device 20 Welding equipment 21 Head 21a electrode 22 Arm 23 Wire 24 Imaging unit 25 Lighting Department 26 Welding control unit 26a Power supply section 26b Gas supply section 26c Control unit 30 Storage device 40 PoE Hub 50 PLC 101 Acquisition Department 102 Teacher data generation device 103 Learning Department 104 Output control section 105 Input section 106 Display section 121 Storage section 1021 Input control unit 1022 Editorial Department 1023 Display control unit 1024 Generation part

Claims

1. A training data generation device for generating training data used to learn a detection model that receives a welding image obtained by photographing a welding target and outputs position information of a plurality of feature points in the welding image, comprising: an input control unit that receives input of one or more of the welding images and feature point information including position information of the feature points for each of the welding images and an evaluation value indicating reliability of the position information; a display control unit that displays, in parallel on a display unit, the welding image in which the feature points are indicated in an editable manner, the feature point information corresponding to the welding image, and an image file list that is a list of storage locations and file names of the one or more welding images that have been input; an editing unit that accepts editing of the feature points from a user on the welding image displayed on the display unit and changes position information of the feature points; a generating unit that generates teacher data including the welding image and the true value of the position information of the feature point by setting the changed position information of the feature point as a true value of the position information of the feature point; A teacher data generating device comprising:

2. The input control unit inputs an image information file in which the welding image and the feature point information are associated with each other, The generation unit generates, as the teacher data, the image information file in which the welding image and the feature point information including the true value of the position information of the feature point and the evaluation value are associated with each other. The teacher data generating device according to claim 1 .

3. The image information file is output by a welding system that captures an image of a welding target to generate a welding image, and generates position information of a plurality of feature points in the welding image and the evaluation value, and can be input to a learning device that learns the detection model. The teacher data generating device according to claim 2 .

4. the display control unit displays the image file list in different display forms for the taught image files to which the true values ​​of the position information of the feature points have been assigned and the image files to which the true values ​​of the position information of the feature points have not been assigned. The teacher data generating device according to claim 1 .

5. the display control unit copies and displays the changed position information of the welding image in which the position information of the feature points has been changed to the position information of the feature points corresponding to the welding image to be edited next in the image file list. The teacher data generating device according to claim 1 .

6. the display control unit selects, from the feature point information, first feature point information having the evaluation value equal to or less than a predetermined threshold, and displays, on the display unit, the first feature point information and the welding image corresponding to the first feature point information as editing targets for position information of the feature points. The teacher data generating device according to claim 1 .

7. The welding image includes a weld pool image. The teacher data generating device according to any one of claims 1 to 6.

8. A training data generation method executed by a training data generation device that generates training data used for learning a detection model that receives a welding image obtained by photographing a welding target and outputs position information of a plurality of feature points in the welding image, the training data generation method comprising: receiving an input of one or more of the welding images and feature point information including position information of the feature points for each of the welding images and an evaluation value indicating reliability of the position information; displaying, in parallel on a display unit, the welding image in which the feature points are indicated in an editable manner, the feature point information corresponding to the welding image, and an image file list which is a list of storage locations and file names of the one or more welding images input; accepting editing of the feature points from a user on the welding image displayed on the display unit and changing position information of the feature points; a step of setting the changed position information of the feature point as a true value of the position information of the feature point, and generating teacher data including the welding image and the true value of the position information of the feature point; A method for generating teacher data comprising the steps of:

9. A program to be executed by a computer of a training data generation device that generates training data used for learning a detection model that receives a welding image obtained by photographing a welding target and outputs position information of a plurality of feature points in the welding image, receiving an input of one or more of the welding images and feature point information including position information of the feature points for each of the welding images and an evaluation value indicating reliability of the position information; displaying, in parallel on a display unit, the welding image in which the feature points are indicated in an editable manner, the feature point information corresponding to the welding image, and an image file list which is a list of storage locations and file names of the one or more welding images input; accepting editing of the feature points from a user on the welding image displayed on the display unit and changing position information of the feature points; a step of setting the changed position information of the feature point as a true value of the position information of the feature point, and generating teacher data including the welding image and the true value of the position information of the feature point; A program for causing the computer to execute the above.

Images