Data preparation device and data preparation method
By performing data augmentation on evaluation raw data to generate expanded data sets, the method addresses the lack of learning data, significantly increasing the amount of training data and enhancing detection accuracy for 3D image models.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
- Filing Date
- 2024-11-05
- Publication Date
- 2026-05-19
AI Technical Summary
Existing methods for generating learning models to detect targets in image data lack sufficient learning data, which hinders accuracy improvement.
A data preparation device and method that performs data augmentation on evaluation raw data to increase the amount of training data by generating expanded data sets, including image data with pasted welding defects, and distributes these data for training and evaluation.
The approach significantly increases the amount of training data and enhances detection accuracy of the learning model, particularly for 3D images, by diversifying the types of welding defects and improving detection efficiency.
Smart Images
Figure 2026081518000001_ABST
Abstract
Description
Technical Field
[0003]
[0001] The present disclosure relates to a data preparation apparatus and a data preparation method for preparing learning data for generating a learning model that detects a detection target for image data.
Background Art
[0002] Patent Document 1 discloses a technique of generating a prediction model using a sample group, generating an objective function using the prediction model, optimizing the generated objective function, and then evaluating the optimization result using a sample group not used for generating the prediction model.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] By the way, in order to generate a learning model that detects a detection target for image data, there are cases where a plurality of sets of learning data each having image data and annotation data indicating the position of the detection target in the image indicated by the image data are used. In such a case, in order to improve the accuracy of the obtained learning model, there is a desire to acquire more learning data.
[0005] The present disclosure has been made in view of such points, and an object thereof is to increase the learning data that can be obtained.
Means for Solving the Problems
[0006] To achieve the above objectives, a first embodiment of the present disclosure is a data preparation device for preparing training data for generating a learning model that detects a target in image data, comprising: a storage unit that stores a plurality of sets of raw data each having image data and annotation data indicating the position of the target in the image shown by the image data, wherein the plurality of sets of raw data include evaluation raw data used to evaluate the learning model; and a data processing unit that acquires the training data by performing data augmentation processing on at least a portion of the evaluation raw data.
[0007] Furthermore, a second embodiment of the present disclosure is a data preparation method for preparing training data for generating a learning model that detects a target in image data, comprising: acquiring a plurality of sets of source data each having image data and annotation data indicating the position of the target in the image shown by the image data; and performing data augmentation processing on at least a portion of the evaluation source data used to evaluate the learning model from the plurality of sets of source data to acquire the training data.
[0008] By performing data augmentation processing on the raw evaluation data, training data can be obtained, which increases the amount of training data that can be acquired compared to when training data is obtained using only the raw data not used for evaluation. [Effects of the Invention]
[0009] According to this disclosure, it is possible to increase the amount of training data that can be acquired. [Brief explanation of the drawing]
[0010] [Figure 1] Figure 1 is a schematic diagram showing the configuration of a learning system equipped with a computer main unit as a data preparation device according to the present disclosure. [Figure 2] Figure 2 is an explanatory diagram illustrating a 3D image based on acquired image data. [Figure 3]Figure 3 is a flowchart showing the operation of the learning system. [Modes for carrying out the invention]
[0011] Embodiments of the present invention will be described below with reference to the drawings. The following description of preferred embodiments is essentially illustrative and is not intended to limit the present invention, its applications, or its uses in any way.
[0012] Figure 1 shows a learning system 1. This learning system 1 comprises a computer main unit 10 as a data preparation device according to the embodiment of this disclosure, an input device 20, and an output device 30. The input device 20 is composed of, for example, a keyboard 21 and a mouse 22. The output device 30 is composed of, for example, a liquid crystal display.
[0013] The computer unit 10 prepares training data and generates a training model using the prepared training data. Specifically, the computer unit 10 comprises a data processing unit 11 and a storage unit 12. The functions of the data processing unit 11 are realized, for example, by a GPU (Graphics Processing Unit).
[0014] The data processing unit 11 includes a data acquisition unit 111, a data expansion unit 112, a data distribution unit 113, a learning unit 114, and an evaluation unit 115.
[0015] The data acquisition unit 111 acquires multiple sets of raw data, each containing acquired image data of a 3D image including the area around the weld, and annotation data corresponding to the acquired image data, and stores them in the storage unit 12. The weld area is the location where a weld mark is formed, for example, a weld bead from arc welding or laser welding, or a dent from a welding gun in resistance welding. The data acquisition unit 111 acquires the acquired image data by receiving it from a 3D sensor (not shown), processing data received from a 3D sensor (not shown), or creating it in response to user input. Hereinafter, "one image data" means the image data for one 3D image. The format of the image data is not limited as long as it represents a 3D image. The image data may be, for example, point cloud data. Also, "number of acquired data" means the number of sets of acquired image data and annotation data. Annotation data specifies the presence or absence of welding defects (flaws) as the target of detection in the 3D image shown by the corresponding image data, the size and position of the rectangular area (bounding box) containing the welding defects in the 3D image shown by the corresponding image data, and the type of welding defect within the rectangular area (e.g., hole, pit, spatter, undercut, protrusion, etc.). Note that annotation data does not need to include information indicating the location of the welding defects (target of detection) in the 3D image shown by the corresponding image data, and is not limited to the data as in this embodiment. Annotation data is identified by user input to the input device 20 while the 3D image based on the acquired image data is output to the output device 30. Specifically, the 3D image is output to the output device 30, the user identifies the rectangular area containing the welding defects within the 3D image by drawing a frame, and assigns a name of the type of welding defect to that rectangular area. Figure 2 shows an example of a 3D image based on acquired image data in a state where welding defects have been identified. In Figure 2, B is the weld bead, W is the workpiece to be welded, and S is the rectangular area containing the welding defects.
[0016] The data expansion unit 112 reads the acquired data obtained by the data acquisition unit 111 from the storage unit 12 and performs data expansion processing on all of the acquired data to obtain expanded data. Expanded data is a pair of expanded image data and annotation data corresponding to the expanded image data. Expanded image data shows a 3D image in which multiple types of welding defects included in the 3D image shown by the acquired image data obtained by the data acquisition unit 111 are pasted onto a good product image in a predetermined ratio. For example, the 3D image shown by the expanded image data has holes, undercuts, pits, spatter, and protrusions pasted on it in a ratio of 1:1:1:1:1. The predetermined ratio is set in advance based on input to the input device 20, etc. Alternatively, the expanded image data may show a 3D image in which multiple types of welding defects included in the 3D image shown by the acquired image data are pasted onto a good product image in a predetermined number. In this case, the predetermined number is set in advance based on input to the input device 20, etc. Hereinafter, "number of expanded data" means the number of pairs of expanded image data and annotation data. A good product image is a 3D image that includes the area around the weld and does not contain any welding defects. A good product image may also be a 3D image represented by one of the acquired image data acquired by the data acquisition unit 111. If the acquired image data acquired by the data acquisition unit 111 does not include image data representing a good product image, a good product image may be acquired by processing the 3D image represented by the acquired image data. The data augmentation process is a process that generates the augmented data. The number of augmented data can be increased by applying one or more of the following processes to the 3D image of the welding defect and pasting it onto the good product image: scaling the vertical and horizontal dimensions by a common or different magnification, rotation, inversion, partial masking, and compositing of two welding defects. The data augmentation unit 112 stores the generated augmented data in the storage unit 12.
[0017] For example, suppose the magnification is 100x, and the 3D image shown by all the acquired image data obtained by the data acquisition unit 111 contains a total of 100 welding defects. In this case, the 3D image shown by all the expanded image data generated by the data expansion unit 112 will contain a total of 10,000 welding defects.
[0018] The data distribution unit 113 reads the acquired data obtained by the data acquisition unit 111 from the storage unit 12, and distributes it into training data to be used for generating a learning model (described later) and evaluation data to be used for evaluating the learning model (described later), and stores them in the storage unit 12. Both the training data and the evaluation data consist of multiple sets of image data and annotation data corresponding to the image data. The data distribution unit 113 distributes the acquired data such that the ratio of the total number of welding defects in the 3D image shown by the training data (image data contained therein) to the total number of welding defects in the 3D image shown by the evaluation data is a predetermined ratio.
[0019] For example, suppose the 3D image represented by all acquired image data acquired by the data acquisition unit 111 contains a total of 100 welding defects, and the predetermined ratio is 7:3. In this case, the data distribution unit 113 distributes the data such that the 3D image represented by all training source data contains a total of 70 welding defects, and the 3D image represented by all evaluation source data contains a total of 30 welding defects. If the data distribution unit 113 cannot make the ratio of the number of welding defects exactly the predetermined ratio, it distributes the data such that the ratio of the number of welding defects is as close to the predetermined ratio as possible.
[0020] The learning unit 114 generates a learning model by performing machine learning using the augmented data acquired by the data augmentation unit 112 and the original learning data distributed by the data distribution unit 113 as learning data. Note that the original evaluation data is not used for generating the learning model. Hereinafter, the number of learning data means the number of pairs of image data and the annotation data corresponding to the image data. The learning model is for detecting a plurality of types of welding defects in the image data. The learning unit 114 is implemented with a known inference engine, for example, YOLO which is an object detection algorithm. The input data of the learning model is image data. The output data of the learning model is for identifying the presence or absence of welding defects in the 3D image based on the image data, the size and position of the rectangular region (bounding box) including the welding defects in the 3D image based on the image data, and the type of welding defects (e.g., holes, pits, spatter, undercut, protrusions, etc.) within the rectangular region. The learning unit 114 stores the parameters specifying the learning model in the storage unit 12.
[0021] For example, assume that a total of 70 welding defects are included in the 3D images represented by all the original learning data, and a total of 10,000 welding defects are included in the 3D images represented by all the augmented image data. In this case, the total number of welding defects included in the 3D images represented by the learning data used for generating the learning model is 10,070.
[0022] The evaluation unit 115 evaluates the learning model generated by the learning unit 114 using the original evaluation data distributed by the data distribution unit 113. Specifically, the evaluation unit 115 inputs the image data included in the original evaluation data into the learning model generated by the learning unit 114. Then, the evaluation unit 115 calculates the total number of correct detections, the total number of false detections, the total number of undetected detections, and the accuracy of the detection position (Intersection over Union) based on the output data of the learning model and the annotation data included in the original evaluation data.
[0023] The positive detection rate, detection rate, and accuracy of the detection position are represented by the following formulas (1) to (3).
[0024] Here, let Pr be the positive detection rate, Re be the detection rate, IoU be the accuracy of the detection position, TP be the total number of positive detections, FP be the total number of false detections, FN be the total number of undetected cases, S1 be the area of the region included in at least one of the rectangular regions specified by the annotation data and the rectangular region specified by the output data of the learning model, and S2 be the area of the region that is the rectangular region specified by the annotation data and is also the rectangular region specified by the output data of the learning model.
[0025] Pr = TP / (TP + FP) ··· (1) Re = TP / (TP + FN) ··· (2) IoU = S2 / S1 ··· (3) Further, the evaluation unit 115 calculates a performance score, which is the sum of the products of three types of evaluation metrics for the learning model and the weights, for a preset weight setting pattern. The three types of evaluation metrics are the positive detection rate, detection rate, and accuracy of the detection position. The weight setting pattern indicates the weight of the positive detection rate, the weight of the detection rate, and the weight of the accuracy of the detection position. Here, let Score be the performance score for the learning model, wp be the weight of the positive detection rate, wr be the weight of the detection rate, and wiou be the weight of the accuracy of the detection position. Then, the performance score is represented by the following formula (4).
[0026] Score = Pr * wp + Re * wr + IoU * wiou ··· (4) When the performance score is greater than or equal to a predetermined threshold, the evaluation unit 115 determines to use the learning model generated by the learning unit 114. When the performance score is less than the predetermined threshold, the evaluation unit 115 determines not to use the learning model generated by the learning unit 114.
[0027] Hereinafter, the operation of the data processing unit 11 will be described with reference to the flowchart of FIG. 3.
[0028] First, in S101, the data acquisition unit 111 acquires multiple sets of acquired 3D image data including the area around the weld and annotation data corresponding to each acquired image data as raw data, and stores them in the storage unit 12. The raw data stored in the storage unit 12 includes training data and evaluation data before sorting. Here, let A be the total number of welding defects included in the 3D image shown by the acquired image data acquired by the data acquisition unit 111.
[0029] Next, in S102, the data augmentation unit 112 reads all the acquired data obtained by the data acquisition unit 111 from the storage unit 12. The data augmentation unit 112 then performs data augmentation processing on all the acquired data to obtain augmented data and stores it in the storage unit 12. In this way, all the acquired data, that is, both the original learning data and the original evaluation data before sorting, are subject to data augmentation processing. The total number of welding defects included in the 3D image shown by the augmented image data is B times the total number of welding defects included in the 3D image shown by the acquired image data, i.e., A*B.
[0030] Next, in S103, the data distribution unit 113 distributes the acquired data obtained in S101 into training data used for generating a learning model by the learning unit 114 and evaluation data used for evaluating the learning model by the evaluation unit 115, and stores them in the storage unit 12. Here, let C:D be the ratio of the total number of welding defects in the 3D image shown by the training data to the total number of welding defects in the 3D image shown by the evaluation data. Then, the total number of welding defects in the 3D image shown by the training data is A*C / (C+D), and the total number of welding defects in the 3D image shown by the evaluation data is A*D / (C+D).
[0031] Next, in S104, the learning unit 114 generates a learning model by performing machine learning using the extended data acquired in S102 and the raw learning data distributed in S103 as learning data. The total number of welding defects in the 3D images shown by the learning data used to generate the learning model is the sum of the total number of welding defects in the 3D images shown by all the raw learning data and the total number of welding defects in the 3D images shown by all the extended image data. In other words, the total number of welding defects in the 3D images shown by the learning data used to generate the learning model is A*B+A*C / (C+D).
[0032] Next, in S105, the evaluation unit 115 evaluates the learning model generated in S104 using the raw evaluation data allocated in S103. Specifically, the evaluation unit 115 calculates the positive detection rate, detection rate, and detection position accuracy (Intersection over Union). Then, the evaluation unit 115 calculates a performance score, which is the sum of the products of the positive detection rate, detection rate, detection position accuracy, and weights, for a pre-set weight setting pattern. If the performance score is above a predetermined threshold, the evaluation unit 115 decides to use the learning model generated by the learning unit 114. If the performance score is below the predetermined threshold, the evaluation unit 115 decides not to use the learning model generated by the learning unit 114. The total number of welding defects included in the 3D image shown by the raw evaluation data used by the evaluation unit 115 to calculate the performance score is A*D / (C+D).
[0033] As described above, in this embodiment, training data is obtained by performing data augmentation processing on the evaluation source data. Therefore, compared to the case where training data is obtained by performing data augmentation processing only on the training source data, the amount of training data that can be obtained can be increased, and the detection accuracy of the learning model can be improved. In addition, since the welding defects included in the source data (raw data) that is subject to data augmentation processing become more diverse, the detection accuracy of the learning model can be improved.
[0034] For example, let's set A to 100, B to 100, and C:D to 7:3. In this case, if data augmentation processing is performed only on the training data, the total number of welding defects in the 3D image shown by the training data will be 7070, which is the sum of A*C / (C+D) and A*B*C / (C+D). In contrast, in the above embodiment, the total number of welding defects in the 3D image shown by the training data will be 10070.
[0035] Furthermore, in the above embodiment, since the images of the training data and the images detected by the training model are 3D images, it becomes easier to detect welding defects and determine the type of welding defect compared to when 2D images are used. Therefore, the training efficiency and detection accuracy can be improved.
[0036] Furthermore, in the above embodiment, data augmentation processing is performed on all of the raw evaluation data stored in the storage unit 12. Therefore, compared to the case where data augmentation processing is performed on only a portion of the raw evaluation data stored in the storage unit 12, the amount of training data that can be acquired can be increased, and the detection accuracy of the learning model can be improved.
[0037] Furthermore, in the above embodiment, data augmentation processing is also performed on the training data. Therefore, compared to the case where data augmentation processing is performed only on the evaluation data, the amount of training data that can be obtained can be increased, and the detection accuracy of the learning model can be improved.
[0038] Furthermore, in the above embodiment, the image data generated by the data augmentation process represents a 3D image in which multiple types of welding defects included in the 3D image shown by the original data are pasted onto a good product image that does not contain the target for detection, in a predetermined ratio. Therefore, by setting the predetermined ratio in a balanced manner, the detection accuracy of the learning model for all welding defects can be improved. Even if the augmented image data represents a 3D image in which multiple types of welding defects included in the 3D image shown by the acquired image data are pasted onto the good product image in a predetermined number, by setting the predetermined number in a balanced manner, the detection accuracy of the learning model for all welding defects can be improved.
[0039] Furthermore, in the above embodiment, since the target of detection is welding defects, the detection accuracy of the learning model for detecting welding defects can be improved.
[0040] In the above embodiment, in S102, the data expansion unit 112 performed the data expansion process on all of the evaluation source data stored in the storage unit 12, but it may also be performed on only a portion of the evaluation source data stored in the storage unit 12.
[0041] Furthermore, although the above embodiment uses welding defects as the target of detection, the present invention can also be applied when the target of detection is something other than welding defects, such as a person.
[0042] Furthermore, in the above embodiment, the multiple types of evaluation indicators used to calculate the performance score in S105 were set to three types: positive detection rate, detection rate, and detection position accuracy. However, as long as there are multiple types of evaluation indicators, there may be two or more types of evaluation indicators. Also, the multiple types of evaluation indicators used to calculate the performance score in S105 may include only one or two of the positive detection rate, detection rate, and detection position accuracy.
[0043] Furthermore, in the above embodiment, in S102, data augmentation processing was performed so that the total number of welding defects included in the 3D image shown by the augmented image data is a predetermined multiple of the total number of welding defects included in the 3D image shown by the acquired image data. However, in S102, data augmentation processing may be performed so that the number of augmented data is a predetermined multiple of the number of acquired data, and in S103, the data may be distributed so that the ratio of the number of training data to the number of evaluation data is a predetermined ratio. In this case, the number of training data used to generate the learning model in S104 will be the sum of the number of training data and the number of augmented data.
[0044] Furthermore, in the above embodiment, the extended image data is a 3D image in which multiple types of welding defects included in the 3D image shown by the acquired image data are attached to the good product image at a predetermined ratio. However, the data extension unit 112 may perform data extension processing so that the extended image data is an image obtained by deforming the 3D image shown by the acquired image data.
[0045] Furthermore, in the above embodiment, the present invention was applied to the generation of a learning model that detects targets in image data representing 3D images. However, the present invention can also be applied to the generation of a learning model that detects targets in image data representing 2D images. [Industrial applicability]
[0046] The data preparation apparatus and method disclosed herein can increase the amount of training data that can be acquired and are useful as a data preparation apparatus and method for preparing training data for generating a learning model that performs detection of targets on image data. [Explanation of Symbols]
[0047] 10. Computer main unit (data preparation device) 11 Data Processing Unit 12 Storage section
Claims
1. A data preparation device for preparing training data to generate a learning model that performs detection of targets on image data, A storage unit that stores multiple sets of source data, each having the aforementioned image data and annotation data indicating the position of the object to be detected within the image shown by the image data, and the multiple sets of source data include evaluation source data used to evaluate the learning model, A data preparation device comprising a data processing unit that acquires the training data by performing data augmentation processing on at least a portion of the evaluation source data.
2. In the data preparation device according to claim 1, The aforementioned image data is a data preparation device that provides 3D image data.
3. In the data preparation device according to claim 1, The data processing unit is a data preparation device that performs the data expansion processing on all of the raw evaluation data.
4. In the data preparation device according to claim 1, The aforementioned multiple sets of source data include the training source data used as training data, The data processing unit is a data preparation device that also performs the data augmentation process on the training source data.
5. In the data preparation device according to claim 1, The aforementioned learning model detects multiple types of the target to be detected. The data augmentation process includes a data preparation device that generates image data showing an image in which the multiple types of detection targets included in the original data are pasted onto an image that does not contain the detection targets, in a predetermined ratio or number.
6. In the data preparation device according to any one of claims 1 to 5, The object to be detected is a data preparation device with welding defects.
7. A data preparation method for preparing training data to generate a learning model that performs detection of targets on image data, Multiple sets of source data are obtained, each containing the aforementioned image data and annotation data indicating the position of the object to be detected within the image shown by the image data. A data preparation method for obtaining the training data by performing data augmentation processing on at least a portion of the evaluation data used to evaluate the learning model from among the multiple sets of original data.