Method for creating learning data, data expansion device, and program for supporting creation of learning data

Abstract

To provide a method for creating highly reliable learning data for detecting internal defects of a cast article from a transmission X-ray image using a segmentation method.SOLUTION: A method includes: preparing transmission X-ray image data in which at least a portion of a cast article is mapped; preparing defect image data consisting of a defect object and a bounding box of the defect object; dividing the transmission X-ray image data into multiple small regions based on the size of the bounding box; determining an area in which the defect object can be placed among the multiple small regions; identifying at least one of the small regions in which the defect object can be placed as an area in which the defect object is placed; placing the defect object in the placement area to generate expanded image data; obtaining positional information of the placed defect object in the expanded image data; and creating learning data including the expanded image data and the positional information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for creating learning data, a data augmentation device, and a program for assisting in the creation of learning data, and particularly relates to a technique for creating learning data by data-augmenting a transmission X-ray image in which defects existing inside a cast article are mapped.

Background Art

[0002] Generally, in the manufacture of products, an inspection process is essential as the final process. Since no added value is added to the manufactured products in the inspection process, automation without human intervention is desired. In recent years, learned data has been created by using supervised machine learning or the like, and the quality of actual products has been inferred by an artificial intelligence (AI) using a segmentation method or the like that uses this learned data, thereby automating the inspection process. In order to improve the accuracy of inspection, it is necessary to prepare a sufficient amount of high-quality learning data for machine learning. However, for products with a low production volume or products with an extremely low defect rate, it is difficult to prepare a sufficient number of learning data based on actual inspection data. In such cases, data augmentation (Data Augmentetion) is performed to artificially create new learning data based on a small amount of actual data.

[0003] For example, Non-Patent Document 1 describes data augmentation of an image including a discolored portion (black dots) generated during manufacturing and air bubbles generated by air mixed in during manufacturing in a plastic part for visual inspection. As a data augmentation method, it is described that an image of an actually obtained defective portion is synthesized with a background image obtained by photographing a portion of the target part surface without defects.

[0004] In addition, Non-Patent Document 2 describes data augmentation of an image including chips that fly and adhere during the manufacturing process in a valve body that is a casting part, in which the image is flipped vertically and horizontally, and slightly rotated, moved, and the brightness is changed.

[0005] Further, Patent Document 1 discloses "an image processing apparatus having a learning unit that performs learning on a learning model using a first set of image data, an inference unit that inputs a second set of image data into the learned model learned by the learning unit to output an inference result, and a processing unit that processes a region of interest at the time of inference for the image data in the second set of image data for which the inference result is correct", and describes a technique for avoiding the creation of learning data including regions that are not necessarily important in subsequent inferences with respect to data augmentation.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Documents

[0007]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0008] In the inspection of defects such as harmful shrinkage cavities and inclusions existing inside a cast article, since an inspector with expertise identifies and determines the image taken with transmission X-rays, automation by a segmentation method is desired. The location of defects occurring inside a cast article is greatly influenced by the structure of the cast article (such as wall thickness distribution), and it is necessary to perform data augmentation considering this. Also, since the position of the mapped defect is usually not digitized in a format processable by a computer, the work of digitizing the defect position information separately by manually reading it (annotation) is also necessary.

[0009] The data augmentation disclosed in Non-Patent Document 1 is obtained by artificially synthesizing targeted parts on the surface of the product where defects can occur, but there is no particular description regarding where to place the defect at a position where a defect can occur. Also, although Non-Patent Document 2 targets cast articles, the swarf to be identified flies in from the outside and adheres to the surface. Therefore, in the data augmentation described in Non-Patent Document 2, it is considered that there is no need to consider the structure of the cast article itself when placing the swarf. Also, in both of the data augmentation methods of Non-Patent Document 1 and Non-Patent Document 2, it is described or suggested that annotation is separately performed artificially. Further, the technique disclosed in Patent Document 1 creates a set of image data that does not include unintended learning data, but requires in advance pseudo-image data for learning and a correct label associated with the pseudo-image.

[0010] The problem to be solved by the present invention is to provide a method for easily creating highly reliable learning data in which defects are not arranged in unintended parts of a cast article, a data augmentation device, and a program for assisting in the creation of learning data for performing detection of defects existing inside a cast article from an image of the cast article taken using X-rays by a segmentation method.

Means for Solving the Problem

[0011] A method for creating training data used to detect defects observed in a transmission X-ray image of a cast article, the method comprising: preparing transmission X-ray image data in which at least a part of the cast article is imaged; preparing defect image data comprising a defect object and a bounding box of the defect object; dividing the transmission X-ray image data into a plurality of small regions based on the size of the bounding box; determining a region in which the defect object can be placed among the plurality of small regions; specifying at least one of the small regions in the placeable region as a region for placing the defect object; placing the defect object in the placing region to generate extended image data; obtaining position information of the placed defect object in the extended image data; and creating training data having the extended image data and the position information. This can avoid placing the defect object at an unintended position, and at the same time, in addition to the extended image data, the position information of the placed defect object is also created, omitting the operation of newly obtaining the position information of the defect object placed in the extended image data. Further, the training data obtained by this method is configured as a set of the extended image data and the position information of the defect object placed in this extended image data, so it is easy to be used for machine learning applied later, and learned data by machine learning can be created efficiently.

[0012] Further, in the determining step, it is preferable to obtain a feature amount based on the brightness of the pixels constituting the small region for each small region, and determine the small region in which the feature amount satisfies a predetermined condition as the placeable region. This can strongly avoid placing the defect object in a part where defects are unlikely to occur or a part where no defects can physically occur.

[0013] Further, the feature amount is based on the %tile value of the brightness of the pixels constituting the small region and the average value of the brightness of the pixels constituting the small region, and the predetermined condition is preferably based on the difference between two different %tile values among the %tile values and the average value. This makes it possible to obtain learning data in which defective objects are further prevented from being arranged in parts where defects are less likely to occur or in parts where physical defects cannot occur at all.

[0014] Further, in the specifying step, it is preferable to specify a plurality of the small regions that are not adjacent to each other among the arrangeable regions. This can prevent unnatural learning data from being created in which a plurality of the same or similar defects are biased to a specific part.

[0015] Further, the generating step preferably further includes a step of adjusting the brightness of the pixels constituting the defective object according to the feature amount of the arranging region. This makes it possible to obtain learning data closer to a more natural transmission X-ray image.

[0016] Further, the generating step preferably further includes a step of applying at least one of rotation, enlargement, or reduction to the defective object. This makes it possible to obtain learning data in which various defective objects are arranged based on one defective object.

[0017] A second embodiment of the present invention is a data augmentation device for creating learning data used to detect defects observed in a transmission X-ray image of a cast article, the device comprising: an acquisition unit that acquires transmission X-ray image data in which at least a part of the cast article is imaged, and defect image data composed of a defect object and a bounding box of the defect object; a division unit that divides the transmission X-ray image data acquired by the acquisition unit into a plurality of small regions; a determination unit that obtains a feature amount based on the brightness of pixels constituting the small regions divided by the division unit for each small region, and determines the small regions in which the feature amount satisfies a predetermined condition; a specification unit that specifies at least one of the small regions determined by the determination unit; a generation unit that generates augmented image data by arranging the defect object in the small region specified by the specification unit; a creation unit that acquires position information of the arranged defect object in the augmented image data generated by the generation unit, and creates learning data having the augmented image data and the position information; a storage unit that stores and reads out the learning data created by the creation unit; and an output unit that outputs the learning data read out from the storage unit. By this means, the method for creating learning data according to the first embodiment of the present invention can be efficiently performed.

[0018] A program for assisting in creating learning data used to detect defects observed in a transmission X-ray image of a cast article, the program including steps of: dividing transmission X-ray image data in which at least a part of the cast article is imaged into a plurality of small regions based on the size of a bounding box of defect image data composed of a defect object and the bounding box of the defect object; determining regions in the plurality of small regions where the defect object can be placed; specifying at least one of the small regions in the placeable regions as a region for placing the defect object; placing the defect object in the placing region to generate extended image data; acquiring position information of the defect object in the extended image data; and creating learning data having the extended image data and the position information A program that causes a computer to execute the steps above.

Advantages of the Invention

[0019] The present invention provides a method for easily creating highly reliable learning data in which the placement of defects on unintended parts of a cast article is avoided, a data augmentation device, and a program for assisting in creating learning data, for obtaining learning data for performing defect detection on the inside of a cast article existing in an image of the cast article taken using X-rays by a segmentation method. As a result, a large number of high-quality learning data can be easily and efficiently created. Furthermore, by training a dataset consisting of the learning data created according to the present invention, learned data capable of highly accurate inference can be obtained.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, the characteristics of the transmission X-ray image mainly targeted by the present invention will be described, and then the embodiments for carrying out the present invention will be described. Note that each of these embodiments is an example for explaining the present invention and is not limited to the described configuration.

[0022] (Transmission X-ray Image and Defect Image of Casting) FIG. 1 is an example of a transmission X-ray image 901 in which a part of a casting article 911 is imaged. The transmission X-ray image 901 is a grayscale image formed by a collection of 256 gradation pixels having 0 to 255 gradations in ascending order of lightness. The lightness (hereinafter referred to as lightness) of the pixel 0 (zero) is the darkest color (black), and the lightness 255 is the brightest color (white).

[0023] The dark part 912 in the cast article 911 is a relatively thick part through which X-rays are less likely to pass, and is imaged with relatively low brightness. On the other hand, the bright part 913 is a part with a relatively small thickness through which X-rays are likely to pass. Also, in the example shown in FIG. 1, a defect 921 (inclusion in this example) that exists inside the cast article 911 and has particularly high brightness is observed. Since the defect 921 (inclusion) has a lower density than the metal material constituting the cast article 911, it is likely to transmit X-rays, and its brightness is mapped to be higher than that of the surroundings. Also, although not illustrated, defects (shrinkage cavities) that are minute cavities inside the cast article caused by solidification shrinkage during casting are similarly mapped with high brightness. The purpose of obtaining such a transmitted X-ray image 901 is to non-destructively identify the defect 921 existing inside the cast article 911 and eliminate defective products. However, the defect 921 to be identified is generally imaged by adjusting the intensity of the X-rays so that its brightness is as high and clear as possible compared to its surroundings. For example, when trying to identify a defect existing in a thick part, in order to obtain an image in which the brightness of the surroundings is also increased so that the brightness of the defect part becomes high, the thin part that is not targeted as the defect identification area becomes an image with higher brightness (closer to white). That is, the magnitude of the brightness of the pixels constituting the cast article in the transmitted X-ray image represents the relative wall thickness distribution of the cast article, not the absolute value of the wall thickness.

[0024] In the example of FIG. 1, an image is illustrated in which the higher the X-ray transmittance, the higher the brightness. However, for example, when using an X-ray sensitive film, the higher the X-ray transmittance of a part, the lower the brightness of the imaging, so the defect is mapped darker than the surrounding part. Therefore, when digitizing a transmitted X-ray film image and applying it to the present invention, the light and dark (black and white) are in an inverted state. However, since it can be treated in the same way in the present invention, the explanation for the case where the light and dark are inverted is omitted.

[0025] Next, some embodiments of the present invention will be described with reference to the drawings. In each of the drawings shown below, the same reference numerals are assigned to the same components, and detailed descriptions of the components described later may be omitted.

[0026] [1] First Embodiment: Method for Creating Learning Data FIG. 2 is a flowchart showing an example of a method for creating learning data according to the first embodiment of the present invention. Each step described in FIG. 2 will be explained while appropriately referring to FIGS. 3 to 6 as well.

[0027] (1) Step of preparing transmission X-ray image data (S1) First, transmission X-ray image data is prepared (S1, which may also be referred to as the S1 step hereinafter. The same applies to the other steps hereinafter). FIG. 3 shows a schematic diagram illustrating an example of transmission X-ray image data. The transmission X-ray image data 1 dealt with in the present invention is an image in which at least a part of the cast article 11 is imaged, but as shown in FIG. 3, it may be an image in which the entire cast article 11 is imaged. The transmission X-ray data 1 is imaged in grayscale, which is a set of pixels (picture elements) having, for example, 256 gradations in the range of 0 to 255 gradations in ascending order of lightness. The cast article 11 illustrated in FIG. 3 has portions 111 to 114 in descending order of wall thickness in the incident direction of the transmission X-ray, and the portion 115 has an opening structure. Therefore, the lightness is mapped to gradually increase in this order, and the opened portion 115 has the maximum lightness (white), similar to the background 12 where the cast article 11 is not imaged. Usually, the position and lightness of each pixel of the transmission X-ray image data 1 are generally acquired or stored in the form of digital data. Therefore, hereinafter, the transmission X-ray image data 1 includes the form visualized as an image as shown in FIG. 3.

[0028] In general, the transmission X-ray image data 1 is obtained from an imaging device, a data server, etc. in the form of digital data of the captured image. However, it may also be obtained by photographing an analog image captured on an X-ray sensitive film with a digital camera or scanning it with an image scanner or the like to convert it into digital data. Further, it is preferable to preliminarily include coordinates for defining position information as annotation information in the transmission X-ray image data 1. For example, the position of the pixels constituting the image can be defined by X-Y coordinates with the point O at the upper left corner of the transmission X-ray image data 1 as the origin.

[0029] (2) Step of preparing defect image data (S2) In addition, defect image data is prepared (S2). FIG. 4 is a schematic diagram showing an example of defect image data. The defect image data 2 is composed of a defect object 21 that forms the main body of the defect image that is the target of the extended data in the present invention, and a bounding box 22 that surrounds the defect object 21. The defect image data 2 is generally obtained by cutting out or copying the periphery including the main body of the target defect image in a transmission X-ray image (not shown) on which the defect image is mapped. At this time, the image of the main body of the defect to be obtained corresponds to the defect object 21, and the rectangular boundary for designating the cutting range corresponds to the bounding box 22. Therefore, the size of the bounding box 22 is set such that the contour 211 of the defect object 21 fits inside the bounding box 22.

[0030] As described above, the defective image data 2 may be obtained by mapping the transmission X-ray image data itself on which the target casting article is mapped, or may be obtained from what is mapped on a transmission X-ray image on which a casting article different from the target casting article is mapped, or may be artificially generated by a generative AI or the like. Since the defective object 21 is arranged in the transmission X-ray image data 1 (see FIG. 3) and used for creating the enlarged image data 4 (see FIG. 6), it is an image of the same grayscale as the transmission X-ray image data 1, or is applied as an image that has been subjected to conversion processing to grayscale. Also, similar to the transmission X-ray image data 1, combining the form of digital data and the form visualized as an image, hereinafter it is referred to as defective image data 2.

[0031] In addition, it is preferable that the defective image data 2 includes not only the brightness of the pixels constituting the defective object 21, but also information defining the relative position and shape with respect to the bounding box 22 of the defective object 21. The information on the relative position with respect to the bounding box 22 of the defective object 21 can be defined, for example, by giving the coordinates corresponding to the positions of any two or more different pixels constituting the defective object 21 when defining the x-y coordinates with the point P at the upper left corner of the rectangular bounding box 22 as the origin. Also, the shape of the defective object 21 can be defined by giving the coordinates of all or a plurality of arbitrary pixels corresponding to the contour 211 of the defective object 21 with respect to the origin P. In this way, by giving the information for defining the position and shape of the defective object 21 together, it is possible to easily obtain the position information of the defective object 21 arranged and enlarged in the transmission X-ray image data 1 in a later step, and the information on the shape after deformation when the defective object 21 itself is deformed such as rotation or reduction, which is preferable.

[0032] (3) Step of dividing into a plurality of small regions (S3) Next, based on the size of the bounding box 22 that constitutes the defect image data 2 obtained in step S2, the transmission X-ray image data 1 obtained in step S1 is divided into a plurality of small regions 3 (S3). An example of dividing the transmission X-ray image data 1 into small parts according to the size of the bounding box 22 is shown in FIG. 5. In FIG. 5, each of the divided parts is collectively referred to as a small region 3. Note that FIG. 5 shows an example of a form in which the entire transmission X-ray image data 1 is divided into small regions 3 without omission or overlap, but a form in which these small regions 3 partially overlap (not shown) may also be used, or a form in which there are small regions (not shown) formed by combining two or more bounding boxes 2 may also be used, or a form in which there are partially existing parts that are not divided by the small regions 3 (omitted parts) may also be used.

[0033] (4) Step of determining an area where placement is possible (S4) Next, among the plurality of small regions 3 divided in step S3, an area 31 where the defect object 2 (see FIG. 4) can be placed is determined (S4). Here, being able to place the defect object 2 means a small region 3 corresponding to a part where a defect is likely to occur or there is a possibility of a defect occurring in an actual cast article. For example, in FIG. 5, there are three small regions 31 shown (there may be other areas where the defect object 2 can be placed, but they are omitted for simplicity of explanation). In other words, small regions 3 corresponding to parts where a defect cannot physically occur or the possibility of a defect occurring is extremely low are not the targets for placing the defect object 21. In FIG. 5, two examples of small regions 3 that are not the placement targets are illustrated. One of them is a small region 32 at a part 115 which is an opening of the cast article 11, and the other is a small region 32 at the outer edge of the cast article 11 near the background 12. That is, step S4 is equivalent to determining small regions 32 where the defect object 21 is not placed. In this way, it is preferable to determine and exclude non-placement areas, that is, to determine in advance the area 31 where placement is possible, in order to obtain highly reliable learning data.

[0034] Generally, it is known that a major reason for defects occurring inside a cast article is due to the wall thickness variation of the cast article (the difference in wall thickness between the relevant part and its peripheral part). Therefore, it is preferable that the area 31 where it is possible to arrange the defect object 21 in FIG. 5 is determined in consideration of this fact. Also, of course, the small area 33 located in the background 12 where the cast article 11 in the transmission X-ray image data 1 is not mapped should not be the target for arranging the defect object 21. As described above, in a cast article imaged by transmission X-rays, a part with a relatively large wall thickness exhibits a small brightness, and a part with a small wall thickness exhibits a large brightness. Therefore, it is possible to determine the area 31 where the defect object 21 can be arranged based on the brightness distribution of the pixels constituting the transmission X-ray image data 1.

[0035] The main part of the present invention is to determine the area 31 where the defect object 21 can be arranged based on the information obtained from the transmission X-ray image data 1 in which the cast article 11 is mapped. As a criterion for determining the area 31 where it is possible to arrange, it is preferable to obtain a feature amount based on the brightness of the pixels constituting the small area 3 for each small area 3 and determine a small area 3 such that the feature amount satisfies a predetermined condition. By using such a predetermined condition that depends only on the feature amount based only on the brightness of the pixels constituting the transmission X-ray image data 1 as a determination criterion, it is efficient because it is possible to determine the area 31 where the defect object 21 can be arranged only with the transmission X-ray image data 11. For example, as the feature amount, there are the %tile value of the brightness of the pixels constituting the small area 3, the average value of the brightness of the pixels constituting the small area 3, and the like. Here, the %tile value means, for example, when the brightness of the pixels is arranged in ascending or descending order, the value located at n% of the whole when counted from the beginning is called the n%tile value. Also, as a predetermined condition, for example, there is a case where the difference between two %tile values of the brightness of the pixels constituting the small area 3 and the average value of the brightness of the pixels constituting the small area 3 are each within a predetermined range. More specifically, for example, by setting the average value in the range of 50 to 200 as a predetermined condition, it is possible to narrow down to the small area 3 where a defect may exist. Further, in combination with this, if the difference between the 25%tile value and the 75%tile value is also within 20 as a predetermined condition, it is possible to exclude the small area 3 with a large wall thickness variation where a defect is unlikely to occur.

[0036] (5) Step of specifying the area to be configured (S5) Next, the area to be actually arranged among the areas 31 where the defective object determined in step S4 can be arranged is specified (S5). Since there are usually a plurality of areas 31 where the defective object can be arranged as exemplified in FIG. 5, at least one of these is specified as the area to be arranged. The specific method may be arbitrary. For example, the areas 31 where the defective object can be arranged may be numbered with different numbers and specified using a random number table, or random numbers may be generated using a computer or the like, and the small area 31 numbered based on this number may be specified. When specifying mechanically and randomly in this way, compared with the case of artificial specification, the bias in the arrangement position of the defective object among the plurality of learning data obtained is reduced, so it can be expected that a more reliable set of learning data can be easily obtained. Details will be described later, and a schematic diagram showing an example of the extended image data is shown in FIG. 6. The area 311 indicated by the broken line in FIG. 6 is an example in which one of the three areas 31 where the defective object 21 can be arranged shown in FIG. 5 is specified as the area 311 for arranging the defective object 21.

[0037] When specifying two or more small areas 31, it is preferable to specify non-adjacent small areas 31. This is based on the technical knowledge that in actual cast articles, defects of the same form are rarely close to each other. That is, this operation makes it difficult for the bias in the arrangement of the defective object to occur in one piece of learning data, and reliable learning data can be obtained. Further, it may be specified in consideration of a predetermined weight based on the feature amount and predetermined conditions used for determination as the area 31 where the defective object can be arranged. By attaching such a weight, it can be expected to obtain more reliable learning data that emphasizes the possibility of occurring in actual cast articles.

[0038] (6) Step of generating extended image data (S6) Next, a defective object 21 is placed in the area 31 specified in step S5 to generate extended image data (S6). FIG. 6 is a schematic diagram showing an example of the extended image data according to the present invention. As shown in FIGS. 4 and 5, since it is divided into small areas 3 based on the size of the bounding box 22 that constitutes the defective image data 2, when placing the other defective object 21 that constitutes the defective image data 2 in the small area 311, it is preferable to place it at a position similar to the relative position with respect to the bounding box 21 because it is easy. That is, since the specified area 311 in FIG. 6 has the same size as the bounding box 22, the defective image data 2 is placed at the position of this specified area 311. Further, when a small area (not shown) formed by combining two or more bounding boxes 22 is specified as the area to be placed, or when an area (not shown) in which the small areas 3 partially overlap is specified as the area to be placed, in any case, the defective object 21 may be placed at any location within the specified placement area. In addition, when placing the defective object 21, the defective object 21 may be overwritten (composited) on the transmission X-ray image data 1. The extended image data 4 referred to in the present invention refers to new image data generated by placing the defective object 21 in this way.

[0039] Also, in the step of generating the extended image data 4, it is preferable to adjust the brightness of the pixels constituting the defective object 21 according to the feature amount of the area 311 where it is to be arranged. Since the actual defect is a cavity or has a low density relative to its surroundings, the defect actually mapped onto the transmission X-ray image is mapped with a higher (brighter) brightness in the direction of the transmitted X-ray. On the other hand, since the X-ray is less likely to penetrate around the mapped defect than the defect, its brightness is mapped lower (darker) than the brightness of the defect. Therefore, the defective object 21 arranged in step S6 is preferably brighter than the surroundings of the position where it is arranged. Also, in view of the object of the present invention to obtain learning data that enables defects that are difficult to detect visually to be identified by segmentation, it is preferable to obtain extended image data 4 in which the difference in brightness between the defective object 21 and the surrounding brightness is small. Therefore, it is preferable to appropriately adjust the brightness of the defective object 21 obtained in step S2, that is, the brightness of the pixels constituting the defective object 21, according to the brightness of the area 311 where it is to be arranged.

[0040] Also, in step S6, it is preferable to further apply at least one of rotation, enlargement, or reduction to the defective object 21 because various learning data can be obtained. It is preferable to perform these deformations before arranging the defective object 21 in the area 311 where it is to be arranged because the step of obtaining the position information (S7) described later becomes efficient. These deformations may be input by the operator to the computer each time, or may be automatically executed by the computer according to a predetermined deformation rule. Note that the defective object (not shown) to which these deformations are applied is preferably contained inside the bounding box 22. Also, it is preferable to acquire in advance information defining the relative position and shape of the defective object to which these deformations are applied with respect to the bounding box 22. The method of acquiring the information on the relative position of the defective object and the shape of the defective object may be the same as in step S2 described above.

[0041] (7) Step of obtaining position information (S7) Furthermore, obtain the position information of the defect object 21 (see FIG. 6) arranged in the extended image data 4 obtained in step S6 (S7). The position information of the defect object 21 may be obtained as text data in the form of the position coordinates (X1, Y1), (X2, Y2),... of two or more pixels constituting the arranged defect object 21, for example, in the X-Y coordinates of the origin O. Also, as described in the explanation of step S2, it is preferable to previously obtain the relative position of the defect object 21 with respect to the bounding box 22 shown in FIG. 4 as text data such as coordinates (x1, y1),... with respect to the origin P. In this case, by specifying the position on the X-Y coordinates of the specified region 311 indicated by the broken line with respect to the origin O (see FIG. 6) of the extended image data 4, the position information of the arranged defect object 21 is also uniquely specified. For this reason, it becomes possible to obtain the position information of the defect object 21 arranged in step S6 when step S5 is completed, and it is preferable because the timing of obtaining the position information of the defect object 21 can be advanced.

[0042] (8) Step of creating learning data (S8) Then, create learning data having the extended image data 4 obtained in step S6 and the position information of the defect object 21 obtained in step S7 (S8). This learning data has a structure including at least the extended image data 4 and the position information of the defect object 21 arranged in the extended image data 4. By preparing a set of learning data consisting of a sufficient number of learning data having this structure and performing supervised machine learning, it is possible to efficiently obtain learned data for detecting defects observed in the transmission X-ray image of the casting article by the segmentation method. The position information of the defect object 21 may be in the form of text data as described above. Since the created learning data is usually stored in a storage device, step S8 also includes a step of storing the learning data. The storage here includes not only the form of electromagnetic recording and storage on a storage medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium, but also the form of recording and storage on a paper medium or the like.

[0043] [2] Second Embodiment: Data Expansion Device Next, with reference to FIGS. 7 and 9, an example of a data expansion device according to the second embodiment of the present invention will be described. FIG. 7 is a schematic diagram showing an example of the configuration of the data expansion device according to the second embodiment of the present invention. FIG. 9 is a schematic diagram showing an example of a data expansion system including the data expansion device 5 of the present invention described in FIG. 7.

[0044] First, the configuration of the data expansion device 5 will be described with reference to FIG. 7. The data expansion device 5 is composed of a main body 50 as its main part, an input device 52 directly connected to the main body 50, a display 53, and the like. Specifically, it is a computer, and for example, in addition to electronic devices such as workstations (WS) and personal computers (PC), electronic devices such as smartphones, tablet terminals, wearable terminals, IoT (Internet of Things) devices, and single-board computers such as Raspberry Pi (registered trademark) may be used. These terminals may have built-in microphones, cameras, and the like.

[0045] As shown in FIG. 7, the configuration of the main body 50 of the data expansion device 5 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a storage device STR (Storage) such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and an I / F (Interface). It may further include a GPU (Graphics Processing Unit). These configurations 501 to 506 are connected to each other by an internal bus 507. And it may further include a housing 500 for housing these.

[0046] The CPU 501 controls the entire data expansion device 5. The RAM 502 is a working area used during the operation of the CPU 501. The ROM 503 stores the operation code of the CPU 501. When the GPU 506 is provided, it processes the data converted by the CPU 501 in parallel to contribute to high-speed arithmetic processing. The STR 504 is an auxiliary storage area for recording or reading information controlled or processed by the CPU 501. The I / F 505 is an interface for transmitting and receiving various information with external devices such as the input device 52 and the display 53 connected to the main body 50.

[0047] Among these I / Fs 505, the I / F 508 performs transmission and reception of various information with other terminals 62, servers 63, etc. via the telecommunication line 61 in the data expansion system 6 shown in FIG. 9, for example.

[0048] The I / F 509 performs transmission and reception of information with the input device 52. A plurality of input devices 52 may be provided. For example, there are input means for commands and numerical values such as a keyboard, image acquisition means such as an image scanner, and pointing devices such as a mouse. Operators who use the data expansion device 5 input control commands and various information of the data expansion device 5 through these input devices 52.

[0049] Also, the I / F 510 performs transmission and reception of various information with the display 53. The display 53 outputs various information stored in the STR 504 which is a storage device, and the processing status of the data expansion device 5, etc. For example, a display is used as the display 53. The display may also have a function as an input device 52 such as a touch panel type. Also, a plurality of displays 53 may be provided.

[0050] Various information including the transmission X-ray image data 1 and the defect image data 2 input to the main body 50 of the data expansion device 5 can be directly input from an operator or the like through the input device 52, or can also be input by a signal transmitted from other terminals 62, servers 63, etc. connected via the telecommunication line 61 in the data expansion system 6 shown in FIG. 9, for example. The various information may also include information for operating and controlling the data expansion device 5.

[0051] Next, the functions of the data augmentation device 5 will be described with reference to FIGS. 3 to 7 and also FIG. 8. FIG. 8 is a schematic diagram showing an example of the functions of the data augmentation device 5 shown in FIG. 7.

[0052] (1) Acquisition unit 551 The acquisition unit 551 acquires transmission X-ray image data 1 including at least a part of the cast article imaged by transmission X-rays (see FIG. 3) and defect image data 2 composed of a defect object 21 and a bounding box 22 of the defect object 21 (see FIG. 4). Moreover, not limited to these, information for adjusting the brightness of the pixels constituting the transmission X-ray image data 1, information for adjusting the size of the bounding box 22, information for adjusting the brightness of the pixels occupied by the defect object 21, information for deforming the defect object 21 by at least any one of rotation, reduction, or enlargement, information for dividing into a plurality of small regions 3, information for determining the area 31 where it can be arranged (see FIG. 5), information for specifying the area 311 to be arranged (see FIG. 6), information for giving a predetermined condition, information for adjusting the brightness of the pixels constituting the defect object 21, and information including at least any one of information for outputting learning data (hereinafter referred to as adjustment information). It preferably has a function of acquiring. As a result, the operator of the data augmentation device 5 can give this adjustment information, and it becomes possible to obtain learning data in which a desired adjustment is reflected by an adjustment unit 559 described later.

[0053] (2) Division unit 552 The dividing unit 552 has a function of dividing the transmission X-ray image data 1 acquired by the acquisition unit 551 into a plurality of small regions 3 (see FIG. 5). The small regions 3 may be divided based on the size of the bounding box 22 acquired together by the acquisition unit 551. Also, the small regions 3 may be divided in a form in which they partially overlap each other (not shown), or may be divided such that there are small regions (not shown) formed by combining two or more bounding boxes 22, or there may be a part where the portion not divided by the small regions 3 (missing portion) exists.

[0054] (3) Determination unit 553 The determination unit 553 has a function of obtaining, for each small region 3 formed by the dividing unit 552, a feature amount based on the brightness of the pixels constituting the small region 3, and determining, as an area 31 where a defective object can be arranged, the small region 3 in which the feature amount satisfies a predetermined condition (see FIG. 5). A plurality of feature amounts may be defined. For example, it may be the % tile value of the brightness of the pixels constituting the small region 3, the average value of the brightness of the pixels constituting the small region 3, or the like. Also, for the predetermined condition, conditions may be defined for each feature amount, and it may be determined to satisfy all or any of these conditions. For example, the difference between two % tile values of the brightness of the pixels constituting the small region 3 falls within a predetermined range and the average value of the brightness of the pixels constituting the small region 3 falls within a predetermined range, and so on.

[0055] (4) Identification unit 554 The identification unit 554 has a function of identifying, as an area 311 for arranging the defective object 22 acquired by the acquisition unit 551, at least one of the arrangeable areas 31 determined by the determination unit 553 (see FIG. 6).

[0056] (5) Generation unit 555 The generation unit 555 has a function of arranging the defective object 21 in the small region 311 identified by the identification unit 554 to generate the extended image data 4 (see FIG. 6). The extended image data 4 may be in a state where the defective object 21 is overwritten and synthesized in the arranged small region 311.

[0057] (6) Creation unit 556 The creation unit 556 has a function of acquiring the position information of the arranged defect object 21 in the extended image data 1 generated by the generation unit 555 and creating learning data including the extended image data 4 and the position information of the defect object 21.

[0058] (7) Output unit 557 The output unit 557 has a function of outputting the learning data created by the creation unit 556. Further, it may have a function of outputting at least any one of not only the learning data but also the transmission X-ray image data 1, the defect object 21, the bounding box 22, the defect image data 2, the feature amount, the predetermined condition, the placeable area 31, the area 311 to be arranged, the extended image data 4, or the position information of the defect object 21 in the extended image data 4. It may also have a function of appropriately outputting the processing status of the data expansion device 5.

[0059] (8) Storage unit 558 The storage unit 558 has a function of storing and reading various information processed by the acquisition unit 551, the classification unit 552, the determination unit 553, the specification unit 554, the generation unit 555, the creation unit 556, the output unit 557, or the adjustment unit 559 described later in the RAM 502 and the STR 504 (see FIG. 7).

[0060] (9) Adjustment unit 559 In the acquisition unit 551 described above, when any adjustment information is further acquired, etc., the adjustment unit 559 may be further provided. The adjustment unit 559 has a function of adjusting at least any one of at least one of the transmission X-ray image data 1 and the defect image data 2, the form of the classification of the plurality of small regions 3 in the classification unit 552, the determination of the placeable area 31 in the determination unit 553, the specification of the area 311 to be arranged in the specification unit 554, and the form of the extended image data 4 in the generation unit 555, based on the adjustment information.

[0061] With the above configuration, the data augmentation device 5 divides the transmission X-ray image data 1 acquired by the input device 52 into a plurality of small regions 3 based on the transmission X-ray image data 1 and the defect image data 2, obtains a feature amount based on the brightness of the pixels constituting the small region 3 for each small region 3, determines a small region 31 whose feature amount satisfies a predetermined condition, identifies at least one small region 311 among the determined small regions 31, arranges a defect object 21 in the identified small region 311 to generate augmented image data 4, acquires the position information of the defect object 21 arranged in the augmented image data 4, and can create, store, and output learning data composed of at least the augmented image data 4 and the position information of the arranged defect object 21.

[0062] [3] Third Embodiment: Program for Assisting in Creation of Learning Data Refer to FIGS. 2 to 8. A program for assisting in the creation of learning data, which is the third embodiment of the present invention, is a program for assisting in the creation of learning data used for detecting defects observed in a transmission X-ray image of a cast article, and includes a step (S3) of dividing the transmission X-ray image data 1 into a plurality of small regions 3 based on the size of the bounding box 22 of the defect image data 2, a step (S4) of determining an area 31 in which the defect object 21 can be arranged in the plurality of small regions 3, a step (S5) of specifying at least one small region 311 in the area 31 where arrangement is possible as an area 311 for arranging the defect object 21, a step (S6) of arranging the defect object 21 in the area 311 to be arranged to generate augmented image data 4, a step (S7) of acquiring the position information of the defect object 21 in the augmented image data 4, and a step (S8) of creating learning data having the augmented image data 4 and the position information of the defect object 21 in the augmented image data 4 are executed by a computer (for example, the data augmentation device 5).

[0063] In addition to this, the program for assisting in creating the learning data of the present invention may cause a computer to execute at least one of the steps (S1) of acquiring transmission X-ray image data 1 in which a part of the cast article is imaged and the step (S2) of acquiring defect image data 2 composed of a defect object 21 and a bounding box 22 of the defect object 21.

Example

[0064] Hereinafter, an example (hereinafter, Example 1) in which learning data is obtained according to the present invention will be described with reference to FIGS. 10 and 11 according to the flowchart shown in FIG. 2. FIG. 10 is the transmission X-ray image data used in Example 1, and FIG. 11 is the extended image data obtained in Example 1. In addition, each step described below was performed by an operator operating a computer (workstation) using a program created in advance by describing it in the programming language Python (registered trademark).

[0065] (1) Preparation of transmission X-ray image data and defect image data (S1, S2) As shown in FIG. 10, transmission X-ray image data 801 in which a part of the cast article 811 is imaged was acquired. In addition, the defect image data was obtained by surrounding and cutting out a defect object 821 (see FIG. 11, not shown in FIG. 10) mapped to another part (not shown) of the transmission X-ray image data 801 with a rectangular bounding box (not shown). At this time, the position information of this defect object with respect to the bounding box (not shown) was also acquired.

[0066] (2) Dividing the transmission X-ray image data into a plurality of small regions (S3) The transmission X-ray image data 801 was divided into small regions (not shown in FIG. 10) with the same size as the size of the bounding box acquired in step S2.

[0067] (3) Determining small regions whose feature amount satisfies a predetermined condition (S4) As feature quantities, the 75% tile value and 25% tile value of the brightness of each pixel constituting the small region, and the average value of the brightness are obtained. As predetermined conditions, it is determined that the difference between the 75% tile value and the 25% tile value is 20 or less, and the average value of the brightness satisfies 50 or more and 200 or less. A small region satisfying this condition was determined as an area where it is possible to arrange. Note that the numerical values of these conditions are brightness values represented in 256 gradations in the 0 to 255 gradation range.

[0068] (4) Specify the area for arranging the defective object (S5) The area 831 for arranging the defective object was specified as one small region 831 indicated by the broken line in FIG. 11. The method of specification was left to the discretion of the operator.

[0069] (5) Arrange the defective object to generate extended image data (S6) The extended image data 804 was generated by arranging and overwriting the defective object 821 in the specified small region 831 (see FIG. 11).

[0070] (6) Obtain the position information of the defective object (S7) Next, the position information of the defective object 821 in the obtained extended image data 804 was obtained.

[0071] (7) Create learning data (S8) Learning data having the obtained extended image data 804 and the position information of the defective object 821 in the extended image data 804 was created and stored in the storage device.

Explanation of symbols

[0072] 1, 801: Transmission X-ray image data 11, 811: Cast article 111, 112, 113, 114, 115: Parts (cast article) 12: Background 2: Defective image data 21, 821: Defective object 211: Contour 22: Bounding box 3, 32, 33: Small area 31: Configurable area (small area) 311, 831: Area to be configured (specified area, configurable area, small area) 4, 804: Extended image data 5: Data expansion device 50: Main body 500: Housing 501: CPU 502: RAM 503: ROM 504: STR 505(508, 509, 510): I / F 506: GPU 507: Internal bus 52: Input device 53: Display 551: Acquisition unit 552: Classification unit 553: Judgment unit 554: Specification unit 555: Generation unit 556: Creation unit 557: Output unit 558: Storage unit 559: Adjustment unit 6: Data expansion system 61: Telecommunication line 62: Other terminal 63: Server 901: Transmission X-ray image 911: Cast article 912: Dark part 913: Bright part 921: Defect

Claims

1. 1. A method for generating training data for use in detecting defects observed in a transmission X-ray image of a cast article, comprising the steps of: preparing radiographic image data of at least a portion of a cast article; providing defect image data comprising a defect object and a bounding box for the defect object; Dividing the transmission X-ray image data into a plurality of small regions based on the size of the bounding box; determining an area among the plurality of small areas in which the defect object can be placed; identifying at least one of the sub-areas of the possible placement area as an area in which the defect object is to be placed; locating the defect object in the location area to generate augmented image data; obtaining position information of the located defect object in the extended image data; A step of creating learning data having the extended image data and the position information. A method for creating learning data, comprising:

2. 2. The method for creating learning data according to claim 1, wherein the determining step determines, for each small region, a feature based on the brightness of the pixels constituting the small region, and determines the small region whose feature satisfies a predetermined condition as the area where placement is possible.

3. 3. The method for creating learning data according to claim 2, wherein the feature amount is based on a percentile value of brightness of the pixels constituting the small region and an average value of the brightness of the pixels constituting the small region, and the predetermined condition is based on a difference between two different percentile values ​​among the percentile values ​​and the average value.

4. 4. The method for creating learning data according to claim 1, wherein the step of specifying comprises specifying a plurality of small areas that are not adjacent to each other within the area in which the image can be placed.

5. A method for creating learning data according to any one of claims 1 to 3, wherein the generating step further includes a step of adjusting the brightness of pixels constituting the defect object according to the feature amount of the area to be placed.

6. 4. The method for creating learning data according to claim 1, wherein the generating step further includes a step of performing at least one of the following transformations on the defect object: rotating, enlarging, or reducing.

7. A data expansion device for creating learning data used to detect defects observed in a transmission X-ray image of a seepage casting article, comprising: an acquisition unit that acquires transmission X-ray image data on which at least a portion of a casting is imaged, and defect image data that is configured of a defect object and a bounding box of the defect object; a division unit that divides the transmission X-ray image data acquired by the acquisition unit into a plurality of small regions; a determination unit that determines, for each small region divided by the division unit, a feature amount based on the brightness of pixels constituting the small region, and determines the small region in which the feature amount satisfies a predetermined condition; an identification unit that identifies at least one of the small regions determined by the determination unit; a generating unit that generates extended image data by arranging the defect object in the small region identified by the identifying unit; a generating unit that obtains position information of the arranged defect object in the extended image data generated by the generating unit, and generates learning data having the extended image data and the position information; a storage unit that stores and reads the learning data created by the creation unit; an output unit that outputs the learning data read from the storage unit A data expansion device comprising:

8. A program for assisting in the creation of learning data used to detect defects observed in a transmission X-ray image of a casting, comprising: A step of dividing the transmission X-ray image data on which at least a portion of the cast article is imaged into a plurality of small regions based on sizes of the bounding boxes of defect image data that are composed of defect objects and bounding boxes of the defect objects; determining an area in which the defect object can be placed within the plurality of small areas; identifying at least one of the small areas in the possible placement area as an area in which the defective object is to be placed; locating the defect object in the location area to generate augmented image data; obtaining position information of the defect object in the extended image data; creating learning data having the extended image data and the position information; A program characterized by causing a computer to execute the above.

Images