Image generation device and object detection device

Abstract

To provide an image generation device capable of acquiring accurate learning samples in X-ray inspection, and an object detection device using the image generation device. [Solution] The image generation device includes a coordinate transformation unit that transforms the coordinates set for an object image OG, which is a sample image obtained by X-ray imaging of an object OB to be detected in an X-ray inspection, according to the imaging conditions of the object OB, and an image synthesis unit that synthesizes the coordinate-transformed sample image (object image OG) with a background image BG to generate a training image LG as training data.

Description

Technical Field

[0001] The present invention relates to an image generation device and an object detection device that perform image generation applicable to X-ray inspection.

Background Art

[0002] For example, as an X-ray inspection device, there is one that makes a determination based on the learning result of a sample image. For an X-ray image taken by flowing an inspection object on a belt conveyor, a determination is made regarding the inspection object based on the degree of overlap of objects (see Patent Document 1).

[0003] Also, for example, there is known a medical image output system that performs conversion of the aspect ratio (vertical-horizontal ratio) for medical images (see Patent Document 2).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1 above, there is no description regarding the aspect ratio of the X-ray image. Also, simply applying the aspect ratio (vertical-horizontal ratio) conversion disclosed in Patent Document 2 to Patent Document 1 does not necessarily result in obtaining accurate learning samples for X-ray inspection.

[0006] The present invention has been made in view of the above points, and an object thereof is to provide an image generation device capable of obtaining accurate learning samples for X-ray inspection and an object detection device using the image generation device.

Means for Solving the Problems

[0007] An image generation device for achieving the above objective includes a coordinate transformation unit that transforms the coordinates set for a sample image obtained by X-ray imaging of an object to be detected in an X-ray inspection, according to the imaging conditions of the object, and an image synthesis unit that generates training data by combining the coordinate-transformed sample image with a background image.

[0008] The image generation device described above converts sample images of objects to be detected in X-ray inspection according to the shooting conditions, then combines them with a background image to generate training data, thereby obtaining accurate samples for training in X-ray inspection.

[0009] In a specific aspect of the present invention, the present invention includes a conveyor that moves an object in one direction during X-ray imaging, an irradiation unit that irradiates the object with X-rays as it is moved by the conveyor, and a line sensor that receives the components of the X-rays that have passed through the object to acquire image data that will become a sample image. In this case, it becomes possible to generate training data according to the conditions of movement by the conveyor.

[0010] In another aspect of the present invention, the coordinate transformation unit changes the aspect ratio of the sample image by scaling the coordinates in the direction of delivery, according to the difference between the delivery speed on the conveyor and the scan rate on the line sensor. In this case, it becomes possible to generate training data corresponding to the discrepancy between the operation of the conveyor and the operation of the line sensor.

[0011] In yet another aspect of the present invention, the coordinate transformation unit performs a coordinate transformation on the coordinates of the sample image using an affine transformation. In this case, the transformation process can be performed simply and accurately.

[0012] In yet another aspect of the present invention, the coordinate transformation unit includes a change in aspect ratio and a rotational movement as coordinate transformations, where the change in aspect ratio for a specific direction is performed before and after the rotational movement, and the ratio of the transformation performed after the rotational movement is inversely proportional to the ratio of the transformation performed before the rotational movement. In this case, coordinate transformations can be performed by a combination of rotation and compression / expansion.

[0013] An object detection device for achieving the above objective comprises: an object detection unit that detects the presence or absence of an object based on the learning results of training data generated by any of the above image generation devices; an imaging unit that performs X-ray imaging of the object to be inspected in an X-ray inspection; and a determination unit that performs detection by the object detection unit on the X-ray image of the object to be inspected acquired by the imaging unit to determine the presence or absence of an object in the object to be inspected.

[0014] The above-described object detection device can perform rapid and accurate X-ray inspection of the object to be inspected based on the training data generated by the image generation device. [Brief explanation of the drawing]

[0015] [Figure 1] (A) is a conceptual perspective view illustrating an image generation device according to one embodiment, and (B) is a conceptual perspective view of an object detection device composed of the image generation device. [Figure 2] This is a block diagram illustrating one example configuration of an image generation device. [Figure 3] This is an image diagram illustrating the appearance of the generated scan image. [Figure 4] (A) and (B) are conceptual diagrams illustrating the processing of individual images of an object, and (C) is a conceptual diagram illustrating the generation of training images through image synthesis. [Figure 5] (A) to (D) are diagrams illustrating the scanning process of an object, and (E) is an image diagram created by image synthesis of one comparative example. [Figure 6] This diagram illustrates an example of how aspect coefficients are calculated. [Figure 7] This is a diagram illustrating an example of preparation processing for an object image. [Figure 8] This is a block diagram illustrating the image generation process. [Figure 9]This is a conceptual diagram for explaining the state of machine learning. [Figure 10] This is a flowchart for explaining a series of processes for an object detection device configured based on an image generation device. [Figure 11] This is a block diagram for explaining the outline of an image generation device.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, referring to FIG. 1 and the like, details of an image generation device according to an embodiment of the present invention and an object detection device using learning data generated by the image generation device will be described. Among the respective figures, FIG. 1(A) is a conceptual perspective view for explaining an image generation device 100 according to an embodiment, and shows a state at the time of acquiring sample learning by the image generation device 100 as a preparation stage for learning. On the other hand, FIG. 1(B) is a conceptual perspective view of an object detection device 500 configured from the image generation device 100. In other words, FIG. 1(A) is a diagram showing a state in the learning preparation stage for configuring the object detection device 500, while FIG. 1(B) is a diagram showing the state of operation of the learned object detection device 500.

[0017] The image generation device 100 shown in Fig. 1(A) includes an X-ray inspection device 10 and a control device 50. It performs X-ray imaging on the object OB to be detected in X-ray inspection to obtain a sample image, and generates learning data (learning image) from the obtained sample image. Further, by using the generated learning data as a learning sample and performing machine learning such as deep learning, it is possible to determine the presence or absence of the object OB during the X-ray inspection of an unknown inspection object based on the learning result using the learning data (learning image). Here, the case where the image generation device 100 functions as an object detection device 500 when the presence or absence of an object can be determined as a result of the above learning will be described. In this case, in the control device 50 before learning, the learning result based on the obtained sample image is stored, and the control device 50 functions as an object detection unit 300 that detects the presence or absence of the object OB based on the result, so that the image generation device 100 functions as an object detection device 500.

[0018] In Fig. 1(A), the X-ray inspection device 10 includes, in addition to a conveyor 20 as a conveyance unit that sends the object OB in one direction during X-ray imaging, an X-ray source 11 that is an irradiation unit that irradiates the object OB with X-rays RL, which are radiation, during the conveyance by the conveyor 20, and an X-ray inspection unit 12 that is a line sensor that receives the component of the X-rays RL from the X-ray source 11 that has passed through the object OB. That is, the X-ray inspection unit 12 obtains image data (X-ray image) that should be a sample image by receiving the component of the X-rays that have passed through the object OB. Hereinafter, the image data (X-ray image) will be referred to as an object image.

[0019] In the X-ray inspection apparatus 10, the X-ray source 11, which emits X-ray RL, is located on the lower side (-Z side) near the center of the shielding box 15, and irradiates the X-ray RL towards the X-ray inspection unit 12. The X-ray inspection unit 12 is a radiation sensor unit that receives the X-ray RL, and in this case, it is a line sensor type. More specifically, the X-ray inspection unit 12 is located on the upper side (+Z side) near the center of the shielding box 15, facing the X-ray source 11 across the transport path of the conveyor 20. The X-ray inspection unit 12 enables line-type scanning in synchronization with transport by the conveyor 20 by arranging, for example, the photodetectors in a line extending in a direction perpendicular to the transport direction (Y direction shown by arrow DD1) (X direction). That is, when the object OB passes near the center of the internal space of the shielding box 15, the object OB is irradiated with X-ray RL, and a two-dimensional inspection of the object OB is performed based on the results received by the X-ray inspection unit 12.

[0020] To perform the above operations in the image generation device 100, the conveyor 20 provided in the X-ray inspection device 10 transports the object OB in a straight line in the direction indicated by arrow DD1 according to the command signal from the control device 50, and the X-ray inspection unit 12 of the X-ray inspection device 10 irradiates the object OB transported into the shielding box 15 with X-ray RL within a predetermined range along the path and detects the transmitted component.

[0021] The control device 50 is composed of a personal computer (PC) having, for example, a CPU, GPU, and various storage devices, and is connected to each part of the X-ray inspection apparatus 10 to control their operation. In this case, the control device 50 is responsible for sending (unloading) the object OB in the direction indicated by arrow DD1 by the conveyor 20, irradiating the object OB with X-rays using the X-ray source 11, which is the irradiation unit, when the object is sent by the conveyor 20, and receiving the detection results in the X-ray inspection unit 12.

[0022] Here, a typical example of an object OB is various dangerous objects such as knives and handguns. In this case, by collecting information on the presence or absence of dangerous objects and learning from it (learning of the dangerous object detection AI), the object detection device 500 functions as a determination device that automatically determines the presence or absence of dangerous objects inside, for example, a bag. As an example here, the control device 50 first prepares multiple necessary patterns of training images (composite images) by combining an X-ray image (object image) of a bag or the like (background image) with an X-ray image of a bag or the like. Then, by performing various machine learning operations such as deep learning based on the training data composed of training images, the control device 50 functions as an object detection unit 300. As described above, the object detection device 500, which is composed of the image generation device 100, performs imaging of luggage BA such as bags by irradiating them with X-rays using the X-ray inspection device 10, as shown in Figure 1(B), and detects the transmitted component of luggage BA, thereby detecting in the object detection unit 300 whether or not an object OBx inside luggage BA is a hazardous material. In other words, the object detection device 500 sees inside luggage BA and collects information regarding the presence or absence of hazardous materials.

[0023] The following will further describe the components and functions of the image generation device 100, referring to the block diagram in Figure 2.

[0024] As shown in the figure and as described above, the X-ray inspection apparatus 10 comprises a conveyor 20, an X-ray source 11, and an X-ray inspection unit 12. In addition to these, the X-ray inspection apparatus 10 also comprises an incoming detection sensor SE and a display unit DP.

[0025] The loading detection sensor SE is installed, for example, near the entrance of the shielded box 15 (see Figure 1) to detect the loading of cargo BA.

[0026] The display unit DP is an output device that, for example, consists of a liquid crystal panel or an organic EL panel, and displays judgment results based on analysis by the control device 50 on the screen.

[0027] Furthermore, when the X-ray inspection device 10 functions as a component of the object detection device 500, the X-ray source 11 and the X-ray inspection unit 12, etc., can be seen as functioning as an imaging unit PH that performs X-ray imaging of the luggage BA, which is the object being inspected in the X-ray inspection.

[0028] As described above, the control device 50 is connected to each part of the X-ray inspection apparatus 10 and controls their operation. In addition to the main control unit 60, it also has various processing units to analyze various data, including image data acquired during inspections in the X-ray inspection apparatus 10. Specifically, in the illustrated example, it comprises an X-ray irradiation control unit 61, an X-ray data detection unit 62, an image processing unit 63, a storage unit 64, and an operation unit 70 as an input device. A display unit DP or equivalent as an output device shown in the X-ray inspection apparatus 10 may also be provided in the control device 50.

[0029] The main control unit 60 is composed of a CPU and the like, and controls the operation of each part that makes up the X-ray inspection apparatus 10, and the processing of each part that makes up the control device 50. Furthermore, it generates (forms) multiple necessary patterns of training images by combining the object image and background image as described above, that is, it performs various processes to prepare training data, and processes to generate artificial intelligence (AI) such as structural data based on the prepared training data. In other words, it performs various processes necessary for the control device 50 to function as the object detection unit 300. When the control device 50 is functioning as the object detection unit 300, the main control unit 60 also functions as a determination unit JD that determines the presence or absence of an object OB (such as a blade) in an object to be inspected, such as luggage BA, as a component of the object detection device 500 (object detection unit 300).

[0030] The X-ray irradiation control unit 61 controls the operation of the X-ray irradiation control unit 61 to irradiate the object OB and bags for background images (package BA as a component of the object detection device 500) transported by the conveyor 20 with X-rays RL from the X-ray source 11 at a predetermined timing.

[0031] The X-ray data detection unit 62 receives the detection results from the X-ray inspection unit 12 of the X-ray inspection apparatus 10 that has been irradiated with X-ray RL, and detects them as scanned image data. In other words, the X-ray data detection unit 62 detects scanned image data that has been scanned row by row as the object OB (or cargo BA) is being transported.

[0032] The image processing unit 63, composed of a GPU and the like, for example, combines scanned image data from the X-ray data detection unit 62 to generate (form) a completed X-ray image of an object. In particular, when operating as a component of the image generation device 100, it generates X-ray images (object images and background images) of the object OB and bags for background images, etc., according to commands from the main control unit 60, performs various processing on the generated images, and generates (forms) training images by combining the object image and background image. Furthermore, when operating as a component of the object detection device 500, it generates (forms) an X-ray image of luggage BA and performs image processing so that the main control unit 60 can determine whether the generated X-ray image contains hazardous materials. In short, the image processing unit 63 is responsible for all aspects of various image processing.

[0033] The storage unit 64 is composed of a storage device and stores various data and programs necessary for processing in each unit, such as the main control unit 60 and the image processing unit 63. The storage unit 64 also stores (stores) various data such as X-ray images acquired by the operation of the X-ray inspection apparatus 10 and judgment results based on analysis by the control device 50.

[0034] The operation unit 70 is composed of input devices such as a keyboard and mouse, and receives various commands from an operator to operate the image generation device 100 or the object detection device 500. In other words, the operation unit 70 performs operations such as starting, initializing, and shutting down various software stored in the control device 50 (object detection unit 300) according to the instructions from the operator.

[0035] Figure 3 shows the scan image generated by the X-ray inspection device 10. In the figure, arrow DD1 indicates the direction of delivery (transport) during scanning. Figures G1 to G3 in Figure 3 show the case where the object detection device 500 detects a hazardous material (a sharp object) as object OBx in the package BA.

[0036] As in the case of the X-ray inspection device 10 described with reference to Figure 1(A), when an object OB is moved on a conveyor belt 20 and photographed, the scan rate of the X-ray inspection unit 12, which is a line sensor, and the speed at which the object moves on the conveyor belt 20 are different. As a result, the image may be stretched in the direction of arrow DD1, as shown in the example images G1 to G3 in Figure 3, and the aspect ratio of the image may differ from that of the actual object. In particular, as is clear from the image images G1 to G3, the way in which the shape of hazardous materials (a knife in the example shown in the figure) contained in the cargo BA changes due to the above stretching will differ depending on its orientation (angle relative to the direction of arrow DD1), and this is thought to have an effect on the preparation stage, i.e., the learning of the hazardous material detection AI.

[0037] Therefore, in the image generation device 100 of this embodiment, coordinates (orthogonal coordinates) are set for the object image, which is a two-dimensional image obtained by X-ray imaging of the object OB to be detected in X-ray inspection, and the set coordinates are transformed on the object image according to the imaging conditions of the object OB, such as the aspect ratio being different from that of the actual object. Then, the coordinate-transformed object image is combined with a background image to generate a training image as training data, thereby obtaining an accurate training sample that takes these circumstances into consideration. Regarding the method of setting the coordinates of the two-dimensional X-ray image, as shown in Figure 3, the transmission (transport) direction during scanning, indicated by arrow DD1, is defined as the y direction (+y direction), and the direction perpendicular to the y direction is defined as the x direction. That is, in line scanning, the direction in which the line extends is the x direction. Regarding the coordinates of each image, for example, a rectangular region is formed to include the object OB during imaging (see Figure 4 described later), and in this case, each side of the rectangle extends along the x direction and the y direction, respectively.

[0038] The following provides a detailed explanation of the generation of training images, which is the preparatory stage for learning, with reference to Figure 4 and other figures. Figures 4(A) and 4(B) are conceptual diagrams illustrating the processing of a single image (blade) of the object OB, and Figure 4(C) is a conceptual diagram illustrating the generation of training images by image synthesis. Figures 5(A) to 5(D) illustrate the scanning process of the object OB, and Figure 5(E) is an image synthesis diagram of a comparative example. Furthermore, Figure 6 illustrates an example of the calculation of the aspect coefficient, which indicates the aspect ratio value in imaging by the X-ray inspection device 10, and Figure 7 illustrates an example of the preparation process for a single image (blade), which is a single image of the object OG, i.e., the object OB.

[0039] First, as shown in Figure 4(A), for the single image of the target object OB (the blade), an image of the target object such as the blade is extracted using a predetermined image processing method, so that it is contained within a rectangular region (an area enclosed by a rectangular frame) RA. Here, the image of the single image (blade) extracted in this way is referred to as the target image OG. As previously described, the rectangular region RA has sides that extend along the x and y directions, respectively. In the example shown, one vertex of the rectangle is set as the origin, and the coordinates for the x and y directions are set accordingly. Figure 4(B) shows the area within the rectangular region RA set as described above that is occupied by the target object, which is filled with hatching in the figure. Here, the hatched area shown in Figure 4(B), i.e., the filled image, is used as the GT (Ground Truth) image in machine learning, and the coordinate information of the GT image is defined as the area where the single image (blade) corresponding to the target object OB truly exists, and learning is performed accordingly.

[0040] Next, as shown in Figure 4(C), a training image LG is generated by combining the object image OG, which contains the GT image information described above, with a background image BG that does not include a single image (of the blade). In this case, the generated training image LG, which includes the background, will contain coordinate information for the GT image that indicates the position and range of the target object such as the blade.

[0041] Regarding the generation of the training images LG described above, by changing various relative positions and orientations (angles) of the object image OG relative to the background image BG during image synthesis, it becomes possible to generate various combinations of images from one object image OG and one background image BG.

[0042] However, as explained with reference to Figure 3, etc., in the case of the embodiment, due to the relationship of the aspect ratio stretching and contraction during shooting, for example, as shown in Figures 5(A) and 5(B), when the object OB, which is a blade, is positioned so that its longitudinal direction is along the direction of arrow DD1 and image is taken, ideally the aspect ratio should be maintained and the image should be like the object image OGi shown in Figure 5(A). However, in reality, the image becomes longer in the longitudinal direction than the actual object (a shape with a longer blade), as shown in the object image OG shown in Figure 5(B). On the other hand, as shown in Figures 5(C) and 5(D), when the object OB is positioned so that its longitudinal direction is along the direction perpendicular to the direction of arrow DD1 and image is taken, the image should be like the object image OGi shown in Figure 5(C). However, in reality, the image becomes longer in the short direction than the actual object (a shape with a longer blade), as shown in the object image OG shown in Figure 5(D). If, for example, as shown in Figure 5(E) as a comparative example (Images G1x to G3x), images created by altering the relative position and orientation (angle) of the object image OG while maintaining its shape, without considering the changes in the object's shape on the image caused by the differences in aspect ratio, are used as training images, then proper training may not be possible.

[0043] In contrast, in this embodiment, in order to avoid such a situation, image synthesis is performed while considering the stretching and contraction of the aspect ratio. Specifically, the aspect ratio of the object image OG, which is to become the sample image, is changed by stretching and contracting the coordinates related to the delivery direction (direction of arrow DD1) according to the difference between the delivery speed on the conveyor 20 and the scan rate of the X-ray inspection unit 12, which is a line sensor.

[0044] The following explanation, with reference to Figure 6, describes an example of calculating the aspect coefficient, which represents the aspect ratio in imaging using the X-ray inspection device 10 with a square sample, as a prerequisite for the considerations described above. As shown in the figure, first, a square with side length d is prepared as the object to be imaged, and it is positioned so that one side is aligned with the transmission direction (direction of arrow DD1). The aspect ratio of the acquired image CA obtained by imaging (scanning) in this state is calculated as the ratio of the length h in the vertical direction along arrow DD1 to the length w in the horizontal direction perpendicular to it, and this is defined as the aspect coefficient α. That is, α = h / w. In this example, length h is greater than length w, so α > 1.

[0045] The following describes an example of preparation processing for the object image OG based on the above, with reference to Figure 7. Here, the coordinates of the object image OG, which is used as a sample image, are transformed using an affine transformation to change the aspect ratio of the sample image, and in addition, rotation and translation are performed.

[0046] First, as shown in state PH1, the object OB (a cutting tool in the illustrated example) is prepared and scanned (imaged) using the X-ray inspection device 10 in Figure 1(A) to obtain the object image OG as shown in state PH2, and the coordinates for x and y are set. In this state, the object image OG obtained has an aspect ratio that is multiplied by α in the y direction compared to the actual object (object OB). Therefore, as shown in state PH3, the entire rectangular region RA containing the object image OG is multiplied by 1 / α in the y direction. In this case, the vertical width (length in the vertical direction) of the rectangular region RA is H and the horizontal width (length in the horizontal direction) is W. Next, as shown in state PH4, the object image OG is translated so that the center of the rectangular region RA becomes the origin. That is, the rectangular region RA containing the object image OG is moved by -W / 2 in the x direction and -H / 2 in the y direction. Next, as shown in state PH5, the rectangular region RA is rotated by an angle θ around the origin, and as shown in state PH6, the rectangular region RA is moved again by +W / 2 in the x direction and +H / 2 in the y direction. Finally, as shown in state PH7, the entire rectangular region RA is multiplied by α in the y direction.

[0047] The above series of processes can be represented using a 2D matrix in x and y coordinates. Let (x, y) be the coordinates of the image before processing, and (x', y') be the coordinates of the image after processing, as shown below. Note that the range to be recognized as a GT image will also change depending on the processing of the target image OG. TIFF2026076463000002.tif23134TIFF2026076463000003.tif45135

[0048] By generating object images OG with various angle θ values ​​as needed, and then appropriately changing their relative position (placement) relative to the background image BG, it becomes possible to obtain a portion of the training images necessary for learning. In other words, the composite images obtained by the above method constitute at least a portion of the training images necessary for machine learning.

[0049] Furthermore, while the above example cited knives as a dangerous item, dangerous items can include a variety of other things besides knives, such as handguns and explosives.

[0050] The following describes an example configuration in which the main control unit 60 performs each of the processes for generating training images by image synthesis as described above, with reference to the block diagram shown in Figure 8. Here, as previously described, command signals are sent from the main control unit 60 to each part, and for example, various command signals are output from the main control unit 60 to the image processing unit 63 (see Figure 2), thereby performing various image processing necessary for image synthesis.

[0051] As shown in Figure 8, the main control unit 60 includes an aspect coefficient calculation unit AC, an object image adjustment unit OA, and an image synthesis unit GC (learning image generation unit GG) to generate training images by the image synthesis described above. Of these, the object image adjustment unit OA includes a coordinate setting unit CS and a coordinate transformation unit CT. In addition to the above, the main control unit 60 also includes a machine learning unit LL for performing machine learning based on the training images generated by image synthesis in the image synthesis unit GC (learning image generation unit GG). Each of these units can be configured by, for example, storing various programs in the storage unit 64 and reading them as needed to perform calculation processing according to the purpose.

[0052] When the aspect coefficient calculation unit AC receives a command to read a square sample as one of the various commands based on the operation of the operation unit 70 by the operator, it calculates the values ​​illustrated in Figure 6 in response and outputs the value of the aspect coefficient α as a result.

[0053] Of the object image adjustment unit OA, the coordinate setting unit CS performs various processes related to the items exemplified in Figures 4(A) and 4(B), and various processes related to the items exemplified in Figure 7, specifically the items mentioned earlier. More specifically, it performs various processes up to the point where the coordinates are set for the object image OG, which is a sample image obtained by X-ray imaging of the object OB to be detected in X-ray inspection. That is, in the above example, the coordinate setting unit CS sets the coordinates of the rectangular region RA containing the object image OG and the range indicating its extent, and sets the range of the GT image that defines the range of the object image OG within the rectangular region RA.

[0054] Of the object image adjustment unit OA, the coordinate transformation unit CT performs various processes for the later items among the items exemplified in Figure 7. More specifically, it transforms the coordinates of the object image OG set in the coordinate setting unit CS according to the shooting conditions of the object OB. That is, in the above example, when shooting the object OB, if there is an extension (magnification) in the transport direction indicated by arrow DD1 as shown by the aspect coefficient α in the y direction, the coordinate transformation unit CT corresponds to shrinking the area including the object image OG by 1 / α in the y direction, translating it to set the rotation center position, rotating it by a predetermined angle (angle θ), translating it again, and extending (magnifying) it by α in the y direction, thereby transforming the coordinates of the object image OG. In the above case, the coordinate transformation unit CT includes at least a change in aspect ratio and a rotation as coordinate transformations, and performs a change in aspect ratio (1 / α times and α times) for a specific direction before and after the rotation, and the ratio of the transformation performed after the rotation is inversely proportional to the ratio of the transformation performed before the rotation.

[0055] The image synthesis unit GC synthesizes the object image OG, which has been transformed as described above, with the background image BG. In this way, the image synthesis unit GC functions as a training image generation unit GG that generates the training image LG.

[0056] The following example illustrates how machine learning based on acquired training images works, referring to the conceptual diagram shown in Figure 9. Various methods can be applied to machine learning; for example, deep learning, which constructs a multi-layered neural network as shown in Figure 9, is one such method. In the illustrated example, there is an input layer (IL), a hidden layer (ML), and an output layer (OL). In the output layer (OL), classification classes are envisioned, such as the classification of types of hazardous materials. That is, in addition to determining whether or not a hazardous material is present, it becomes possible to classify the hazardous material as a knife, a handgun, etc., and then display these classifications as probabilities.

[0057] During the learning phase, machine learning is performed based on training images (sample images) composed of composite images and the GT images contained within them. Specifically, these image data are processed from the input layer IL, and the layers from the input layer IL to the hidden layer ML and output layer OL are constructed to improve the accuracy of the judgment results. In the completed learning state, the X-ray image acquired of an unknown object, i.e., the luggage BA to be detected as exemplified in Figure 1(B), is processed from the input layer IL, and the system determines which of the pre-trained classifications it belongs to.

[0058] The following describes a series of processes for the object detection device 500, which is configured based on the image generation device 100, with reference to the flowchart shown in Figure 10. Here, we will explain the entire process from generating training images by image synthesis in the image generation device 100, to generating a hazardous material detection AI (generation of a learning model) using machine learning based on the generated training images, and finally to detecting hazardous materials by the object detection device 500, which is configured based on the image generation device 100.

[0059] First, in the image generation device 100, the control device 50 performs imaging using the X-ray inspection device 10 in order to acquire image data of the object OB alone, i.e., the object image OG, and the background image BG that does not include the object OB (step S1).

[0060] Next, the control device 50 creates a GT image (filled image) with respect to the object image OG in step S1, that is, a single image of object OB (step S2).

[0061] Meanwhile, the control device 50 performs a process to take an image of a square sample in the X-ray inspection device 10 (step S3) and calculate the aspect coefficient α as a characteristic of the X-ray inspection device 10 (step S4).

[0062] Subsequently, the control device 50 uses the aspect coefficient α calculated in step S4 to perform a coordinate transformation on the object image OG using an affine transformation, i.e., rotation, scaling (extension), and reduction (step S5).

[0063] Next, the control device 50 generates a training image LG by combining the background image BG from step S1 with the object image OG converted in step S5, i.e., the composite image data after various processing (step S6).

[0064] As described above, the image generation device 100 functions as a device for generating training images LG. Furthermore, the image generation device 100 performs machine learning based on the training images LG generated above, and various other training images that are separately created or acquired as needed, to generate a learning model. As an example here, a hazardous materials detection AI capable of determining the presence or absence of knives, handguns, etc., is created through this machine learning (step S7). With the creation of the hazardous materials detection AI, the image generation device 100 is configured to form an object detection device 500 for detecting hazardous materials. At this time, the control device 50 functions as an object detection unit 300 by having a learning model composed of composition data, etc.

[0065] In the object detection device 500 configured as described above, based on the created hazardous material detection AI, it is detected whether or not the luggage BA, etc. to be inspected contains a hazardous material, which is the object OB (Step S8).

[0066] The outline of the image generation device 100 in this embodiment will be described below with reference to the block diagram shown as Figure 11. As shown in the figure and as previously described, the image generation device 100 comprises a coordinate transformation unit CT and an image synthesis unit GC.

[0067] In this embodiment, when a sample image SG (e.g., object image OG) is obtained by X-ray imaging of the object OB to be detected in X-ray inspection, coordinate settings are assigned to it. The coordinate transformation unit CT then transforms the set coordinates according to the imaging conditions of the object OB. In the example described above, a characteristic of the imaging conditions is that when imaging is performed by the X-ray inspection device 10, the image is stretched in the transmission direction, and the aspect coefficient α (>1) is calculated. The coordinate transformation unit CT performs a coordinate transformation that rotates the object image OG while applying a coordinate stretching transformation related to the transmission direction according to the aspect coefficient α, so that even if the orientation (angle) of the object image OG as the coordinate-transformed sample image SG is changed from the original object image OG, the shape on the image will be in accordance with the characteristics of the X-ray imaging described above.

[0068] Furthermore, the image synthesis unit GC generates training data by combining the object image OG, which is a sample image SG transformed as described above, with the background image BG. In this case, even if the position and angle (orientation) of the object image OG (sample image SG) relative to the background image BG are changed, it is possible to create a composite image that serves as accurate sample data, taking into account the characteristics of X-ray imaging, as training data.

[0069] As described above, the image generation device 100 of this embodiment includes a coordinate transformation unit CT that transforms the coordinates set for a sample image SG (object image OG) obtained by X-ray imaging of the object OB to be detected in X-ray inspection, according to the imaging conditions of the object OB, and an image synthesis unit BG that generates training data by combining the coordinate-transformed sample image SG (object image OG after coordinate transformation) with a background image BG. In the image generation device 100, the sample image SG of the object to be detected in X-ray inspection is transformed according to the imaging conditions and then combined with the background image BG to generate training data, thereby obtaining appropriate training samples for X-ray inspection.

[0070] 〔others〕 This invention is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit.

[0071] Firstly, although the above embodiment uses hazardous materials as the object of determination, it is not limited to this and can be applied to the determination of various other objects.

[0072] Furthermore, in the above embodiment, the object detection device 500 is configured directly from the image generation device 100. However, this is not limited to this configuration. For example, the image generation device 100 may perform the generation of training images, and the machine learning model based on the generated training images, or the learning model such as a hazardous material detection AI obtained as a result of the learning, may be incorporated into another device equivalent to the image generation device 100.

[0073] Furthermore, various interpretations of the image generation device are possible. For example, it is possible to consider only the part of the image generation device 100 with the above configuration that performs image processing for image generation (for example, the control device 50 or a part thereof in the above example), excluding the device that captures X-ray images, as the image generation device.

[0074] Furthermore, as described above, a characteristic of the imaging situation is that when imaging is performed by the X-ray inspection device 10, the image is stretched in the transmission direction, and processing is performed corresponding to the aspect coefficient α. However, the aspect coefficient α may not always be constant, but rather, if, for example, the scan rate of the line sensor or the speed at which the belt conveyor moves can be adjusted (changed) as appropriate, the value of the aspect coefficient α may change according to the degree of adjustment. In such a case, for example, the generated training image may be changed according to the change in the value of the aspect coefficient α.

[0075] Furthermore, the machine learning methods used are not limited to the example given above; various other methods can be employed. [Explanation of Symbols]

[0076] 10...X-ray inspection device, 11...X-ray source, 12...X-ray inspection unit, 15...Shielding box, 20...Conveyor, 50...Control device, 60...Main control unit, 61...X-ray irradiation control unit, 62...X-ray data detection unit, 63...Image processing unit, 64...Storage unit, 70...Operation unit, 100...Image generation device, 300...Object detection unit, 500...Object detection device, AC...Aspect coefficient calculation unit, BA...Luggage, BG...Background image, BG...Image synthesis unit, CA...Captured image, CS...Coordinate setting CT...Coordinate transformation unit, DD1...Arrow, DP...Display unit, GC...Image synthesis unit, GG...Training image generation unit, IL...Input layer, JD...Decision unit, LG...Training image, LL...Machine learning unit, ML...Intermediate layer, OA...Object image adjustment unit, OB...Object, OBx...Object, OG...Object image, OL...Output layer, PH...Imaging unit, PH1-PH7...State, RA...Rectangular area, RL...X-ray, SE...Input detection sensor, SG...Sample image, α...Aspect coefficient, θ...Angle

Claims

1. A coordinate transformation unit that transforms the coordinates set for a sample image obtained by X-ray imaging of the object to be detected in an X-ray inspection, according to the imaging conditions of the object, An image synthesis unit generates training data by combining the coordinate-transformed sample image with a background image. An image generation device equipped with the following features.

2. In X-ray imaging, a conveyor is used to move the object in one direction, An irradiation unit that irradiates the object with X-rays when it is sent by the conveyor, A line sensor that acquires image data to become the sample image by receiving the X-ray component that has passed through the object, The image generating apparatus according to claim 1, comprising:

3. The image generation apparatus according to claim 2, wherein the coordinate transformation unit changes the aspect ratio of the sample image by scaling the coordinates in the direction of transmission according to the difference between the transmission speed on the conveyor and the scan rate of the line sensor.

4. The image generation apparatus according to claim 1, wherein the coordinate transformation unit performs a coordinate transformation on the coordinates of the sample image by affine transformation.

5. The image generating apparatus according to claim 1, wherein the coordinate transformation unit includes a change in aspect ratio and a rotational movement as coordinate transformations, the change in aspect ratio for a specific direction is performed before and after the rotational movement, and the ratio of the transformation performed after the rotational movement is inversely proportional to the ratio of the transformation performed before the rotational movement.

6. An object detection unit that detects the presence or absence of an object based on the learning results of the learning data generated by the image generation device according to any one of claims 1 to 5, The imaging unit performs X-ray imaging of the object being inspected during an X-ray inspection, The X-ray image of the object to be inspected, acquired by the imaging unit, is detected by the object detection unit, and a determination unit determines whether or not the object is present in the object to be inspected. An object detection device equipped with the following features.

Images