Luggage inspection system, luggage inspection method and luggage inspection program

Abstract

To provide technique for maintaining the precision of object detection at a predetermined standard or higher.SOLUTION: A luggage inspection is constituted which comprises: an X-ray image acquisition unit which acquires an X-ray image; a colored image acquisition unit which acquires a colored image formed by coloring objects included in the X-ray image in colors of materials of the objects, respectively; a corrected image generation unit which generates a corrected image by inputting the colored image to a corrected image generation model so machine-learnt as to receive an input of the colored image and generate the corrected image having different objects colored in different colors; an object detection unit which detects the objects by inputting the corrected image to an object detection model so machine-learnt as to receive an input of the corrected image and detect the objects included in the corrected image; and a model update unit which updates the corrected image generation model and the object detection model when detection precision of the objects by the object detection model is smaller than a predetermined standard.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a baggage inspection system, a baggage inspection method, and a baggage inspection program. [Background technology]

[0002] There is known a baggage inspection system that inspects the baggage of participants at airports, event venues, etc. For example, Patent Document 1 discloses a technology for recognizing an object using a learning model in an X-ray inspection device capable of taking images from multiple directions. It also discloses a technology for determining the material of an object and generating a color image that distinguishes the material by color and the density by darkness. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-13217 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, it was sometimes difficult to distinguish between objects. For example, assume that a dangerous object such as a blade is the detection target. In this case, if there are multiple objects to be detected, the colors and shading of each object are similar to each other, making it difficult to distinguish them. In addition, even if there is an object other than a dangerous object, if there is a metal object, it becomes difficult to distinguish it from the object to be detected. Furthermore, when objects overlap, there is no boundary between the different objects, and the image becomes connected, making it difficult to distinguish between the objects. One solution to this problem is to correct the colors so that each object has a different color, but it has traditionally been difficult to automatically correct the colors so that objects can be easily distinguished from one another, which has made it difficult to detect objects from the corrected image. The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a technique for correcting colors so as to improve the accuracy of object detection. [Means for solving the problem]

[0005] In order to achieve the above-mentioned object, the baggage inspection system includes an X-ray image acquisition unit that acquires an X-ray image indicating the amount of transmission of X-rays irradiated to a baggage to be inspected; a colored image acquisition unit that acquires a colored image in which objects included in the X-ray image are colored with the color of each material of the objects; a corrected image generation unit that inputs the colored image and generates a corrected image using a corrected image generation model that has been machine-learned to input the colored image and generate corrected images in which different objects are colored with different colors; an object detection unit that inputs the corrected image and detects the objects using an object detection model that has been machine-learned to input the corrected image and detect the objects included in the corrected image; and a model update unit that updates the corrected image generation model and the object detection model when the detection accuracy of the object by the object detection model is less than a predetermined standard.

[0006] That is, in the baggage inspection system, an X-ray image of a baggage is colored in the color of each material of the object, and the colored image after coloring is input to a correction image generation model and corrected so that each object has a different color. Furthermore, the corrected image after correction is input to an object detection model to perform object detection. In this way, according to a baggage inspection system capable of executing object detection for baggage inspection using a correction image generation model and an object detection model, it is possible to automatically perform both color correction to the color of each object and detection of the object after correction. Furthermore, by advancing machine learning, the accuracy of object detection can be improved. Furthermore, it is possible to prevent the accuracy of object detection from becoming smaller than a predetermined standard, and to maintain the accuracy of object detection at or above a predetermined standard. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of a baggage inspection system. [Diagram 2]FIG. 1 is a diagram for illustrating a colored image. [Diagram 3] FIG. 2 is a diagram illustrating a generative model and a discriminative model. [Figure 4] FIG. 11 is a diagram illustrating an example of object detection and a relationship between the correct answer value. [Diagram 5] FIG. 1 is a diagram illustrating an example of object detection. [Figure 6] 1 is a flowchart of a machine learning process. [Figure 7] FIG. 1 is a diagram for explaining IoU. [Figure 8] 13 is a flowchart of an inspection process. [Figure 9] 13 is a flowchart of a machine learning model update process. [Figure 10] FIG. 13 is a diagram for explaining updating teacher data. [Figure 11] 13 is a flowchart of an inspection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Here, the embodiments of the present invention will be described in the following order. (1) Baggage inspection system configuration: (2) Machine learning system configuration: (3) Inspection process: (4) Machine learning model update process: (5) Other embodiments:

[0009] (1) Baggage inspection system configuration: FIG. 1 is a block diagram showing the configuration of a baggage inspection system. The baggage inspection system according to this embodiment is used to inspect baggage carried by a user at a security checkpoint in an airport, an event venue, or the like. In this embodiment, the baggage inspection system includes an image analysis system 10 and a machine learning system 100. In this embodiment, the image analysis system 10 analyzes an X-ray image taken by an X-ray imaging device 40 and performs various analysis processes. The analysis processes include a process of detecting an object included in the X-ray image (i.e., an object included in the baggage), and the like. The image analysis system 10 includes a control unit 20, a storage medium 30, an X-ray imaging device 40, a display unit 41, an input unit 42, and a communication unit 50.

[0010] The X-ray imaging device 40 is a device that irradiates X-rays on luggage and captures an X-ray image showing the amount of X-rays that have penetrated the luggage. The X-ray imaging device 40 includes an X-ray output unit (not shown), an X-ray sensor, a housing that surrounds the X-ray output unit and the X-ray sensor, and a transport unit that transports the luggage to a location in the housing where the X-rays are irradiated. That is, when a user places his / her luggage on the transport unit, the luggage is transported into the housing by the transport unit. In the housing, the luggage is irradiated with X-rays output from the X-ray output unit, and the X-rays that have penetrated the luggage are detected by the X-ray sensor. X-ray image data showing the X-ray image, which is the detection result, is output to the control unit 20.

[0011] The display unit 41 is a display capable of displaying any image. In this embodiment, the display unit 41 provides various information to an inspector who is an operator of the image analysis system 10. The input unit 42 is an input device that inputs various information in response to an operation by an operator, such as a keyboard, a mouse, or a touch panel. In this embodiment, the input unit 42 is an input device that is operated by an inspector who is an operator of the image analysis system 10.

[0012] The communication unit 50 is a device that communicates with the machine learning system 100. The mode of communication is not limited, and wired communication or wireless communication can be used. The control unit 20 can transmit various information to the machine learning system 100 via the communication unit 50, and can receive various information from the machine learning system 100. The machine learning system 100 and the image analysis system 10 may be installed in different locations. Furthermore, one machine learning system 100 may be able to communicate with multiple image analysis systems 10.

[0013] The control unit 20 includes a CPU, RAM, ROM, etc. (not shown), and can execute various programs recorded in a storage medium 30, etc. In this embodiment, an X-ray image taken by an X-ray imaging device 40 is processed by an image analysis system 10, and objects included in the X-ray image are detected using an object detection model trained by machine learning. If a dangerous object is included in the detected objects, the image analysis system 10 issues a warning.

[0014] The object detection model is machine-trained in advance so that it can detect objects contained in X-ray images, but during the operation of the image analysis system 10, the object detection accuracy may decrease from the initial operation. If the object detection accuracy decreases, dangerous objects that should be detected may be overlooked, or a false report may be generated in which an object that is not dangerous is deemed to be a dangerous object. Therefore, in this embodiment, the object detection accuracy is monitored and managed so that it does not decrease below a predetermined standard.

[0015] In this embodiment, the control unit 20 executes the image analysis program 21 to perform the process including the management of the object detection model as described above. To perform the process, the image analysis program 21 includes the functions of the X-ray image acquisition unit 21a, the colored image acquisition unit 21b, the corrected image generation unit 21c, the object detection unit 21d, the output control unit 21e, and the model update unit 21f. The X-ray image acquisition unit 21a causes the control unit 20 to execute the function of acquiring an X-ray image showing the amount of penetration of the X-ray irradiated to the baggage to be inspected. That is, the control unit 20 controls the X-ray imaging device 40 to irradiate the baggage with X-rays and to cause the X-ray sensor to detect the X-ray that has penetrated the baggage. The X-ray imaging device 40 outputs the X-ray image showing the detection result of each baggage as X-ray image data 30a. The control unit 20 acquires the X-ray image data 30a and stores it in the storage medium 30.

[0016] The X-ray image represented by the X-ray image data 30a is expressed by a gradation value indicating the amount of transmission of the X-ray irradiated to the baggage, and therefore, for example, the image is an image in which the object is expressed in shades of gray. In addition, when the baggage to be inspected is a bag, etc., since multiple objects are overlapped within the baggage, it is often difficult to distinguish each object in the X-ray image expressed in shades of gray. Therefore, in this embodiment, the objects included in the X-ray image are colored according to the color of their material.

[0017] The colored image acquisition unit 21b is a function of acquiring a colored image in which an object included in an X-ray image is colored in the color of each material of the object. That is, the control unit 20 converts the color of each object included in the X-ray image into the color of each material based on the X-ray image by the function of the colored image acquisition unit 21b. Such color conversion can be performed by various methods. For example, a method is irradiating X-rays of different energies to a baggage in the X-ray imaging device 40, acquiring the difference between X-ray images taken with X-rays of different energies, thereby emphasizing objects made of materials with different mass absorption coefficients and identifying objects made of the materials. In this case, a specific type of material is colored in a specific color, and other colors are colored in a color different from the specific color. Also, coloring may be performed based on the shade of gray in the X-ray image. In this case, the gray density in the X-ray image is divided into a plurality of levels, and the images belonging to each level are colored in a different color. Of course, these coloring methods may be combined, or other methods may be adopted.

[0018] When the coloring is performed as described above, colored image data 30b representing the colored image is stored in the storage medium 30. In this embodiment, the colored image is an image colored in a chromatic color. That is, the X-ray image is a gray image expressed by a gradation value of one channel, whereas the colored image is a color image expressed by a gradation value of three channels. The control unit 20 colors each object in a color that is predetermined for each material of the object. For example, a color corresponding to the material is predetermined, such as blue for metal and red for resin, and the object is colored in the color for each material. Of course, the shade of the color, such as blue or red, may be adjusted according to the gradation value of the X-ray image before coloring.

[0019] FIG. 2 is a diagram for illustrating a colored image. In FIG. 2, the image of the object is expressed in gray due to color restrictions on the drawing, but in actual examples, the image of the object is often colored in a chromatic color. The bag shown in FIG. 2 contains metal tools and blades, and a rectangular object made of resin. In this example, the appearance of the metal tools and blades and the rectangular object made of resin being colored in different colors is expressed by the difference in gray color.

[0020] When the coloring is performed as described above, the control unit 20 stores colored image data 30b representing the colored image in the storage medium 30. The colored image represented by the colored image data 30b makes it easier to distinguish between objects of different materials in the image by color. That is, it makes it easier to distinguish between metal objects (hazardous materials) to be detected and non-metallic objects such as resin. However, if objects of the same material overlap or are adjacent to each other in the luggage, it can still be difficult to distinguish between the two.

[0021] Therefore, in this embodiment, the colored image is corrected to generate a corrected image colored with the color of each object. The corrected image generating unit 21c causes the control unit 20 to execute a function of inputting the colored image to the corrected image generation model 30c and generating a corrected image.

[0022] The corrected image generation model 30c is a model that has been machine-learned to input the colored image represented by the colored image data 30b and generate a corrected image in which different objects are colored in different colors. Specifically, the corrected image generation model 30c is information indicating an equation for deriving a correspondence between the two data when the gradation value in each channel for each pixel of the input image data is set as an input value and the gradation value in each channel for each pixel of the image data of the generated image is set as an output value.

[0023] The corrected image generation model 30c changes the hue and saturation of the color in the colored image for each object, thereby coloring the object with the color of each object. Such coloring by the corrected image generation model 30c is realized by the fact that the teacher data of the corrected image generation model 30c is a pair of images corresponding to the change in color before and after coloring (details will be described later).

[0024] In this embodiment, the corrected image generation model 30c is created in advance by the machine learning system 100 before the image analysis system 10 is operated, and is stored in the storage medium 30. Furthermore, in this embodiment, additional machine learning is performed on the corrected image generation model 30c created in advance, and the corrected image generation model 30c can be updated during the operation process. The creation and update of the corrected image generation model 30c will be described later, but in this embodiment, machine learning is performed so that the detection accuracy of the object based on the corrected image is increased. When the X-ray image data 30a is output from the X-ray imaging device 40 and the colored image data 30b is generated based on the X-ray image data 30a, the control unit 20 inputs the colored image data 30b into the corrected image generation model 30c to generate a corrected image. The corrected image data 30d indicating the corrected image is stored in the storage medium 30.

[0025] The object detection unit 21d causes the control unit 20 to execute a function of inputting a corrected image to the object detection model 30e and detecting an object. The object detection model 30e is a model trained by machine learning to input the corrected image indicated by the corrected image data 30d and detect an object included in the corrected image. Specifically, the object detection model 30e is a model trained by machine learning to use the gradation value in each channel for each pixel of the input corrected image as an input value, and to output the position of a detection box, which is a rectangular frame surrounding an image of the object, and the reliability of a class indicating the type of object photographed within the detection box.

[0026] In this embodiment, the object detection model 30e is created in advance by the machine learning system 100 before the image analysis system 10 is operated, and is stored in the storage medium 30. Furthermore, in this embodiment, additional machine learning is performed on the object detection model 30e created in advance, and the object detection model 30e can be updated during operation. The creation and update of the object detection model 30e will be described later, but in this embodiment, machine learning is performed so that the detection accuracy of the object based on the corrected image is increased. In this embodiment, the control unit 20 performs object detection using the object detection model 30e for each of the corrected image data 30d, and stores the object detection results in the storage medium 30 as detection data 30f.

[0027] The output control unit 21e causes the control unit 20 to execute a function of outputting various information. In this embodiment, the output of the object detection result and the output of information for updating the machine learning model are performed. Specifically, the control unit 20 controls the display unit 41 to output information indicating the object detection result. As a result, an inspector as an operator of the image analysis system 10 can visually confirm the object detection result. Various modes can be used for outputting the object detection result. In this embodiment, a rectangular frame surrounding the object detected in the corrected image is superimposed on the corrected image, and information indicating the detected object is displayed on the display unit 41 in a state in which it is associated with the frame. When the object detection result is displayed on the display unit 41 by the output control unit 21e, the inspector can grasp the object detection result based on the display content.

[0028] The destination of the information for updating the machine learning model is the machine learning system 100. In this embodiment, when the corrected image generation model 30c and the object detection model 30e are updated, the colored image data 30b, the corrected image data 30d, and the detection data 30f (including the object detection result by the inspector described later) generated after the image analysis system 10 starts operating are referenced. Then, the output control unit 21e controls the communication unit 50 to transmit these pieces of information to the machine learning system 100. It is sufficient that these pieces of information are transmitted so that they can be referenced when the machine learning model is updated, and they may be transmitted whenever they are updated, or may be transmitted collectively when the update is performed.

[0029] The model update unit 21f causes the control unit 20 to execute a function of updating the corrected image generation model 30c and the object detection model 30e when the object detection accuracy by the object detection model 30e is lower than a predetermined standard. That is, the control unit 20 monitors the detection accuracy by the object detection model 30e, and when there is a possibility that the detection accuracy will become lower than the predetermined standard, updates the corrected image generation model 30c and the object detection model 30e to prevent a decrease in the detection accuracy.

[0030] In this embodiment, the inspector himself / herself also detects the object in order to evaluate the object detection accuracy. That is, in this embodiment, the object detection accuracy by the inspector is estimated to be higher than the object detection accuracy by the object detection model 30e. In addition, it is desired that the object detection accuracy by the object detection model 30e is equal to or higher than the object detection accuracy by the inspector. Therefore, if the object detection result by the inspector is regarded as the correct answer, it is possible to evaluate whether the object detection result by the object detection model 30e is accurate or not. The object detection by the inspector is used when determining whether the machine learning model should be updated. Therefore, the object detection by the inspector may be performed continuously during the operation of the image analysis system 10, or may be performed in a specific case, for example, when performing a process to determine whether the machine learning model should be updated.

[0031] When detecting an object, the inspector visually checks the corrected image displayed on the display unit 41 and identifies the object contained in the corrected image at his / her own discretion. Then, the inspector operates the input unit 42 to input the position and type of the detected object. The information may be input in various ways, for example, a configuration in which the position is specified by a rectangular frame surrounding the image of the object and the type of the object is specified by selecting a class indicating a predetermined type of object, etc. Of course, other ways may be used, for example, a way in which the correctness of the detection result by the object detection model 30e is input. When the detection result is input by the inspector, the control unit 20 associates the input information with information indicating that it is a detection result by the inspector, and adds it to the detection data 30f.

[0032] The control unit 20 refers to the detection data 30f and specifies the object detection accuracy based on the object detection result by the inspector and the object detection result by the object detection model 30e. The detection accuracy can be evaluated, for example, by a match rate indicating the ratio at which the object detection result by the object detection model 30e matches the object detection result by the inspector. In this embodiment, a lower limit value of the detection accuracy (e.g., 80%) is determined in advance as a predetermined standard. Therefore, the control unit 20 compares the detection accuracy with the predetermined standard, and if the detection accuracy is smaller than the predetermined standard, it considers that the machine learning model should be updated. When updating is performed, the control unit 20 transmits information instructing the machine learning system 100 to update the machine learning model via the communication unit 50.

[0033] When the instruction is given, the machine learning system 100 updates the corrected image generation model 30c and the object detection model 30e based on the information transmitted under the control of the output control unit 21e. That is, the corrected image generation model 30c and the object detection model 30e are machine-learned based on the updating teacher data including the colored image data 30b indicating the colored image acquired after the operation of the baggage inspection system is started, the corrected image data 30d indicating the corrected image generated by inputting the colored image indicated by the colored image data 30b into the corrected image generation model 30c, and the object detection result by the inspector indicated by the detection data 30f. The corrected image generation model 30c and the object detection model 30e obtained by the machine learning are transmitted to the image analysis system 10 via the communication unit 500. The control unit 20 acquires the corrected image generation model 30c and the object detection model 30e and updates these models by storing them in the storage medium 30.

[0034] In the above configuration, the corrected image generation model 30c and the object detection model 30e are machine-learned to improve the object detection accuracy. Therefore, it is possible to automatically perform color correction to obtain the color of each object and detection of the object after correction. Furthermore, by advancing machine learning, the accuracy of object detection can be improved. Furthermore, according to the above configuration, when the object detection accuracy by the object detection model 30e is lower than a predetermined standard, the corrected image generation model 30c and the object detection model 30e are updated. Therefore, even if the detection accuracy by the object detection model 30e temporarily becomes lower than the predetermined standard, the accuracy of object detection can be maintained at or above the predetermined standard by updating.

[0035] Furthermore, in this embodiment, the object detection accuracy of the object detection model 30e can be evaluated based on the inspection result of the object by the inspector. Also, the teacher data for updating can be defined based on the inspection result of the object by the inspector. Therefore, after the baggage inspection system starts operating, it is possible to update the corrected image generation model 30c and the object detection model 30e based on the actual X-ray image obtained after the operation starts.

[0036] Furthermore, in this embodiment, the corrected image is generated so that the object has its own color. Therefore, the detection target object is easily distinguishable. As a result, the detection accuracy of the object detection model 30e can be easily improved.

[0037] (2) Machine learning system configuration: Next, a configuration of the machine learning system 100 will be described. The machine learning system 100 includes a control unit 200, a storage medium 300, a display unit 410, and a communication unit 500. The display unit 410 is a display device that displays images. The communication unit 500 is a device that communicates with the image analysis system 10.

[0038] The control unit 200 includes a CPU, RAM, ROM, a GPU (Graphics Processing Unit), etc. (not shown), and can execute various programs recorded in a storage medium 300, etc. In this embodiment, the control unit 200 can execute a management program 210 for generating a corrected image generation model 30c and an object detection model 30e and outputting various information. When the management program 210 is executed, the control unit 20 functions as a machine learning unit 210a, an object detection unit 210b, and an output control unit 210c.

[0039] The machine learning unit 210a causes the control unit 200 to execute a machine learning process that optimizes a training model that forms a neural network. In this embodiment, the learning targets are a corrected image generation model 30c and an object detection model 30e. The corrected image generation model 30c may take various forms, but in this embodiment, it is assumed that a model called pix2pix is ​​machine-learned.

[0040] FIG. 3 is a diagram showing a schematic diagram of a corrected image generation model in this embodiment. In this embodiment, a generation model G and a discrimination model D are prepared as training models, and are alternately trained by competing with each other in an adversarial manner. The generation model G is a model that inputs an image and generates another image, and when machine learning is completed, the generation model G becomes a corrected image generation model 30c. Here, an image input to the generation model G is called an input image Ii, and an image output from the generation model G is called a generated image Ig. In machine learning, the input image Ii and the generated image Ig generated from the input image Ii are paired. In addition, a plurality of input images Ii are prepared. Therefore, for each input image Ii, there is a paired generated image Ig. In the upper center of FIG. 3, an arrow indicates that the input image Ii and the generated image Ig are paired. In FIG. 3, the colored image Ic and the generated image Ig are expressed in gray due to color restrictions on the drawing, but in the colored image Ic, metal objects are expressed in almost the same color, and in the generated image Ig, metal objects are expressed in different colors for each object.

[0041] Two image pairs are input to the discrimination model D. The input image pair is a pair of an input image Ii and a generated image Ig, or a pair of two images associated by the teacher data 300a. The teacher data 300a is defined by preparing a plurality of pairs of an input image Ii in which an object is colored in the color of each material, and a target image It in which the hue and saturation of the color included in the input image Ii are changed for each object and colored in the color of each object. In this embodiment, the input image Ii included in the teacher data 300a is also an image input to the generation model G during machine learning. The input image Ii is also a colored image Ic in which an X-ray image is colored by the function of the colored image acquisition unit 21b.

[0042] On the other hand, the target image It is an image in which an object is colored in a color for each object, and is generated in advance. The target image It can be generated, for example, as follows. Images included in the colored image are classified into images for each object, and the color of each pixel constituting the image of each object is converted to another color with at least one of a hue and a saturation different from the others. At this time, the conversion is performed so that the colors of the pixels constituting the same object become the same color. That is, the colors are converted so that the same object becomes the same color and different objects become different colors. Note that, in this embodiment, objects of any material included in the colored image are colored in the color for each object. Therefore, both objects made of a specific material that is the material of the hazardous object to be detected and objects that are not made of a specific material are colored for each object.

[0043] In the teacher data 300a, multiple pairs of an input image Ii and a target image It are also prepared. In the lower center of Fig. 3, arrows indicate how the input image Ii and the target image It are associated with each other in the teacher data 300a.

[0044] The teacher data 300a may be generated by various methods. For example, X-ray images of a plurality of packages are actually taken by the X-ray imaging device 40, and the X-ray images are converted into colored images Ic to prepare a plurality of colored images Ic. Furthermore, objects are identified from the colored images Ic, and a target image It is generated by performing coloring for each object.

[0045] As described above, by utilizing the fact that the target image It corresponding to the input image Ii is defined by the teacher data 300a, the discrimination model D is trained to discriminate whether or not the two input image pairs are a combination included in the teacher data 300a. That is, when the input image pair is the input image Ii and the target image It, the correct answer label is defined as real, and when the input image pair is the pair of the input image Ii and the generated image Ig, the correct answer label is defined as fake.

[0046] The discrimination model D is trained using real and fake prepared as correct labels so that when the input image pair is a pair of an input image Ii and a target image It (correct label is true), the value of the discrimination signal output approaches a value indicating real. Also, when the input image pair is a pair of an input image Ii and a generated image Ig (correct label is false), the discrimination model D is trained so that the value of the discrimination signal output approaches a value indicating fake.

[0047] Furthermore, the generative model G is trained so that when a pair of an input image Ii and a generated image Ig is input to the discrimination model D, the value of the discrimination signal output is close to a value indicating real (the generated image Ig is mistaken for the target image It). In other words, training is performed so that the generated image Ig is close to the target image It so that it cannot be distinguished from the target image It when a colored image Ic is input to the generative model G as the input image Ii. Since the target image It is an image in a state where the objects included in the colored image Ic have been converted to have the color of each object, when the generated image Ig is trained so that it cannot be distinguished from the target image It, the generated image Ig can be said to be a corrected image Ia in a state where the objects included in the colored image Ic have been corrected to have the color of each object.

[0048] In this embodiment, when learning the generative model G, machine learning is further performed so that the detection accuracy of an object based on the corrected image Ia generated by the generative model G is improved. That is, machine learning is performed so that the difference between the detection result of the object included in the generated image Ig and the correct value indicating the object included in the input image is reduced. When the machine learning is completed and the generated image Ig and the target image It cannot be distinguished from each other, the generated image Ig can be said to be a corrected image Ia colored with the color of each object, and therefore machine learning is performed so that the detection accuracy of the object based on the corrected image Ia generated by the generative model G is improved.

[0049] In order to perform the above-described machine learning, the teacher data 300a is prepared in advance before the machine learning is performed. As described above, the teacher data 300a includes a plurality of pairs of the input image Ii and the target image It, but in this embodiment, the teacher data 300a further includes a corrective value of the object detection as information for generating the corrected image Ia so as to improve the object detection accuracy.

[0050] The correct answer value is information indicating the position of an object contained in the target image It in the image and a class that is the type of the object. The correct answer value may be in various formats, but in this embodiment, the correct answer value is configured by information indicating the position and class of a correct answer box surrounding the object. In FIG. 3, the correct answer box Bt in the target image It is illustrated by a dashed line. The position of the rectangle can be expressed, for example, by the coordinates of two diagonally opposite vertices. The class can be expressed, for example, by a numerical value associated with the type of dangerous object to be detected, such as a wrench, pliers, or blade.

[0051] FIG. 4 is a diagram showing a schematic diagram of the object detection model 30e in this embodiment. The object detection model 30e in this embodiment is a single shot multibox detector (SSD). The input image Ii, generated image Ig, and teacher data 300a shown on the left side of FIG. 4 are the same as the images shown in FIG. 3. The object detection model 30e is a model that inputs an image and outputs the position of a detection box indicating an object detected in the image and the reliability of a class indicating the type of object photographed in the detection box. In this embodiment, when a generated image Ig by the generation model G is input to the object detection model 30e, the detection result for each object in the generated image Ig is output as the position of the detection box and the reliability of the class. In FIG. 4, the detection box B1 to be output is illustrated on the right side of the object detection model 30e.

[0052] FIG. 5 is a diagram for explaining an example of object detection. In the example in this embodiment, an object detection result associated with each of the default boxes previously set in the image is output. In the example shown in FIG. 5, an example of a generated image Ig is shown at the top of the figure, and 16 default boxes are set so as to overlap the generated image Ig. When object detection is performed by the function of the object detection unit 210b, an object detection result for the object included in each of the default boxes is output. Although the output format is not limited, in this example, a default box, a detection box which is a rectangular frame surrounding an image of a detected object, and a reliability of a class indicating the type of object photographed in the detection box are output. In FIG. 5, a default box is illustrated by a solid line, and a detection box is illustrated by a dashed line.

[0053] The format of the information indicating the detection result is not limited, but for example, in the case of the default box DB1 in FIG. 5, the default box DB1 is expressed by information indicating the center of gravity coordinates and the width and height of the box. The detection box B1 is also expressed by information indicating the center of gravity coordinates and the width and height of the box, but the detection box B1 is expressed by the difference between the center of gravity coordinates of the default box DB1 and the center of gravity coordinates of the detection box B1, the difference between the width of the default box DB1 and the width of the detection box B1, and the difference between the height of the default box DB1 and the height of the detection box B1. The reliability of the class is expressed by a value that increases as the possibility that the detected object is a wrench, pliers, blade, etc. increases. Therefore, if the maximum value of the reliability of the class is equal to or greater than the reference value, it can be estimated that an object of the class has been detected. The size and number of the default boxes are not limited to the example shown in FIG. 5, and may be more or less, or default boxes of different sizes may be used in combination.

[0054] In machine learning, a plurality of generated images Ig are prepared as shown in FIG. 4. Therefore, a detection result exists for each generated image Ig. In this embodiment, machine learning is performed by utilizing the fact that the correct answer value of the object included in the target image It is included in the teacher data 300a.

[0055] Specifically, by comparing the detection result of an object obtained by inputting the generated image Ig to the object detection model 30e with the correct answer value included in the teacher data 300a, the accuracy of object detection based on the generated image Ig can be evaluated. In FIG. 5, the correct answer box corresponding to the correct answer value of the target image It is illustrated in the lower part by a dashed line. In the example shown in FIG. 5, by comparing the detection box B1 of the generated image Ig with the correct answer box Bt, it is possible to evaluate how close the detection result by the detection box B1 is to the correct answer value. The object detection model 30e is machine-learned using the evaluation. That is, the machine learning is performed so that the detection result of the object included in the input generated image Ig becomes close to the correct answer value included in the teacher data 300a.

[0056] When the teacher data 300a is prepared, machine learning can be performed using the teacher data 300a. Fig. 6 is a flowchart showing the machine learning process. In this embodiment, the machine learning process is performed in advance before the operation of the image analysis system 10 is started.

[0057] When the machine learning process is started, the control unit 200 initializes a variable Cycle for counting cycles to 0 by the function of the machine learning unit 210a, and sets the learning conditions (step S100). Here, the learning conditions include various settings necessary for learning, such as the definition of the training model of the generative model G, the discriminative model D, and the object detection model 30e, that is, the settings of the filter, activation function, loss function, and initial values ​​of variable parameters that indicate the model. In addition, the learning conditions include settings of the number of training data and test data extracted from the teacher data 300a, the image size to be handled, and the threshold value to be compared with Cycle.

[0058] Next, the control unit 200 acquires the teacher data 300a by the function of the machine learning unit 210a (step S105). Next, the control unit 200 outputs a generated image Ig by the generative model G by the function of the machine learning unit 210a (step S110). That is, the control unit 200 acquires an input image Ii by referring to the teacher data 300a, and inputs it to the generative model G at the current time point to acquire a generated image Ig.

[0059] Next, the control unit 200 performs object detection using the function of the object detection unit 210b (step S115). The function of the object detection unit 210b is the same as that of the object detection unit 21d. The control unit 200 inputs the target image It to the current object detection model 30e and performs object detection. The control unit 200 also inputs the generated image Ig to the current object detection model 30e and performs object detection.

[0060] Next, the control unit 200 acquires a loss l_loss related to image evaluation based on the target image It and the generated image Ig using the function of the machine learning unit 210a (step S120). The loss can be acquired, for example, by a method similar to the well-known DCGAN (Deep Convolutional Generative Adversarial Networks), and may indicate the difference between the generated image Ig and the target image It.

[0061] Next, the control unit 200 acquires losses E_loss0 and E_loss1 related to the accuracy of the detection result from the object detection result by using the function of the machine learning unit 210a (step S125). That is, the control unit 200 acquires an index for comparing the object detection result based on the target image It acquired in step S115 with the correct answer value, and defines the loss E_loss0. Also, the control unit 200 acquires an index for comparing the object detection result based on the generated image Ig acquired in step S115 with the correct answer value, and defines the loss E_loss1. In the former case, the accuracy of object detection by the object detection model 30e can be evaluated, and in the latter case, the detection accuracy when object detection is performed by the object detection model 30e on an image corrected by the corrected image generation model 30c can be evaluated.

[0062] The loss may be defined in various ways. For example, the loss can be defined by obtaining an index for evaluating the position accuracy and the class estimation accuracy between the correct answer and the object detection result. In this embodiment, the IoU (Intersection over Union) is obtained for the evaluation. Figures 7A and 7B are diagrams for explaining IoU. In Figures 7A and 7B, the correct answer box Bt is indicated by a dashed line, and the detection box SB is indicated by a dashed line.

[0063] In Fig. 7A, the overlapping portion where the correct answer box Bt and the detection box SB overlap is shown in gray, and in Fig. 7B, the joining portion, which is a portion included in at least one of the correct answer box Bt and the detection box SB, is shown in gray. IoU is defined as the area of ​​the overlapping portion / the area of ​​the joining portion. If the IoU is large, the accuracy of object detection is estimated to be high, but if the IoU is small, the accuracy of object detection is estimated to be low.

[0064] Here, if the IoU is equal to or greater than a threshold (e.g., 0.5), the detection box is called a positive box, and if the IoU is smaller than the threshold, the detection box is called a negative box. The control unit 200 obtains the IoU for each detection box obtained for each default box by comparing it with the ground truth box Bt.

[0065] Furthermore, the control unit 200 calculates the error between the position of the correct answer box Bt indicated by the correct answer value and the position of the detection box SB for the positive box, and substitutes the error into an activation function such as an error output control function to obtain a loss for evaluating the position accuracy. Furthermore, the control unit 200 evaluates the error between the class indicated by the correct answer value and the reliability of the class corresponding to the detection box SB for the positive box and the negative box, for example, by cross entropy, to obtain a loss for evaluating the class estimation accuracy. These losses are multiplied by a predetermined coefficient and normalized. The control unit 200 obtains the above-mentioned IoU and the loss based on the IoU based on the object detection result obtained from the target image It for all of the target images It, and obtains the sum of the losses as E_loss0. Furthermore, the control unit 200 obtains the above-mentioned IoU and the loss based on the IoU based on the object detection result obtained from the generated image Ig for all of the generated images Ig obtained from the input image Ii, and obtains the sum of the losses as E_loss1.

[0066] Next, the control unit 200 acquires a synthesis loss for the corrected image generation model using the function of the machine learning unit 210a (step S130). That is, the control unit 200 acquires the sum of the losses acquired in steps S120 and S125. Next, the control unit 200 adjusts parameters based on the synthesis loss using the function of the machine learning unit 210a (step S135). A publicly known algorithm can be used as the algorithm for adjusting the parameters. At this time, machine learning of the discriminative model D is also performed using a publicly known algorithm. That is, the control unit 200 fixes one of the generative model G and the discriminative model D and performs machine learning of the other, and further fixes the other with the learned parameters and performs machine learning of one.

[0067] According to the above configuration, the generative model G is trained so that the generated image Ig and the target image It cannot be distinguished from each other. When the generated image Ig and the target image It cannot be distinguished from each other, the generated image Ig can be said to be the corrected image Ia. Furthermore, in this embodiment, since the synthesis loss acquired in step S130 includes a loss related to the accuracy of the object detection result, the generative model G is trained so as to improve the accuracy of object detection based on the generated image Ig. Therefore, according to this embodiment, the generative model G is trained so as to improve the accuracy of object detection based on the corrected image Ia.

[0068] In this embodiment, the losses E_loss0 and E_loss1 and their composite sum are used when machine learning the generative model G, but of course the losses E_loss0 and E_loss1 and their composite sum may also be used when machine learning the discriminative model D. Furthermore, when calculating the composite loss in step S130, a predetermined offset value may be added to the weighted sum of the values ​​acquired in S120 and S125.

[0069] Next, the control unit 200 acquires a composite loss for the object detection model by using the function of the machine learning unit 210a (step S140). That is, the control unit 200 acquires the loss acquired in step S125. When calculating the loss, normalization and the like equivalent to step S125 are performed, but the coefficients, offset values, and the like at that time may be the same as or different from those in step S125.

[0070] Next, the control unit 200 adjusts parameters based on the loss for the object detection model by using the function of the machine learning unit 210a (step S145). A publicly known algorithm can be used as the parameter adjustment algorithm. That is, the control unit 200 performs machine learning on the object detection model 30e using various methods known as SSD learning algorithms. According to the above configuration, the object detection model 30e is trained so that the object detection result based on the generated image Ig approaches the correct value of the object included in the target image It.

[0071] Next, the control unit 200 determines whether the loss is smaller than a predetermined threshold value (step S150). That is, the control unit 200 determines whether the loss after parameter adjustment is smaller than the threshold value. If it is not determined that the loss after parameter adjustment is smaller than the threshold value, the control unit 200 increments the variable Cycle (step S155). Then, the control unit 200 determines whether the variable Cycle is equal to or greater than a maximum value (step S160), and if it is not determined that the variable Cycle is equal to or greater than the maximum value, the control unit 200 repeats the processes from step S105 onwards.

[0072] On the other hand, if it is determined in step S150 that the synthesis loss is smaller than a predetermined threshold value, or if it is determined in step S160 that the variable Cycle is equal to or greater than the maximum value, the control unit 200 records the machine learning model (step S165). That is, the control unit 200 stores the trained generation model G as the corrected image generation model 30c in the storage medium 300. The control unit 200 also stores the trained object detection model 30e in the storage medium 300. Furthermore, the corrected image generation model 30c and the object detection model 30e are transmitted to the image analysis system 10 and are also stored in the storage medium 30. Before storing the corrected image generation model 30c in step S165, it may be determined that learning is complete using test data. If learning does not proceed as planned, machine learning may be performed again by changing the learning conditions, for example.

[0073] According to the above process, machine learning is performed so that the difference between the detection result of an object included in the corrected image Ia outputted from inputting the colored image Ic to the corrected image generation model 30c and the correct value becomes small. As a result, the corrected image generation model 30c and the object detection model 30e can be generated so that object detection can be performed with high accuracy.

[0074] (3) Inspection process: Next, the inspection process executed in the image analysis system 10 will be described with reference to the flowchart shown in FIG. 8. The inspection process is executed after the corrected image generation model 30c and the object detection model 30e are recorded in the storage medium 30. When a user places his / her baggage on the transport section of the X-ray imaging device 40, the baggage is transported into the housing and an X-ray image is taken. When the X-ray image is taken, X-ray image data 30a indicating the X-ray image is output from the X-ray imaging device 40. When the X-ray image data 30a is output from the X-ray imaging device 40 and stored in the storage medium 30, the inspection process is executed.

[0075] When the inspection process is started, the control unit 20 acquires an X-ray image by using the function of the X-ray image acquisition unit 21a (step S200). That is, the control unit 20 stores X-ray image data 30a indicating the X-ray image output from the X-ray imaging device 40 in the storage medium 30. When material identification is performed based on X-ray images of different energies, X-ray image data 30a indicating multiple X-ray images of the same baggage taken with different energies is acquired.

[0076] Next, the control unit 20 performs material identification based on the X-ray image by using the function of the colored image acquisition unit 21b (step S205). That is, the control unit 20 classifies the image of the object included in the X-ray image into images of each material based on the X-ray image acquired in step S200. For example, when the material is identified based on X-ray images of different energies, the image of the object is classified into materials based on the difference between the X-ray images. Next, the control unit 20 acquires a colored image by using the function of the colored image acquisition unit 21b (step S210). That is, the control unit 20 colors the X-ray image acquired in step S200 with the color of each material based on the material identification result in step S205. The acquired colored image is stored in the storage medium 30 as colored image data 30b.

[0077] Next, the control unit 20 inputs the colored image acquired in step S210 to the correction image generation model 30c by the function of the correction image generation unit 21c (step S215). Next, the control unit 20 acquires a correction image by the function of the correction image generation unit 21c (step S220). That is, the control unit 20 regards the generated image Ig output from the correction image generation model 30c as the correction image Ia, and stores the correction image data 30d indicating the correction image Ia in the storage medium 30.

[0078] Next, the control unit 20 detects an object included in the corrected image Ia by using the function of the object detection unit 21d (step S225). That is, the control unit 20 inputs the corrected image Ia acquired in step S220 to the object detection model 30e, generates detection data 30f indicating the object detection result, and stores it in the storage medium 30.

[0079] Next, the control unit 20 displays the corrected image Ia by the function of the output control unit 21e (step S230). That is, the control unit 20 controls the display unit 41 to display the corrected image Ia indicated by the corrected image data 30d. With the above configuration, the inspector can visually recognize the corrected image Ia colored in the color of each object.

[0080] Next, the control unit 20 accepts input of the detection result by the inspector through the function of the model update unit 21f (step S235). That is, the inspector detects an object included in the corrected image displayed on the display unit 41, and inputs the detection result by operating the input unit 42. The control unit 20 acquires the input detection result, associates it with information indicating that it is a detection result by an inspector, and adds it to the detection data 30f.

[0081] Next, the control unit 20 acquires the matching rate of the detection result by using the function of the model update unit 21f (step S240). Specifically, the control unit 20 refers to the detection data 30f and acquires the detection result of the object by the object detection model 30e and the detection result of the object by the inspector. The matching rate is the ratio of the detection result of the object by the object detection model 30e to the detection result of the object by the inspector that matches, and can be acquired by various methods. For example, a method can be adopted in which the number of objects detected by the inspector is used as the denominator, and the number of objects whose positions and classes match in the detection result by the object detection model 30e is used as the numerator. Of course, when determining whether the positions of the objects match, a deviation of a certain value or less may be allowed and may be considered to match. In addition, the degree of matching of each object may be evaluated so that the higher the degree of matching between the position and class of the object in the detection result by the object detection model 30e and the detection result by the inspector, the higher the matching rate.

[0082] When the matching rate of the detection result is acquired, the control unit 20 determines whether or not the matching rate is equal to or higher than a predetermined standard by the function of the model update unit 21f (step S245). That is, the control unit 20 determines whether or not the detection accuracy by the object detection model 30e is equal to or higher than a desired accuracy.

[0083] If it is determined in step S245 that the matching rate is equal to or greater than the predetermined standard, the control unit 20 uses the function of the output control unit 21e to display the object detection result (step S250). That is, if the matching rate is equal to or greater than the predetermined standard, the control unit 20 determines that the detection result by the object detection model 30e is reliable, and controls the display unit 41 to display the object detection result by the object detection model 30e indicated by the detection data 30f. As a result, the inspector can visually confirm the object detection result. Of course, the display manner is an example, and the detection result by the inspector may be displayed, or a process may be performed in which an object for which the detection results by the object detection model 30e and the inspector do not match.

[0084] Next, the control unit 20 refers to the detection data 30f and determines whether or not a specific type of object exists (step S255). That is, the control unit 20 identifies a class indicating the type of object from the detection result indicated by the detection data 30f, and determines whether or not a class indicating a predetermined specific type exists. The specific type is the type of object that is the target of a warning, and in this embodiment, it corresponds to a dangerous object such as a blade.

[0085] If it is determined in step S255 that a specific type of object is present, the control unit 20 outputs a warning using the function of the output control unit 21e (step S260). Specifically, the control unit 20 controls the display unit 41 to display warning information indicating that the baggage contains a specific type of object. In response to the warning, the inspector takes a predetermined action, such as requesting the user to disclose the baggage. If it is not determined in step S255 that a specific type of object is present, the control unit 20 skips step S260. When the inspection process is completed, the control unit 20 waits, and when the next X-ray image is acquired, it executes the processes from step S200 onwards.

[0086] On the other hand, if it is not determined in step S245 that the matching rate is equal to or higher than the predetermined standard, the control unit 20 displays the detection result by the inspector by the function of the output control unit 21e (step S265). That is, if the matching rate is smaller than the predetermined standard and the object detection accuracy is not equal to or higher than the desired accuracy, the control unit 20 considers that the detection result by the inspector is more reliable than the detection result by the object detection model 30e. Then, the control unit 20 controls the display unit 41 to display the object detection result by the inspector indicated by the detection data 30f. As a result, the inspector can visually confirm the object detection result by himself. In this case, the inspector himself judges whether or not a specific type of object exists. Of course, the display manner is an example, and the detection result by the object detection model 30e may be displayed, or a process may be performed in which an object whose detection result does not match that of the object detection model 30e and the inspector is highlighted.

[0087] Next, the control unit 20 determines whether the number of times that the matching rate does not reach the predetermined standard or more is a predetermined number of times or more (step S270). That is, each time the matching rate does not reach the predetermined standard or more, the number of times is counted up. Also, the predetermined number of times is a value for checking the number of times that the detection accuracy does not reach the desired accuracy or more, and in this embodiment, when the number of times that the matching rate does not reach the predetermined standard or more is a predetermined number of times or more, it is considered that the detection accuracy does not reach the desired accuracy or more and the machine learning model should be updated.

[0088] In step S270, if it is determined that the number of times that the matching rate does not reach the predetermined standard or more is the predetermined number of times or more, the control unit 20 executes a machine learning model update process (step S275) by the function of the model update unit 21f, and updates the corrected image generation model 30c and the object detection model 30e. Specifically, the control unit 20 instructs the machine learning system 100 to update the machine learning model. As a result, the machine learning system 100 executes a machine learning model update process shown in FIG. 9, which will be described later. Note that, when the machine learning model update process is performed, the count number of times that the matching rate does not reach the predetermined standard or more is initialized to 0. In step S270, if it is not determined that the number of times that the matching rate does not reach the predetermined standard or more is the predetermined number of times or more, the control unit 20 skips step S275.

[0089] (4) Machine learning model update process: FIG. 9 is a flowchart of the machine learning model update process. The machine learning model update process is started by the control unit 20 instructing the machine learning system 100 to update in the above-mentioned step S275. However, before the start of the machine learning model update process, the machine learning system 100 collects the colored image data 30b, the corrected image data 30d, and the detection data 30f from the image analysis system 10. That is, when the operation of the image analysis system 10 is started, the colored image data 30b, the corrected image data 30d, and the detection data 30f increase every time a baggage is inspected. The image analysis system 10 transmits the colored image data 30b, the corrected image data 30d, and the detection data 30f to the machine learning system 100 at any timing. The machine learning system 100 executes the machine learning model update process based on the colored image data 30b, the corrected image data 30d, and the detection data 30f collected after the last update.

[0090] The machine learning model update process is similar to the machine learning process shown in Fig. 6, but is partially different. In addition, the teacher data used in the machine learning model update process is not the teacher data 300a but the update teacher data 300b. In the machine learning model update process, step S107 is executed instead of step S105 shown in Fig. 6, and the update teacher data 300b is generated in step S107.

[0091] FIG. 10 is a diagram for explaining the updating teacher data 300b. The updating teacher data 300b is generated based on the colored image data 30b, the corrected image data 30d, and the detection data 30f collected after the image analysis system 10 has started to operate. In FIG. 10, a plurality of colored images Ic represented by the colored image data 30b acquired based on the X-ray image taken after the operation has started is shown in the upper part of the drawing. In addition, in FIG. 10, a plurality of corrected images Ia represented by the corrected image data 30d generated by correcting the colored image Ic are shown in the lower part of the drawing. Furthermore, in FIG. 10, the result of the object detection by the inspector represented by the detection data 30f generated based on the corrected image Ia is shown by a dashed rectangle superimposed on the corrected image Ia.

[0092] Since the corrected image Ia is generated from the colored image Ic, when the original colored image Ic and the generated corrected image Ia are paired, each of the multiple colored images Ic is paired with each of the corrected images Ia, resulting in multiple pairs. Each pair is associated with the detection result of the object included in the corrected image Ia belonging to each pair. Therefore, when the above-mentioned pairs of the colored image Ic and the corrected image Ia and the detection results of each image are set, a data set similar to the teacher data 300a shown in FIG. 4 is generated. In this embodiment, the control unit 200 stores the data set generated in this manner in the storage medium 300 as updating teacher data 300b.

[0093] When the updating teacher data 300b is generated in step S107 by the above process, the process from step S110 onwards is executed, which is almost the same as the machine learning process shown in Fig. 6. However, since the teacher data referred to in the machine learning model update process is the updating teacher data 300b, the control unit 200 sets the corrected image Ia indicated by the corrected image data 30d as the target image It. The generated image Ig is the corrected image Ia generated by inputting the colored image Ic indicated by the colored image data 30b to the corrected image generation model 30c being updated.

[0094] Then, the control unit 200 goes through the machine learning process of steps S110 to S160, and in step S165 updates the existing corrected image generation model 30c with the generation model G under training. Also, updates the existing object detection model 30e with the object detection model 30e under training. Furthermore, the control unit 200 transmits the corrected image generation model 30c and the object detection model 30e to the image analysis system 10 via the communication unit 500, and updates the corrected image generation model 30c and the object detection model 30e stored in the storage medium 30.

[0095] According to the above process, even if the accuracy of object detection by the initial corrected image generation model 30c and object detection model 30e temporarily decreases, the corrected image generation model 30c and object detection model 30e are updated based on the newly acquired X-ray image and the detection result by the inspector, so that the accuracy of object detection can be restored. Therefore, compared with a configuration in which the corrected image generation model 30c and the object detection model 30e are not updated, it is possible to reduce the maintenance work of the baggage inspection system. In addition, in the baggage inspection system according to this embodiment, a color display device that expresses colors with gradation values ​​of three channels is used for the display unit 41, 410. Therefore, in the above embodiment, the colored image is a color image expressed with gradation values ​​of three channels. This embodiment is an example, and for example, it is possible to configure the baggage inspection system using a display device capable of displaying more channels. In this case, an image that matches the number of channels and gradation level that can be displayed is adopted as the colored image. Therefore, images that match the same number of channels and gradation level are adopted for each image such as the corrected image and the target image.

[0096] (5) Other embodiments: The above embodiment is an example for implementing the present invention, and various other embodiments can be adopted. For example, the image analysis system 10 and the machine learning system 100 may be realized by multiple devices. Furthermore, machine learning may be performed by the image analysis system 10, or may be performed by other devices, such as a cloud server or a portable terminal. If machine learning can be performed in the image analysis system 10, additional learning can be performed using, for example, X-ray images taken at each site where the image analysis system 10 is installed.

[0097] When machine learning is possible on a portable terminal, the terminal can be used as the machine learning system 100, and machine learning can be performed at any location. Therefore, machine learning can be performed at the site where the image analysis system 10 is installed or in its vicinity, and the learned corrected image generation model 30c and object detection model 30e can be transferred to the image analysis system 10 by any means such as a removable memory, wired communication, or wireless communication, and the corrected image generation model 30c and object detection model 30e can be updated. With this configuration, even if the image analysis system 10 is not connected to a computer such as the machine learning system 100 so as to be able to communicate remotely, the corrected image generation model 30c and object detection model 30e can be updated, and luggage can be inspected using the updated corrected image generation model 30c and object detection model 30e.

[0098] Furthermore, at least some of the X-ray image acquisition unit 21a, the colored image acquisition unit 21b, the corrected image generation unit 21c, the object detection unit 21d, the output control unit 21e, and the model update unit 21f may be present in a plurality of separate devices. Also, at least some of the machine learning unit 210a, the object detection unit 210b, and the output control unit 210c may be present in a plurality of separate devices. Of course, some of the configurations of the above-mentioned embodiment may be omitted, and the order of processing may be changed or omitted. For example, if a corrected image generation model is machine-learned so as to input an X-ray image and generate a corrected image in which different objects are colored in different colors, a baggage inspection system that omits the colored image acquisition unit can be constructed.

[0099] The X-ray image acquisition unit only needs to be able to acquire an X-ray image showing the amount of transmission of X-rays irradiated to the luggage to be inspected. Therefore, the X-ray imaging device that irradiates the luggage with X-rays and takes the X-ray image may be integrated with the luggage inspection system or may be configured as a separate device. The luggage to be inspected is an inspection unit that may include any object, and the inspection unit may be in any form. For example, the luggage to be inspected may be a bag that may include multiple objects, or luggage carried in a cloth or resin bag or luggage carried by hand without being placed in a bag or the like may be the inspection target. Of course, objects other than luggage may be the inspection target, but the luggage inspection system is a system in which luggage on which multiple objects may overlap can be the inspection target, and when luggage on which multiple objects may overlap is the inspection target, inspection by object detection using a machine learning model is particularly useful.

[0100] An X-ray image only needs to show the amount of transmitted X-rays. In other words, since the amount of transmitted X-rays that penetrate an object included in a package varies depending on the type and thickness of the object's material, an X-ray image is an image obtained by detecting transmitted X-rays attenuated according to the type and thickness of the material by an X-ray sensor.

[0101] The colored image acquisition unit may acquire a colored image in which an object included in an X-ray image is colored in the color of the object's material. The method for coloring an X-ray image is not limited to the above-mentioned method, and various methods may be used. For example, a colored image may be generated using a machine learning model that inputs a grayscale X-ray image and outputs a colored image colored in the color of the material. Coloring in the color of each material may be performed by various methods. For example, the material to be detected may be colored to distinguish it from other materials, or a difference in the amount of X-ray transmission may be considered as a difference in the material, and coloring may be performed to emphasize the difference in the amount of transmission.

[0102] The corrected image generating unit may input a colored image to a correction image generation model that has been machine-learned to input a colored image and generate a corrected image in which different objects are colored in different colors. The corrected image generation model may be machine-learned to convert a colored image into a corrected image, and the input and output formats are not limited, and images of various types may be input and output. That is, the size of the colored image and the corrected image is not limited, and the colored image may be an image that has been subjected to various pre-processing processes. The formats of the colored image and the corrected image may be the same or different.

[0103] The corrected image generation model generated by machine learning may be a model that converts a colored image into a corrected image, and various configurations other than those described above may be adopted. That is, the machine learning model may be an image-image conversion model obtained by various aspects of DCGAN or their derivatives, conditional GAN, etc.

[0104] The object detection unit may input the corrected image to an object detection model that has been machine-learned to input the corrected image and detect an object contained in the corrected image, and may detect the object. The object detection model may input the corrected image and machine-learn to detect the object, and the input and output formats are not limited, and images of various types may be input, and object detection results may be output in various types. That is, the size of the corrected image is not limited, and the corrected image may be an image that has been subjected to various pre-processing processes. The output mode of object detection is not limited to a combination of a frame and a label indicating a detected object.

[0105] The object detection model generated by machine learning may be a model that outputs the result of object detection based on the corrected image, and various configurations other than the above-mentioned configuration may be adopted. That is, the machine learning model is not limited to the above-mentioned SSD, and various models such as YOLO (You Only Look Once) and region-based convolutional neural networks R-CNN (Region Based Convolutional Neural Networks) can be used. Also, although FIG. 5 shows an example in which the generated image Ig is divided into 16 non-overlapping default boxes, other configurations may be adopted. For example, multiple default box groups of different sizes may be further added. Also, an example in which the default boxes overlap each other may be added.

[0106] The mode of machine learning for generating the corrected image generation model or the object detection model is not limited, and machine learning may be performed by appropriately selecting various elements such as the number of layers and the number of nodes constituting the neural network, the type of activation function, the type of loss function, the type of gradient descent method, the type of optimization algorithm for the gradient descent method, the presence or absence of mini-batch learning and the number of batches, the learning rate, the initial value, the type or presence or absence of an overlearning suppression method, the presence or absence of a convolution layer, the size of the filter in the convolution operation, the type of filter, the type of padding or stride, the type or presence or absence of a pooling layer, the presence or absence of a fully connected layer, and the presence or absence of a recursive structure. Of course, learning may be performed by other machine learning, such as a support vector machine, clustering, or reinforcement learning. Furthermore, machine learning may be performed in which the structure of the model (for example, the number of layers, the number of nodes per layer, etc.) is automatically optimized. Various modes may also be adopted as the mode of various parameters. For example, the range of the identification signal output from the identification model D is not limited to 0 to 1, and may be a range of 0 to 100, etc.

[0107] The correction image may be an image in which the object of a specific material to be detected is colored in the color of the object. Therefore, in the above embodiment, both the object of the specific material and the object of a non-specific material are colored in the color of the object, but the object of a non-specific material does not need to be colored in the color of the object.

[0108] Such a configuration can be realized by using the following teacher data 300a. Specifically, when the teacher data 300a is generated, a color corresponding to a specific material is extracted from a colored image in which an object is colored in the color of each material, and the image expressed in that color is classified into an image for each object, and the hue and saturation of the color of each object are expanded and converted into another color. As a result, the teacher data 300a is defined by preparing a plurality of pairs of a target image It colored in the color of each object and an original colored image. According to the above configuration, a corrected image is generated so that an object made of a specific material (such as a material of a hazardous material) to be detected has a color corresponding to the object. Therefore, the object to be detected is easily distinguished. As a result, the detection accuracy by the object detection model 30e can be easily improved.

[0109] The model update unit may update the corrected image generation model and the object detection model when the object detection accuracy by the object detection model is smaller than a predetermined standard. That is, the model update unit may update the corrected image generation model and the object detection model by a model that can be estimated to improve the detection accuracy when the detection accuracy is smaller than a predetermined standard. Examples of models that can be estimated to improve the detection accuracy include a model in which additional machine learning is performed using unlearned teacher data prepared in advance, and a model in which additional machine learning is performed using unlearned teacher data generated based on X-ray images acquired during the operation of the baggage inspection system. The predetermined standard is a minimum accuracy that the object detection accuracy should satisfy, and may be determined in advance. The predetermined standard may also change depending on various conditions, such as the purpose, location of use, and time of day of the baggage inspection system.

[0110] The correction image generation model and the object detection model may be updated simultaneously or one at a time at different times. Each model may be updated so as to improve the object detection accuracy, and re-learning may be performed using unlearned teacher data that was not used in machine learning before the update, or by adding unlearned teacher data that was not used in machine learning before the update to existing teacher data. Re-learning may be performed in the baggage inspection system, or in a device different from the baggage inspection system.

[0111] The input unit may input an object detected by an inspector based on at least one of an X-ray image, a colored image, and a corrected image. Therefore, the input unit is not limited to a configuration in which an inspector visually recognizes a corrected image inputs an object detected by the inspector, as in the above-described embodiment, and may be input by an inspector visually recognizes either an X-ray image or a colored image, or may be input by an inspector visually recognizes multiple images. The manner in which the detected object is input is not limited and may be various. In other words, the input may be performed so as to include at least the detected object and the label of the object, so that the detected object can be a label of the teacher data.

[0112] In addition, in the above embodiment, the output of the discriminative model D is one output, but the aspect of the discriminative model D is not limited to this aspect. For example, a model may be defined so that two outputs are obtained from the discriminative model D, and a value representing a false accuracy is output from one, and a value representing a true accuracy is output from the other. In this case, when the discriminative model D and the generative model G are trained, the difference between the vector consisting of two outputs and the corresponding target vector is back-propagated, and training is performed in the same manner as in the above embodiment.

[0113] In the above embodiment, the object detection accuracy of the object detection model 30e is determined by comparing the object detection result by the inspector with the object detection result by the object detection model 30e, but the object detection accuracy may be defined by various methods. For example, the object detection accuracy of the object detection model 30e may be evaluated based on a colored image prepared in advance and a correct answer value for the type of object contained in the colored image.

[0114] Fig. 11 is a flow chart showing an example of an inspection process for evaluating the accuracy of object detection based on a colored image prepared in advance and a correct answer value for the type of object contained in the colored image. A part of the inspection process shown in Fig. 11 is similar to the inspection process shown in Fig. 8. In Fig. 11, the same processes as those in Fig. 8 are indicated by the same reference numerals. That is, in the inspection process shown in Fig. 11, steps S200 to S225 are similar to those in Fig. 8.

[0115] In the inspection process shown in Fig. 11, steps S230 to S245 shown in Fig. 8 are skipped after step S225. Therefore, in the inspection process shown in Fig. 11, when an object included in the corrected image is detected in step S225, the detection result of the object is displayed in step S250. Then, in step S255, it is determined whether or not a specific type of object exists based on the object detection result, and if it is determined that a specific type of object exists, a warning is output in step S260.

[0116] When a warning is output or when it is not determined that a specific type of object exists, the control unit 20 determines whether or not the inspection has been performed a reference number of times or more by the function of the model update unit 21f (step S262). That is, each time an object is detected by the object detection model 30e, the number is counted up as the number of inspections. In this embodiment, when the number of inspections after the machine learning model is updated becomes equal to or greater than the reference number of times, a process of determining whether or not to perform the update is started. Therefore, when it is determined in step S262 that the inspection has been performed a reference number of times or more, the control unit 20 executes the process of step S264 and subsequent steps. When it is not determined that the inspection has been performed a reference number of times or more, the control unit 20 skips the process of step S264 and subsequent steps. Note that, in this embodiment as well, when the machine learning model is updated, the number of inspections is initialized to 0.

[0117] In step S264, the control unit 20 inputs a colored image prepared in advance to the pre-update correction image generation model 30c by using the function of the correction image generation unit 21c. Here, the colored images are a plurality of colored images prepared in advance before the machine learning model update process is performed, and the position of an object included in each colored image is associated with a class indicating the type of the object. Furthermore, the colored images are not used in the machine learning of the pre-update correction image generation model 30c and the object detection model 30e, that is, they are unlearned data.

[0118] The colored image may be an image in which an X-ray image of a baggage containing an object is colored with the color of each material, but it is more preferable that the colored image is an image that makes it easy to evaluate the detection accuracy of the object by the object detection model 30e. For example, a colored image that is likely to cause erroneous recognition of an object by the object detection model 30e corresponds to an image that makes it easy to evaluate the detection accuracy. More specifically, the detection accuracy is easily evaluated by using an image in which multiple objects are adjacent to each other, an image in which multiple objects are overlapping, or an image containing multiple types of objects.

[0119] In step S264, when each of the plurality of colored images prepared in advance is input to the pre-update correction image generation model 30c, a plurality of correction images are generated. The control unit 20 acquires the plurality of correction images by the function of the correction image generation unit 21c (step S266). That is, the control unit 20 stores data indicating the correction images output from the correction image generation model 30c in the storage medium 30 by the process of step S264.

[0120] Next, the control unit 20 detects objects included in the plurality of corrected images by using the function of the object detection model 30e (step S268). That is, the control unit 20 inputs each of the plurality of corrected images acquired in step S266 to the object detection model 30e, generates data indicating the detection result of the object included in each corrected image, and stores the data in the storage medium 30 in association with the colored image.

[0121] Next, the control unit 20 judges whether the misjudgment rate is equal to or lower than a reference threshold value by the function of the model update unit 21f (step S272). The misjudgment rate is the ratio of the detection results of the object in step S268 that are erroneous. Therefore, the control unit 20 compares the correct answer values ​​of the position and type of the object associated with the multiple colored images input to the corrected image generation model 30c in step S264 with the position and type of the object detected by the corrected image generation model 30c before the update in step S268, and counts the detection results that differ from the correct answer values ​​as errors. Then, the control unit 20 obtains the misjudgment rate by dividing the number of errors by the number of objects associated with the multiple colored images input to the corrected image generation model 30c in step S264.

[0122] The reference threshold is a value that is determined in advance as an upper limit of the misjudgment rate. Therefore, if it is determined in step S272 that the misjudgment rate is not equal to or less than the reference threshold, the control unit 20 determines that the corrected image generation model 30c and the object detection model 30e need to be updated, and executes step S275. Step S275 may be the same as the process shown in FIG. 9. On the other hand, if it is determined in step S272 that the misjudgment rate is equal to or less than the reference threshold, the control unit 20 determines that it is not necessary to update the corrected image generation model 30c and the object detection model 30e, and skips step S275.

[0123] Furthermore, as in the present invention, when the object detection accuracy by the object detection model is smaller than a predetermined standard, the method of updating the corrected image generation model and the object detection model can also be applied as a program or method. In addition, the above-mentioned system, program, and method can be realized as a single device or a plurality of devices, and include various aspects. In addition, it can be modified as appropriate, such as being partly software and partly hardware. Furthermore, the invention can also be established as a recording medium of a program that controls the system. Of course, the recording medium of the software may be a magnetic recording medium or a semiconductor memory, and can be considered in exactly the same way in any recording medium developed in the future. [Explanation of symbols]

[0124] 10...image analysis system, 20...control unit, 21...image analysis program, 21a...X-ray image acquisition unit, 21b...colored image acquisition unit, 21c...corrected image generation unit, 21d...object detection unit, 21e...output control unit, 21f...model update unit, 30...storage medium, 30a...X-ray image data, 30b...colored image data, 30c...corrected image generation model, 30d...corrected image data, 30e...object detection model, 30f...detection data, 40...X-ray imaging device, 41...display unit, 42...input unit, 50...communication unit, 100...machine learning system, 200...control unit, 210...management program, 210a...machine learning unit, 210b...object detection unit, 210c...output control unit, 300...storage medium, 300a...teaching data, 300b...teaching data for update, 410...display unit, 500...communication unit

Claims

1. an X-ray image acquisition unit for acquiring an X-ray image showing an amount of transmission of X-rays irradiated to a baggage to be inspected; a colored image acquisition unit for acquiring a colored image of an object included in the X-ray image, the colored image being colored with a color corresponding to a material of the object; a corrected image generation unit that inputs the colored image and generates the corrected image for a corrected image generation model that has been machine-learned to input the colored image and generate corrected images in which different objects are colored in different colors; an object detection unit that inputs the corrected image and detects the object using an object detection model that has been machine-learned to input the corrected image and detect the object included in the corrected image; a model update unit that updates the corrected image generation model and the object detection model when the detection accuracy of the object by the object detection model is lower than a predetermined standard; A baggage inspection system comprising:

2. an input unit for inputting the object detected by an inspector based on at least one of the X-ray image, the colored image, and the corrected image, The model update unit is updating the original model using the corrected image generation model and the object detection model, which are machine-learned based on updating training data including the colored image acquired after the start of operation of the baggage inspection system, the corrected image generated by inputting the acquired colored image into the corrected image generation model, and the object detection result input by the inspector; 2. The baggage inspection system according to claim 1.

3. The corrected image is the colored image is an image in which different objects are colored in different colors by changing at least one of the hue, saturation, and brightness of a color corresponding to a specific material in the colored image for each object; 3. A baggage inspection system according to claim 1 or 2.

4. an X-ray image acquisition step of acquiring an X-ray image showing the amount of X-ray transmitted through the baggage to be inspected; a colored image acquisition step of acquiring a colored image of an object included in the X-ray image, the colored image being colored in the color of each material of the object; a corrected image generation step of inputting the colored image to a corrected image generation model that has been machine-learned to generate corrected images in which the different objects are colored in different colors; and an object detection process in which the corrected image is input to an object detection model that has been machine-learned to detect the object included in the corrected image, and the object is detected; a model updating step of updating the corrected image generation model and the object detection model when the detection accuracy of the object by the object detection model is lower than a predetermined standard; A baggage inspection method including:

5. Computer, an X-ray image acquisition unit for acquiring an X-ray image showing the amount of X-rays transmitted through the baggage to be inspected; a colored image acquisition unit that acquires a colored image of an object included in the X-ray image, the colored image being colored in a color corresponding to the material of the object; a corrected image generation unit that inputs the colored image and generates the corrected image for a corrected image generation model that has been machine-learned to input the colored image and generate corrected images in which different objects are colored in different colors; an object detection unit that inputs the corrected image and detects the object using an object detection model that has been machine-learned to input the corrected image and detect the object included in the corrected image; a model update unit that updates the corrected image generation model and the object detection model when the detection accuracy of the object by the object detection model is lower than a predetermined standard; This serves as a baggage inspection program.

Images