X-ray Image Processing Apparatus and X-ray Image Processing Method

Abstract

To provide an X-ray image processor capable of displaying X-ray images including an article recognition area allowing a user to easily check and recognize surely.SOLUTION: The X-ray image processor determines whether each of multiple articles with overlapping recognition regions is arranged without overlapping in an X-ray image and whether postures of the multiple articles are frontal. When determining that the multiple articles are arranged without overlapping and the postures of the multiple articles are frontal, the X-ray image processor determines whether to integrate the recognition areas of the multiple articles in the X-ray image into one recognition area based on a determination result whether the combination of categories of the multiple articles is included in the combination of categories indicated by predetermined relation information. When determining to integrate the recognition areas of the multiple articles into one recognition area, the X-ray image processor draws one recognition area integrated into the X-ray image and displays the same on a display unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an X-ray image processing apparatus and an X-ray image processing method.

Background Art

[0002] An X-ray baggage inspection apparatus is used for baggage inspection at airports, large-scale event venues, and the like. Generally, an X-ray baggage inspection apparatus generates a grayscale image indicating the X-ray transmission amount and a color image in which the materials contained in the baggage are determined and colored for each material. An inspector visually checks the X-ray image generated by the X-ray baggage inspection apparatus to confirm whether the baggage contains dangerous goods. When the inspector discovers a dangerous good, the baggage is opened for inspection.

[0003] In order for an inspector to check an X-ray image to confirm whether the baggage contains dangerous goods, it is necessary to receive advanced training. Therefore, for example, it is difficult to temporarily secure a large number of inspectors at a large-scale event or the like from the viewpoints of prior training and cost. Thus, in order to reduce the load on the inspector as much as possible, attempts have been made to automate the discovery of dangerous goods.

[0004] As one measure to automate image recognition, there is an image recognition technology that utilizes deep learning by AI (artificial intelligence). Deep learning is widely used in applications such as video analysis and is spreading because high recognition accuracy can be obtained. For example, Japanese Unexamined Patent Application Publication No. 2018-4363 (Patent Document 1) describes that "the X-ray automatic determination apparatus according to the present invention determines whether an article is safe based on whether the area of a portion where the X-ray transmission amount is less than the transmission amount threshold and the material is continuous in the pixel region of the X-ray image exceeds the area threshold." (See the abstract).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When photographing a luggage containing a large amount of prohibited items or a luggage containing a plurality of prohibited items at the same position, in the technology described in Patent Document 1, a large number of detection areas indicating the positions of the prohibited items are displayed, or the detection areas overlap, which requires time for the inspector to visually check the prohibited items. Therefore, one aspect of the present invention is to display an X-ray image including an article recognition area that is easy for the user to confirm and surely easy to recognize.

Means for Solving the Problems

[0007] In order to solve the above problems, one aspect of the present invention adopts the following configuration. The X-ray image processing apparatus has a processor and a memory, and is connected to a display device. The memory holds X-ray image information indicating an X-ray image, a category of an article included in the X-ray image, a recognition area where the article is recognized in the X-ray image, and at least one of the area and the material amount of the article, relationship information indicating a combination of categories of articles, and overlap / front determination information indicating a category of an article and a condition indicating at least one of the area and the material amount of the article. The processor determines, based on a determination result as to whether each of a plurality of articles whose recognition areas overlap in the X-ray image satisfies a condition corresponding to the category of the article indicated by the overlap / front determination information, whether the plurality of articles are arranged without overlapping and whether the postures of the plurality of articles are frontal. When it is determined that the plurality of articles are arranged without overlapping and the postures of the plurality of articles are frontal, based on a determination result as to whether the combination of the categories of the plurality of articles is included in the combination of categories indicated by the relationship information, it is determined whether to integrate the recognition areas of the plurality of articles in the X-ray image into one recognition area. When it is determined to integrate the recognition areas of the plurality of articles into one recognition area, the integrated one recognition area is drawn in the X-ray image and displayed on the display device.

Advantages of the Invention

[0008] According to one aspect of the present invention, an X-ray image including an article recognition area that is easy for a user to check and is surely easy to recognize is displayed.

[0009] Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Best Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted that this embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention.

[0012] In this embodiment, an X-ray image processing apparatus and an X-ray image processing method will be described. In the present disclosure, the "front" of an article means a predetermined surface that can be seen from a direction in which it is easy for a general inspector to identify the article when viewed. Also, in the present disclosure, the "overlap" of an article means that a plurality of articles are transmitted and appear at the same position with respect to the imaging direction. In a photograph taken with a digital camera or the like, only the article closest to the viewer is shown, but an X-ray image is characterized in that even if articles overlap in the imaging direction, each overlapping article is shown transparently. In this embodiment, it is assumed that an article recognition method capable of detecting each overlapping article is used even when articles overlap.

Examples

[0013] FIG. 1 is a schematic diagram showing a configuration example of an X-ray inspection apparatus having an X-ray image processing apparatus. The X-ray inspection apparatus 100 is, for example, an apparatus used as a baggage inspection apparatus in airport security inspections. The X-ray inspection apparatus 100 includes, for example, an X-ray apparatus main body (hereinafter also referred to as the "apparatus main body") 101, an X-ray image processing apparatus 102, a display apparatus 103, and an input apparatus 104. The X-ray image processing apparatus 102 is connected to the apparatus main body 101, the display apparatus 103, and the input apparatus 104.

[0014] The apparatus main body 101 includes an irradiation mechanism that irradiates X-rays and an X-ray imaging mechanism that images an object such as a baggage and measures the amount of X-ray transmission. The apparatus main body 101 outputs data on the amount of X-ray transmission (hereinafter also simply referred to as "transmission amount data" or "transmission amount"). The apparatus main body 101 includes a transport mechanism having a belt conveyor for transporting baggage. The transport mechanism is controlled by a control unit included in the apparatus main body 101 to drive and stop the belt conveyor.

[0015] The transport mechanism of the apparatus main body 101 is provided with, for example, two types of X-ray sensors (hereinafter also simply referred to as "sensors") that measure the X-ray transmission amount, and each of the two types of sensors acquires different types of data. For example, one sensor acquires low-energy data, and the other sensor acquires high-energy data.

[0016] The X-ray image processing apparatus 102 can determine the material of the object for each pixel of the X-ray image based on the difference between the high-energy data and the low-energy data acquired by the two sensors. The sensor only needs to be able to acquire X-ray data capable of determining the material, and the detection method by the sensor is not limited. For example, a backscattering type material determination sensor or the like can be used as the sensor. Furthermore, the X-ray inspection apparatus 100 may be provided with sensors at a plurality of locations such as the side surface (vertical surface) and the ceiling surface (horizontal surface) of the area through which the article (for example, included in the luggage) passes. By providing the sensors in this way, the X-ray inspection apparatus 100 can photograph the article simultaneously from the upper and lateral two directions.

[0017] The X-ray image processing apparatus 102 is constituted by a computer such as a personal computer (PC). The X-ray image processing apparatus 102 determines whether the article (for example, included in the luggage) is safe or dangerous based on the article recognition result of the X-ray image photographed by the apparatus main body 101. When the X-ray image processing apparatus 102 determines that the luggage contains a dangerous article (alert target) as a result of article recognition, it lights a display lamp installed in the apparatus main body 101 or near the apparatus main body 101 to notify the user (for example, an inspector) of the X-ray image processing apparatus 102 that the article is an alert target. Note that a display device 103 or an audio output unit of the X-ray image processing apparatus 102 may be used for the notification that the article is an alert target.

[0018] The X-ray image processing apparatus 102 of this embodiment has a feedback function for feeding back to the user whether the detection result is correct or there is a detection omission, and can learn a model for image recognition using the feedback information accumulated by the feedback function.

[0019] The display device 103 displays the X-ray image captured by the apparatus main body 101, so that the user can visually confirm the X-ray image and the article detection information displayed on the screen. Further, the display device 103 displays the processing result by the X-ray image processing apparatus 102. Note that the X-ray inspection apparatus 100 may include a plurality of display devices 103. For example, when the X-ray inspection apparatus 100 includes two display devices 103, the two display devices 103 may display images captured from different directions or display different types of images (for example, a color image and a grayscale image). The input device 104 is a device that receives input from an operator, such as a keyboard or a mouse.

[0020] Note that at least a part of the devices included in the X-ray inspection apparatus 100 may be integrated. Specifically, in FIG. 1, an example is shown in which the apparatus main body 101 and the X-ray image processing apparatus 102 are separate devices, but the X-ray image processing apparatus 102 may be built in the apparatus main body 101.

[0021] FIG. 2 is a block diagram showing a configuration example of the X-ray image processing apparatus 102. The X-ray image processing apparatus 102 is configured by, for example, a computer having a CPU (Central Processing Unit) 201, a memory 202, a display I / F (interface) 203, an input I / F 204, a communication device 205, and an auxiliary storage device 210.

[0022] The CPU 201 includes a processor and executes a program loaded in the memory 202. The memory 202 includes a ROM (Read Only Memory), which is a non-volatile memory element, and a RAM (Random Access Memory), which is a volatile memory element. The ROM stores unchanging programs (e.g., BIOS (Basic Input / Output System)). The RAM is a high-speed and volatile memory element such as a DRAM (Dynamic Random Access Memory), and temporarily stores the program executed by the CPU 201 and the data used during the execution of the program.

[0023] The auxiliary storage device 210 is a large-capacity and non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores the OS (Operating System) 211, the program executed by the CPU 201, and the data used during the execution of the program. That is, the program is read from the auxiliary storage device 210, loaded into the memory 202, and executed by the CPU 201. Note that part or all of the information stored in the auxiliary storage device 210 may be stored in the memory 202, or part or all of the information stored in the memory 202 may be stored in the auxiliary storage device 210.

[0024] Part or all of the program executed by the CPU 201 may be provided to the X-ray image processing apparatus 102 from a removable medium (such as a CD-ROM or a flash memory), which is a non-transitory storage medium, or from an external computer equipped with a non-transitory storage device via a network, and stored in the non-volatile auxiliary storage device 210, which is a non-transitory storage medium. Therefore, the X-ray image processing apparatus 102 preferably has an interface for reading data from the removable medium.

[0025] The display I / F 203 is an interface that outputs the display result to the display device 103. The input I / F 204 is an interface that receives the input from the input device 104 or receives the input of the X-ray transmission amount data from the apparatus main body 101. The communication device 205 is a network interface device that controls communication with other devices according to a predetermined protocol. Also, the communication device 205 may include a serial interface such as USB (Universal Serial Bus).

[0026] The auxiliary storage device 210 holds the X-ray image acquisition program 212, the article recognition program 213, the overlap / front determination program 214, the region relationship determination program 215, the table update program 216, the display image generation program 217, and the screen display program 218, which are programs respectively.

[0027] The X-ray image acquisition program 212 acquires the X-ray transmission amount data photographed by the apparatus main body 101 via the input I / F 204 (for example, a dedicated interface or a general-purpose screen input terminal such as a VGA terminal). Note that the X-ray inspection apparatus 100 used at an airport security inspection site or the like generally performs photographing from one direction or two directions.

[0028] The X-ray image acquisition program 212 acquires, as an X-ray image, a color image in which the material information of the baggage and the density information of the article are visualized from the X-ray transmission amount data, or a grayscale image in which the portion with less X-ray transmission amount is darkened and the portion with more transmission amount is lightened. When the apparatus main body 101 is provided with a plurality of sensors and each of the plurality of sensors performs photographing from different directions, the X-ray image acquisition program 212 acquires each of the X-ray images obtained by photographing from different directions by the plurality of sensors.

[0029] The object recognition program 213 recognizes objects included in the X-ray image acquired by the X-ray image acquisition program 212 on a pixel-by-pixel basis, using, for example, deep learning segmentation technology or the like. The overlap / front determination program 214 determines whether the recognized object overlaps with other objects and whether the posture of the recognized object is frontal based on information such as the area and density of the region of the recognized object.

[0030] The region relationship determination program 215 determines whether a plurality of objects recognized in overlapping regions on the X-ray image are recognized as the same object or as different objects. Further, the region relationship determination program 215 determines whether the characteristics of the recognition regions are similar based on, for example, the area and density of each of the plurality of objects recognized in the overlapping regions on the X-ray image.

[0031] The table update program 216 acquires information such as the relationship between object categories and the position of components from feedback information from the user and X-ray images accumulated during operation, and updates the information in various tables.

[0032] The display image generation program 217 generates an image showing information such as a rectangle indicating the position where an object is recognized in the captured X-ray image and the name of the recognized object. The screen display program 218 displays the image generated by the display image generation program 217 on the display device 103.

[0033] For example, the CPU 201 functions as an X-ray image acquisition unit by operating according to the X-ray image acquisition program 212 loaded into the memory 202, and functions as an object recognition unit by operating according to the object recognition program 213 loaded into the memory 202. The relationship between the program and the functional unit is the same for other programs included in the auxiliary storage device 210.

[0034] Note that part or all of the functions realized by the functional units implemented by the CPU 201 and the program in FIG. 2 may be realized by hardware such as, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0035] In addition, the auxiliary storage device 210 holds X-ray image data 220, an article recognition model 221, feedback data 222, component relationship data 223, a category relationship table 224, overlap data 225, and an overlap / front determination table 226. The X-ray image data 220 stores the X-ray image acquired by the X-ray image acquisition program 212 and information for identifying the luggage photographed in the X-ray image.

[0036] The article recognition model 221 is a model that outputs an article included in each pixel of the X-ray image when the X-ray image is input. Note that, for one article in the X-ray image, article recognition models learned in various directions such as the front and the side may be applied. However, since the X-ray image processing apparatus 102 of the present embodiment can not only identify the articles included in the X-ray image but also determine whether the posture of the identified article is the front, the article recognition model 221 learned with the front image and the side image of the article as images of different articles may be used. By using such an article recognition model 221, for example, the identification between the front and the side of an article such as a smartphone, whose appearance is significantly different between the front and the side, can be more accurately reflected.

[0037] The feedback data 222 holds feedback from the user of the X-ray image processing apparatus 102. The component relationship data 223 holds information indicating conditions for an article of a certain category recognized in an overlapping region in the X-ray image to be a component of an article of another category. Note that when it is determined that an article of a certain category recognized in the X-ray image is a component of an article of another category, the recognition regions of these articles are merged (integrated into one recognition region).

[0038] The category relationship table 224 holds information indicating conditions for considering combinations of articles of a plurality of categories recognized in overlapping regions within the X-ray image as the same article. When it is determined that combinations of articles of a plurality of categories recognized within the X-ray image can be considered as the same article, the recognition regions of the articles of the plurality of categories are merged.

[0039] The overlap data 225 holds information regarding an article whose recognition region has been merged in the X-ray image. The overlap / front determination table 226 holds information indicating conditions for determining whether a plurality of articles recognized in overlapping regions within the X-ray image overlap and are positioned on the front side or the back side in the shooting direction, and whether the postures of the plurality of articles are frontal.

[0040] Note that, in the present embodiment, the information used by the X-ray image processing apparatus 102 may be represented in any data structure regardless of the data structure. For example, a data structure appropriately selected from a table, a list, a database, or a queue can store the information.

[0041] Note that the X-ray image processing apparatus 102 is a computer system configured on one physical computer or on a plurality of computers configured logically or physically, and may operate in separate threads on the same computer, or may operate on a virtual computer constructed on a plurality of physical computer resources.

[0042] Figure 3 is an example of the component relationship data 223. The component relationship data 223 includes, for example, a component definition table 2231 and an article number table 2232. The component definition table 2231 includes, for example, an ID column 223a, a category column 223b, a component column 223c, a relationship column 223d, an area column 223e, and a center of gravity position column 223f. Note that the component definition table 2231 may not include at least one of the area column 223e and the center of gravity position column 223f.

[0043] The ID column 223a holds an ID that identifies a record in the constituent definition table 2231. The category column 223b holds information indicating the category of an article (hereinafter also referred to as the whole article) that includes the constituent. The constituent column 223c holds information indicating an article that can be a constituent of the article in the category indicated by the category column 223b. The relationship column 223d holds information (``constituent'') indicating the relationship between the article in the category indicated by the category column 223b and the article indicated by the constituent column 223c.

[0044] The area column 223e indicates a condition regarding the area in an X-ray image for the article indicated by the constituent column 223c to be a constituent of the whole article indicated by the category column 223b. For example, in the example of FIG. 3, for the ``battery'' to be a constituent of the ``smartphone'', the area of the ``battery'' in the X-ray image needs to be ``xx% or more and yy% or less'' of the area of the ``smartphone''. Note that the condition regarding the area indicated by the area column 223e may be indicated not by the ratio between the area of the article in the category indicated by the category column 223b and the area of the article indicated by the constituent column 223c, but by the absolute amount of at least one of the area of the article in the category indicated by the category column 223b and the area of the article indicated by the constituent column 223c.

[0045] The center-of-gravity position column 223f indicates a condition regarding the center-of-gravity position in an X-ray image for the article indicated by the constituent column 223c to be a constituent of the article in the category indicated by the category column 223b. For example, in the example of FIG. 3, for the ``battery'' to be a constituent of the ``smartphone'', the center-of-gravity position of the ``battery'' in the X-ray image needs to be ``(XX,YY)''. Note that in the example of FIG. 3, the center-of-gravity position column 223f indicates the center-of-gravity position as an absolute position, but it may also indicate the center-of-gravity position as a relative range based on the position of the article of the category. For example, for the ``battery'' to be a constituent of the ``smartphone'', the condition may be defined in the center-of-gravity position column 223f such that the center-of-gravity position of the ``battery'' in the X-ray image is included in a circle with a predetermined radius centered on the center-of-gravity position of the ``smartphone''.

[0046] The item quantity table 2232 includes, for example, an ID column 223g, a category column 223h, a component column 223i, and a quantity column 223j. The ID column 223g holds an ID for identifying a record in the item quantity table 2232. The category column 223h holds information indicating the category of an item including components. The component column 223i holds information indicating an item that is a component of the item in the category indicated by the category column 223h.

[0047] The quantity column 223j indicates a condition regarding the quantity of a component for the item indicated by the component column 223i to be a component of the item in the category indicated by the category column 223h. For example, in the example of FIG. 3, for the "battery" to be a component of the "smartphone", the number of "batteries" displayed superimposed on the "smartphone" in the X-ray image needs to be "1 piece".

[0048] FIG. 4 is an example of a category relationship table 224. The category relationship table 224 includes, for example, an ID column 224a, a first category column 224b, a second category column 224c, an area column 224d, a metal quantity column 224e, a light metal quantity column 224f, and an organic matter quantity column 224g. Note that the category relationship table 224 may not include at least one of the area column 224d, the metal quantity column 224e, the light metal quantity column 224f, and the organic matter quantity column 224g.

[0049] The ID column 224a holds information for identifying a record in the category relationship table 224. The first category column 224b and the second category column 224c hold information indicating conditions for regarding items of different categories as items of the same category even if they are recognized as items of different categories.

[0050] The area column 224d indicates the conditions regarding the area in the X-ray image for the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c to be regarded as articles in the same category. For example, in the example of FIG. 4, in order for "hair dyeing" and "spray" to be regarded as articles in the same category, the area of "spray" in the X-ray image needs to be "XX% or more and YY% or less" of the area of "hair dyeing".

[0051] The metal amount column 224e indicates the conditions regarding the metal amount for the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c to be regarded as articles in the same category. For example, in the example of FIG. 4, in order for "hair dyeing" and "spray" to be regarded as articles in the same category, the metal amount of "spray" and the metal amount of "hair dyeing" in the X-ray image need to be "XX or more and YY or less".

[0052] The light metal amount column 224f indicates the conditions regarding the light metal amount in the X-ray image for the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c to be regarded as articles in the same category. For example, in the example of FIG. 4, in order for "hair dyeing" and "spray" to be regarded as articles in the same category, the light metal amount of "spray" and the light metal amount of "hair dyeing" in the X-ray image need to be "XX or more and YY or less".

[0053] The organic matter amount column 224g indicates the conditions regarding the organic matter amount in the X-ray image for the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c to be regarded as articles in the same category. For example, in the example of FIG. 4, in order for "hair dyeing" and "spray" to be regarded as articles in the same category, the organic matter amount of "spray" and the organic matter amount of "hair dyeing" in the X-ray image need to be "XX or more and YY or less".

[0054] Note that the condition indicated by the area column 224d may be indicated by the absolute amount of at least one of the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c, rather than the relative amount of the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c.

[0055] In addition, the conditions indicated by the metal amount column 224e, the light metal amount column 224f, and the organic matter amount column 224g may be indicated by the absolute amount of one of the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c, or may be indicated by the relative amount of the articles in the category indicated by the first category column 224b and the articles in the category indicated by the second category column 224c.

[0056] FIG. 5 is an example of an overlap / front determination table 226. The overlap / front determination table 226 includes, for example, a category column 226a, an area column 226b, a maximum long side value column 226c, an average transmittance value column 226d, a lower limit transmittance value column 226e, a metals ratio column 226f, and an organic matter ratio column 226g. Note that the overlap / front determination table 226 only needs to include at least one of the area column 226b, the maximum long side value column 226c, the average transmittance value column 226d, the lower limit transmittance value column 226e, the metals ratio column 226f, and the organic matter ratio column 226g in addition to the category column 226a.

[0057] The category column 226a holds information indicating the category of the article. The area column 226b indicates the condition regarding the area in the X-ray image of the article for which the article indicated by the category column 226a does not overlap with other articles in the X-ray image. The maximum long side value column 226c indicates the condition regarding the maximum long side value in the X-ray image of the article for which the article indicated by the category column 226a does not overlap with other articles in the X-ray image.

[0058] The average transmittance value column 226d indicates the condition regarding the average transmittance value in the X-ray image of the article indicated by the category column 226a, because the article does not overlap with other articles in the X-ray image. The lower limit transmittance value column 226e indicates the condition regarding the lower limit transmittance value in the X-ray image of the article indicated by the category column 226a, because the article does not overlap with other articles in the X-ray image.

[0059] The metals ratio column 226f indicates the condition regarding the metals ratio in the X-ray image of the article indicated by the category column 226a, because the article does not overlap with other articles in the X-ray image. The organic matter ratio column 226g indicates the condition regarding the organic matter ratio in the X-ray image of the article indicated by the category column 226a, because the article does not overlap with other articles in the X-ray image.

[0060] FIG. 6 is a flowchart showing an example of article recognition processing and display processing by the X-ray image processing apparatus 102. Before the processing of FIG. 6 starts, the article recognition model 221, the component relationship data 223, the category relationship table 224, and the overlap / front determination table 226 are set in advance.

[0061] The X-ray image acquisition program 212 acquires a RAW data or transmittance data, which is transmittance data acquired by a sensor of the apparatus main body 101, as an X-ray image (a screen signal for display on the display device 103) that has been imaged (S301).

[0062] Here, when the input from the apparatus main body 101 to the X-ray image processing apparatus 102 is RAW data of two types of high and low energies, the X-ray image acquisition program 212 determines the material of each pixel of the X-ray image from the difference information of the transmission amount data of the two types of high and low energies. For example, the X-ray image acquisition program 212 determines the material of each pixel by classifying each pixel into one of four types: metal, inorganic substance, organic substance, or others, based on the difference information of the transmission amounts of the two types of high and low energies. Then, the X-ray image acquisition program 212 uses the material information and the transmission amount of either the high or low energy of the two types of energies to generate a color image that shows the material information in color and the transmission amount in the shade of the color (the part with a large density is dark and the part with a small density is light).

[0063] Here, when the X-ray image data acquired by the X-ray image acquisition program 212 from the apparatus main body 101 is not divided for each object to be photographed, that is, when a plurality of objects are captured in one image data, the X-ray image acquisition program 212 divides the image data into units of objects by the first to third division methods shown below.

[0064] The first division method is used when a screen signal for display on the display device 103 is input to the X-ray image processing apparatus 102 from the VGA output terminal or the like of the apparatus main body 101. First, the X-ray image acquisition program 212 acquires the screen signal and determines whether the display content of the object has changed using the degree of change of the frame image. When the photographing of the object is completed, the object is displayed statically on the screen.

[0065] When the X-ray image acquisition program 212 determines a dark portion where the transmission amount is equal to or less than a predetermined threshold as the object area, it can determine the end of the photographing of the object and specify the object area at the end of the photographing by determining whether the object area has changed. Note that the X-ray image acquisition program 212 may determine the end of the photographing using information such as a counter that is incremented for each photographing displayed on the display device 103 or an icon indicating that photographing is in progress, and determine the image at the timing when the photographing is completed as the object image.

[0066] The second splitting method will be described. The X-ray image acquisition program 212 acquires a screen signal or RAW data, and based on the acquired screen signal or RAW data, obtains an integrated value of the transmission amount for each predetermined number of lines or for each predetermined time. The X-ray image acquisition program 212 determines that there is a package in a portion where the integrated value of the transmission amount is less than a predetermined threshold value (i.e., the color is dark), and determines that there is no package in a portion where the integrated value of the transmission amount is equal to or greater than the predetermined threshold value (i.e., the color is light), and splits the image into package units.

[0067] In addition, if the X-ray image acquisition program 212 determines the transmission amount for one line, there is a possibility of misjudgment due to noise or the like. Therefore, when it is determined that the integrated value of the transmission amount obtained by taking a moving average is less than the predetermined threshold value, it can be determined that there is a package, and when it is determined that the value is equal to or greater than the predetermined threshold value, it can be determined that there is no package. Further, since a portion with a light color may be included in the middle of a package, the X-ray image acquisition program 212 may determine a location where a portion with a light color continues for a predetermined amount as the space between packages.

[0068] The third splitting method is used when a package detection sensor is provided at the package insertion port of the apparatus main body 101. In a general apparatus main body 101, for example, a sensor capable of detecting shielding is provided at a position where a light beam is shielded by a package. The X-ray image acquisition program 212 can obtain the time of the package start point (imaging start time) by adding the time from the time when the sensor detects the package to the time when the tip of the package arrives at the line sensor when the sensor detects the package.

[0069] In addition, the X-ray image acquisition program 212 can obtain the time of the end point of the package (the end time of shooting) by adding the time from when the sensor stops detecting the package until the end of the package arrives at the line sensor when the sensor stops detecting the package. If the input from the apparatus main body 101 to the X-ray image processing apparatus 102 is a screen signal, the X-ray image acquisition program 212 can identify the X-ray image at the time of the end point of the package and the package area from the information on the shooting time by obtaining the time of the start point of the package and the time of the end point of the package. If the input from the apparatus main body 101 to the X-ray image processing apparatus 102 is RAW data, the X-ray image acquisition program 212 can generate an X-ray image of the package area by using the RAW data from the start time to the end time as the RAW data of one package.

[0070] In addition, when the apparatus main body 101 capable of shooting X-ray images from two directions, the vertical direction and the horizontal direction, is used, the X-ray image acquisition program 212 applies the above three division methods to, for example, the X-ray image taken from the vertical direction and the X-ray image taken from the horizontal direction.

[0071] The X-ray image acquisition program 212 registers the acquired X-ray image in the X-ray image data 220 so that the X-ray image acquired by the X-ray image acquisition program 212 can be used for the article recognition process in step S302 or when the user refers to past X-ray images.

[0072] Next, the article recognition program 213 acquires the one-directional or two-directional X-ray image acquired in step S301 from the X-ray image data 220, and recognizes the category of the article included in each pixel by using the article recognition model 221 and / or the segmentation process of deep learning (S302). Note that, as the article recognition result for each pixel, information indicating the category of the identified article (it may be recognized that only one category of article is included in each pixel, or it may be recognized that articles of a plurality of categories are included), or information indicating that there is no article is obtained.

[0073] Here, for the recognition of an article, libraries such as "Fully Convolutional Instance-aware Semantic Segmentation" and "Mask R-CNN", which are known as libraries of OSS (Open Source Software), can be used. Note that as long as the article included in a region of a predetermined shape such as a pixel unit or a rectangular region can be identified, the recognition method is not limited to the above-described method.

[0074] Here, when the X-ray image acquisition program 212 acquires X-ray images in two directions respectively, the article recognition program 213 performs article recognition processing on each of the two X-ray images, namely, the X-ray image taken from the first direction and the X-ray image taken from the second direction.

[0075] For example, when a smartphone is in the vertical and horizontal directions, if the front of the smartphone is photographed from the vertical direction, then the side of the smartphone is photographed from the horizontal direction. The article recognition program 213 has a high accuracy in determining from the X-ray image in which the front of the smartphone is photographed that the smartphone is included in the X-ray image, but has a low accuracy in determining from the X-ray image in which the side of the smartphone is photographed that the smartphone is included in the X-ray image. Also, when there is an overlap of articles in the photographed direction, it is difficult to discriminate the articles. Thus, since the recognition result may only be obtained for one image, the recognition results of the two photographed images often differ.

[0076] Note that by the article recognition program 213 performing article recognition using an article recognition model 221 in which models in a plurality of directions of the article (for example, a model of the front of the article and a model of the side of the article) are learned as separate models, it is possible to discriminate which direction the article included in the X-ray image is close to being photographed from.

[0077] Also, for example, when an article is photographed at an angle between the front and side of the article, the article recognition program 213 may select the closer one of the front and side of the article. Further, the article recognition program 213 may determine whether the posture of the photographed article is the front or the side using other methods, such as the area of the article and the degree of similarity by template matching with the template images of the front and side prepared in advance, for the result recognized using the article recognition model 221 that combines the front and side. For example, the article recognition program 213 may determine whether the posture of the article is the front or the side, even if it determines like a smartphone (front) or a smartphone (side) at the time of article recognition, for the result recognized using the article recognition model 221 that combines the front and side.

[0078] Also, the article recognition model 221 learned from a color image and the article recognition model 221 learned from a grayscale image are prepared in advance. When the article recognition program 213 determines that the overlap of the articles is large, that is, when it determines that the area of the dark part of the image is larger than a predetermined threshold, etc., recognition can also be performed using the article recognition model 221 learned from a grayscale image. Thereby, the influence of the color change indicating the type of material due to the overlap of the articles can be avoided.

[0079] Next, the region relationship determination program 215 determines whether there is a region where articles of a plurality of categories are recognized as overlapping, using the degree of overlap of the articles of the pixels recognized as including the article (S303). When the region relationship determination program 215 determines that there is no region where articles of a plurality of categories are recognized as overlapping (S304: NO), it generates region information indicating the regions where each article is recognized and the categories of each article without merging the recognition regions of any article (S312).

[0080] When the area relationship determination program 215 determines that there is an area where articles of a plurality of categories are recognized as overlapping (S304: YES), the overlap / front determination program 214 refers to the overlap / front determination table 226 to determine whether all the articles of the plurality of categories recognized as overlapping in the same area are arranged overlappingly, and whether the postures of all the articles of the plurality of categories recognized as overlapping in the same area are not frontal (S305).

[0081] Note that the state where articles of a plurality of categories recognized as overlapping in the same area are arranged overlappingly means that all the articles included in the articles of the plurality of categories are simply arranged in front of or behind the shooting direction, and no arbitrary combination of the articles included in the articles of the plurality of categories can be regarded as articles of the same category, nor can the arbitrary combination be regarded as a combination of a whole object and a constituent object.

[0082] For example, in step S305, the overlap / front determination program 214 determines whether all the conditions indicated by the record corresponding to the category in the overlap / front determination table 226 are satisfied for each of the articles of the plurality of categories. When none or only one of the categories of the articles of the plurality of categories satisfies all the conditions, the overlap / front determination program 214 determines that there is an overlap among all the articles of the plurality of categories and / or the postures of all the articles are not frontal. Also, when the overlap / front determination program 214 determines that there are a plurality of categories of the articles of the plurality of categories that satisfy all the conditions, it determines that there is no overlap and the postures are frontal for the combinations of the articles of the plurality of categories that satisfy all the conditions.

[0083] In the overlapping / front determination table 226, the standard area of articles in each category and the amount of each standard material constituting the articles in each category are defined based on, for example, past statistical data and the like. Therefore, the area relationship determination program 215 can determine whether the articles in each category are within the range of the standard area and material amount by making a determination using the overlapping / front determination table 226, and thus can determine whether there is an overlap and whether the posture of the article is front.

[0084] Note that the overlapping / front determination process in step S305 may be executed in other steps, such as when the article recognition is performed using the article recognition models 221 separated into a front model and a side model in step S302 where the article recognition process is performed.

[0085] FIG. 7 is a schematic diagram showing an example of the overlap of articles when articles in a plurality of categories are recognized in the same area. The X-ray image 701 is an X-ray image taken by a smartphone alone. The X-ray image 702 is an X-ray image taken by a smartphone in a state of overlapping with another article. Since the smartphone in the X-ray image 702 is overlapping with another article, it appears darker in color than the smartphone in the X-ray image 701.

[0086] Also, the X-ray image 703 is an X-ray image of a smartphone taken from an oblique direction. Since the front of the smartphone is not photographed in the X-ray image 703 and it is photographed in a shape different from the front of the smartphone, the area and material of the smartphone in the X-ray image 703 are different from those of the smartphone photographed from the front.

[0087] Return to the description of FIG. 6. If the overlapping / front-facing determination table 226 determines that all articles of a plurality of categories recognized as overlapping in the same area are arranged overlappingly and / or that the postures of all articles of a plurality of categories recognized as overlapping in the same area are not front-facing (S306: NO), the process proceeds to step S312 without merging the areas of the articles of the plurality of categories recognized as overlapping in the same area. If the areas where the articles are recognized are merged when there is an overlap or the posture is not front-facing, the risk of grouping different articles as the same article increases. Therefore, the areas of the articles of the plurality of categories recognized as overlapping in the same area are not merged.

[0088] If the overlapping / front-facing determination program 214 determines that there is a combination of articles of a plurality of categories recognized as overlapping in the same area that have no overlap and are front-facing (S306: YES), the area relationship determination program 215 determines whether the combination of articles of the categories included in the combination includes a combination of articles that can be regarded as the same category as those recognized as a different category, whether the combination of articles includes a combination of articles in the relationship between the whole article and the constituent article, or whether neither combination is included (S307).

[0089] Details of the process in step S307 will be described. First, for example, the area relationship determination program 215 calculates, for each combination of two articles included in a plurality of categories of articles that have no overlap and are front-facing, the ratio of the number of pixels where the article of the other category overlaps the pixels where the article of the one category is recognized, out of the total number of pixels where the article of the one category is recognized. When the ratio is equal to or higher than a predetermined threshold (for example, 90%), the area relationship determination program 215 presumes that the article of the one category is a constituent of the article of the other category.

[0090] Also, for example, the area relationship determination program 215 calculates the IoU (Intersection over Union) between the pixels in the area where an article of one category is recognized and the pixels in the area where an article of another category is recognized for each combination of two articles included in a plurality of categories of articles that do not overlap and have a front-facing posture. For example, the IoU between an article of category A and an article of category B is calculated as IoU = (A ∩ B) / (A ∪ B) (A ∩ B indicates the number of pixels in the common part of the area where the article of category A is recognized and the area where the article of category B is recognized, and A ∪ B indicates the number of pixels where at least one of the article of category A or the article of category B is recognized). When the calculated IoU is equal to or greater than a predetermined threshold, the area relationship determination program 215 infers that the article of one category and the article of the other category are articles of the same category although they are recognized as articles of different categories. Note that when the area relationship determination program 215 determines that both the condition for inferring that two categories of articles are in the relationship of a whole object and a constituent object and the condition for inferring that they are articles of the same category although they are recognized as articles of different categories are satisfied, the area relationship determination program 215 may infer that the two categories of articles are, for example, in the relationship of a whole object and a constituent object but are not articles of the same category. Also, in this case, the area relationship determination program 215 may infer that the two articles are articles of the same category but are not in the relationship of a whole object and a constituent object.

[0091] Also, when the area relationship determination program 215 determines that none of the combinations of two articles included in a plurality of categories of articles that do not overlap and have a front-facing posture can be regarded as articles of the same category even though they are recognized as different categories, nor are they combinations of articles in the relationship of a whole object and a constituent object, it determines that all combinations of articles are combinations of different articles.

[0092] FIG. 8 is an explanatory diagram showing an overview of the relationship of the article recognition regions. Region 801 is a region where an article of category A is recognized as being located, and region 802 is a region where an article of category B is recognized as being located. Further, it is assumed that the article recognized in region 801 and the article recognized in region 802 do not overlap, and the postures of the articles recognized in region 801 and the article recognized in region 802 are determined to be frontal. Since region 802 is included in region 801 (i.e., since the above ratio is 100%), the region relationship determination program 215 infers that the article of category B is a component of the article of category A.

[0093] Also, region 803 is a region where an article of category C is recognized as being located, and region 804 is a region where an article of category D is recognized as being located. Further, it is assumed that the article recognized in region 803 and the article recognized in region 804 do not overlap, and the postures of the articles recognized in region 803 and the article recognized in region 804 are determined to be frontal. Since the IoU between region 803 and region 804 is equal to or greater than a predetermined threshold value, the region relationship determination program 215 infers that the articles of one category and the articles of the other category, although recognized as articles of different categories, are articles of the same category.

[0094] Note that the region relationship determination program 215 may perform the determination of inferring whether there is a relationship between the whole article and the component as described above, and the determination of inferring that articles recognized as articles of different categories are articles of the same category, in combination with the determination in step S304. In this case, when the region relationship determination program 215 infers that there is at least one of a combination of articles inferred to have a component relationship or a combination of articles inferred to be articles of the same category although recognized as articles of different categories, it transitions to step S305, and when it infers that there is no combination of any articles, it transitions to step S312 without merging the recognition regions of any articles.

[0095] Return to the description of the process in step S307. First, the determination of whether there is a relationship between the whole object and the component will be described. The area relationship determination program 215 determines whether a combination of articles presumed to be in a relationship between the whole object and the component is in a relationship between the whole object and the component, with reference to the component relationship data 223. Specifically, when the area relationship determination program 215 presumes that an article in category B is a component of an article in category A, a record in which the value in the category column 223b is category A and the value in the component column 223c is category B is included in the component definition table 2231, and when the recognized article in category A and the article in category B satisfy the conditions indicated by the area column 223e and the center of gravity position column 223f in the record, it is determined that the article in category B is a component of category A.

[0096] Note that the area relationship determination program 215 may determine that the article in category B is a component of category A when either a record in which the value in the category column 223b is category A and the value in the component column 223c is category B is included in the component definition table 2231, or the recognized article in category A and the article in category B satisfy the conditions indicated by the area column 223e and the center of gravity position column 223f in the record.

[0097] By the area relationship determination program 215 executing the determination of whether there is a relationship between the whole object and the component by the above-described process, in the case where a component is built in an article of a certain category, etc., the article and the component are not detected separately, but can be detected together as a whole object. In particular, for example, although a smartphone with a built-in battery as a component and an airbag with a built-in gas cylinder as a component are not prohibited articles, a single battery or a single gas cylinder may be a prohibited article. In such a case, by grouping articles in a component relationship, it is possible to suppress the occurrence of a situation where a prohibited article alert is erroneously output.

[0098] In addition to determining the relationship between the whole object and the component by referring to the above-described component relationship data 223, the area relationship determination program 215 may further execute the determination of the relationship between the whole object and the component by referring to the article quantity table 2232.

[0099] Specifically, for the combination of articles determined to be in a component relationship by referring to the above-described component relationship data 223, the area relationship determination program 215 determines that the combination of the articles is in a relationship between the whole object and the component when the number indicated by the number column 223j in the record having the values of the category column 223h and the component column 223i corresponding to the component relationship matches the number of components included in the component relationship, and determines that it is not in a relationship between the whole object and the component when they do not match.

[0100] For example, according to the article quantity table 2232 in FIG. 3, since the number of batteries of a smartphone is 1, when two batteries are detected in the area where the smartphone is recognized, there may be a battery different from the battery built in the smartphone as a component, so the area relationship determination program 215 does not group the recognition areas of the smartphone and the two batteries (it may group the smartphone and one battery and not group the other battery, or it may not group the smartphone and the two batteries at all).

[0101] Conversely, when a smartphone as a whole object is recognized but the battery, which is a component of the smartphone, is not recognized, the area relationship determination program 215 may add a virtual detection area for prompting attention that the battery has been missed in detection on the recognition area of the smartphone. Among articles, although there are articles that are not treated as prohibited articles when there is only one, there are articles that are treated as prohibited articles when they are larger than the specified size or there are more than the specified number. Therefore, when a detection omission is suspected in the determination of the relationship as described above, the area relationship determination program 215 can suppress the oversight due to the quantity limit by displaying an area indicating the possibility of detection omission within the area of the article with a dashed rectangle or the like.

[0102] Next, a determination for grouping articles recognized as belonging to different categories into articles of the same category will be described. The area relationship determination program 215 determines, with reference to the category relationship table 224, whether a combination of articles that are presumed to be groupable into articles of the same category although recognized as articles of different categories can actually be grouped.

[0103] Specifically, when the area relationship determination program 215 presumes that an article of category A and an article of category B can be grouped together, if a record in which the combination of the value in the first category column 224b and the value in the second category column 224c is the combination of the article of category A and the article of category B is included in the category relationship table 224, and the recognized article of category A and the article of category B satisfy the conditions indicated by the area column 224d, the metal amount column 224e, the light metal amount column 224f, and the organic matter amount column 224g in the record, it is determined that the article of category A and the article of category B can be grouped together.

[0104] Note that when the area relationship determination program 215 presumes that an article of category A and an article of category B can be grouped together, it may also be determined that the article of category A and the article of category B can be grouped together if either the combination of the value in the first category column 224b and the value in the second category column 224c is the combination of the article of category A and the article of category B and the record is included in the category relationship table 224, or the recognized article of category A and the article of category B satisfy the conditions indicated by the area column 224d, the metal amount column 224e, the light metal amount column 224f, and the organic matter amount column 224g in the record.

[0105] By the area relationship determination program 215 executing the determination to group articles of different categories into articles of the same category by the above-described process, articles that are actually the same article but are recognized as articles of a plurality of different categories can be grouped together, reducing the burden on the user.

[0106] Specifically, for example, even if the categories of gas cylinders and spray cans are defined as prohibited items, only one of the categories is output without outputting both the categories of gas cylinders and spray cans, thereby reducing the burden on the user. In particular, when the number of defined categories is large, there is a high possibility that a plurality of categories are recognized for one article, but the area relationship determination program 215 can output the plurality of categories by grouping them into one category.

[0107] Also, in the above-described process, the area relationship determination program 215 can group, with a high probability, only when they are the same article, even when recognized in an area where hair dye and a spray can that can be grouped as articles of the same category overlap, by considering the conditions indicated by the area column 224d, the metal amount column 224e, the light metal amount column 224f, and the organic matter amount column 224g.

[0108] In step S305, the overlap / front determination program 214 performs overlap / front determination on articles of categories that may be grouped into other categories. Therefore, the conditions based on the area and material amount of the articles in the overlap / front determination table 226 are desirably looser than the conditions based on the area and material quality of the articles in the category relationship table 224 and the component definition table 2231. The area and features registered in the category relationship table 224 may be statistically generated from feedback results indicating merging of articles of a plurality of categories input by the user in the past, data with high reliability of article recognition, and the like.

[0109] When the area relationship determination program 215 determines that there is no combination of articles that can be regarded as the same category although recognized as different categories, and no combination of articles in the relationship between the whole article and the component (S308: different articles), the process proceeds to step S312 without merging the recognition areas of any articles.

[0110] When the area relationship determination program 215 determines that there is at least one of a combination of articles that can be regarded as the same category although recognized as different categories, and a combination of articles in a relationship between a whole article and a component article (S308: Component article / different category), it executes area feature determination processing (S309).

[0111] When article A and component B of article A which is a whole article are recognized, even if component B is in a part of the area of article A, it is conceivable that component B may be accidentally under article A. Therefore, in step S309, for example, when the area relationship determination program 215 determines that the position of component B with respect to article A which is a whole article is included in a predetermined standard position range, and the area ratio of component B with respect to article A is included within a predetermined standard range, it determines that the area features of the area where article A is recognized and the area where component B is recognized are similar.

[0112] When the area relationship determination program 215 determines that there is no combination with similar area features among a combination of articles that can be regarded as the same category although recognized as different categories, and a combination of articles in a component relationship (S310: NO), it transitions to step S312 without merging the recognition areas of any articles.

[0113] When the area relationship determination program 215 determines that there is a combination with similar area features among a combination of articles that can be regarded as the same category although recognized as different categories, and a combination of articles in a component relationship (S310: YES), it merges the recognition areas of the combinations with similar area features among each combination of articles of a combination of articles that can be regarded as the same category although recognized as different categories, and / or a combination of articles in a component relationship (S311), and transitions to step S312.

[0114] Note that when the region relationship determination program 215 merges the recognition regions of combinations of articles that can be regarded as the same category although recognized as different categories and have similar region characteristics, for example, it selects one of the recognition regions of the combinations before merging as the recognition region after merging, and determines the category of the recognition region after merging from one of the categories of the combinations before merging.

[0115] Specifically, for example, when the region relationship determination program 215 merges the recognition region of an article of category A and the recognition region of an article of category B, it selects the recognition region of the article with a higher reliability of the article recognition result among these two recognition regions as the recognition region after merging, and determines the category of the article with the higher reliability as the category of the recognition region after merging. By adopting the category of the recognition region with high reliability, the region relationship determination program 215 can appropriately select the category name indicating the characteristics of the recognition region.

[0116] Also, for example, when the region relationship determination program 215 merges the recognition region of an article of category A and the recognition region of an article of category B, it may select the recognition region of the article of the category with a higher predetermined priority as the recognition region after merging and determine the category with the higher priority as the category of the recognition region after merging, or it may select the recognition region of the randomly selected category as the recognition region after merging and determine the randomly selected category as the category of the recognition region after merging.

[0117] In addition, when the region relationship determination program 215 merges the recognition regions of combinations of articles that are in the relationship of a whole object and a constituent object and have similar region characteristics, it selects the recognition region of the whole object as an option for the recognition region after merging and determines the category of the whole object as the category of the recognition region after merging.

[0118] Note that the processes of step S310 and step S311 may be omitted. In this case, in step S308, when it is determined that there is at least one of a combination of articles that can be regarded as the same category among those recognized as different categories and a combination of articles in a constituent relationship, the recognition regions of each of these combinations are merged.

[0119] Next, the table update program 216 registers, as accumulation information on the overlapping pattern, information indicating a combination of categories of articles whose recognition regions overlap, the IoU of the recognition regions in the combination, the area of the articles included in the combination and the amount of each material, and information such as the area ratio of the constituent to the whole article of the constituent merged as a combination of the whole article and the constituent, and the position of the constituent in the overlapping data 225 (S313).

[0120] For example, when the table update program 216 refers to the overlapping data 225 and determines that in an X-ray image of an airbag with a built-in gas cylinder, the airbag and the gas cylinder are recognized at the same position every time, the table update program 216 estimates that the gas cylinder is a constituent of the airbag, and newly defines, in the constituent relationship data 223, the relationship in which the airbag is the main article and the gas cylinder is the constituent.

[0121] Also, for example, when the table update program 216 refers to the overlapping data 225 and estimates that articles of categories with an IoU value close to 1 (for example, a predetermined value or more such as 0.8 or more) are recognized as the same article as articles of two categories, the table update program 216 newly defines, in the category relationship table 224, that the articles of these categories can be treated as articles of the same category.

[0122] Note that the process of newly defining the relationship between the main article and the constituent in the constituent relationship data 223 and the process of newly defining in the category relationship table 224 that articles of multiple categories can be treated as articles of the same category do not necessarily have to be performed completely automatically. For example, candidates for newly added definitions may be periodically presented to the user, and new definitions may be added according to the user's instructions.

[0123] In this way, the table update program 216 accumulates the overlapping pattern in the overlapping data 225 based on the actually obtained X-ray image, and updates the component relationship data 223 and the category relationship table 224 based on the accumulated overlapping pattern, so that the component relationship data 223 and the category relationship table 224 can be optimized based on the goods handled in the actual operation.

[0124] Note that even when the table update program 216 accumulates information about the combination of categories already registered in the component relationship data 223 and the category relationship table 224 in the overlapping data 225, the image features such as the area and the material quantity may change due to the appearance of new products, etc. Therefore, it is desirable to update the information about the area and the material quantity in the component relationship data 223 and the category relationship table 224.

[0125] Next, the display image generation program 217 generates an image to be displayed on the display device 103 (S314). The screen display program 218 displays the image generated in step S314 on the display device 103 (S315).

[0126] FIG. 9 is an explanatory diagram showing an example of a display image displayed on the display device 103. The display image 901 is an X-ray image in which six articles that are recognized as both hair dyes and sprays are photographed. In the display image 902, all of the six articles are recognized as both hair dyes and sprays, and it includes six frames indicating the recognition areas of the hair dyes and six names of hair dyes, as well as six frames indicating the recognition areas of the sprays and six names of sprays.

[0127] In the display image 903, although hair dyeing and spray belong to different categories, they are determined to be grouped into the same category. A merged frame (assuming here that the hair dyeing frame is selected as the merged frame) that combines the frame indicating the recognition area of hair dyeing and the frame indicating the recognition area of spray, and the name of hair dyeing, which is the name corresponding to the merged frame, are included. The display image 903 is a display image included in the display screen displayed in step S315 when an X-ray image in which six articles recognized as both hair dyeing and spray are photographed is input to the X-ray image processing apparatus 102.

[0128] Note that when the display image generation program 217 determines that articles of the same category are arranged adjacent to each other (for example, arranged within a predetermined distance), it may be good to generate an image that combines the frames of the recognition areas of the articles of the same category arranged adjacent to each other and also displays the number of the combined recognition area frames, as in the display image 904.

[0129] The display image 905 is an X-ray image of a smartphone with a built-in battery. In the display image 906, the smartphone and the battery are recognized as different articles, and a frame indicating the recognition area of the smartphone and a frame indicating the recognition area of the battery are respectively included.

[0130] In the display image 907, it is determined that the battery is a component of the smartphone, and the frame indicating the recognition area of the battery as a component and the name "battery" are not included, while the frame indicating the recognition area of the smartphone as the main body and the name "smartphone" are included. The display image 907 is a display image included in the display screen displayed in step S315 when an X-ray image of a smartphone including the battery as a component is input to the X-ray image processing apparatus 102.

[0131] In addition, the display image 907 may further include text, icons, etc. indicating that the smartphone includes a battery as a component. Also, when an item such as a battery, whose classification as a prohibited item depends on its quantity, is included in the X-ray image, if the recognized quantity of the item exceeds a predetermined value (i.e., when it is treated as a prohibited item), a frame indicating the recognition area of the item may be included in the display image 907, and if the recognized quantity of the item is less than or equal to the predetermined value, the frame indicating the recognition area of the item may not be included in the display image 907.

[0132] Note that in the example of FIG. 9, the frame indicating the recognition area of the item is displayed as a rectangle, but the shape of the frame is not limited to a rectangle. For example, the shape of the frame may be the contour of the item recognized in the item recognition process.

[0133] Finally, the table update program 216 executes feedback processing (S316) and ends the process. In the feedback processing, when the user feeds back that the recognition result of the item is incorrect or there is a recognition omission, the table update program 216 registers feedback information in the X-ray image data 220 and the feedback data 222. Details of the feedback processing will be described later with reference to FIGS. 10 to 13.

[0134] Note that the feedback processing in step S316 may be omitted. If the inspection continues, after step S316, the process may transition to step S301 to start processing for a new X-ray image.

[0135] FIG. 10 is a flowchart showing an example of the feedback processing. The screen display program 218 displays a feedback screen on the display device 103 (S901). Note that the feedback screen and the screen displayed in step S315 may be the same.

[0136] The table update program 216 receives an input from the user via the input device 104 for the position in the X-ray image where feedback is required (S902). If the table update program 216 determines that the position where feedback is required has not been input by the user (S903: NO), the feedback process ends. When the table update program 216 determines that the position where feedback is required has been input by the user (S903: YES), it determines the type of the input feedback (S904).

[0137] For example, there are a first type of feedback for summarizing combinations of articles recognized as different-category articles although recognized at overlapping positions, a second type of feedback for adding combinations of articles recognized as different-category articles at overlapping positions as combinations of a whole article and a constituent, and a third type of feedback for separating a constituent included in a whole article from the whole article, etc.

[0138] Next, the table update program 216 displays, on the feedback screen, a display area for inputting the confirmation result of the feedback content according to the type of feedback obtained in step S904 (S905), and receives an input of the confirmation result from the user via the input device 104 (S906).

[0139] The table update program 216 stores the input confirmation result in the feedback data 222 (S907). The table update program 216 executes the reflection of the feedback data for updating the constituent relationship data 223 or the category relationship table 224 according to the type of feedback and the confirmation result (S908), and ends the feedback process.

[0140] Note that when the type of feedback is the first type, the table update program 216 adds the combination of the categories to the category relationship table 224 in step S908.

[0141] At this time, the table update program 216 may obtain, from the X-ray image data 220, the past X-ray images recognized in the area where the articles of the combination overlap, and based on the obtained X-ray images, determine conditions such as the area and the amount of material to be stored in the category relationship table 224. Also, when only one X-ray image recognized in the area where the articles of the combination overlap is included in the X-ray image data 220, the table update program 216 may use the upper limit value and the lower limit value obtained by increasing or decreasing a predetermined value by a predetermined allowable amount to determine conditions such as the area and the amount of material to be stored in the category relationship table 224.

[0142] Also, when the type of feedback is the second type, the table update program 216 adds the combination of the whole object and the component to the component relationship data 223. Also, when the type of feedback is the third type, the table update program 216 deletes the combination of the whole object and the component from the component relationship data 223.

[0143] Note that the table update program 216 may not delete the combination of the whole object and the component from the component relationship data 223 when it has received the third type of feedback only once. Specifically, for example, the table update program 216 deletes the combination of the whole object and the component from the component relationship data 223 only when it has received the third type of feedback at a predetermined frequency or more, or when the third type of feedback is specified by the user. When the combination of the whole object and the component is not deleted although the third type of feedback has been received, the area and the amount of material of the component of the article of the category of the combination are accumulated in the feedback data 222.

[0144] As a result, the table update program 216 can obtain statistical information on the accumulated area and material quantity of the articles in the category of the combination, and based on the statistical information, conditions (such as area and material quantity conditions) for not grouping the articles in the category of the combination as an overall object and a constituent object may be additionally registered in the constituent relationship data 223. In this case, even if an article in the category of the combination defined as a combination of an overall object and a constituent object in the constituent relationship data 223 is recognized, if the article in the category of the combination satisfies the condition, the articles in the combination will not be grouped as a combination of an overall object and a constituent object, so that unnecessary feedback for the user can be suppressed.

[0145] FIGS. 11, 12, and 13 are examples of a feedback screen. The feedback screen 1100 includes an X-ray image display area 1101 and a representative name selection area 1102. In the X-ray image display area 1101, an X-ray image including a frame indicating recognition areas of two categories is displayed. In the example of FIG. 11, only one article recognized as both hair dye and spray is displayed in the X-ray image, but since the combination of hair dye and spray is not defined in the category relationship table 224, a frame indicating the recognition area of hair dye and a frame indicating the recognition area of spray are respectively displayed.

[0146] The arrow included in the X-ray image in the X-ray image display area 1101 indicates a cursor. When there is an area in the X-ray image where articles of two categories are recognized as overlapping, the IoU of the recognition areas of the two categories of articles is equal to or greater than a predetermined threshold, and the combination of the two categories is not defined in the category relationship table 224, when the overlapping area is selected, it is determined that the first type of feedback has been input.

[0147] In the example of FIG. 11, when the overlapping area between the frame indicating the recognition area of "hair dyeing" and the frame indicating the recognition area of "spray" is selected by the user using the cursor, it is determined that the first type of feedback has been input, and the representative name selection area 1102 is displayed as the area for inputting the confirmation result of the feedback.

[0148] In the representative name selection area 1102, when one of "spray" or "hair dyeing" is selected, the feedback type and the result of the selection are stored in the feedback data 222, and in the category relationship table 224, the combination of "spray" and "hair dyeing" (for example, it is preferable that the one selected in the representative name selection area 1102 among "spray" or "hair dyeing" is stored in the first category column 224b), the material quantity and area of "spray" and "hair dyeing" in the X-ray image of the X-ray image display area 1101 are added.

[0149] The feedback screen 1200 of FIG. 12 includes, for example, an X-ray image display area 1201 and a component determination area 1202. In the example of FIG. 12, the frame indicating the recognition area of the battery is included in the frame indicating the recognition area of the smartphone, but since the combination of the smartphone as the whole object and the battery as the component is not defined in the component relationship data 223, the frame indicating the recognition area of the smartphone and the frame indicating the recognition area of the battery are each displayed.

[0150] The arrow included in the X-ray image of the X-ray image display area 1201 indicates the cursor. When there is an area in the X-ray image where articles of two categories are recognized as overlapping, and the ratio of the number of pixels of the article of the other category to the number of pixels of the article of one of the two categories that overlap is equal to or greater than a predetermined value, and the combination of the two categories is not defined in the component relationship data 223, when the overlapping area is selected, it is determined that the second type of feedback has been input.

[0151] In the example of FIG. 12, when the overlapping area between the frame indicating the recognition area of the "smartphone" and the frame indicating the recognition area of the "battery" is selected by the user using the cursor, it is determined that the second type of feedback has been input, and the component determination area 1202 is displayed as an area for inputting the confirmation result of the feedback.

[0152] In the component determination area 1202, when "Yes" is selected, the feedback type and the result of the selection are stored in the feedback data 222, and in the component relationship data 223, the combination where the "smartphone" is the whole object and the "battery" is the component, and the material quantity and area of the "smartphone" and the "battery" in the X-ray image of the X-ray image display area 1201 are added. In the component determination area 1202, when "No" is selected, the feedback type and the result of the selection are stored in the feedback data 222.

[0153] The feedback screen 1300 in FIG. 13 includes, for example, an X-ray image display area 1301 and a component cancellation area 1302. In the example of FIG. 13, although a battery including a smartphone is displayed in the X-ray image, since the combination where the smartphone is the whole object and the battery is the component is defined in the component relationship data 223, the frame indicating the recognition area of the battery is not displayed, and only the frame indicating the recognition area of the smartphone is displayed.

[0154] The arrow included in the X-ray image in the X-ray image display area 1301 indicates the cursor. When there is an article in the category determined to be in the relationship of the whole object and the component in the X-ray image, when the area of the whole object (or the component) is selected, it is determined that the third type of feedback has been input.

[0155] In the example of FIG. 13, when the area within the frame indicating the recognition area of the "smartphone" is selected by the user using the cursor, it is determined that the third type of feedback has been input, and the component cancellation area 1302 is displayed as an area for inputting the confirmation result of the feedback.

[0156] In the component release area 1302, when "Yes" is selected, the feedback type and the result of the selection are stored in the feedback data 222. Further, in the component release area 1302, when "Yes" is selected at a frequency equal to or higher than a predetermined value, the combination in which "smartphone" is the whole object and "battery" is the component is deleted from the component relationship data 223. Also, in the component release area 1302, when "No" is selected, the feedback type and the result of the selection are stored in the feedback data 222.

[0157] According to the X-ray image processing apparatus 102 of the present embodiment, when a plurality of categories of articles are recognized in an overlapping area on the X-ray image, if the plurality of categories can be regarded as the same, or if the articles of the plurality of categories are in a relationship of a whole object and a component, the merged recognition area and the merged category name of the plurality of category articles are displayed.

[0158] This makes it easier for the user to check the recognition area, and thus improves the inspection efficiency. Further, in the X-ray image processing apparatus 102, even if a large number of categories are defined as the categories of the articles to be recognized, if the recognition areas are merged, it is possible to prevent the frames indicating the recognition areas from being displayed doubly.

[0159] Also, the X-ray image processing apparatus 102 can distinguish whether the article recognized in the X-ray image is a component of another article or an article that is simply overlapped (i.e., located on the front side or the back side) and not a component of another article, and thus can display the recognition area, thereby preventing the user from overlooking.

[0160] In the present embodiment, an example in which a luggage is photographed from one direction to generate an X-ray image has been mainly described, but the luggage may be photographed from a plurality of directions. When the luggage is photographed from a plurality of directions to generate an X-ray image, the X-ray image processing apparatus 102 may also utilize information on whether the same position in a plurality of directions is detected in the overlap / front determination.

[0161] Specifically, for example, assume that a smartphone is photographed from two directions (e.g., from above and from the left), and X-ray images of the upper surface and the side surface (elongated shape) of the smartphone are obtained. Further, when another article is placed overlapping the smartphone vertically, in the side image, the thickness may be greater than that of the actual smartphone, or another article may be seen with a slight gap from the smartphone. Therefore, the X-ray image processing apparatus 102 determines whether there is an overlap by taking into account both the images from above and from the side. Also, for example, when the apparatus main body 101 captures a CT (Computed Tomography) image as an X-ray image, since the X-ray image processing apparatus 102 can accurately estimate the three-dimensional positional relationship of the article from the CT image, the presence or absence of overlap of the article and whether it is the front may be determined using this three-dimensional positional relationship.

[0162] Further, the X-ray image processing apparatus 102 may condition the area and the material amount in the category relationship table in two directions. Specifically, for example, when a smartphone is photographed from two directions (e.g., from above and from the left) and X-ray images of the upper surface and the side surface (elongated shape) of the smartphone are obtained, conditions for determining overlap and front are defined for the area and the material amount obtained from the upper surface image and the area and the material amount obtained from the side surface image, respectively. At this time, for example, for each of the upper surface image and the side surface image of the smartphone, if the conditions for the area and the material amount are satisfied, the X-ray image processing apparatus 102 may determine that the smartphone is in the front and there is no overlap.

[0163] Also, in this embodiment, an example in which imaging using X-rays is performed is shown. However, as long as it is an electromagnetic wave that can penetrate an article for imaging, for example, other electromagnetic waves such as terahertz waves may be used instead of X-rays.

[0164] Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace other configurations for a part of the configuration of each embodiment.

[0165] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, by an integrated circuit. Also, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, file, etc. for realizing each function can be placed in a memory, a recording device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, a DVD.

[0166] Also, the control lines and information lines show those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In practice, it may be considered that almost all the configurations are interconnected.

Description of Reference Numerals

[0167] 102 X-ray image processing apparatus, 103 display apparatus, 104 input apparatus, 201 CPU, 202 memory, 205 communication apparatus, 210 auxiliary storage apparatus, 212 X-ray image acquisition program, 213 article recognition program, 214 overlap / front determination program, 215 region relationship determination program, 216 table update program, 217 display image generation program, 218 screen display program, 220 X-ray image data, 221 article recognition model, 222 feedback data, 223 component relationship data, 224 category relationship table, 225 overlap data, 226 overlap / front determination table

Claims

1. An X-ray image processing apparatus, comprising a processor and a memory, connected to a display device, wherein the memory holds X-ray image information indicating an X-ray image, a category of an article included in the X-ray image, a recognition region where the article is recognized in the X-ray image, and at least one of the area and the material quantity of the article, holds relationship information indicating a combination of categories of articles, and holds overlap / front determination information indicating a category of an article and a condition indicating at least one of the area and the material quantity of the article, wherein the processor determines whether each of a plurality of articles whose recognition regions overlap in the X-ray image satisfies the condition corresponding to the category of the article indicated by the overlap / front determination information, and determines whether the plurality of articles are arranged without overlapping and whether the postures of the plurality of articles are frontal, when it is determined that the plurality of articles are arranged without overlapping and the postures of the plurality of articles are frontal, determines whether to integrate the recognition regions of the plurality of articles in the X-ray image into one recognition region based on a determination result as to whether the combination of the categories of the plurality of articles is included in the combination of categories indicated by the relationship information, and when it is determined to integrate the recognition regions of the plurality of articles into one recognition region, draws the integrated one recognition region in the X-ray image and displays it on the display device. An X-ray image processing apparatus.

2. The X-ray image processing apparatus according to claim 1, wherein the relationship information includes category relationship information indicating a combination of categories that can be regarded as the same article although they are articles of different categories, wherein the processor when it is determined that the plurality of articles are arranged without overlapping, the postures of the plurality of articles are frontal, and the IoU of the recognition regions of the plurality of articles is equal to or greater than a predetermined value, determines whether to integrate the recognition regions of the plurality of articles in the X-ray image into one recognition region based on a determination result as to whether the combination of the categories of the plurality of articles is included in the combination of categories indicated by the category relationship information. An X-ray image processing apparatus.

3. The X-ray image processing apparatus according to claim 2, connected to an input device, wherein the processor The plurality of articles are arranged without overlapping, the postures of the plurality of articles are frontal, the IoU of the recognition regions of the plurality of articles is equal to or greater than a predetermined value, and when it is determined that the combination of the categories of the plurality of articles is not included in the combination of categories indicated by the category relationship information, an input indicating whether to include the combination of the categories of the plurality of articles in the category relationship information is received via the input device, An X-ray image processing apparatus that updates the category relationship information based on the input.

4. The X-ray image processing apparatus according to claim 1, The relationship information indicates a combination of categories of articles and a condition indicating at least one of the area and the material quantity of the articles in the combination of categories, The processor, When it is determined that the plurality of articles are arranged without overlapping and the postures of the plurality of articles are frontal, An X-ray image processing apparatus that determines whether to integrate the recognition regions of the plurality of articles in the X-ray image into one recognition region based on a determination result as to whether the combination of the categories of the plurality of articles is included in the combination of categories indicated by the relationship information and whether the plurality of articles satisfy the conditions indicated by the relationship information.

5. An X-ray image processing method by an X-ray image processing apparatus, The X-ray image processing apparatus, Has a processor and a memory, Is connected to a display device, The memory, Holds X-ray image information indicating an X-ray image, the category of an article included in the X-ray image, a recognition region in which the article is recognized in the X-ray image, and at least one of the area and the material quantity of the article, Relationship information indicating a combination of categories of articles, Overlap / front determination information indicating a category of an article and a condition indicating at least one of the area and the material quantity of the article, The X-ray image processing method, Based on a determination result as to whether each of the plurality of articles whose recognition regions overlap in the X-ray image satisfies the condition corresponding to the category of the article indicated by the overlap / front determination information, the processor determines whether the plurality of articles are arranged without overlapping and the postures of the plurality of articles are frontal, When the processor determines that the plurality of articles are arranged without overlapping and the postures of the plurality of articles are frontal, based on the determination result as to whether the combination of the categories of the plurality of articles is included in the combination of the categories indicated by the relationship information, it is determined whether to integrate the recognition regions of the plurality of articles in the X-ray image into one recognition region. An X-ray image processing method, wherein when the processor determines to integrate the recognition regions of the plurality of articles into one recognition region, the integrated one recognition region is drawn in the X-ray image and displayed on the display device.

Images