Display device

Abstract

To provide a display device which allows a user to recognize a detected content from a transmitted image in which the inside of a container is transmitted without receiving a special training.SOLUTION: A transmission image acquisition part 111 acquires a transmission image from an X-ray inspection device 2. A content detection part 112 detects a container and a content from the transmission image by using a learned model 121. A type identification part 113 identifies the type of the detected content. An attitude identification part 114 identifies the attitude of the detected content in the container. A graphic acquisition part 115 acquires graphic data of the identified type from a graphic DB 122. A hazard degree determination part 116 determines the hazard degree of the detected content by referring to the graphic DB 122. A synthesized image generation part 117 generates a synthesized image in which a graphic representing the shape of the content is superposed on the transmission image by using the acquired graphic data. A display control part 118 displays the generated synthesized image on a display part 15.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a display device technology for detecting and displaying contents of a container such as baggage from a transmitted image captured through the inside of the container. [Background technology]

[0002] Baggage inspections at airports and other facilities are generally carried out by personnel who visually inspect the X-ray images to make a judgment. To support the personnel's visual judgment, devices that learn detection targets such as dangerous objects in advance and display detected detection targets in an emphasized manner are becoming widespread.

[0003] Patent Document 1 discloses a two-dimensional and three-dimensional CT system that reconstructs and displays the inside of an object by calculation using projection data obtained by transmitting the object from a small number of directions.

[0004] Patent Document 2 discloses an inspection device in which a detection unit is virtually divided into a plurality of regions, and the acquisition status of the detection result of the sample for each region is displayed on a display unit.

[0005] Patent document 3 discloses an enclosure inspection device that irradiates the front or back of an envelope with a specific near-infrared wavelength using a near-infrared LED, and captures a transmitted image formed by absorbing or reflecting the light that passes through the envelope. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2000-215304 A [Patent Document 2] Patent Publication No. 2022-49924 [Patent Document 3] JP 2018-58258 A Summary of the Invention [Problem to be solved by the invention]

[0007] X-ray baggage inspection devices generally display X-ray transmission images as inspection results. However, X-ray transmission images are often difficult to understand for untrained personnel. In response to this, as shown in the above-mentioned patent documents, devices that detect dangerous objects from X-ray transmission images and display a rectangle circumscribing the shape of the dangerous object or display a character string indicating the name of the dangerous object are also being considered. However, even if the dangerous object is highlighted by the rectangle or character string, it may be difficult for staff members who view the transmission image to intuitively understand what type of dangerous object has been detected. Therefore, when a baggage inspection device that displays X-ray transmission images is introduced, costs are incurred for training staff members who handle the device.

[0008] An object of the present invention is to provide a display device that enables a user, without special training, to grasp the contents detected from a transmitted image of the inside of a container. [Means for solving the problem]

[0009] In a first aspect, the present invention provides a display device that detects contained items by type from a transmitted image of the inside of a container, and arranges a figure representing the shape of each detected item according to the position at which the item was detected, and superimposes it on the transmitted image or displays it at the position of the transmitted image.

[0010] According to the display device of the first aspect, a user can recognize the contained item detected from the transmitted image through the inside of the container without receiving any special training.

[0011] In the display device of the first aspect, a configuration in which the graphic corresponding to each posture of the contained item is displayed may be adopted as a second aspect.

[0012] According to the display device of the second aspect, even if the contained item looks different depending on the position, the user can check the figure according to the shape seen in each position.

[0013] In the display device of the first aspect, a configuration may be adopted in which the detected contents are judged to be a hazardous item or not, and the graphic is displayed in a manner according to the judgment result.

[0014] According to the display device of the third aspect, the user can confirm the display according to the judgment of whether or not the object is a dangerous object.

[0015] In the display device according to any one of the first to third aspects, a configuration may be adopted as a fourth aspect in which the graphic corresponding to the contained item is generated from the transmission image and displayed.

[0016] According to the display device of the fourth aspect, it is not necessary to store in advance a figure corresponding to the contained item. [Brief description of the drawings]

[0017] [Figure 1] 1 is a diagram showing an example of the overall configuration of a display system 9 according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of an X-ray inspection device 2. [Diagram 3] FIG. 1 shows an example of the configuration of a display device 1. [Figure 4] FIG. 1 is a diagram for explaining a multilayer perceptron. [Diagram 5] A figure showing an example of a trained model 121. [Figure 6] FIG. 13 is a diagram showing an example of a graphic DB 122. [Figure 7] FIG. 2 is a diagram showing an example of the functional configuration of the display device 1. [Figure 8] 4 is a flowchart showing an example of the operation flow of the X-ray inspection device 2. [Figure 9] FIG. 4 is a flowchart showing an example of the operation flow of the display device 1. [Figure 10] FIG. 11 is a diagram showing an example of a composite image to be displayed. [Figure 11] FIG. 13 is a diagram showing an example of a graphic DB 122 in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] <Embodiment> <Overall Configuration of Display System> FIG. 1 is a diagram showing an example of the overall configuration of a display system 9 according to an embodiment of the present invention. This display system 9 irradiates an X-ray on a container J to be inspected, and detects the contents stored in the container J from a transmission image generated based on the transmitted light and displays it to the user. The display system 9 shown in FIG. 1 includes a display device 1 and an X-ray inspection device 2. The display device 1 and the X-ray inspection device 2 are connected to be communicable, for example, by wire.

[0019] The X-ray inspection device 2 is an inspection device that generates the above-described transmission image and supplies it to the display device 1. The display device 1 is a display device that detects the contents stored in the container J from the transmission image supplied from the X-ray inspection device 2 and displays it to the user.

[0020] <Configuration of X-ray Inspection Device> FIG. 2 is a diagram showing an example of the configuration of the X-ray inspection device 2. This X-ray inspection device 2 includes a control board 21, an interface 23, a photosensor 24, an X-ray generator 25, an X-ray sensor 26, a front conveyor 27, and a rear conveyor 28. In addition, a front conveyor controller 270 for controlling the front conveyor 27 and a rear conveyor controller 280 for controlling the rear conveyor 28 are respectively provided on the front conveyor 27 and the rear conveyor 28.

[0021] The control board 21 is a control device for controlling the X-ray inspection device 2. The control board 21 is connected to the interface 23, the photosensor 24, the X-ray generator 25, the X-ray sensor 26, the front conveyor controller 270, and the rear conveyor controller 280, for example, by a bus or the like. The control board 21 controls the front conveyor 27 via the front conveyor controller 270 and controls the rear conveyor 28 via the rear conveyor controller 280.

[0022] The photosensor 24 is a device that detects the presence or absence of a container J before inspection, and includes, for example, an infrared LED (light emitting diode) and a photodiode. The photosensor 24 has an infrared LED provided on the side of the preceding conveyor 27 that irradiates infrared rays into the space above the preceding conveyor 27. The photosensor 24 detects a container J being transported on the preceding conveyor 27 by receiving the infrared rays with a photodiode provided on the side opposite to the infrared LED.

[0023] When the photodiode detects that the infrared rays have been blocked, the photosensor 24 sends a signal indicating that to the control board 21. Based on the signal sent from the photosensor 24, the control board 21 determines that the infrared rays have been blocked and that a container J is present.

[0024] The X-ray generator 25 is installed at the top of the transport path midway from the front-stage conveyor 27 to the rear-stage conveyor 28. Under the control of the control board 21, the X-ray generator 25 generates X-rays and irradiates the container J passing directly underneath with the X-rays from above.

[0025] When the container J is detected by the photosensor 24, the control board 21 controls the X-ray generator 25 to generate X-rays at the timing when the container J passes directly under the X-ray generator 25.

[0026] The X-ray sensor 26 is a sensor that detects X-rays irradiated by the X-ray generator 25 and transmitted through the container J. The X-ray sensor 26 is a line sensor that extends in a direction that intersects, for example, approximately perpendicularly with the conveying direction of the front-stage conveyor 27. The X-ray sensor 26 may have, for example, an image sensor such as a scintillator that converts X-rays into visible light, a CCD (Charge Coupled Device), or a CMOS (Complementary Metal Oxide Semiconductor). While the container J is conveyed across the group of sensors arranged in a straight line of the X-ray sensor 26, the X-ray sensor 26 sequentially converts the transmitted light into signals, thereby generating a transmission image capturing the container J on a plane.

[0027] The interface 23 is a communication circuit for communicatively connecting the X-ray inspection apparatus 2 to an external device such as the display device 1. The control board 21 supplies the transmission image generated by the X-ray sensor 26 to the display device 1 via the interface 23.

[0028] <Display device configuration> Fig. 3 is a diagram showing an example of the configuration of the display device 1. The display device 1 is a display device that acquires a transmission image from the X-ray inspection device 2, detects an object contained in a container J shown in the transmission image, and displays a graphic representing the shape of the object. The display device 1 shown in Fig. 3 has a processor 11, a memory 12, an interface 13, and a display unit 15.

[0029] The processor 11 reads and executes a computer program (hereinafter simply referred to as a program) stored in the memory 12 to control each part of the display device 1. The processor 11 is, for example, a CPU (Central Processing Unit).

[0030] The interface 13 is a communication circuit for communicatively connecting to the X-ray inspection apparatus 2. The processor 11 acquires the above-mentioned transmission image from the X-ray inspection apparatus 2 connected via the interface 13.

[0031] The display unit 15 is a device that displays images under the control of the processor 11, and has a display screen such as a liquid crystal display.

[0032] The memory 12 is a storage means for storing an operating system, various programs, data, etc., which are loaded into the processor 11. The memory 12 includes a random access memory (RAM) or a read only memory (ROM). The memory 12 may include a solid state drive, a hard disk drive, etc.

[0033] Moreover, the memory 12 shown in FIG. 3 stores a trained model 121 and a graphic DB 122.

[0034] Fig. 4 is a diagram for explaining a multi-layer perceptron. Fig. 5 is a diagram showing an example of a trained model 121. The trained model 121 stored in the memory 12 is a copy of a parameter group generated by performing deep learning using a learning computer having a CPU, a GPU (Graphics Processing Unit), etc. This deep learning may be performed by the processor 11 of the display device 1, or may be performed by another learning device.

[0035] The learning device obtains training data in which a transmission image obtained by irradiating a container containing a previously prepared object with X-rays corresponds to the type of the object / container as a correct answer label. Next, the learning device inputs the obtained training data to, for example, the input layer (left end) of a multilayer perceptron (MLP) as shown in FIG. 4. Then, the learning device obtains a result from the output layer (right end) of the multilayer perceptron as to what type of object / container is placed in the transmission image, and in what attitude and position.

[0036] Based on the difference between this result and the correct label described above, the learning device performs processing such as the so-called backpropagation method, thereby updating the weight coefficients of each node of the multilayer perceptron and generating a learned model.

[0037] 5, in the trained model 121, for example, weight coefficients of the nodes are described according to a combination of a layer number and a node number indicating the node of the above-mentioned multilayer perceptron. By using the trained model 121, the processor 11 of the display device 1 can analyze the acquired unknown transparent image and identify the container J shown in the transparent image and the contents contained therein.

[0038] 6 is a diagram showing an example of the graphic DB 122. The graphic DB 122 is a database that stores graphic data indicating the shape of an object (i.e., container J and contained items) by type. The graphic data is digital data indicating the shape of an object, and is, for example, data representing an illustration or the like of a simplified drawing of the object. The graphic data may be raster image data such as a bitmap, or vector image data such as SVG (Scalable Vector Graphics).

[0039] The graphic DB 122 shown in Fig. 6 stores multiple records that associate items such as type ID, type name, hazard level, and graphic data. The type ID is identification information that identifies the type of object. The type name is a character string or the like indicating the name of the type of object. The hazard level is a numerical value indicating the possibility that the object is a hazardous object, for example a real number from 0 to 1. The graphic data is image data equivalent to an illustration or the like in which an illustrator or the like has previously drawn the shape of an object of the corresponding type.

[0040] <Functional configuration of the display device> Fig. 7 is a diagram showing an example of a functional configuration of the display device 1. The processor 11 of the display device 1 shown in Fig. 7 executes a program read from the memory 12 to function as a transmission image acquisition unit 111, a contained object detection unit 112, a type identification unit 113, a posture identification unit 114, a graphic acquisition unit 115, a risk determination unit 116, a composite image generation unit 117, and a display control unit 118.

[0041] The transmission image acquisition unit 111 acquires a transmission image from the X-ray inspection apparatus 2 via the interface 13 .

[0042] The contained item detection unit 112 uses the trained model 121 to detect the container J and the contained items contained in this container J from the transmission image acquired by the transmission image acquisition unit 111.

[0043] The type identification unit 113 identifies the type of the contained item detected by the contained item detection unit 112. Therefore, this display device 1 is an example of a display device that detects the contained items by type from a transmitted image of the inside of the container.

[0044] The orientation identifying unit 114 identifies the orientation of the contained item in the container J detected by the contained item detecting unit 112 .

[0045] The graphic acquisition unit 115 acquires graphic data of the type identified by the type identification unit 113 from the graphic DB 122 .

[0046] The danger level determination unit 116 refers to the graphic DB 122 to determine the danger level of the contained item detected by the contained item detection unit 112 .

[0047] The composite image generating unit 117 uses the graphic data acquired by the graphic acquiring unit 115 to generate a composite image by superimposing a graphic representing the shape of the contents on the transparent image acquired by the transparent image acquiring unit 111 .

[0048] In addition, the composite image generation unit 117 performs conversions such as translation, rotation, inversion, enlargement, and reduction on the figure indicating the shape of the container according to the posture of the container specified by the posture identification unit 114, and then superimposes the converted figure on the transmission image. Therefore, this display device 1 is an example of a display device that arranges the figure representing the shape of the detected container corresponding to each detected container according to the position where the container is detected and superimposes the figure on the transmission image.

[0049] Note that the composite image generation unit 117 may change the display mode of the figure indicating the shape of the container according to the determination result of the danger level of the container by the danger level determination unit 116. In addition to superimposing the "figure indicating the shape of the container" on which a predetermined conversion has been performed according to the posture on the transmission image as described above, the composite image generation unit 117 may display the figure at the position of the transmission image. In this case, the composite image generation unit 117 does not necessarily need to superimpose this "figure indicating the shape of the container" on the transmission image. For example, the composite image generation unit 117 may generate a composite image in which the "figure indicating the shape of the container" is arranged at a position corresponding to the position of the transmission image of the container described above. Also, although the figure indicating the shape of the container has been subjected to a predetermined conversion according to the posture in the transmission image of the container, the composite image generation unit 117 does not necessarily need to perform the above conversion. In this case, the processor 11 does not necessarily need to function as the posture identification unit 114.

[0050] The display control unit 118 causes the display unit 15 to display the composite image generated by the composite image generation unit 117. Therefore, this display device 1 is an example of a display device that superimposes a figure representing the shape of the container on the transmission image or displays the figure at the position of the transmission image.

[0051] Also, when changing the display mode of the figure according to the determination result by the danger level determination unit 116, this display device 1 determines whether the detected container is a dangerous object, and displays the figure representing the shape of the container in a mode corresponding to the determination result.

[0052] <Operation of the X-ray inspection device> 8 is a flow diagram showing an example of the flow of operations of the X-ray inspection apparatus 2. When a power switch (not shown) or the like is pressed, the control board 21 of the X-ray inspection apparatus 2 operates the front-stage conveyor 27 to start transporting an object placed thereon (step S101).

[0053] The control board 21 receives a detection signal from the photosensor 24 and determines whether or not an object has been detected (step S102). While it is determined that an object has not been detected (step S102; NO), the control board 21 continues this determination process.

[0054] When it is determined that an object has been detected (step S102; YES), the control board 21 causes the X-ray generator 25 to generate X-rays and irradiate them toward the detected object (step S103).

[0055] The control board 21 measures the time from the start of X-ray irradiation by referring to the built-in clock, and judges whether or not a predetermined specified time has elapsed (step S104). If it is judged that the specified time has not elapsed (step S104; NO), the control board 21 returns to step S103 and repeats the process.

[0056] When it is determined that the specified time has elapsed (step S104; YES), the control board 21 instructs the X-ray generator 25 to stop emitting X-rays (step S105). The control board 21 also instructs the rear-stage conveyor controller 280 to start transport on the rear-stage conveyor 28 (step S106).

[0057] Then, the control board 21 supplies the transmission image generated by the X-ray sensor 26 to the display device 1 via the interface 23 (step S107).

[0058] The control board 21 judges whether or not there is an instruction from the display device 1 to stop the device due to the discovery of a dangerous object (step S108). If it is judged that there is an instruction to stop (step S108; YES), the control board 21 stops the transport of the front-stage conveyor 27 and controls a warning device (not shown) to issue a warning to the user (step S109). On the other hand, if it is judged that there is no instruction to stop (step S108; NO), the control board 21 ends the process without performing the process of step S109.

[0059] <Display device operation> 9 is a flow diagram showing an example of the operation flow of the display device 1. The processor 11 of the display device 1 acquires a transmission image from the X-ray inspection device 2 (step S201), refers to the trained model 121 stored in the memory 12 (step S202), and detects the contents from the acquired transmission image (step S203).

[0060] Processor 11 identifies the type and orientation of the detected contents (step S204), extracts graphic data associated with this type from graphic DB 122, and superimposes and synthesizes the data on the transparent image to generate a composite image (step S205). When the composite image is generated, processor 11 causes display unit 15 to display the composite image generated in step S205 (step S206).

[0061] FIG. 10 is a diagram showing an example of a composite image to be displayed. The display device 1 acquires a transmission image shown in FIG. 10(a) from the X-ray inspection device 2. The display device 1 detects objects (containers and contents) from this transmission image using the trained model 121. Then, graphic data showing the shapes of each of the detected objects is read from the graphic DB 122, and the graphics shown by the graphic data are superimposed on this transmission image. The graphics showing the objects are subjected to processing such as affine transformation according to the positions and orientations at which the objects are detected. This processing generates a composite image shown in FIG. 10(b). In this composite image shown in FIG. 10(b), illustrations showing the shapes of a backpack B0, a game machine B1, a smartphone B2, a bottle B3, and a knife B4 are arranged according to the detected positions and orientations.

[0062] Next, processor 11 determines whether or not the contained item displayed in the composite image is a dangerous item (step S207).

[0063] When the processor 11 determines that the displayed contents are dangerous goods (step S207; YES), the processor 11 issues a transport stop command to the X-ray inspection device 2 (step S208). On the other hand, when the processor 11 determines that the above-mentioned contents are not dangerous goods (step S207; NO), the processor 11 ends the process without issuing the command of step S208.

[0064] As described above, this display system 9 detects the contents stored in the container by using a machine learning model or the like, and displays an illustration or the like associated with the type of the contents by superimposing it on the transparent image. Therefore, a user such as an attendant who sees this display can see the shape of the contents contained in the transparent image in the illustration or the like, and can understand what has been detected as the contents without receiving special training to interpret the transparent image.

[0065] The configurations, shapes, sizes, and layouts described in the above embodiments are merely schematic to the extent that the present invention can be understood and practiced. Therefore, the present invention is not limited to the described embodiments, and may be modified in various forms without departing from the scope of the technical ideas set forth in the claims.

[0066] <Modification> The above is a description of the embodiment, but the contents of this embodiment may be modified as follows. In addition, the following modifications may be combined.

[0067] <1> In the above-described embodiment, the display device 1 acquires graphic data associated with the detected contents from the graphic DB 122, converts the graphic represented by the graphic data according to the posture of the contents, and then superimposes the graphic data on the transparent image. However, in the graphic DB 122, the contents may be associated with different graphic data for each posture.

[0068] 11 is a diagram showing an example of a graphic DB 122 in a modified example. This graphic DB 122 stores blades classified by their posture. That is, this graphic DB 122 distinguishes between blades viewed from at least three directions, namely, sideways, the blade tip side, and the handle side, and stores graphic data showing the shape of each blade.

[0069] When the processor 11 of the display device 1 detects the contained item from the transparent image, it may refer to the graphic DB 122 to identify graphic data showing the shape of the detected contained item. In this case, the processor 11 detects the contained items by distinguishing their postures and superimposes the corresponding graphics on the transparent image or displays them at the position of the transparent image. Therefore, even if the same dangerous item looks completely different depending on the viewing direction, the trained model 121 does not need to store the "apparent shapes" in association with a common dangerous item. In other words, the display device 1 in this modified example is an example of a display device that displays graphics corresponding to each posture of the contained item.

[0070] <2> In the above-described embodiment, the processor 11 executes step S207 after step S206. However, the processor 11 may execute step S206 after step S207, and change the display mode of step S206 for each stored item depending on the result of the risk level determination in step S207. For example, the handle of the knife B4 is shown hidden behind the smartphone B2 in FIG. 10(a), but the display device 1 may display it in front as shown in FIG. 10(b) when generating a composite image.

[0071] Furthermore, the display device 1 may generate a composite image by superimposing only the contained items whose danger level is equal to or higher than the threshold on the transparent image. In this case, the contained items whose danger level is less than the threshold are not displayed, so that the user who sees the display of the display device 1 can consciously grasp only the dangerous items contained in the container.

[0072] <3> In the above embodiment, when the display device 1 detects a contained item, it obtains graphic data showing the shape of the contained item from the graphic DB 122. However, graphic data may be generated each time a contained item is detected. For example, the graphic DB 122 may store three-dimensional data such as polygon data showing the shape of the contained item in a three-dimensional space for each contained item. In this case, when the display device 1 detects a contained item from a transparent image, it may identify the posture of the contained item with respect to the plane of the transparent image, and generate a figure seen from an angle corresponding to the posture from the above-mentioned three-dimensional data. That is, the display device 1 in this modification is an example of a display device that generates and displays a figure corresponding to the contained item from a transparent image. According to the display device 1 in this modification, even if the contained item looks different depending on the viewing direction (viewpoint), a figure is generated for each viewpoint, so that the difference between the contained item shown in the transparent image and the figure superimposed in the composite image is suppressed compared to a case without this configuration. <4> In the above-described embodiment, the display device 1 and the X-ray inspection device 2 each use a CPU as a processor, but other processors may be used. The other processor may be a programmable logic device, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), or may include these. [Explanation of symbols]

[0073] 1...display device, 11...processor, 111...transmitted image acquisition unit, 112...content detection unit, 113...type identification unit, 114...posture identification unit, 115...graphic acquisition unit, 116...risk determination unit, 117...synthetic image generation unit, 118...display control unit, 12...memory, 121...model, 122...graphic DB, 13...interface, 15...display unit, 2...X-ray inspection device, 21...control board, 23...interface, 24...photo sensor, 25...X-ray generator, 26...X-ray sensor, 27...front conveyor, 270...front conveyor controller, 28...rear conveyor, 280...rear conveyor controller, 9...display system, B0...backpack, B1...game console, B2...smartphone, B3...bottle, B4...knife.

Claims

1. A display device which detects the contents of a container by type from a transmitted image of the inside of the container, and which arranges a figure representing the shape of each detected item according to the position at which the item was detected, and superimposes the figure on the transmitted image or displays the figure at the position of the transmitted image.

2. The graphic corresponding to each posture of the contained item is displayed. The display device according to claim 1 .

3. It is determined whether the detected contents are dangerous or not, and the graphic is displayed in a manner according to the result of the determination. The display device according to claim 1 .

4. The figure corresponding to the contained item is generated from the transmitted image and displayed. The display device according to claim 1 .

Images