Information processing device, inspection system, and information processing method

Abstract

This invention provides an information processing device, an inspection system, and an information processing method that enable efficient retrieval of information regarding inspected packages. [Solution] According to the embodiment, the information processing device has a storage unit and a processor. The storage unit stores a database that associates a package image with package data for each package. When a search of the database using a search image is requested, the processor outputs search results that include package data corresponding to a package image similar to the search image selected from the database. When a search of the database using search text data is requested, the processor outputs search results that include package images corresponding to package data similar to the search text data selected from the database.

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an inspection system, and an information processing method.

Background Art

[0002] Conventionally, at inspection sites such as customs, an inspection operation is performed using an inspection system to inspect the contents within inspection targets such as luggage. In the inspection system, based on an image (luggage image) taken by irradiating electromagnetic waves such as X-rays onto the luggage to be inspected and the content of a notice (application form) in which the applicant indicates the destination and contents in advance, the inspection of the luggage is carried out. In recent years, a notice (application form) indicating the destination and contents of the luggage to be inspected at customs or the like is input into the inspection system as luggage data (customs electronic data, EAD (Electronic Advance Data)) including digitized text data.

[0003] In the inspection system as described above, it is possible to save the inspection results of each piece of luggage in a database together with the luggage image and EAD. If there is a database that stores the inspection results together with the luggage image and EAD, there is a demand to search for the inspection results of past similar pieces of luggage based on the photographed image or EAD of the specified piece of luggage. However, in the conventional inspection system, there is a problem that it is possible to search for a piece of luggage in which the words in the EAD, which is text data, exactly match, but it is not possible to search for an EAD expressed in another word with a similar content.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem that this invention aims to solve is to provide an information processing device, an inspection system, and an information processing method that can efficiently retrieve information about inspected packages. [Means for solving the problem]

[0006] According to the embodiment, the information processing device includes a storage unit and a processor. The storage unit stores a database that associates a package image with package data for each package. When a search of the database using a search image is requested, the processor outputs search results that include package data corresponding to package images similar to the search image selected from the database. When a search of the database using search text data is requested, the processor outputs search results that include package images corresponding to package data similar to the search text data selected from the database. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the overall configuration of an inspection system including a data server as an information processing device according to the embodiment. [Figure 2] Figure 2 is a schematic diagram showing an example of the configuration of a system to be placed in an inspection facility, which includes an inspection device connected to a data server as an information processing device according to the embodiment. [Figure 3] Figure 3 is a block diagram showing an example configuration of an inspection device connected to a data server, which is an information processing device according to the embodiment. [Figure 4] Figure 4 is a block diagram showing an example configuration of a data server as an information processing device according to the embodiment. [Figure 5] Figure 5 is a flowchart illustrating the registration process of captured images of luggage and EAD feature quantities by a data server, which is an information processing device according to the embodiment. [Figure 6] Figure 6 schematically shows the process by which a data server, acting as an information processing device according to this embodiment, generates captured images of luggage and feature quantities of the EAD. [Figure 7]Figure 7 is a flowchart illustrating the search process by which a data server, as an information processing device according to this embodiment, searches a database based on the captured image. [Figure 8] Figure 8 is a schematic diagram illustrating the process by which a data server, acting as an information processing device according to this embodiment, searches for an EAD corresponding to a captured image similar to the image being searched. [Figure 9] Figure 9 shows an example of how search results are displayed when a data server, acting as an information processing device according to the embodiment, searches a database based on the captured image. [Figure 10] Figure 10 is a flowchart illustrating the search process by which a data server, as an information processing device according to the embodiment, searches a database based on the search EAD. [Figure 11] Figure 11 is a schematic diagram illustrating the process by which a data server, as an information processing device according to the embodiment, searches for captured images corresponding to EADs similar to the searched EAD. [Figure 12] Figure 12 shows an example of how search results are displayed when a data server, acting as an information processing device according to the embodiment, searches a database based on the search EAD. [Figure 13] Figure 13 is a schematic diagram illustrating an example of how a data server, acting as an information processing device according to the embodiment, calculates features that are similar in a pair of captured images and an EAD (Exploratory Image Data). [Figure 14] Figure 14 schematically shows an example in which a data server, as an information processing device according to the embodiment, searches for an EAD from a pair of captured images when the feature quantities of the EAD are similar. [Figure 15] Figure 15 schematically shows an example in which a data server, as an information processing device according to the embodiment, searches for a captured image from a search EAD when the feature quantities of the pair of captured images and the EAD are similar. [Modes for carrying out the invention]

[0008] The embodiments will be described below with reference to the drawings. First, the configuration of the inspection system 1 according to this embodiment will be described. Figure 1 is a diagram illustrating an example of the overall configuration of the inspection system 1 according to this embodiment. Inspection System 1 is a system for inspecting packages as objects of inspection. For example, Inspection System 1 is a system installed at customs offices, etc., that inspects packages (cargo) including goods for export and import. Packages subject to inspection may be cargo such as imported or exported goods, delivery items including international mail, or hand luggage. Packages subject to inspection by Inspection System 1 according to this embodiment only need to have package data (e.g., Electronic Access Data (EAD)) that shows information about the package declared by the declarant.

[0009] In the following embodiments, the package data for the package to be inspected is described as electronic customs data (EAD) indicating the contents and destination of the package. The EAD is assigned identification information corresponding to the identification information attached (written) to the package and includes information indicating the contents of the package. For example, the EAD includes information such as the package identification information, recipient information (recipient address, name), sender information (sender address, name), the name of each item (contents), the unit price of each item, the country of origin of each item, the weight of each item, item identification information for each item (e.g., HS code), the number of each item, the type of each item, the total value of the items, and the total weight.

[0010] In the configuration example shown in Figure 1, the inspection system 1 according to this embodiment includes an inspection device 11, an imaging device 12, a display device 13, an operating device 14, a data server (information processing device) 16, and a search device 17, etc. An external system 18 is also connected to the data server 16 of the inspection system 1 in a communication manner.

[0011] In the inspection area where an inspector inspects actual luggage, an inspection device 11, a photographing device 12, a display device 13, an operating device 14, etc. are installed. The inspection device 11 acquires information such as a luggage image which is a photographed image (X-ray image) of the luggage to be inspected by the photographing device 12 and EAD which is luggage data indicating the contents of the luggage. The inspection device 11 provides information regarding the luggage to be inspected to the inspector who conducts the inspection of the luggage.

[0012] The inspector inspects whether the luggage to be inspected contains objects to be detected by referring to the information regarding the luggage presented in the inspection area. As objects to be detected, for example, dangerous goods, substances whose handling is prohibited, substances whose import or export into a predetermined area (for example, within a country) is prohibited, goods with inappropriate tax amounts, etc. are assumed. Note that taxable goods are goods for which taxes (for example, customs duties) of an amount determined by a tax rate set by a predetermined tax rule are collected. The tax amount for taxable goods is indicated to the inspector by, for example, a table of tax amounts.

[0013] The inspection device 11 presents information regarding the luggage to be inspected to the inspector by displaying on the display device 13 information including the photographed image of the luggage by the photographing device 12 and the EAD of the luggage, etc. The inspector conducts an inspection of the luggage by referring to the information such as the photographed image of the luggage and the EAD displayed on the display device 13 by the inspection device 11. For example, the inspector determines whether to conduct an opening inspection of opening the luggage and inspecting the contents by referring to the information displayed on the display device 13. The inspector conducts inspections such as whether there are objects to be detected in the luggage and whether the tax amount for the goods (taxable goods) in the luggage is appropriate for the luggage determined to be subject to an opening inspection. <N

[0014] In addition, the inspection device 11 supplies information such as the photographed image of the luggage taken by the photographing device 12 and the inspection result of the luggage by the inspector to the data server 16. For example, the inspection device 11 supplies the photographed image of the luggage taken by the photographing device 12 to the data server 16 in association with the identification information of the luggage. Further, the inspector inputs the inspection result for the luggage by operating the operation device 14 or an information input device (not shown). The inspection device 11 supplies the inspection result input by the inspector to the data server 16 in association with the identification information of the luggage.

[0015] The data server 16 is an example of an information processing device. The data server 16 is composed of a computer such as one or more server devices. The data server 16 is communicatively connected to the inspection device 11, the search device 17, and the external system 18. The data server 16 has a database 16a that stores various types of information related to the luggage.

[0016] In the configuration example shown in FIG. 1, the data server 16 acquires an EAD as declaration data indicating the destination and contents of the luggage from the external system 18. The data server 16 stores the EAD acquired from the external system 18 in the database 16a and outputs it in response to a request from the inspection device 11. Further, the data server 16 acquires information such as the photographed image of the luggage and the inspection result of the luggage from the inspection device 11. The data server 16 stores the information such as the photographed image of the luggage and the inspection result of the luggage acquired from the inspection device 11 in the database 16a in association with the EAD of the luggage.

[0017] Furthermore, the data server 16 calculates the feature quantities of the captured images and EADs of the packages to be stored in the database 16a, and stores the calculated feature quantities of the captured images and EADs in the database 16a. The data server 16 has a search function that searches the database 16a based on the feature quantities of the captured images or EADs of the packages. In the configuration example shown in Figure 1, when a captured image or EAD to be queried is specified by the inspection device 11 or the search device 17, the data server 16 searches the database 16a for captured images or EADs of previously inspected packages that are similar to the captured image or EAD to be queried.

[0018] In this embodiment, the features include those that can be extracted by methods known at the time of filing. For example, the difference in brightness in an image can be considered as a set and extracted as a feature. Alternatively, the brightness within the grayscale of an image can be considered, and the gradient intensity can be normalized and combined to extract a feature. Furthermore, a method of extracting features using a trained model, which has been trained to adjust the parameters for feature extraction by repeatedly training, including deep learning, is also acceptable.

[0019] For example, the data server 16 extracts images of previously inspected packages from the database 16a that are similar to the images of packages specified by the inspection device 11 or the search device 17, and performs a search process to output the extracted images of previously inspected packages or the EADs corresponding to those images. The data server 16 also extracts EADs of previously inspected packages from the database 16a that are similar to the EADs of packages specified by the inspection device 11 or the search device 17, and performs a search process to output the extracted EADs of previously inspected packages or the images of packages corresponding to those EADs.

[0020] External system 18 is a system that manages customs electronic data (EAD), which is declaration data indicating the contents of declarations (e.g., notices) regarding goods subject to inspection. For example, external system 18 accepts goods such as delivery items and obtains EADs for accepted goods from the declarant. External system 18 stores and manages the EADs obtained from the declarant and supplies the EADs to the data server 16 of inspection system 1, which performs the inspection of the goods.

[0021] Figure 2 shows an example of the configuration of a system installed in an inspection area where the inspection system 1 according to the embodiment inspects the packages to be inspected. As shown in Figure 2, the inspection device 11, conveyor C, camera 12, display device 13, and operating device 14 are installed in an inspection area where the necessity of opening and inspecting packages is determined. The inspection device 11 is connected to the camera 12, display device 13, and operating device 14 via an interface. The inspection device 11 is also connected to the data server 16 shown in Figure 1. The inspection system 1 may also have an information input device (not shown) for inspectors to input the inspection results obtained from opening and inspecting packages.

[0022] In the configuration example shown in Figure 2, the conveyor C is a device that transports the packages M to be inspected. The conveyor C transports the packages M to be inspected to the image capture position (reading position) of the imaging device 12. For example, the conveyor C transports packages M supplied by an inspector. Alternatively, the conveyor C may be configured to transport packages M supplied by a robotic arm or the like.

[0023] The imaging device 12 photographs the package M to be inspected by irradiating it with electromagnetic waves as it is transported on the conveyor belt C. The imaging device 12 only needs to acquire an image in which the items inside the package M to be inspected can be visually identified. The imaging device 12 may acquire two-dimensional image data or three-dimensional image data as the image. Furthermore, the imaging device 12 shall acquire image data that includes an image (captured image) of the package M to be inspected irradiated with electromagnetic waves, and physical property information indicating the physical properties of each part of the captured image (e.g., pixels or voxels). The imaging device 12 supplies the image data of the package M to the inspection device 11. A pixel is a pixel that makes up two-dimensional image data. A voxel is the smallest unit of data that makes up three-dimensional data and represents a value in a normal grid unit.

[0024] The imaging device 12 is, for example, an X-ray CT scanner (CT scanner). An example of an imaging device 12, an X-ray CT scanner, acquires three-dimensional X-ray image data as an image by irradiating X-rays around the cargo M being transported by the conveyor C. The X-ray CT scanner, as imaging device 12, also acquires image data that includes a three-dimensional X-ray image (image) of the cargo M taken using X-rays, and physical property information indicating the physical properties of each constituent unit (voxel) that makes up the X-ray image. The X-ray CT scanner, as imaging device 12, supplies the image data acquired from the cargo M to the inspection device 11. However, the imaging device 12 is not limited to an X-ray CT scanner.

[0025] The inspection device 11 consists of a computer and other components equipped with interfaces for connecting to each device. The inspection device 11 has functions such as acquiring information about luggage M from the data server 16, acquiring image data of luggage M from the camera 12, displaying information about luggage M and image data on the display device 13, acquiring information such as inspection results entered by the inspector using the operating device 14, and outputting inspection results entered by the inspector.

[0026] The inspection device 11 displays information about the package to be inspected on the display device 13 and accepts input of inspection results from the inspector via the operating device 14. For example, the inspection device 11 identifies the package M to be inspected being transported by the conveyor C and obtains information such as customs electronic data and screening results as information about the identified package M from the data server 16. The inspection device 11 also obtains data (image data) as a result of photographing the package M by the photographing device 12 from the photographing device 12.

[0027] The inspection device 11 displays information about the package M obtained from the data server 16, as well as images of the package M taken by the camera 12, on the display device 13, and accepts input of the inspection results for the package from the inspector to the operation device 14. The inspector performs an inspection of the package M, including determining whether or not an open inspection is necessary, using the information displayed on the display device 13 as a reference, and inputs the inspection results into the inspection device 11 using the operation device 14.

[0028] The inspection device 11 acquires the inspection results for the package M entered by the inspector using the operating device 14. In addition, if an open inspection is performed on the package M, the inspection device 11 acquires the inspection results of the open inspection for the package M entered using the operating device 14 or an information input device (not shown) installed at the location where the open inspection is performed.

[0029] When the inspection device 11 obtains the inspection result for the package M, it transmits the obtained inspection result to the data server 16. For example, the inspection device 11 adds information such as the identification information of the package M to the inspection result of the package M and supplies it to the data server 16. As a result, the data server 16 stores the inspection result in association with the captured image and EAD of the package. The inspection result of the package M may also include the identification information of the inspector who performed the inspection of the package M.

[0030] The display device 13's display content is controlled by the inspection device 11. The display device 13 displays information about the package M to be inspected. For example, the display device 13 may not only display the captured image of the package M to be inspected and the EAD of package M, but may also display a captured image of a previously inspected package similar to the captured image of the package M, or the EAD corresponding to that previously inspected package (EAD of the previously inspected package). In addition, the display device 13 may not only display the captured image of the package M to be inspected and the EAD of package M, but may also display the EAD of a previously inspected package similar to the EAD of the package M, or a captured image corresponding to that previously inspected package (previously inspected captured image).

[0031] The operating device 14 generates an operation signal in response to the operator's input and supplies the operation signal to the inspection device 11. The operating device 14 consists of an operation device such as a keyboard and a pointing device. Alternatively, the operating device 14 may be configured as a touch panel or the like on the display screen of the display device 13.

[0032] Next, the configuration of the control system in the inspection device 11 of the inspection system 1 according to this embodiment will be described. Figure 3 is a block diagram showing an example of the configuration of the control system in the inspection device 11 of the inspection system 1 according to the embodiment. As shown in Figure 3, the inspection device 11 includes a processor (second processor) 31, ROM 32, RAM 33, storage unit 34, communication unit 35, display interface (I / F) 36, operation interface (I / F) 37, and image interface (I / F) 39.

[0033] The processor 31 performs arithmetic processing. The processor 31 is, for example, a CPU (Central Processing Unit). The processor 31 functions as a processing unit that performs various operations by executing programs stored in the ROM 32 or storage unit 34 using the RAM 33.

[0034] ROM32 is a read-only, non-volatile memory. ROM32 stores program data and control data, among other things. RAM33 is a volatile memory that functions as working memory. RAM33 temporarily stores data.

[0035] The memory unit 34 is a rewritable non-volatile memory. The memory unit 34 is composed of a hard disk drive (HDD), a solid-state drive (SSD), etc. The memory unit 34 stores information such as program data, setting values ​​as control data, and the results of inspection processing. For example, the memory unit 34 stores image data of the luggage to be inspected acquired from the imaging device 12 and information about the luggage acquired from the data server 16.

[0036] The communication unit 35 is a communication interface for communicating with the data server 16. The processor 31 obtains the EAD (Expected Address) of packages and other information from the data server 16 by communicating with the data server 16 via the communication unit 35. The processor 31 also transmits data such as the inspection results of packages to the data server 16 via the communication unit 35.

[0037] The display interface 36 is an interface for connecting to the display device 13. The display interface 36 only needs to be compatible with the interfaces provided by the display device 13. The processor 31 controls the display content to be displayed on the display device 13 via the display interface 36.

[0038] The operation interface 37 is an interface for connecting to the operating device 14. The operation interface 37 only needs to correspond to the interfaces provided by the operating device 14. The processor 31 acquires information input by the operating device 14 via the operation interface 37.

[0039] The image interface 39 is an interface for connecting to the imaging device 12. The image interface 39 can be any interface that is compatible with the imaging device 12, such as an X-ray CT scanner. The processor 31 acquires image data of the X-ray images (hereinafter referred to as X-ray images) captured by the X-ray CT scanner, which is the imaging device 12, via the image interface 39. The processor 31 may also control the imaging operation of the imaging device 12 on the luggage M via the image interface 39.

[0040] Furthermore, the inspection device 11 may also be equipped with a device for notifying inspectors of warnings or alerts. For example, the inspection device 11 may be equipped with a speaker that outputs sound as a warning or alert, or it may be equipped with a light-emitting device that outputs light as a warning or alert.

[0041] Next, the configuration of the data server 16 as an information processing device in the inspection system 1 according to this embodiment will be described. Figure 4 is a block diagram showing an example configuration of a data server 16 as an information processing device in the inspection system 1 according to this embodiment. The data server 16 is an information processing device that manages information in the entire inspection system 1. The data server 16 is composed of one or more computers. In the configuration example shown in Figure 4, the data server 16 has a processor (first processor) 41, ROM 42, RAM 43, storage unit 44, communication unit 45, and external communication unit 46.

[0042] The processor 41 performs arithmetic processing. The processor 41 is, for example, a CPU. The processor 41 functions as a processing unit that performs various operations by executing programs stored in the ROM 42 or storage unit 44 using the RAM 43.

[0043] ROM42 is a read-only non-volatile memory. ROM42 stores program data and control data, etc. RAM43 is a volatile memory that functions as working memory. RAM43 temporarily stores data.

[0044] The storage unit 44 is a rewritable non-volatile memory. The storage unit 44 is composed of a hard disk drive (HDD), a solid-state drive (SSD), etc. The storage unit 44 stores information such as program data, setting values ​​as control data, and data acquired from the inspection device 11 or an external system 18. The storage unit 44 also has a database 16a. For each package, the database 16a stores the image captured by the imaging device 12, the EAD, the inspection result, the feature quantities of the captured image, and the feature quantities of the EAD.

[0045] The communication unit 45 includes a communication interface for communicating with the inspection device 11. For example, the processor 41 supplies information about the package to the inspection device 11 via the communication unit 45 and obtains the inspection results for the package from the inspection device 11 (including the determination of whether or not an open inspection is necessary). The communication unit 45 also includes a communication interface for communicating with the search device 17. For example, the processor 41 communicates with the search device 17 via the communication unit 45 to search for a photographic image or EAD similar to the photographic image or EAD specified by the search device 17, and supplies the search results to the search device 17.

[0046] The external communication unit 46 includes a communication interface for communicating with the external system 18. The external communication unit 46 is an example of an acquisition unit that acquires the EAD of the package to be inspected by the inspection system 1. For example, the processor 41 acquires the EAD of the package by communicating with the external system 18 via the external communication unit 46.

[0047] Next, we will describe the feature data registration process in which the data server 16, which is an information processing device according to the embodiment, registers the feature data of the captured image of the package (hereinafter referred to as the X-ray image) and the feature data of the EAD into a database. Figure 5 is a flowchart illustrating the feature quantity registration process by the data server 16 as an information processing device according to this embodiment. Figure 6 is a schematic diagram showing the feature quantities (feature vectors) calculated from a pair of X-ray images and EADs. The feature registration process shown in Figure 5, as shown in Figure 6, calculates feature quantities (feature vectors) from the paired X-ray images and EADs (X-ray images and EADs for each package), and for each package, the X-ray image, EAD, the feature quantities of the X-ray image, and the feature quantities of the EAD are linked and registered in the database 16a.

[0048] First, the processor 41 of the data server 16 extracts the X-ray images and EADs (pairs of X-ray images and EADs) of the packages for which the features of the X-ray images and EADs are to be registered. For example, the processor 41 of the data server 16 extracts the X-ray image and EAD pairs of packages for which the features of the X-ray images and EADs have not been registered in the database 16a, and then performs the process of registering the features of those X-ray images and EADs.

[0049] For example, the processor 41 may perform feature registration processing when a pair of X-ray images and EADs is registered in the database 16a, or it may extract pairs of X-ray images and EADs for which features have not yet been registered from the database 16a at a predetermined timing or at the instruction of the administrator and perform feature registration processing.

[0050] This section describes the process of registering feature quantities for each pair of n X-ray images and EADs. The processor 41 registers the feature quantities of the n X-ray images and the EADs in the database 16a by repeating the ST12-14 process shown in Figure 5 n times.

[0051] The processor 41 of the data server 16 first obtains the i(1 to n)th X-ray image (Img_i) and EAD (Ead_i) to be registered as features (step ST11). For example, the processor 41 reads the ith X-ray image (Img_i) and EAD (Ead_i) to be registered as features from the database 16a.

[0052] When the processor 41 acquires an X-ray image (Img_i), it encodes the X-ray image (Img_i) by executing a program (image encoder) for calculating the features of the X-ray image, and calculates the feature vector Fimg_i of the X-ray image (step ST12).

[0053] The Image Encoder executed by the processor 41 encodes X-ray images so that the feature quantities (feature vectors) of similar X-ray images are close in value. The feature quantities of X-ray images are calculated, for example, by histograms, bag-of-feature analysis of local features, or by methods using deep learning. An Image Encoder, as a program that applies these methods for calculating image feature quantities (feature vectors), is installed in the memory unit 44 and executed by the processor 41.

[0054] Furthermore, when the processor 41 extracts EAD(Ead_i), it encodes EAD(Ead_i) by executing a program (text encoder, EAD encoder, EAD Encoder) to calculate the features of EAD(Ead_i), and calculates the feature vector Fead_i of EAD(Ead_i) (step ST13).

[0055] The EAD Encoder executed by processor 41 calculates EAD features (feature vectors) composed of text data using machine learning. The EAD Encoder executed by processor 41 only needs to calculate EAD features in such a way that the distance between EAD features containing similar words, even if they are not exact matches, is small. An EAD Encoder program that applies this machine learning method for calculating EAD features is installed in the memory unit 44 and executed by processor 41.

[0056] For example, the EAD Encoder executed by processor 41 could employ a method that uses Large Language Models (LLMs) to calculate features (feature vectors). By using feature calculation with large language models, it is possible to encode similar words so that their features are similar, and features can be calculated that allow EADs containing similar words, even if they are not exact matches, to be searched as similar EADs.

[0057] Furthermore, EAD is data that contains text data for multiple items. For this reason, the text data for each item included in the EAD may be converted into features using the text encoder described above, and these sets of features may be used as the features of the EAD. Alternatively, the features of the EAD may include features for combinations of the text data for each item included in the EAD.

[0058] The processor 41 calculates the feature vector Fimg_i of the X-ray image (Img_i) and the feature vector Fead_i of the EAD (Ead_i), and stores the pair of feature quantities (Fimg_i, Fead_i) in the database 16a, associating them with the package (step ST14). As a result, the database 16a stores information for each package, including the X-ray image, EAD, features of the X-ray image, and features of the EAD.

[0059] In step ST15, if there are any pairs for which features have not been registered (i > n), the processor 41 increments i (i = i + 1) and returns to step ST11. If there are no pairs for which features have not been calculated (i = n), the feature generation process is terminated.

[0060] As shown in Figure 6, the feature data registration process described above allows for the calculation of feature data for each pair of X-ray images (package images) and EADs (package data). The feature data of the X-ray image in the Image feature space and the feature data of the EAD in the EAD feature space can then be associated and stored in the database. In other words, using the database with the registered feature data as described above, it becomes possible to identify the EAD corresponding to the feature data of a specific X-ray image, or to identify the X-ray image corresponding to the feature data of a specific EAD.

[0061] Next, we will describe a search process in which the data server 16, as an information processing device according to the embodiment, searches for an EAD corresponding to an X-ray image of an inspected package that is similar to the specified captured image (X-ray image). Figure 7 is a flowchart illustrating a search process (image-based search process) in which the data server 16, as an information processing device according to the embodiment, searches the database for EADs corresponding to X-ray images similar to a specified X-ray image (search image). Figure 8 is a schematic diagram showing a search process that searches for EADs corresponding to X-ray images with similar features to the features of the search image.

[0062] The image-based search process shown in Figure 7, as shown in Figure 8, selects inspected X-ray images with similar feature quantities to the X-ray image used as the search image, and outputs search results that include the EAD corresponding to the selected X-ray image.

[0063] The processor 41 of the data server 16 receives a search request for the database 16a from the inspection device 11 or the search device 17. As one of the functions for searching the database 16a, the processor 41 has a function to perform an image-based search process that searches the database 16a for EADs corresponding to X-ray images similar to a specified image (X-ray image).

[0064] In image-based search processing, the processor 41 receives a search request from the inspection device 11 or the search device 17 that includes a specified X-ray image (search X-ray image) and search conditions. For example, the processor 41 obtains an X-ray image of the package to be inspected from the inspection device 11 as the search X-ray image, and obtains the number of X-ray images to be selected based on the search X-ray image and the number of EADs (search count) as search conditions.

[0065] When the processor 41 obtains the search X-ray image specified in the search request, it calculates the feature quantity (feature vector) Fq of the search X-ray image (step ST21). The processor 41 calculates the feature vector Fq of the search X-ray image by executing an Image Encoder, which is a program that calculates feature vectors from X-ray images. As shown in Figure 8, the search X-ray image is encoded by the Image Encoder into a feature vector Fq that is represented in the Image feature space.

[0066] The processor 41 calculates the feature vector Fq of the searched X-ray image and then calculates the distance (similarity) between the feature vector Fq and the feature vector of the X-ray image of each package (previously inspected package) registered in the database 16a (step ST22). The distance between the feature vector Fq of the searched X-ray image and the feature vector of the X-ray image of each package registered in the database 16a corresponds to the similarity between the searched X-ray image and the X-ray images of previously inspected packages. In other words, the processor 41 selects X-ray images similar to the searched X-ray image in order of proximity to the feature vector of the searched X-ray image in the Image feature space.

[0067] The processor 41 calculates the distance between the feature vector Fq of the search X-ray image and the feature vector of the X-ray image of each package (previously inspected package), and identifies the search results for the search X-ray image based on the calculated distance (step ST23). For example, when the processor 41 searches for n EADs with similar X-ray images, it selects n feature vectors of X-ray images in the Image feature space in order of proximity to the feature vector of the search X-ray image. The processor 41 identifies n EADs as search results that correspond to the feature vectors of the n EADs that correspond to the feature vectors of the n X-ray images selected in order of proximity to the feature vector of the search X-ray image.

[0068] In the example shown in Figure 8, the data server 16 selects the feature vector of the X-ray image that is closest in distance to the feature vector of the searched X-ray image. The data server 16 identifies the feature vector of the EAD that corresponds to the feature vector of the X-ray image that is closest in distance, and uses that EAD as the search result for the database 16a based on the searched X-ray image.

[0069] When the processor 41 of the data server 16 identifies an EAD as a search result from the search X-ray image, it outputs the search results including the identified EAD to the inspection device 11 or search device 17 that requested the search (step ST24). The processor 41 may also output, as a search result, not only the EAD identified from the search X-ray image, but also X-ray images of previously inspected packages that are determined to be similar to the search X-ray image, and the inspection results of those previously inspected packages.

[0070] Furthermore, the processor 41 may output multiple similar X-ray images and EADs of packages as search results. For example, if the search condition specifies searching for n EADs, the processor 41 selects n X-ray images similar to the searched X-ray image (two X-ray images with close feature vector distances) and outputs n EADs corresponding to the selected n X-ray images as inspection results.

[0071] As shown in Figure 8, the search process based on X-ray images as described above selects X-ray images from the database whose feature vectors are close in distance to the feature vector of the searched X-ray image, and outputs the EAD corresponding to the selected X-ray image as the search result. This makes it possible to search for X-ray images of previously inspected packages that are similar to a specific X-ray image, and furthermore, to output the EAD corresponding to the similar X-ray image as the search result.

[0072] Furthermore, the inspection system 1 as a whole can display search results from the data server 16 on a display device when the inspection device 11 or the search device 17 requests an X-ray image search from the data server 16. For example, the inspection device 11 may request an X-ray image search of the package to be inspected from the data server 16 and display the search results from the data server 16 on the display device 13. Alternatively, the search device 17 may request an X-ray image search specified by the operator from the data server 16 and display the search results from the database 16a on a display device (not shown) provided by the search device 17.

[0073] Figure 9 shows an example of a display on a display device that shows X-ray images and EADs of previously inspected packages that are similar to the searched X-ray image retrieved by the data server 16 based on the searched X-ray image. The display screen shown in Figure 9 shows the searched X-ray image and search criteria, and as a search result for database 16a, three X-ray images similar to the searched X-ray image and three EADs paired with the searched X-ray image are displayed.

[0074] Here, as an example of the display processing of search results by the data server 16, we will describe an example of the operation in which the search results are displayed on the display device 13 when the X-ray image of the luggage to be inspected by the inspection device 11 is used as the search X-ray image. While inspecting a package, the inspection device 11 acquires an X-ray image of the package to be inspected from the imaging device 12 via the image interface 39. When the processor 31 of the inspection device 11 acquires an X-ray image of the package to be inspected, it displays the acquired X-ray image and the EAD of the package (for example, the EAD acquired from the data server 16 based on the package identification information) on the display device 13.

[0075] In this case, the processor 31 of the inspection device 11 requests the data server 16 to search the database 16a using the X-ray image of the package to be inspected as the search X-ray image. The processor 31 may also request the data server 16 to perform a search using the search X-ray image in response to instructions from the inspector. In this case, the processor 31 uses the operating device 14 to request the data server 16 to perform a search using the X-ray image specified by the inspector (the X-ray image of the package to be inspected) as the search X-ray image, and the search conditions specified by the inspector.

[0076] Furthermore, the processor 31 may request the data server 16 to perform a search using the X-ray image of the package to be inspected as the search X-ray image, without instructions from the inspector. For example, when the processor 31 acquires an X-ray image of the package to be inspected from the imaging device 12, it may request the data server 16 to perform a search using the X-ray image of the package as the search X-ray image. In this case, the processor 31 should request a search using pre-set search conditions.

[0077] After requesting a search using a search X-ray image, the processor 31 of the inspection device 11 retrieves EADs from the data server 16 as search results, corresponding to past inspected X-ray images similar to the X-ray image of the package to be inspected. Upon obtaining the search results, the processor 31 of the inspection device 11 displays the X-ray image and EAD of the package to be inspected, as well as the EADs corresponding to past inspected X-ray images similar to the search X-ray image, on the display device 13. This allows inspectors to refer not only to the X-ray image and EAD of the package to be inspected, but also to EADs corresponding to past X-ray images similar to the X-ray image of the package to be inspected.

[0078] Furthermore, if the processor 31 obtains X-ray images similar to the searched X-ray image as part of the search results, it also displays X-ray images of previously inspected packages that are similar to the searched X-ray image on the display device 13. This allows inspectors to refer not only to the X-ray image and EAD of the package being inspected, but also to past X-ray images that are similar to the X-ray image of the package being inspected.

[0079] Furthermore, if the processor 31 obtains inspection results for packages with X-ray images similar to the searched X-ray image as part of the search results, it may also display the inspection results for packages with X-ray images similar to the searched X-ray image on the display device 13. Additionally, if the processor 31 obtains inspection results for packages with X-ray images similar to the searched X-ray image as part of the search results, it may also provide guidance or warnings corresponding to the inspection results.

[0080] For example, if the processor 31 finds that an X-ray image of a package similar to the searched X-ray image contains an object to be seized, it may display a warning on the display device 13 along with the X-ray image of that package. The processor 31 may also determine a warning level based on the package inspection results and change the display state (e.g., background color) of the EAD and X-ray image as search results according to the warning level based on the inspection results.

[0081] In this way, by having the inspection device provide guidance based on the inspection results of packages with X-ray images similar to the searched X-ray image, inspectors performing the package inspection can see not only the X-ray image and EAD of the package being inspected, but also the inspection results, guidance, and warnings for packages with X-ray images similar to the X-ray image of the package being inspected.

[0082] Next, we will describe the search process by which the data server 16, as an information processing device according to the embodiment, searches for X-ray images corresponding to EADs similar to the search EAD or keyword (search text data). Figure 10 is a flowchart illustrating the search process (EAD-based search process) in which the data server 16, as an information processing device according to this embodiment, searches for X-ray images corresponding to a search EAD or an EAD similar to a keyword. Figure 11 is a schematic diagram showing the search process for searching for X-ray images corresponding to an EAD with feature quantities similar to the feature quantities of the search EAD. The EAD search process shown in Figure 10, as shown in Figure 11, selects EADs of inspected packages that have similar (closely located) features to the search EAD or keyword features, and outputs the X-ray images corresponding to the selected EADs as search results.

[0083] The processor 41 of the data server 16 has a function to perform a search process to search for X-ray images corresponding to EADs similar to the specified EAD or keyword (text data) in the database 16a, as one of the functions for searching the database 16a.

[0084] The processor 41 receives a search request from the inspection device 11 or the search device 17, specifying a search EAD or keyword and search conditions. For example, the processor 41 obtains the EAD of the package to be inspected from the inspection device 11 as the search EAD, and obtains the number of X-ray images of the inspected package to be searched based on the search EAD as search conditions. The processor 41 also obtains a keyword specified by the inspector from the inspection device 11 as the search keyword, and obtains the number of X-ray images of the inspected package to be searched based on the search keyword as search conditions.

[0085] When the processor 41 obtains a search EAD or keyword (hereinafter referred to as the search EAD), it calculates the feature vector Fq of the search EAD (step ST31). The processor 41 calculates the feature vector Fq of the search EAD by executing a program that acts as a text encoder (EAD encoder, EAD Encoder) that calculates feature vectors from the EAD. As shown in Figure 11, the search EAD is encoded by the EAD Encoder into a feature vector Fq that is represented in the EAD feature space.

[0086] The processor 41 calculates the feature vector Fq of the search EAD, and then calculates the distance (similarity) between the feature vector Fq and the feature vector of the EAD of each package (previously inspected package) registered in the database 16a (step ST32). The distance between the feature vector Fq of the search EAD and the feature vector of the EAD of each package registered in the database 16a corresponds to the similarity between the search EAD and the EAD of the previously inspected package.

[0087] The processor 41 calculates the distance between the feature vector Fq of the search EAD and the feature vector of the EAD of each package (previously inspected package), and then selects X-ray images corresponding to EADs similar to the search EAD based on the calculated distance (step ST23). For example, the processor 41 selects EADs similar to the search EAD in order of proximity to the feature vector of the search EAD in the EAD feature space as shown in Figure 11, and then selects X-ray images corresponding to the selected similar EADs from the database 16a. In the example shown in Figure 11, the data server 16 selects the feature vector of the EAD that is closest in distance to the feature vector of the search EAD. The data server 16 selects the feature vector of the X-ray image corresponding to the feature vector of the EAD that is closest in distance to the feature vector of the search EAD.

[0088] Furthermore, if the search condition specifies one search term, the processor 41 selects one EAD that is closest in distance to the feature vector of the searched EAD, and then selects an X-ray image corresponding to that selected EAD. If the search condition specifies n search terms, the processor 41 selects n EADs in order of proximity to the feature vector of the searched EAD, and then selects n X-ray images corresponding to those selected EADs.

[0089] When the processor 41 of the data server 16 selects an X-ray image to be used as the search result for the search process using the search EAD, it outputs the search results, including the selected X-ray image, to the inspection device 11 or search device 17 that requested the search (step ST34). Here, the processor 41 may output not only the X-ray image selected from the search EAD, but also EADs of previously inspected packages that have been determined to be similar to the search EAD, or inspection results of inspected packages with similar EADs, as search results. For example, if the number of searches specified in the search conditions is n, the processor 41 may output inspection results that include n EADs similar to the search EAD and n EADs corresponding to the n similar EADs.

[0090] As shown in Figure 11, the search process using EADs as described above selects EADs with feature vectors close in distance to the searched EAD from the database, and outputs the X-ray images corresponding to the selected EADs as search results. This makes it possible to search for EADs of previously inspected packages that are similar to a specific EAD, and furthermore, to output X-ray images corresponding to similar EADs as search results.

[0091] Furthermore, if the inspection device 11 or the search device 17 requests a search using EAD from the data server 16, it can display the search results from the data server 16 on the display device. For example, the inspection device 11 may request a search using EAD for the package to be inspected from the data server 16 and display the search results from the data server 16 on the display device 13. Alternatively, the search device 17 may supply a search using EAD specified by the operator to the data server 16 and display the search results from the database 16a on a display device (not shown) provided by the search device 17.

[0092] Figure 12 shows an example of a display showing the EADs and X-ray images of previously inspected packages that are similar to the searched EAD, which were searched by the data server 16 based on the searched EAD. The display screen shown in Figure 12 shows the search EAD and search criteria, and the search results show three EADs similar to the search EAD and three X-ray images paired with similar EADs.

[0093] Here, as an example of the process for displaying search results, we will describe an example of the operation in which the inspection device 11 displays the search results on the display device 13 when the EAD of the package to be inspected is used as the search EAD. While inspecting a package, the inspection device 11 acquires an X-ray image of the package to be inspected from the imaging device 12 via the image interface 39. When the processor 31 of the inspection device 11 acquires an X-ray image of the package to be inspected, it displays the acquired X-ray image and the EAD of the package (for example, the EAD acquired from the data server 16 based on the package identification information) on the display device 13.

[0094] In this case, the processor 31 of the inspection device 11 requests the data server 16 to perform a search using the EAD of the package as the search EAD or using a keyword specified by the inspector as the search keyword. For example, when requesting a search using the search EAD from the data server 16 in accordance with the inspector's instructions, the processor 31 uses the operating device 14 to request the data server 16 to perform a search using the EAD (the EAD of the package) or keyword specified by the inspector as the search EAD or search keyword, and the search conditions specified by the inspector.

[0095] Furthermore, the processor 31 may request a search using the EAD of the package to be inspected without instructions from the inspector. For example, when the identification information of the package to be inspected is identified, the processor 31 may request the data server 16 to perform a search using the EDA of the package as the search EAD. In this case, the processor 31 should request a search using pre-set search conditions.

[0096] After requesting a search using a search EAD or search keyword, the processor 31 of the inspection device 11 retrieves inspection results from the data server 16, including X-ray images corresponding to previously inspected EADs similar to the EAD (or keyword specified by the investigator) of the package to be inspected. Here, the inspection device 11 retrieves not only X-ray images corresponding to similar EADs as search results from the data server 16, but also inspection results for similar EADs and packages with similar EADs.

[0097] When the processor 31 of the inspection device 11 obtains search results, it displays on the display device 13 not only the X-ray image and EAD of the package to be inspected, but also X-ray images corresponding to past inspected EADs similar to the searched EAD (or keyword). This allows the inspector to refer not only to the X-ray image and EAD of the package to be inspected, but also to X-ray images corresponding to past EADs similar to the EAD of the package to be inspected or keywords specified by the inspector.

[0098] Furthermore, if the processor 31 obtains an EAD similar to the search EAD (or keyword) as a search result, it also displays the EADs of previously inspected packages that are similar to the search EAD on the display device 13. This allows inspectors to refer not only to the X-ray image and EAD of the package to be inspected, but also to past EADs similar to the EAD of the package to be inspected or keywords specified by the inspector.

[0099] Furthermore, if the processor 31 obtains inspection results for packages with EADs similar to the search EAD (or keyword) as part of the search results, it may also display the inspection results for packages with EADs similar to the search EAD on the display device 13. In addition, if the processor 31 obtains inspection results for packages with EADs similar to the search EAD (or keyword) as part of the search results, it may also provide guidance or warnings according to the inspection results.

[0100] For example, if the processor 31 finds that an inspection result for a package with an EAD similar to the searched EAD contains the object to be seized, it may display a warning on the display device 13 along with the EAD of that package. Alternatively, the processor 31 may determine a warning level based on the inspection results for packages with EADs similar to the searched EAD, and change the display state (e.g., background color) of the search results, such as the EAD and X-ray images, according to the warning level based on the inspection results.

[0101] In this way, by having the inspection device provide guidance based on the inspection results of packages with X-ray images similar to the searched X-ray image, inspectors performing the package inspection can see not only the X-ray image and EAD of the package being inspected, but also the inspection results, guidance, and warnings for packages with X-ray images similar to the X-ray image of the package being inspected.

[0102] As described above, the data server, as an information processing device according to the embodiment, stores in a database the captured image (X-ray image), which is the image of the package, and the EAD, which is the text data, associated with each package. When a database search using a specified X-ray image is requested, it outputs search results that include the EAD corresponding to an X-ray image similar to the specified X-ray image. When a database search using a specified EAD or keyword is requested, it outputs search results that include the X-ray image corresponding to an EAD similar to the specified EAD or keyword.

[0103] As a result, the data server, which is an information processing device according to the embodiment, can search the database for EADs of packages similar to a specified package in the X-ray image, or search the database for X-ray images of packages similar to a specified package in the EAD.

[0104] Furthermore, the data server, as an information processing device according to the embodiment, outputs search results including EADs corresponding to X-ray images with features similar to those of the specified X-ray image when a database search using a specified X-ray image is requested, and outputs search results including X-ray images with EADs corresponding to those with features similar to those of the specified EAD or keyword when a database search using a specified EAD or keyword is requested.

[0105] As a result, the data server as an information processing device according to the embodiment can search for X-ray images corresponding to EADs with feature quantities similar to the specified EAD or keyword feature quantities, and can search for X-ray images of packages that include not only articles or articles indicated by keywords included in the EAD, but also articles similar to articles or articles indicated by keywords included in the EAD (articles with similar words).

[0106] Furthermore, in the inspection system according to this embodiment, the inspection device requests the data server to perform a database search using X-ray images of the package to be inspected, the data server searches for EADs corresponding to X-ray images similar to the X-ray images of the package to be inspected and supplies the search results to the inspection device, and the inspection device displays the search results from the data server on a display device.

[0107] As a result, according to the inspection system according to the embodiment, the inspection device can display an EAD (Exploratory Image Display) or the like, which corresponds to an X-ray image similar to the X-ray image of the luggage to be inspected, on a display device, and the inspector can perform the inspection while visually checking this information.

[0108] Next, I will explain the first modified example. In the embodiment described above, the data server 16 encodes X-ray images into feature quantities (feature vectors) in the Image feature space using an Image Encoder, and encodes EADs into feature quantities (feature vectors) in the EAD feature space using an EAD Encoder. In contrast, in the first modified example, the luggage image and text data of each luggage are encoded so that they become similar feature vectors in the same feature space.

[0109] The data server according to the first modification uses an image encoder and an EAD encoder as encoders that convert X-ray images and EADs into the same feature space, and uses a large number of pairs of X-ray images and EADs for each package to train the package image and text data for each package to have similar feature vectors in the same feature space.

[0110] Such learning is achieved through machine learning that processes multimodal information of images and language and associates images with text. For example, CLIP (Contrastive Language Image Pretraining) can be used to create an Image Encoder and EAD Encoder that produce similar feature vectors for X-ray images (images of luggage) and EAD text data.

[0111] Figure 13 schematically illustrates an example in which pairs of X-ray images and EADs for each package are transformed into similar features (feature vectors) in the same feature space. As shown in Figure 13, the Image Encoder and EAD Encoder each convert the X-ray image, which is the image of the luggage, and the EAD (text data), which is the luggage data, into similar features. The data server 16 in the first modified example calculates the features of the X-ray image and EAD for each luggage by executing the Image Encoder and EAD Encoder as shown in Figure 13, and registers the calculated features of the X-ray image and EAD, linked to the luggage, in the database 16a.

[0112] As shown in Figure 13, the data server 16 can directly search for EADs (EADs of similar past inspected packages) registered in the database 16a from the searched X-ray image, or directly search for X-ray images (X-ray images of similar past inspected packages) registered in the database 16a from the searched EAD.

[0113] Figure 14 schematically shows an example in which the data server 16 according to the first modified example searches for EADs registered in the database 16a from the searched X-ray images. As shown in Figure 14, when a search X-ray image is provided, the data server 16 calculates the features of the search X-ray image by running an Image ENCODER that has been trained so that the pair of X-ray images and EADs have similar features. The data server 16 identifies similar EADs (EADs of previously inspected packages) by searching the database 16a for features similar to those of the search X-ray image calculated by the Image ENCODER.

[0114] In the first modification, since the features of the X-ray image and the EAD are calculated as feature vectors in the same feature space, the features of the searched X-ray image can be directly compared with the features of the EAD registered in database 16a, and an EAD with features similar to those of the searched X-ray image can be identified.

[0115] Figure 15 schematically shows an example in which the data server 16 according to the first modified example searches for X-ray images registered in the database 16a from the search EAD. As shown in Figure 15, when a search of the database 16a is requested using the search EAD, the data server 16 calculates the features of the search EAD by executing an EAD ENCODER that has been trained to ensure that the pair of X-ray images and EADs have similar features. The data server 16 identifies X-ray images of packages similar to the package in the search EAD (X-ray images of previously inspected packages) by searching the database 16a for X-ray images with similar features to the search EAD calculated by the EAD ENCODER.

[0116] In the first modification, since the features of the X-ray image and the EAD are calculated as feature vectors in the same feature space, the features of the searched EAD can be directly compared with the features of the X-ray images registered in the database 16a, and an X-ray image with features similar to those of the searched EAD can be identified.

[0117] The data server according to the first modification converts the features of the X-ray image and EAD into feature vectors in the same feature space, and calculates the features of the X-ray image and EAD using an encoder that has been trained so that the feature vectors of the corresponding X-ray image and EAD for each package are close together. As a result, the data server according to the first modification makes it possible to directly search for EADs with similar features from a searched X-ray image, or to directly search for X-ray images with similar features from a searched EAD.

[0118] Furthermore, according to the first modification, it is possible to directly search for EADs of similar packages from the searched X-ray image, or to directly search for X-ray images of similar packages from the searched EAD. Therefore, even for packages where either an X-ray image or an EAD is not available in the database, it becomes possible to search for X-ray images or EADs of similar packages.

[0119] Furthermore, the data server 16 according to the first modified example can also check whether the distance between the feature quantities of the X-ray image and the feature quantities of the EAD of a single package is within a predetermined standard value. In other words, the data server 16 according to the first modified example is configured such that the Image ENCODER and EAD ENCODER are made to have similar feature quantities in pairs of X-ray images and EADs (X-ray image and EAD for each package) through machine learning. Using these encoders, the data server 16 according to the first modified example can check whether the X-ray image and EAD are consistent by determining whether the distance between the feature quantities of the X-ray image and the feature quantities of the EAD for a given package is within a predetermined threshold value.

[0120] For example, in the first modified example, the processor 31 of the data server 16, upon acquiring an X-ray image and an EAD for a single package, converts the X-ray image of the package into a feature vector using an Image ENCODER, and converts the EAD of the package into a feature vector using an EAD ENCODER. The processor 31 of the data server determines whether the distance between the feature quantities of the X-ray image and the feature quantities of the EAD of the package is within a predetermined reference value. If the distance between the feature quantities of the X-ray image and the EAD of the package is greater than the predetermined reference value, the processor 31 outputs an alert indicating that there may be an inconsistency between the X-ray image and the EAD of the package.

[0121] As a specific example of operation, when the processor 31 acquires an X-ray image of the package to be inspected from the inspection device 11, it can check whether the distance between the feature quantities of the X-ray image and the feature quantities of the package's EAD is within a predetermined standard value. In this case, if the distance between the feature quantities of the X-ray image of the package being inspected from the inspection device 11 and the feature quantities of the package's EAD is greater than the predetermined standard value, the processor 31 outputs a warning to the inspection device 11. This allows the inspection device 11 to notify an alert in response to a warning from the data server 16 if the X-ray image of the package being inspected and the package's EAD do not match.

[0122] Next, I will explain the second variation. In the embodiment described above, the data server 16 calculates feature quantities from a single X-ray image of the entire package captured by the imaging device 12. In contrast, the data server 16 according to the second modification calculates feature quantities not only from a single X-ray image of the entire package, but also from images of the divided regions (divided images) after dividing the X-ray image into multiple regions.

[0123] The data server 16 according to the second modification stores multiple segmented images, each derived from an X-ray image of a single package, in a database 16a, associating each segmented image with the EAD of the package. In this case, the data server 16 can perform the same processing as in the above-described embodiment, even if the image corresponding to the EAD of a single package is a segmented image derived from an X-ray image.

[0124] As a specific example of operation, the data server 16 according to the second modification may generate segmented images by dividing the X-ray image of a single package into image regions that are presumed to contain items within the package. In this case, when the data server 16 receives a search request specifying an image of a specific object as the search image, it selects segmented images similar to the image of the specific object. This makes it possible for the data server 16 to identify packages whose X-ray images contain segmented images similar to the image of the specific object.

[0125] The configuration of the inspection system according to the embodiment described above is merely an example of an inspection system, and several devices may be implemented as a single device, or one device may be implemented as multiple devices. For example, the inspection device may perform some or all of the processing performed by the data server, which is the information processing device described above.

[0126] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0127] 1...Inspection system, 11...Inspection device, M...Luggage (object to be inspected), C...Conveyor, 12...Photography device, 13...Display device, 14...Operation device, 16...Data server (information processing device), 16a...Database, 17...Search device, 31...Processor (second processor), 32...ROM, 33...RAM, 34...Storage unit, 35...Communication unit, 36...Display interface, 37...Operation interface, 39...Image interface, 41...Processor (first processor), 42...ROM, 43...RAM, 44...Storage unit, 45...Communication unit, 46...External communication unit.

Claims

1. In an information processing device that manages information related to luggage, A storage unit that stores a database that associates each package with a package image and package data, A processor that, when a database search using a search image is requested, outputs search results including luggage data corresponding to luggage images similar to the search image selected from the database, and when a database search using search text data is requested, outputs search results including luggage images corresponding to luggage data similar to the search text data selected from the database. An information processing device having

2. The luggage images stored in the aforementioned database are images captured by a camera during luggage inspection. The package data stored in the aforementioned database is electronic data including text data indicating the contents of the package. The information processing apparatus according to claim 1.

3. The processor calculates the feature quantities of the package image and the feature quantities of the package data for each package, and registers the calculated feature quantities of the package image and the feature quantities of the package data in the database, linking them to each package. When a database search is requested using a search image, luggage data corresponding to luggage images with features similar to those of the search image is selected from the database. When a database search is requested using search text data, luggage images corresponding to luggage data with features similar to those of the search text data are selected from the database. The information processing apparatus according to claim 1.

4. The processor comprises an image encoder that converts luggage images into features in an image feature space, and a text encoder that converts luggage data into features in a text feature space. The information processing apparatus according to claim 3.

5. The text encoder is configured so that the feature quantities of similar words become similar through machine learning. The information processing apparatus according to claim 4.

6. The aforementioned machine learning model is a large-scale language model. The information processing apparatus according to claim 5.

7. The processor registers the feature vectors of the luggage image and luggage data for each luggage, calculated using an encoder configured to learn a large number of luggage image and luggage data combinations and convert the luggage image and luggage data for each luggage into similar feature vectors in the same feature space, into the database. When a database search is requested using a search image, luggage data with feature vectors similar to the feature vectors of the search image is selected from the database; and when a database search is requested using search text data, luggage images with feature vectors similar to the feature vectors of the search text data are selected from the database. The information processing apparatus according to claim 1.

8. The aforementioned database stores multiple segmented images, obtained by dividing the package image for each package, in association with the package data. The processor, when a search of the database using a search image is requested, outputs search results including luggage data corresponding to segmented images similar to the search image selected from the database. The information processing apparatus according to claim 1.

9. In an inspection system including an information processing device and an inspection device, The aforementioned information processing device is A storage unit that stores a database that associates each package with a package image and package data, The system includes a first processor that, when a database search using a search image is requested, outputs search results including luggage data corresponding to luggage images similar to the search image selected from the database, and when a database search using search text data is requested, outputs search results including luggage images corresponding to luggage data similar to the search text data selected from the database. The inspection device, A display interface for connecting a display device, An image interface that acquires images of the luggage to be inspected as captured images taken by a camera, The system includes a second processor that requests the information processing device to search a database using the luggage image acquired by the image interface as the search image, and displays the search results output by the information processing device together with the luggage image to be inspected on a display device. Inspection system.

10. The database of the aforementioned information processing device further stores the inspection results of the packages. When the inspection device requests a database search using a search image, the first processor of the information processing device outputs the inspection results of luggage images similar to the search image, including them in the search results. The second processor of the inspection apparatus further provides guidance corresponding to the inspection results of the packages included in the search results output by the information processing device. The inspection system according to claim 9.

11. An information processing method using an information processing device that manages information related to luggage, A database is stored in the storage unit that associates the image of each package with the package data. When a database search using a search image is requested, the system outputs search results that include luggage data corresponding to luggage images similar to the search image selected from the database. When a database search using search text data is requested, the system outputs search results that include images of luggage corresponding to luggage data similar to the search text data selected from the database. Information processing methods.

12. Furthermore, for each package, we calculate the feature quantities of the package image and the feature quantities of the package data. The calculated features of the package image and the features of the package data are linked for each package and registered in the database. When a database search using a search image is requested, the system selects luggage data from the database that corresponds to luggage images with features similar to those of the search image. When a database search is requested using search text data, the system selects a package image from the database that corresponds to package data with features similar to those of the search text data. The information processing method according to claim 11.

13. By learning from a large number of combinations of luggage images and luggage data, an encoder is configured to convert the luggage image and luggage data for each luggage into similar feature vectors in the same feature space. Using this encoder, the feature vectors for the luggage image and luggage data of each luggage are calculated. The calculated feature vectors of the package image and the feature vectors of the package data are linked to the package and registered in the database. When a database search using a search image is requested, luggage data with feature vectors similar to the feature vectors of the search image is selected from the database. When a database search is requested using search text data, an image of luggage with a feature vector similar to the feature vector of the search text data is selected from the database. The information processing method according to claim 11.

Images