Image retrieval system

Abstract

In some examples, a method for generating an image retrieval system configured to rank a plurality of images of cargo from an image dataset in response to a query corresponding to an image of cargo of interest generated using penetrating radiation is disclosed. The method can involve obtaining a plurality of annotated training images comprising cargo, each of the training images being associated with an annotation indicative of a type of cargo in the training image, and training the image retrieval system by applying a deep learning algorithm to the obtained annotated training images. The training can involve applying a feature extraction convolutional neural network to the annotated training images, and applying an aggregated generalized mean pooling layer associated with image spatial information.

Description

Technical Field

[0001] This invention relates to, but is not limited to, generating an image retrieval system configured to sort multiple images of a cargo from an image dataset in response to a query corresponding to an image of a cargo of interest generated using penetrating radiation. The invention also relates to, but is not limited to, sorting multiple images of a cargo from an image dataset based on an inspection image corresponding to a query. The invention further relates to, but is not limited to, generating an apparatus configured to sort multiple images of a cargo from an image dataset generated using penetrating radiation. The invention also relates to, but is not limited to, corresponding apparatus and computer programs or computer program products. Background Technology

[0002] Penetrating radiation can be used to generate inspection images of containers holding goods. In some examples, a user may want to detect objects on the inspection image that correspond to the goods of interest. Detecting such objects can be difficult. In some cases, objects may not be detected at all. When detection from the inspection image is unclear, the user can manually inspect the container, which can be time-consuming. Summary of the Invention

[0003] Aspects and embodiments of the invention are set forth in the appended claims. These and other aspects and embodiments of the invention are also described herein.

[0004] Any feature in one aspect of the invention may be applied to other aspects of the invention in any suitable combination. In particular, the method aspect may be applied to the apparatus and computer program aspects, and vice versa.

[0005] Furthermore, features implemented in hardware can often be implemented in software, and vice versa. Any references to software and hardware features in this document should be interpreted accordingly. Attached Figure Description

[0006] Embodiments of the invention will now be described by way of example with reference to the accompanying drawings, wherein:

[0007] Figure 1 A flowchart illustrating an example method according to this disclosure is shown;

[0008] Figure 2 This schematically illustrates the configuration for implementation. Figure 1 Example systems and example devices for example methods;

[0009] Figure 3 Exemplary inspection images according to this disclosure are illustrated;

[0010] Figure 4 Explanation is shown Figure 1A flowchart detailing the example methods;

[0011] Figure 5 Explanation is shown Figure 1 A flowchart detailing the example methods;

[0012] Figure 6 This schematically illustrates the configuration to implement, for example Figure 1 An example image retrieval system with example methods;

[0013] Figure 7 Explanation is shown Figure 1 A flowchart detailing the example methods;

[0014] Figure 8 Explanation is shown Figure 1 A flowchart detailing the example methods;

[0015] Figure 9 A flowchart illustrating another example method according to this disclosure is shown; and

[0016] Figure 10 A flowchart illustrating another example method according to this disclosure is shown.

[0017] In the accompanying drawings, similar elements have the same reference numerals.

[0018] Detailed description of exemplary embodiments

[0019] This disclosure discloses an example method for generating an image retrieval system configured to sort multiple images of goods from an image dataset. The sorting is performed in response to a query corresponding to an image of the goods of interest generated using penetrating radiation (e.g., X-rays, but other penetrating radiation is conceivable). As a non-limiting example, the goods of interest can be any type of goods (such as food, industrial products, medicines, or cigarettes).

[0020] This disclosure also discloses an example method for sorting multiple images of goods from an image dataset based on an inspection image corresponding to a query.

[0021] This disclosure also discloses an example method for generating an apparatus configured to sort multiple images of cargo from an image dataset generated using penetrating radiation.

[0022] This disclosure also discloses the corresponding equipment and computer programs or computer program products.

[0023] Image retrieval systems enable operators of inspection systems to benefit from existing image datasets and / or existing textual information (such as expert reports) and / or codes associated with sorted images. Image retrieval systems can enable enhanced inspection of goods of interest.

[0024] Image retrieval systems can enable operators of inspection systems to benefit from the automatic output of textual information (such as cargo description reports, scanning process reports) and / or codes associated with the cargo of interest.

[0025] Figure 1 The illustration shows the method for generating according to this disclosure. Figure 6 The flowchart illustrates an example method 100 of the image retrieval system 1. Figure 2 A device 15, configurable by method 100, is shown in response to an inspection image 1000 of the cargo 11 of interest generated using penetrating radiation. Figure 3 and 6 (As shown in the image) The corresponding query sorts multiple images of the cargo from image dataset 20 generated using penetrating radiation. Image 1000 can be examined, for example, by using penetrating radiation generated by device 15.

[0026] Figure 1 Method 100 generally includes:

[0027] In S1, multiple annotated training images 101, including cargo 110, are obtained. Figure 3 and 6 As shown in the figure, each of the training images 101 is associated with an annotation indicating the type of goods 110 in the training image 101; and

[0028] In S2, the image retrieval system 1 is trained by applying the deep learning algorithm 30 to the obtained annotated training image 101.

[0029] As shown later in the diagram, method 200... Figure 9 In more detail, the configuration of device 15 involves, for example, storing the image retrieval system 1 at device 15 in S32. In some examples, the image retrieval system 1 can be obtained at S31 (e.g., via...). Figure 1 The image retrieval system 1 is generated as in method 100. In some examples, obtaining the image retrieval system 1 in S31 may include receiving the image retrieval system 1 from another data source.

[0030] As described above, the image retrieval system 1 uses a deep learning algorithm to derive from training images 101 and is configured to produce outputs corresponding to goods of interest 11 in the inspection images 1000. In some examples and as described in more detail below, the output may correspond to a sorting of multiple images of goods from image dataset 20. Dataset 20 may include at least one of the following: one or more training images 101 and multiple inspection images 1000.

[0031] In the image retrieval system 1, the memory 151 of device 15 (e.g.) is stored. Figure 2 After that (as shown), the image retrieval system 1 is arranged to more easily generate output, even though the process 100 used to derive the image retrieval system 1 from the training image 101 may be computationally intensive.

[0032] After the device 15 is configured, the device 15 can provide an accurate output corresponding to the goods 11 by applying the image retrieval system 1 to the inspection image 1000. Figure 10 The sorting process (as described later) is shown in process 300.

[0033] Computer systems and testing equipment

[0034] Figure 2 The illustration schematically shows a configuration to at least partially implement Figure 1 Example computer system 10 and device 15 of example method 100. In particular, in a preferred embodiment, computer system 10 executes a deep learning algorithm to generate an image retrieval system 1 to be stored on device 15. Although a single device 15 is shown for clarity, computer system 10 may communicate and interact with multiple such devices. Training images 101 themselves can be obtained using images acquired using device 15 and / or other similar devices and / or other sensors and data sources.

[0035] In some examples, such as Figure 4 As shown, obtaining training images 101 in S1 may include retrieving annotated training images from an existing image database (such as dataset 20 in a non-limiting example) in S11. Alternatively or additionally, obtaining training images 101 in S1 may include generating annotated training images 101 in S12. In some examples, the generation in S12 may include:

[0036] Irradiating one or more containers containing goods with penetrating radiation, and

[0037] Detect radiation from one or more irradiated containers.

[0038] In some instances, one or more devices configured to inspect containers are used to perform irradiation and / or detection.

[0039] In some examples, the training image 101 may be obtained in different environments, for example, using similar devices (or equivalent sets of sensors) installed in different (but preferably similar) environments, or in a controlled test configuration in a laboratory environment.

[0040] Figure 2 The computer system 10 includes a memory 121, a processor 12, and a communication interface 13.

[0041] System 10 can be configured to communicate with one or more devices 15 via interface 13 and link 30 (e.g., Wi-Fi connection, but other types of connections are also conceivable).

[0042] The memory 121 is configured to at least partially store data, for example, for use by the processor 12. In some examples, the data stored on the memory 121 may include dataset 20 and / or data such as training image 101 (and data used to generate training image 101) and / or data for deep learning algorithms.

[0043] In some examples, the processor 12 of system 10 can be configured to at least partially perform Figure 1 Method 100 and / or Figure 9 Method 200 and / or Figure 10 At least some of the steps in method 300.

[0044] Figure 2 The detection device 15 includes a memory 151, a processor 152, and a communication interface 153 that allows connection to the interface 13 via a link 30 (e.g., a Wi-Fi connection, but other types of connections are conceivable).

[0045] In a non-limiting example, device 15 may also include means 3 for use as an inspection system, as described in more detail later. Means 3 may be integrated into device 15 or connected to other parts of device 15 via a wired or wireless connection.

[0046] In some examples, such as Figure 2 As shown, this disclosure can be applied to the inspection of a physical container 4 containing cargo 11 of interest. Alternatively or additionally, at least some of the methods of this disclosure may include obtaining an inspection image 1000 by irradiating one or more physical containers 4 configured to contain cargo with penetrating radiation and detecting the radiation from the irradiated one or more physical containers 4.

[0047] In other words, device 3 can be used to obtain multiple training images 101 and / or obtain inspection images 1000.

[0048] In some examples, the processor 152 of device 15 can be configured to at least partially perform Figure 1 Method 100 and / or Figure 9 Method 200 and / or Figure 10 At least some of the steps in method 300.

[0049] Generative Image Retrieval System

[0050] Return to reference Figure 1 An image retrieval system 1 is constructed by applying a deep learning algorithm to training images 101. Any suitable deep learning algorithm can be used to construct the image retrieval system 1. For example, a method based on convolutional deep learning algorithms can be used.

[0051] An image retrieval system 1 is generated based on the training images 101 obtained in S1.

[0052] The learning process is typically computationally intensive and may involve a large number of training images 101 (such as thousands or tens of thousands of images). In some examples, the processor 12 of system 10 may include greater computing power and storage resources than the processor 152 of device 15. Therefore, the image retrieval system 1 generation is performed at least partially at computer system 10, remotely from device 15. In some examples, at least steps S1 and / or S2 of method 100 are performed by the processor 12 of computer system 10. However, if sufficient processing power is available locally, the image retrieval system 1 learning can be performed (at least partially) by the processor 152 of device 15.

[0053] The deep learning steps involve inferring image features based on training image 101 and encoding the detected features in the form of image retrieval system 1.

[0054] The training images 101 are annotated, and each training image 101 is associated with an annotation indicating the type of goods 110 in the training image 101. In other words, the properties of goods 110 are known in the training images 101. In some examples, domain experts may manually annotate the training images 101 using ground truth annotations (e.g., the type of goods in the image).

[0055] In some examples, the generated image retrieval system 1 is configured to detect at least one image in the image dataset 20, which includes goods most similar to the goods of interest 11 in the examined images 1000. In some examples, multiple images in the dataset 20 are detected and sorted based on the similarity of their goods to the goods of interest 11 (e.g., as a non-limiting example, multiple images can be sorted from most similar to least similar, or from least similar to most similar).

[0056] In this disclosure, the similarity between goods can be based on the Euclidean distance between the features of the goods.

[0057] As described in more detail below, Euclidean distance can be used in the loss function associated with the image retrieval system 1 applied to the training image 101. It was considered.

[0058] As described in more detail below and Figure 2 As shown, the features of the goods can be derived from one or more compact vector representations 21 of images (such as training image 101 and / or inspection image 1000). In some examples, the one or more compact vector representations of the image may include at least one of a feature vector f, a descriptor matrix V, and a final image representation FIR. In some examples, the one or more compact vector representations 21 of the image may be stored in the memory 121 of system 10.

[0059] In other words, during the training performed by S2, the image retrieval system 1 is configured to learn a metric problem that allows Euclidean distance to capture the similarity between features of the goods.

[0060] During the training performed by S2, image retrieval system 1 and parameter functions ( The correlation, and the training performed in S2, enable image retrieval system 1 to find a function that minimizes the loss function. The learnable parameters make:

[0061] (Equation 1)

[0062] have: as well as There are three images. as well as Make: Including anchor point images Goods similar to those, and Including anchor point images Different goods

[0063] yes European style The norm, and d is the dimension of the image vector representation, and d can be chosen by the operator training the system.

[0064] N is the number of images in the image dataset, and

[0065] It is a hyperparameter that controls the margin between similar and different images, and it can be selected by the operator of the training system.

[0066] like Figure 5 and 6 As shown, in S2, the training image retrieval system 1 includes applying a feature extraction convolutional neural network 1001 (referred to as CNN 1001) to the annotated training images 101 in S21. CNN 1001 includes multiple convolutional layers to generate image feature tensors. .

[0067] As a non-limiting example, the feature extraction CNN 1001 may include at least one of a CNN called AlexNet, VGG, and ResNet. In some examples, the feature extraction CNN 1001 is fully convolutional.

[0068] The image retrieval system 1 trained in S2 also includes tensors generated in S22. An Aggregate Generalized Average (AgGeM) pooling layer 1002, which is associated with image spatial information, is applied.

[0069] like Figure 7 As shown, the application in S22 includes applying a generalized average pooling layer 1011 in S221 to generate multiple embedding vectors. , so that: ,

[0070] have:

[0071]

[0072] It has the following characteristics: P is a set of positive integers p representing the pooling parameters of the generalized average pooling layer.

[0073] The tensor With feature map of Each feature map is activated by applying a feature extraction CNN 1001 to the training image 101, and H and W are the height and width of each feature map, respectively.

[0074] K is the number of feature maps in the final convolutional layer of the feature extraction CNN 1001.

[0075] x is a tensor generated from Features, and

[0076] It is a tensor of The base number.

[0077] The application in S22 also includes, in S222, applying the weights α of the hierarchical layer 1012 associated with the attention mechanism to the generated multiple embedding vectors. To aggregate the multiple embedding vectors For each pooling parameter p belonging to P, the weight α is as follows: ,

[0078] have:

[0079] The weights α and parameters θ can be learned by the image retrieval system 1 to minimize the loss function. .

[0080] The aggregation performed in S222 is configured to generate a feature vector f such that: ,

[0081] have:

[0082] .

[0083] Return to reference Figure 5 and Figure 6 In S2, training the image retrieval system 1 may include, in S23, directing the generated tensor... Unordered feature pooling layer 1003 associated with image texture information.

[0084] like Figure 8 As shown, applying the unordered feature pooling layer 1003 in S23 may include using a Gaussian mixture model (GMM) to generate unordered image descriptors of image features.

[0085] The application of S23 can include tensors in S231. Image features , Mapped to a structure with diagonal variance A set of clusters of the Gaussian mixture model such that:

[0086] .

[0087] have: identity matrix

[0088] yes The size of cluster k, and

[0089] It represents the variance in the k-th cluster. The reciprocal of the smoothing factor, The image retrieval system can learn to minimize the loss function. .

[0090] The application of S23 can also include S232 by combining with features Associated weights Assigned to center c k Using cluster k to apply the soft assignment algorithm, such that:

[0091] ,

[0092] have: It is a vector representing the center of the k-th cluster. Image retrieval systems are learnable and can minimize a loss function. ,

[0093] For an index k=k' in the range from 1 to K, and same,

[0094] It is a hyperparameter representing the number of clusters to be included in the set of multiple clusters in the Gaussian mixture model.

[0095] The application of S23 can also include generating a descriptor matrix V in S233, such that:

[0096] .

[0097] The hyperparameter M can be selected by the operator who trained system 1.

[0098] like Figure 5 and 6 As shown, in some examples, applying the Aggregate Generalized Average (AgGeM) pooling layer 1002 in S22 and applying the unordered feature pooling layer 1003 (e.g., GMM layer) in S23 can be performed in parallel.

[0099] like Figure 5 and 6 As shown, the training in S2 also includes applying the bilinear model layer 1004 to the combined output associated with the aggregated generalized average pooling layer 1002 and the unordered feature pooling layer 1003 in S24.

[0100] In some examples, bilinear model layer 1004 can be associated with a bilinear function. Related, such that:

[0101]

[0102] It has: a t It is a vector of dimension I and associated with the output of the unordered feature pooling layer 1003.

[0103] b s It is a vector of dimension J and associated with the output of the aggregated generalized average pooling layer 1002, and

[0104] ωij is configured to balance a t and b s The interaction weights, ωij, are learnable by the image retrieval system 1 to minimize the loss function. .

[0105] like Figure 6 As shown, it can be done by... 2. Normalization layer 1005 and / or fully connected layer 1006 are applied to the descriptor matrix V to obtain vector a. t .

[0106] like Figure 6 As shown, it can be achieved by (such as) 2. Normalization layers and / or batch normalization layers (e.g., 1007) and / or fully connected layers (e.g., 1008) are applied to the feature vector f to obtain the vector b. s .

[0107] like Figure 5 and 6 As shown, S2 may also include in S25 applying at least one normalization layer 1009 (e.g., 2. A normalization layer is applied to the combined output associated with the aggregated generalized average pooling layer 1002 and the unordered feature pooling layer 1003. Alternatively or additionally, S2 may also include applying a fully connected layer 1010 to the combined output associated with the aggregated generalized average pooling layer 1002 and the unordered feature pooling layer 1003 in S26.

[0108] At least one normalization layer 1009 is applied in S25 and / or a fully connected layer 1010 is applied in S26 so that the final image representation FIR of the image can be obtained.

[0109] In some examples, each of the training images 101 is also associated with a code of a coordinated Goods Description and Decoding System (HS). The HS includes hierarchical sections and chapters corresponding to the types of goods in the training images 101.

[0110] In some examples, training image retrieval system 1 in S2 also includes a loss function in image retrieval system 1. The HS is considered in terms of its hierarchical structure and chapters.

[0111] In some examples, the loss function of an image retrieval system is as follows:

[0112] (Equation 1')

[0113] have: It is the HS code of the training image corresponding to the query.

[0114] It is the HS code of the training image, which shares the same hierarchical chapters and / or hierarchical chapters as the training image corresponding to the query.

[0115] It is the HS code of the training image, which has different hierarchical portions and / or hierarchical sections than the training image corresponding to the query.

[0116] η is a parameter function associated with image retrieval system 1, and η is a loss function that can be learned by image retrieval system 1 to minimize the loss function. , parameters

[0117] These are parameters that control the importance of the hierarchical structure assigned to the HS code during training, and

[0118] δ is a hyperparameter that controls the boundaries between similar and different HS codes, and it can be chosen by the operator of the training system.

[0119] In some examples, the S2 training image retrieval system 1 also includes the application of a hardness-aware depth metric learning (HDML) algorithm.

[0120] Other structures can also be conceived for image retrieval system 1. For example, deeper structures can be conceived, and / or structures similar to those in other systems can be conceived. Figure 6 A structure of the same shape as the one shown will produce vectors or matrices with dimensions different from those already discussed (e.g., vector f, matrix V, vector b). s Vector a t And / or the final image represents FIR).

[0121] Return to reference Figure 6 The scoring layer 1012 of the AgGeM layer 1002 may include two convolutions and softplus activation. In some examples, the size of the last of the two convolutions may be 1×1. Other architectures are also envisioned for the scoring layer 1012.

[0122] In some examples, each of the training images 101 is also associated with textual information corresponding to the type of goods in the training image 101. In some examples, the textual information may include at least one of the following: a report describing the goods (e.g., an existing expert report) and a report describing the parameters of the inspection of the goods (such as radiation dose, radiation energy, inspection equipment type, etc.).

[0123] Equipment manufacturing

[0124] like Figure 9 As shown, the method 200 for generating sorting device 15 (which is configured to sort multiple images of cargo from an image dataset generated using penetrating radiation) may include:

[0125] In S31, an image retrieval system 1 generated according to method 100 in any aspect of this disclosure is obtained; and

[0126] In S32, the obtained image retrieval system 1 is stored in the memory 151 of device 15. In S32, image retrieval system 1 can also be stored in detection device 15. Image retrieval system 1 can be created and stored using any suitable representation, such as a data description including data elements specifying sorting criteria and their sorting outputs (e.g., sorting based on the Euclidean distance of image features relative to the query image features). Such a data description can be encoded, for example, using XML or using a custom binary representation. Then, when image retrieval system 1 is applied, the data description is interpreted by processor 152 running on device 15.

[0127] Alternatively, the deep learning algorithm can directly generate the image retrieval system 1 as executable code (e.g., machine code, virtual machine bytecode, or an interpretable script). This can be in the form of code routines that device 15 can call to apply the image retrieval system 1.

[0128] Regardless of the representation of the image retrieval system 1, the image retrieval system 1 effectively defines a ranking algorithm (including a set of rules) based on the input data (i.e., the inspected image 1000 that defines the query).

[0129] After image retrieval system 1 is generated, it is stored in memory 151 of device 15. Device 15 may be temporarily connected to system 10 to communicate the generated image retrieval system (e.g., as a data file or executable code), or the communication may be performed using a storage medium (e.g., a memory card). In a preferred method, the image retrieval system is communicated from system 10 to device 15 via network connection 30 (this may include transmission over the Internet from the central location of system 10 to the local network where device 15 is located). Image retrieval system 1 is then installed on device 15. The image retrieval system may be installed as part of a firmware update for the device software, or it may be installed independently.

[0130] The installation of image retrieval system 1 can be performed once (e.g., during manufacturing or installation) or repeatedly (e.g., as a periodic update). The latter approach allows for improvements in the classification performance of the image retrieval system over time as new training images become available.

[0131] The image retrieval system is applied for sorting.

[0132] The ranking of images from dataset 20 is based on image retrieval system 1.

[0133] After the device 15 has been configured with the image retrieval system 1, the device 15 can use the image retrieval system 1 to sort multiple images of goods from the image dataset 20 based on the locally obtained inspection image 1000.

[0134] In some examples, the image retrieval system 1 effectively defines a ranking algorithm for performing the following steps: extracting features from the query (i.e., examining image 1000), calculating the distance between the image features of dataset 20 and the image features of the query, and ranking the images of dataset 20 based on the calculated distance.

[0135] Typically, the image retrieval system 1 is configured to extract features of the goods 11 of interest from the inspected image 1000 in a manner similar to the feature extraction performed during training in S2.

[0136] Figure 10 A flowchart illustrating an example method 300 for sorting multiple images of goods from image dataset 20 is shown. Method 300 is performed by device 15 (e.g., Figure 2 (As shown).

[0137] Method 300 includes:

[0138] In S41, inspection image 1000 is obtained;

[0139] In S42, the image retrieval system 1, generated according to any aspect of the present disclosure, is applied to the obtained image 1000; and

[0140] In S43, based on the application, multiple images of goods from image dataset 20 are sorted.

[0141] It should be understood that in order to sort multiple images in dataset 20 in S43, device 15 may be connected to system 10 at least temporarily, and device 15 may access memory 121 of system 10.

[0142] In some examples, at least a portion of the dataset 20 and / or portions of one or more compact vector representations 21 of the images (e.g., feature vector f, descriptor matrix V, and / or final image representation FIR) may be stored in the memory 151 of the device 15.

[0143] In some examples, sorting multiple images in S43 includes: outputting a sorted list of images, which includes goods corresponding to the goods of interest in the inspected images.

[0144] In some examples, as a non-limiting example, the sorted list may be a subset of the image dataset 20, such as 1 image of the dataset or 2, 5, 10, 20 or 30 images of the dataset.

[0145] In some examples, sorting multiple images in S43 may also include: outputting at least a portion of the code of a coordinated commodity description and decoding system (HS), the HS including hierarchical sections and chapters corresponding to the type of goods in each of the multiple sorted images.

[0146] In some examples, the sorting in S43 may further include: outputting at least partial text information corresponding to the cargo type in each of the plurality of sorted images. In some examples, the text information may include at least one of the following: a report describing the cargo and a report describing the parameters of the cargo inspection.

[0147] Further details and embodiments

[0148] This disclosure may be advantageous, but is not limited to, customs and / or security applications.

[0149] This disclosure is generally applied to cargo inspection systems (e.g., sea or air cargo).

[0150] Figure 2 The device 3 is used as an inspection system, which is configured to inspect the container 4, for example, by transmitting inspection radiation through the container 4.

[0151] As a non-limiting example, the container 4, configured to hold goods, can be placed on a vehicle. In some examples, the vehicle may include a trailer configured to carry the container 4.

[0152] Figure 2 The device 3 may include a source 5 configured to generate inspection radiation.

[0153] Radiation source 5 is configured to inspect goods through the material of the wall of container 4 (typically steel), for example, for the detection and / or identification of goods. Alternatively or additionally, a portion of the inspection radiation may be transmitted through container 4 (the material of container 4 is therefore transparent to radiation), while another portion of the radiation may be reflected at least partially by container 4 (referred to as "backscattering").

[0154] In some examples, device 3 can be mobile and transportable from one location to another (device 3 may include a motor vehicle).

[0155] In Source 5, electrons are typically accelerated at voltages between 100 keV and 15 MeV.

[0156] In a mobile inspection system, the power of the X-ray source 5 can be, for example, between 100 keV and 9.0 MeV, typically such as 300 keV, 2 MeV, 3.5 MeV, 4 MeV or 6 MeV, to achieve steel penetration capability (e.g., between 40 mm and 400 mm, typically such as 300 mm (12 inches)).

[0157] In a static inspection system, the power of the X-ray source 5 can be, for example, between 1 MeV and 10 MeV, typically, for example, 9 MeV, to obtain steel penetration capability (e.g., between 300 mm and 450 mm, typically, for example, 410 mm (16.1 inches)).

[0158] In some examples, source 5 can emit continuous X-ray pulses. The pulses can be emitted at a given frequency, including between 50 Hz and 1000 Hz, for example, about 200 Hz.

[0159] Based on some examples, the detector can be mounted on a rack, such as Figure 2 As shown. The frame, for example, is shaped like an inverted "L". In a mobile inspection system, the frame may include an electro-hydraulic boom, which can be in a retracted position (not shown) in transport mode and in an inspection position (…). Figure 2 Operation. The boom can be operated by a hydraulic actuator (e.g., a hydraulic cylinder). In a static inspection system, the frame may include a static structure.

[0160] It should be understood that the inspection radiation source may include other penetrating radiation sources, such as ionizing radiation sources (e.g., gamma rays or neutrons) as non-limiting examples. The inspection radiation source may also include sources unsuitable for activation by a power source, such as radioactive sources (e.g., using Co60 or Cs137). In some examples, the inspection system includes detectors (e.g., X-ray detectors, optionally gamma and / or neutron detectors) adapted, for example, to detect the presence of radioactive gamma and / or neutron-emitting materials within the cargo simultaneously with, for example, X-ray inspection. In some examples, detectors may be positioned to receive radiation reflected by container 4.

[0161] In the context of this disclosure, container 4 can be any type of container, such as a holder or box. Thus, as a non-limiting example, container 4 can be a pallet (e.g., a pallet of European standard, American standard or any other standard) and / or a train wagon and / or a tank and / or a vehicle trunk and / or a “transport container” (e.g., a tank or ISO container or a non-ISO container or a unit loading device (ULD) container).

[0162] In some examples, one or more memory elements (e.g., the memory of one of the processors) may store data used for the operations described herein. This includes memory elements capable of storing software, logic, code, or processor instructions that are executed to perform the activities described in this disclosure.

[0163] A processor can execute any type of instructions associated with data to perform the operations detailed herein. In one example, a processor can transform an element or entry (e.g., data) from one state or thing to another. In another example, the activities outlined herein can be implemented with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor), and the elements identified herein can be some type of programmable processor, programmable digital logic (e.g., field-programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), ASIC including digital logic, software, code, electronic instructions, flash memory, optical disc, CD-ROM, DVD ROM, magnetic card or optical card, other types of machine-readable media suitable for storing electronic instructions, or any suitable combination thereof.

[0164] As one possibility, a computer program, computer program product, or computer-readable medium is provided, comprising computer program instructions to cause a programmable computer to perform any one or more of the methods described herein. In example implementations, at least some portions of the processor-related activities may be implemented in software. It should be understood that, if desired, the software components of this disclosure may be implemented in ROM (Read-Only Memory). If desired, the software components may typically be implemented in hardware using conventional techniques.

[0165] In the context of this disclosure, other variations and modifications of the system will be apparent to those skilled in the art, and the various features described above may or may not have the advantages of having the other features described above. The above embodiments should be understood as illustrative examples, and other embodiments are contemplated. It should be understood that any feature described with respect to any embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other embodiment, or in any combination with any other embodiment. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the appended claims.

Claims

1. A method for generating an image retrieval system, the image retrieval system being configured to sort multiple images of a cargo from an image dataset in response to a query corresponding to an image of a cargo generated using penetrating radiation, the method comprising: Obtain multiple annotated training images of goods, each of which is associated with an annotation indicating the type of goods in the training image; as well as The image retrieval system is trained by applying a deep learning algorithm to the obtained annotated training images, the training comprising: A convolutional neural network (CNN) with multiple convolutional layers is applied to the annotated training images to generate tensors of image features. ,as well as The generated tensor The application includes using aggregated generalized average AgGeM pooling layers associated with image spatial information, the application comprising: Apply a generalized average pooling layer to generate multiple embedding vectors , so that: ， have: ， have: P is a set of positive integers p representing the pooling parameters of the generalized average pooling layer. The tensor With feature map of Each activation, the feature map being obtained by applying the feature extraction CNN to the training image, has: H and W being the height and width of each element in the feature map, respectively. K is the number of feature maps in the final convolutional layer of the feature extraction CNN. x is a tensor generated from Features, and It is the tensor of The cardinality, and By applying the weights α of the scoring layer associated with the attention mechanism to the generated multiple embedding vectors To aggregate the multiple embedding vectors For each pooling parameter p belonging to P, the weight α makes: ， The weights α and parameters θ of the image retrieval system are learnable to minimize the associated loss function. The aggregation is configured to generate a feature vector f such that: ， have: ， Each of the training images is also associated with a code of a coordinated product description and decoding system (HS), which includes hierarchical portions and sections corresponding to the type of the goods in the training images. The training of the image retrieval system further includes: considering the hierarchical portion and chapters of the HS in the loss function associated with the image retrieval system. Wherein, the loss function of the image retrieval system Make: ， have: as well as There are three images. as well as Make: Including anchor point images Goods similar to the aforementioned goods, and Including anchor point images The goods mentioned are different from the goods mentioned above. yes European style The norm, and d is the dimension of the image vector representation, and d can be chosen by the operator training the system. N is the number of images in the image dataset. These are hyperparameters that control the boundaries between similar and dissimilar images, and they can be selected by the operator training the system. It is the HS code corresponding to the training image in the query. It is the HS code of a training image that shares the same hierarchical portion and / or hierarchical section as the training image corresponding to the query. The HS codes of training images that are different from the hierarchical portions and / or hierarchical sections corresponding to the training images in the query. It is a parameter function associated with the image retrieval system, where η is a parameter that the image retrieval system can learn to minimize the loss function. , These are parameters that control the importance of the hierarchical structure assigned to the HS code during training, and δ is a hyperparameter that controls the boundary between similar and different HS codes, and it can be selected by the operator training the system.

2. The method according to claim 1, wherein training the image retrieval system further comprises: The generated tensor Apply unordered feature pooling layers that are associated with image texture information.

3. The method according to claim 2, wherein applying the unordered feature pooling layer comprises: The application of using a Gaussian Mixture Model (GMM) to generate unordered image descriptors for the image features includes: The tensor The image features , Mapped to a structure with diagonal variance A set of clusters of the Gaussian mixture model such that: . have: identity matrix yes The size of the cluster k, and It represents the variance in the k-th cluster. The reciprocal of the smoothing factor, The image retrieval system is learnable to minimize the loss function. By using the features described Associated weights Assigned to center c k The clustering k is used to apply the soft assignment algorithm, such that: ， have: The vector representing the center of the k-th cluster. The image retrieval system is learnable to minimize the loss function. For an index k=k' in the range from 1 to K, and same, This is a hyperparameter representing the number of clusters to be included in the set of multiple clusters of the Gaussian mixture model, and it can be selected by the operator training the system. Generate a descriptor matrix V such that: 。 4. The method according to claim 3, wherein, The application of the aggregated generalized average pooling layer and the application of the unordered feature pooling layer are executed in parallel.

5. The method according to claim 4, wherein, The training further includes applying a bilinear model layer to the combined output associated with the aggregated generalized average pooling layer and the unordered feature pooling layer, wherein the bilinear model layer is associated with a bilinear function. Related, such that: Where: a t It is a vector of dimension I and associated with the output of the unordered feature pooling layer. b s It is a vector of dimension J and associated with the output of the aggregated generalized average pooling layer, and ωij is configured to balance a t and b s The weights of the interactions between them, ωij, are learnable by the image retrieval system to minimize the loss function.

6. The method according to claim 5, wherein, The vector a t By 2. A normalization layer and / or a fully connected layer are applied to the descriptor matrix V to obtain it.

7. The method according to claim 5 or 6, wherein, The vector b s It is obtained by applying a normalization layer and / or a fully connected layer to the feature vector f.

8. The method of claim 4, further comprising: Apply at least one of the following to the combined output associated with the aggregated generalized average pooling layer and the unordered feature pooling layer: At least one normalization layer, and / or Fully connected layer.

9. The method of claim 8, wherein generating the image retrieval system further comprises: Obtain the final image representation of the image.

10. The method of claim 1, wherein training the image retrieval system further comprises: Applying a hardness-aware depth metric learning HDML algorithm, or The feature extraction CNN includes at least one of AlexNet, VGG, and ResNet, or The feature extraction CNN is a fully convolutional one.

11. The method according to claim 1, wherein, The scoring layer consists of two convolutions and softplus activation.

12. The method of claim 11, wherein the size of the last of the two convolutions is 1×1, or in, The weight α has a value in the range [0, 1].

13. The method according to claim 1, wherein, Each of the training images is also associated with textual information of the type corresponding to the goods in the training image.

14. The method of claim 13, wherein the text information includes at least one of: a report describing the goods and a report describing the parameters of the inspection of the goods.

15. The method according to claim 1, wherein, Obtaining the annotated training images includes: Retrieve the annotated training images from an existing image database; and / or Generating the annotated training images includes: Irradiating one or more containers containing goods with penetrating radiation, and Detect radiation from one or more irradiated containers.

16. The method of claim 15, wherein the irradiation and / or the detection are performed using one or more devices configured to inspect the container.

17. The method according to claim 1, wherein, The image retrieval system is configured to detect at least one image in the image dataset, the at least one image including a cargo most similar to the cargo of interest in the inspected image, the similarity between the cargoes being based on the Euclidean distance between the cargoes being considered in the loss function of the image retrieval system applied to the training images.

18. The method according to claim 1, wherein, The method is performed on a computer system separate from the device configured to inspect the container.

19. A method comprising: Obtain an inspection image of the cargo of interest generated using penetrating radiation, the inspection image corresponding to the query; The image to be examined is applied to an image retrieval system generated by the method according to any one of the preceding claims; as well as Based on the application, multiple images of goods from an image dataset are sorted.

20. The method according to claim 19, wherein, Sorting the plurality of images includes: outputting a sorted list of images, the images including goods corresponding to the goods of interest in the inspected images.

21. The method of claim 20, wherein the sorted list is a subset of the image dataset.

22. The method of claim 20, wherein the dataset comprises at least one of the following: one or more training images and a plurality of inspection images.

23. The method according to claim 19 or 20, wherein, The sorting of the plurality of images also includes: outputting at least a portion of the code of a coordinated product description and decoding system HS, the HS including hierarchical portions and sections corresponding to the type of goods in each of the plurality of sorted images.

24. The method according to claim 19 or 20, wherein, Sorting the plurality of images further includes: outputting at least a portion of textual information corresponding to the type of goods in each of the plurality of sorted images.

25. The method of claim 24, wherein the text information includes at least one of: a report describing the goods and a report describing the parameters of the inspection of the goods.

26. A method for generating an apparatus configured to sort multiple images of cargo from an image dataset generated using penetrating radiation, the method comprising: Obtain an image retrieval system generated by the method according to any one of claims 1 to 18; as well as The obtained image retrieval system is stored in the memory of the device.

27. The method according to claim 26, wherein, The storage includes: transmitting the generated image retrieval system to the device via a network, the device receiving and storing the image retrieval system, or... The image retrieval system is generated, stored, and / or transmitted in one or more of the following forms: Data representation of the image retrieval system; Executable code for applying the image retrieval system to one or more inspected images.

28. The method according to claim 26 or 27, wherein, The goods of interest include at least one of the following: Threats; and / or Contraband.

29. An apparatus configured to sort multiple images of cargo from an image dataset generated using penetrating radiation, the apparatus comprising a memory storing an image retrieval system generated by the method according to any one of claims 1 to 18.

30. The device of claim 29, further comprising a processor, wherein the memory of the device further comprises instructions that, when executed by the processor, enable the processor to perform the method of any one of claims 19 to 25.

31. A computer program product comprising instructions that, when executed by a processor, enable the processor to perform the method according to any one of claims 1 to 28, or the instructions being stored on a memory of the device according to claim 29 or claim 30.

Images