Decomposition of inspection image of cargo

Abstract

A method of generating a decomposer 3 configured to decompose an input image of a cargo comprising a container, the input cargo image being generated using penetrating radiation, the decomposition dec

Description

Field of the invention The invention relates but is not limited to generating a decomposer configured to decompose an input image of a cargo comprising a container, the input cargo image being generated using penetrating radiation. The invention also relates but is not limited to decomposing an input image of a cargo comprising a container. The invention also relates but is not limited to producing a device configured to decompose an input image of a cargo comprising a container. The invention also relates but is not limited to corresponding devices and computer programs or computer program products. Background Inspection images of cargo comprising containers may be generated using penetrating radiation. In some examples, a user may want to detect objects corresponding to a cargo of interest on the inspection images. Detection of such objects may be difficult. In some cases, the object may not be detected at all. In cases where the detection is not clear from the inspection images, the user may inspect the container manually, which may be time consuming for the user. Summary of the Invention Aspects and embodiments of the invention are set out in the appended claims. These and other aspects, as well as embodiments which are helpful in understanding the invention as set out in the claims are also described herein. Any feature in one aspect of the disclosure may be applied to other aspects of the disclosure, in any appropriate combination. In particular, method aspects may be applied to device and computer program aspects, and vice versa. Furthermore, features implemented in hardware may generally be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly. Brief Description of Drawings Embodiments of the present disclosure will now be described, by way of example, with reference to the accompanying drawings, in which: Figure 1 shows a flow chart illustrating an example method according to the disclosure; Figure 2 schematically illustrates an example system and an example device configured to implement the example method of Figure 1; Figures 3A and 3B illustrate example input images according to the disclosure; Figure 4A schematically shows steps of the example method of Figure 1; Figure 4B schematically shows steps of the example method of Figure 1; Figure 5A schematically shows steps of the example method of Figure 1; Figure 5B schematically shows steps of the example method of Figure 1; Figure 6 shows a flow chart illustrating another example method according to the disclosure; and Figure 7 shows a flow chart illustrating another example method according to the disclosure. In the figures, similar elements bear identical numerical references. Description of Example Embodiments Overview The disclosure discloses an example method for generating a decomposer configured to decompose an input image of a cargo comprising a container into two images. A first image is an image of the container of the cargo, in which the container is represented empty. A second image is an image of a load of the cargo contained in the container of the cargo, in which the load is represented without the container. The input image of the cargo corresponds to an inspection image obtained by transmission of radiation through the cargo. Embodiments of the disclosure helps a user of a cargo inspection system in determining whether a container under inspection is empty, by checking whether the container contains a load, regardless of the type of inspection system generating the input image and regardless of the type of container of the cargo (i.e., whether the container is an ISO container or is a non-ISO container of various sizes, etc.) Generating the decomposer comprises training the decomposer by applying a machine learning algorithm to both input images of non-empty containers and input images of empty containers. For each training input image, the training comprises generating a first image of the container represented as empty and a second image of the load represented without the container and recombining the generated first image and the generated second image to obtain a reconstructed image. The training also comprises comparing the obtained reconstructed image and the input image, the decomposer being trained to minimize a reconstruction loss between the obtained reconstructed image and the input image. The disclosure also discloses an example method for decomposing an input image of a cargo comprising a container, by applying the decomposer after the decomposer has been trained for the decomposition. The disclosure also discloses an example method for producing a device configured to decompose an input image of a cargo comprising a container. The disclosure also discloses corresponding devices and computer programs or computer program products. Detailed Description of Example Embodiments Figure 1 shows a flow chart illustrating an example method 100 according to the disclosure for generating a decomposer 3. The decomposer 3 is illustrated in Figure 2. Figure 2 shows a device 15 configurable by the method 100 to decompose an input image of a cargo comprising a container into two images. A first image corresponds to an image of the container of the cargo, the container being represented empty in the first image. A second image corresponds to an image of a load of the cargo contained in the container of the cargo, the load being represented without the container in the second image. Non-limiting examples of input cargo images 1000 are shown in Figures 3A and 3B, with an empty container (Figure 3B) and a non-empty container (Figure 3A). In Figures 3A and 3B, the container type of the non-empty container is the same as the type of container of the empty container. However, it should be understood that, in the context of the disclosure, the non-empty containers are not necessarily of the same type as that of the empty containers. In the present disclosure, the term “container" should be construed broadly as any holder or any box or any receptacle suitable for containing a load: such as a palette (for example a palette of European standard, of US standard or of any other standard), and / or any “shipping container”, such as an ISO container, a Unit Load Device (ULD) container or a non-ISO container, and / or any trailer or any boot or any wagon or any tank of any type of vehicle, such as a train or a truck as non-limiting examples. For example, the container may be a trailer or boot comprising walls made out of metal or not, refrigerated or not. The input cargo images 1000 of Figures 3A and 3B are generated using penetrating radiation as explained in greater detail below. Computer system and detection device Figure 2 schematically illustrates an example computer system 10 and the device 15 configured to implement, at least partly, the example method 100 of Figure 1. In particular, in a preferred embodiment, the computer system 10 executes the deep learning algorithm to generate the decomposer 3 to be stored on the device 15. Although a single device 15 is shown for clarity, the computer system 10 may communicate and interact with multiple such devices. The computer system 10 of Figure 2 comprises a memory 121, a processor 12 and a communications interface 13. The system 10 may be configured to communicate with one or more devices 15, via the interface 13 and a link 30 (e.g. Wi-Fi connectivity, but other types of connectivity may be envisaged). The memory 121 is configured to store, at least partly, data, for example for use by the processor 12. In some examples the data stored on the memory 121 may comprise the training images (and the data used to generate the training images) and / or the deep learning algorithm. The detection device 15 of Figure 2 comprises a memory 151, a processor 152 and a communications interface 153 (e.g. Wi-Fi connectivity, but other types of connectivity may be envisaged) allowing connection to the interface 13 via the link 30. In a non-limiting example, the device 15 may also comprise an apparatus 50 acting as an inspection system, as described in greater detail later. The apparatus 50 may be integrated into the device 15 or connected to other parts of the device 15 by wired or wireless connection. In some examples, as illustrated in Figure 2, the disclosure may be applied for inspection of cargo 4 containing a container 11. To obtain the input images for training, the method may further comprise irradiating, using penetrating radiation, cargo comprising a container, and detecting radiation from the irradiated cargo. The training input images may thus correspond to actual observed images. The irradiating and / or the detecting may be performed using one or more devices configured to inspect cargo. The training may be performed at a computer system separate, optionally remote, from a device configured to inspect cargo. In some examples, the processor 152 of the device 15 may be configured to perform, at least partly, at least some of the steps of the method 100 of Figure 1 and / or the method 200 of Figure 6 and / or the method 300 of Figure 7. In some examples, the processor 12 of the system 10 may be configured to perform, at least partly, at least some of the steps of the method 100 of Figure 1 and / or the method 200 of Figure 6 and / or the method 300 of Figure 7. Generating the decomposer In Figure 1, the method 100 comprises training, at S1, the decomposer by applying a machine learning algorithm to both input images of non-empty containers and input images of empty containers. The learning process is typically computationally intensive and may involve large volumes of training images. In some examples, the processor 12 of the system 10 shown in Figure 2 may comprise greater computational power and memory resources than the processor 152 of the device 15. The decomposer 3 generation is therefore performed, at least partly, remotely from the device 15, at the computer system 10. However, if sufficient processing power is available locally then the decomposer learning could be performed (at least partly) by the processor 152 of the device 15. In Figure 1, the training S1 comprises, for each input image of a non-empty container containing a load, generating, at S11, a first image of the container represented as empty and a second image of the load represented without the container, recombining, at S12, the generated first image and the generated second image to obtain a reconstructed image, and comparing, at S13, the reconstructed image and the input image. In Figure 4A, for an input image 1000 of a non-empty container containing a load, the decomposer 3 generates a first image 1001 of the container represented as empty and a second image 1002 of the load represented without the container. In Figure 4A, the decomposer 3 recombines the generated first image 1001 and the generated second image 1002 to obtain a reconstructed image 1003. In Figure 4A, the decomposer 3 compares the reconstructed image 1003 and the input image 1000. In Figure 1, the training S1 comprises, for each input image of an empty container not containing any load, generating, at S14, a first image of the container represented as empty and a second blank image, recombining, at S15, the generated first image and the generated second blank image to obtain a reconstructed image, and comparing, at S16, the reconstructed image and the input image. In Figure 4B, for an input image 1000 of an empty container not containing any load, the decomposer 3 generates a first image 1001 of the container represented as empty and a second blank image 1002. In Figure 4B, the decomposer 3 recombines the generated first image 1001 and the generated second blank image 1002 to obtain a reconstructed image 1003. In Figure 4B, the decomposer 3 compares the reconstructed image 1003 and the input image 1000. The decomposer is trained to minimize a reconstruction loss between the reconstructed image and the input image. The reconstruction loss may comprise a similarity metric of Lp-norm (p being an integer greater or equal to 1), such as an average absolute deviation or a least mean square distance. The machine learning algorithm may comprise a convolutional neural network. As schematically shown in Figure 1 and Figure 5A, the training S1 may further comprise, for each input image of an empty container, comparing, at S17 shown in Figure 1, the generated second blank image 1002 and a reference blank image 1004 as shown in Figure 5A. The decomposer 3 is trained to minimize a reconstruction loss between the generated second blank image 1002 and the reference blank image 1004. The reconstruction loss may comprise a similarity metric of Lp-norm (p being an integer greater or equal to 1), such as an average absolute deviation or a least mean square distance. The input images 1000 of the non-empty containers containing a load, an example of which is shown at Figure 3A, correspond to a first plurality of observed images generated using penetrating radiation. Alternatively or additionally, the input images 1000 of the empty containers not containing any load, an example of which is shown at Figure 3B, correspond to a second plurality of observed images generated using penetrating radiation. As already stated, the disclosure applies to non-empty containers which are not necessarily of the same type as that of the empty containers. During the training, the containers of the input images of non-empty containers are not necessarily of the same type as that of the input images of empty containers. The decomposer is thus trained to generate realistic first image of the container represented as empty for each input image of a non-empty container, as explained below. As schematically shown in Figure 1 and Figure 5B, the training S1 may further comprise, inputting, at S18 represented in Figure 1, the generated first image 1001 of the container represented as empty for each input image 1000 of a non-empty container into a discriminator 40 of a generative adversarial network, GAN, the generator of the GAN being the decomposer 3, as shown in Figure 5B. As shown in Figure 5B, the generated first image 1001 of the container represented as empty corresponds to a “fake image” input for the discriminator 40. As schematically shown in Figure 1 and Figure 5B, the training S1 may further comprise, inputting, at S19 represented in Figure 1, an input image 1000 of an empty container not containing any load into the discriminator 40 of the GAN, as shown in Figure 5B. As shown in Figure 5B, the input image 1000 corresponds to a “real image” input for the discriminator 40. The decomposer 3 and the discriminator 40 of Figures 5B compete based on a GAN loss function, the decomposer being trained to minimize the GAN loss between the “fake image” and the “real image”. The GAN loss and the reconstruction loss previously mentioned are combined to have the total loss. The decomposer 3 is arranged to produce the output more easily, after it is stored in a memory 151 of the device 15 (as shown in Figure 2), even though the process 100 for generating the decomposer 3 from the training images may be computationally intensive. After it is configured, the device 15 may provide an accurate output of a decomposition of an input image into two images, by applying the decomposer to an input inspection image 1000. The decomposition process is illustrated (as process 300) in Figure 7 (described later). Device manufacture As illustrated in Figure 6, the method 200 of producing the device 15 configured to decompose an input image of a cargo comprising a container comprises: obtaining, at S31, a decomposer generated by the method 100 according to any aspects of the disclosure; and storing, at S32, the obtained decomposer 3 in the memory 151 of the device 15. In some examples, the decomposer 3 may be obtained at S31 (e.g. by generating the decomposer 3 as in the method 100 of Figure 1). In some examples, obtaining the decomposer 3 at S31 may comprise receiving the decomposer 3 from another data source. The decomposer 3 may be stored, at S32, in the detection device 15. The decomposer 3 may be created and stored using any suitable representation, for example as a data description comprising data elements specifying decomposition conditions and their decomposition outputs. Such a data description could be encoded e.g. using XML or using a bespoke binary representation. The data description is then interpreted by the processor 152 running on the device 15 when applying the decomposer 3. Alternatively, the deep learning algorithm may generate the decomposer 3 directly as executable code (e.g. machine code, virtual machine byte code or interpretable script). This may be in the form of a code routine that the device 15 can invoke to apply the decomposer 3. After the decomposer is generated, the decomposer 3 is stored in the memory 151 of the device 15. The device 15 may be connected temporarily to the system 10 to transfer the generated decomposer 3 (e.g. as a data file or executable code) or transfer may occur using a storage medium (e.g. memory card). In a preferred approach, the decomposer 3 is transferred to the device 15 from the system 10 over the network connection 30 (this could include transmission over the Internet from a central location of the system 10 to a local network where the device 15 is located). The decomposer 3 is then installed at the device 15. The decomposer 3 could be installed as part of a firmware update of device software, or independently. Installation of the decomposer 3 may be performed once (e.g. at time of manufacture or installation) or repeatedly (e.g. as a regular update). The latter approach can allow the decomposition performance of the decomposer 3 to be improved over time, as new training images become available. Applyinq the decomposer to perform decomposition Figure 7 shows a flow chart illustrating an example method 300 for decomposing an input image of a cargo comprising a container. The method 300 is performed by the device 15 (as shown in Figure 2). The method 300 comprises: obtaining, at S41, an input image 1000, the input image being generated using penetrating radiation; decomposing, at S42, the input image 1000 into two images, a first image of the container of the cargo, the container being represented empty in the first image, and a second image of a load of the cargo contained in the container of the cargo, the load being represented without the container in the second image. It should be understood that in order to decompose at S42 the input image 1000 into two images, the device 15 may be connected, at least temporarily, to the system 10, and the device 15 may access the memory 121 of the system 10. The decomposed two images may be displayed to a user of the decomposer and device on a man / machine interface conventionally comprising a display and input means, such as a keyboard and / or a mouse and / or a tactile function of the display. The display of the decomposed two images helps the user in determining whether the corresponding container under inspection is empty. Further details and examples The disclosure may be advantageous but is not limited to customs and / or security applications. The disclosure typically applies to cargo inspection systems (including road, sea or air cargo). The apparatus 50 of Figure 2, acting as an inspection system, is configured to inspect the container 11, e.g. by transmission of inspection radiation through the container 11. In other words, the apparatus 50 may be used to acquire the inspection image 1000. The apparatus 50 of Figure 2 may comprises a source 5 configured to generate the inspection radiation. The radiation source 5 is configured to cause the inspection of the cargo through the material (usually but not necessarily steel) of walls of the container 11, e.g. for detection and / or identification of the cargo. Alternatively or additionally, a part of the inspection radiation may be transmitted through the container 11 (the material of the container 11 being thus transparent to the radiation), while another part of the radiation may, at least partly, be reflected by the container 11 (called “back scatter”). In the source 5, electrons are generally accelerated under a voltage comprised between 100keV and 15MeV. In mobile inspection systems, the power of the X-ray source 5 may be e.g., between 100keV and 9.0MeV, typically e.g., 300keV, 2MeV, 3.5MeV, 4MeV, or 6MeV, for a steel penetration capacity e.g., between 40mm to 400mm, typically e.g., 300mm (12in). In static inspection systems, the power of the X-ray source 5 may be e.g., between 1MeV and 10MeV, typically e.g., 9MeV, for a steel penetration capacity e.g., between 300mm to 450mm, typically e.g., 410mm (16.1in). In some examples, the source 5 may emit successive X-ray pulses. The pulses may be emitted at a given frequency, comprised between 50 Hz and 1000 Hz, for example approximately 200 Hz. It should be understood that the inspection radiation source may comprise sources of other penetrating radiation, such as, as non-limiting examples, sources of ionizing radiation, for example gamma rays or neutrons. The inspection radiation source may also comprise sources which are not adapted to be activated by a power supply, such as radioactive sources, such as using Co60 or Cs137. In some examples, the inspection system comprises detectors, such as X-ray detectors, optional gamma and / or neutrons detectors, e.g., adapted to detect the presence of radioactive gamma and / or neutrons emitting materials within the cargo, e.g., simultaneously to the X-ray inspection. In some examples, detectors may be placed to receive the radiation reflected by the container. In some examples, the apparatus 50 may be mobile and may be transported from a location to another location (the apparatus 50 may comprise an automotive vehicle). According to some examples, detectors may be mounted on a gantry, as shown in Figure 2. The gantry for example forms an inverted “L”. In mobile inspection systems, the gantry may comprise an electro-hydraulic boom which can operate in a retracted position in a transport mode (not shown on the Figures) and in an inspection position (Figure 2). The boom may be operated by hydraulic actuators (such as hydraulic cylinders). In static inspection systems, the gantry may comprise a static structure. In some examples, one or more memory elements (e.g., the memory of one of the processors) can store data used for the operations described herein. This includes the memory element being able to store software, logic, code, or processor instructions that are executed to carry out the activities described in the disclosure. A processor can execute any type of instructions associated with the data to achieve the operations detailed herein in the disclosure. In one example, the processor could transform an element or an article (e.g., data) from one state or thing to another state or thing. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor) and the elements identified herein could be some type of a programmable processor, programmable digital logic (e.g., a field programmable gate array (FPGA), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM)), an ASIC that includes digital logic, software, code, electronic instructions, flash memory, optical disks, CD-ROMs, DVD ROMs, magnetic or optical cards, other types of machine-readable mediums suitable for storing electronic instructions, or any suitable combination thereof. As one possibility, there is provided a computer program, computer program product, or computer readable medium, comprising computer program instructions to cause a programmable computer to carry out any one or more of the methods described herein. In example implementations, at least some portions of the activities related to the processors may be implemented in software. It is appreciated that software components of the present disclosure may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. Other variations and modifications of the system will be apparent to the skilled in the art in the context of the present disclosure, and various features described above may have advantages with or without other features described above. The above embodiments are to be understood as illustrative examples, and further embodiments are envisaged. It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in 5 combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims.

Claims

1. A method for generating a decomposer configured to decompose an input image of a cargo comprising a container, the input cargo image being generated using penetrating radiation, the decomposition decomposing the input image into two images:a first image of the container of the cargo, the container being represented empty in the first image, anda second image of a load of the cargo contained in the container of the cargo, the load being represented without the container in the second image,the method comprising:training the decomposer by applying a machine learning algorithm to both input images of non-empty containers and input images of empty containers, the non-empty containers not being necessarily of the same type as that of the empty containers, the training comprising:for each input image of a non-empty container containing a load,generating a first image of the container represented as empty and a second image of the load represented without the container,recombining the generated first image and the generated second image to obtain a reconstructed image, andcomparing the reconstructed image and the input image, the decomposer being trained to minimize a reconstruction loss between the reconstructed image and the input image, andfor each input image of an empty container not containing any load,generating a first image of the container represented as empty and a second blank image,recombining the generated first image and the generated second blank image to obtain a reconstructed image, andcomparing the reconstructed image and the input image, the decomposer being trained to minimize a reconstruction loss between the reconstructed image and the input image.

2. The method of claim 1, further comprising, for each input image of an empty container:comparing the generated second blank image and a reference blank image, the decomposer being trained to minimize a reconstruction loss between the generated second blank image and the reference blank image.

3. The method of claim 1 or claim 2, further comprising:inputting the generated first image of the container represented as empty for each input image of a non-empty container into a discriminator of a generative adversarial network, GAN, the generator of the GAN being the decomposer, wherein the generated first image of the container represented as empty corresponds to a “fake image" input for the discriminator,inputting each input image of an empty container not containing any load into the discriminator of the GAN, wherein the input image corresponds to a “real image” input for the discriminator,wherein the decomposer and the discriminator compete based on a GAN loss function,the decomposer being trained to minimize the GAN loss between the “fake image” and the “real image”.

4. The method of claim 3, wherein the GAN loss and / or the reconstruction loss comprises a similarity metric of Lp-norm, p being an integer greater or equal to 1, such as an average absolute deviation or a least mean square distance.

5. The method of any preceding claim, wherein the input images of the non-empty containers containing a load correspond to a first plurality of observed images generated using penetrating radiation, and / orwherein the input images of the empty containers not containing any load correspond to a second plurality of observed images generated using penetrating radiation.

6. The method of any preceding claim, further comprising obtaining the input images, optionally comprising:irradiating, using penetrating radiation, cargo comprising a container, and detecting radiation from the irradiated cargo.

7. The method of claim 6, wherein the irradiating and / or the detecting are performed using one or more devices configured to inspect cargo.

8. The method of any preceding claim, wherein the machine learning algorithm comprises a convolutional neural network.

9. A method according to any preceding claim, wherein the method is performed at a computer system separate, optionally remote, from a device configured to inspect cargo.

10. A method for decomposing an input image of a cargo comprising a container, the method comprising:obtaining an input image, the input image being generated using penetrating radiation;decomposing the input image into two images:a first image of the container of the cargo, the container being represented empty in the first image, anda second image of a load of the cargo contained in the container of the cargo, the load being represented without the container in the second image.

11. The method of claim 10, performed by a decomposer generated by the method according to any one of claims 1 to 9, orwherein obtaining the input image comprises:irradiating cargo comprising a container, using penetrating radiation; and detecting radiation from the irradiated cargo.

12. A method of producing a device configured to decompose an input image of a cargo comprising a container, the method comprising:obtaining a decomposer generated by the method according to any one of claims 1 to 9; andstoring the obtained decomposer in a memory of the device.

13. The method according to claim 12, wherein the storing comprises transmitting the generated decomposer to the device via a network, the device receiving and storing the decomposer, orwherein the decomposer is generated, stored and / or transmitted in the form of one or more of:a data representation of the decomposer;executable code for applying the decomposer to one or more input images.

14. A device configured to decompose an input image of a cargo comprising a container, the device comprising a memory storing a decomposer generated by the method according to any one of claims 1 to 9, optionally further comprising a processor, and wherein the memory of the device further comprises instructions which, when executed by the processor, enable the processor to perform the method of any one of claims 10 to 13.

15. A computer program or a computer program product comprising instructions which, when executed by a processor, enable the processor to perform the method according to any one of claims 1 to 13 or to provide the device according to claim 14.

Images