Method of operating a delivery vehicle

Abstract

The present disclosure relates to a method of operating a delivery vehicle, comprising the steps of generating a reference image dataset of a package, analyzing the reference image dataset to determine an identification dataset of the package, and temporarily storing the identification dataset, generating a partial image dataset of the package in the delivery vehicle, a spatial localization dataset indicative of spatial localization of the parcel is determined, the partial image dataset is analyzed using the spatial localization dataset to determine an allocation dataset for the parcel, and the allocation dataset is compared to the identification dataset to re-identify the parcel.

Description

Technical Field

[0001] The present invention relates to a method for operating a delivery vehicle, a computer program product configured to execute such a method, a system for operating such a delivery vehicle, and a delivery vehicle for such a system. Background Art

[0002] Packages (e.g., postal packages from a delivery service) are delivered to customers by delivery vehicles (e.g., vans). Various studies have shown that when delivering packages, the last-mile transportation cost accounts for more than 50% of the total transportation cost. Most of the costs occur during loading and unloading packages onto the delivery vehicle.

[0003] In many cases, the delivery is not automated. For example, at a distribution center, delivery agents manually scan all packages with a handheld device (e.g., a barcode reader) and load all packages onto the delivery vehicle. Then, during the delivery phase, the delivery agent has to find the correct location of the packages in the delivery vehicle. This is time-consuming and leads to a decrease in delivery efficiency, and thus an increase in costs. Summary of the Invention

[0004] The object of the present disclosure is to show that improved methods can be achieved herein.

[0005] Another object of the present disclosure is achieved by a method for operating a delivery vehicle, the method comprising the steps of: generating a reference image dataset of packages, analyzing the reference image dataset to determine an identification dataset of packages and temporarily storing the identification dataset, generating a partial image dataset of packages in the delivery vehicle, determining a spatial positioning dataset indicating the spatial positioning of packages, analyzing the partial image dataset using the spatial positioning dataset to determine an allocation dataset of packages, comparing the allocation dataset with the identification dataset to identify packages.

[0006] Thus, in a first step, at a distribution center of a delivery service, images of all sides (i.e., all six sides of a package having a cubic basic shape) of packages are recorded by one or more cameras and temporarily stored as a reference image dataset.

[0007] In a further step, the thus-formed reference image dataset is analyzed in the distribution center to determine an identification dataset for each side of the package. The identification dataset is a machine-readable binary data string that encodes the data of the reference image dataset. A trained Siamese neural network can be used to generate the identification dataset of the package from the reference image dataset, which will be explained in detail below. The reference image dataset can be analyzed in a cloud service, and the reference image dataset is transferred to the cloud service before the analysis. However, in contrast, the reference image dataset can also be analyzed in the distribution center or in the delivery vehicle.

[0008] Then, the package is loaded onto the delivery vehicle, and the identification dataset is temporarily stored in the cloud service or in the delivery vehicle. This can be carried out at different times (i.e., successively), or it can also be carried out simultaneously. Then, the delivery vehicle leaves the distribution center with the package.

[0009] Then, for example, when the delivery vehicle is driving towards the recipient, partial image datasets of the sides of the package are generated in the delivery vehicle by a camera in the loading space of the delivery vehicle. However, instead of recording the image data of all sides of the package, only the images of, for example, two or three sides are recorded.

[0010] Then, the partial image datasets are analyzed to first determine the spatial orientation of the package (i.e., the direction of the package in space), and then to determine the assignment dataset of the package. Like the identification dataset, the assignment dataset is a machine-readable binary data string that encodes the data of the partial image datasets. The assignment dataset of the package is generated from the partial image datasets using the same trained Siamese neural network as was used to generate the identification dataset in the distribution center.

[0011] Then the assignment dataset is compared with the identification dataset to re-identify the package. In other words, in the case of multiple packages in the delivery vehicle, the corresponding assignment datasets are compared with the corresponding identification datasets to determine those pairs that match each other best.

[0012] Thus, the method allows the corresponding package to be identified based on the partial image datasets by comparison with the reference image dataset. In other words, the fact is utilized that only a part of the data used for the initial identification is necessary for re-identifying the package.

[0013] According to one embodiment, the method comprises the following steps: Once a package is identified by comparing an allocation data set with an identification data set, a position data set related to the current position of the package in the delivery vehicle is generated. To this end, the current position of the package in the delivery vehicle is detected and assigned to the identified package (e.g., in a cloud service). The position data set then encodes the position of the package in the delivery vehicle (e.g., in the form of a shelf number and a position in the shelf). The position data set is then wirelessly transmitted from the cloud service to the delivery vehicle and from the delivery vehicle to the handheld device of the delivery agent, or directly from the cloud service to the handheld device. The handheld device is configured to output the position data set in the form of text and / or a voice message. In other words, the handheld device assists the delivery agent in finding the position of the package based on the position data set.

[0014] According to another embodiment, the identification data set and the allocation data set are based on image data. This is data obtained by one or more cameras (e.g., CMOS cameras), and in each case the data is converted into a machine-readable binary data string. Thus, they are suitable for machine processing.

[0015] According to another embodiment, symbol data is also determined and the symbol data is analyzed by analyzing a reference image data set and a partial image data set. The symbol data can include, on the one hand, the dimensions of the package. The symbol data can also be data that can be immediately understood by the delivery agent (e.g., an address label with details related to the recipient and / or the sender) or other details (e.g., a package that needs to be handled with care due to particularly fragile contents, or an advertising label of the sender (e.g., a tape or package tape with the sender's symbol and / or trademark)). However, unlike the image data, the symbol data is not converted into a machine-readable form but remains in a format that can be immediately understood by the delivery agent. In other words, the symbol data can be OCR-converted so that, in addition to the encoded image data in machine-readable form, there is also text data, and they are compared with each other. This simplifies and improves the reliability of package re-identification.

[0016] The invention also includes a computer program product configured to execute such a method, a system for operating such a delivery vehicle, and a delivery vehicle for such a system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will now be explained with reference to the accompanying drawings, in which: Figure 1 The components of a system for operating a delivery vehicle are shown in schematic form; Figure 2 The loading space of the delivery vehicle is shown in schematic form; Figure 3 More details of the image data analysis are shown; Figure 4 Shows more details of the image data analysis; Figure 5 Shows more details of the image data analysis; Figure 6 Shows in a schematic form Figure 1 The sequence of the operating method of the system shown. Detailed implementation

[0018] First refer to Figure 1 .

[0019] Shows the system 2 for operating the delivery vehicle 4.

[0020] Among the components of the system 2, the cloud service 6, the control unit 8, the distribution center camera array 10 and the handheld device 12 are as Figure 1 shown.

[0021] In this exemplary embodiment, the cloud service 6 is a high-performance computing center with high computing power and storage capacity. It serves as the memory and data processing unit of the system 2. Different from this exemplary embodiment, the system 2 may include multiple sub-cloud services that are connected to a central computing center for data transmission. Further different from this exemplary embodiment, a memory other than the cloud service 6 may also be used.

[0022] In this exemplary embodiment, the control unit 8 is an embedded system unit located in the delivery vehicle 4. The control unit 8 has the necessary computing power and hardware elements (e.g., CPU, GPU, RAM, memory, CAN), is configured for wireless data transmission (4G, LTE, 5G), and has corresponding interfaces for this purpose (e.g., Ethernet interface, HDMI, USB, Wi-Fi).

[0023] In this exemplary embodiment, the distribution center camera array 10 in the distribution center 24 includes multiple cameras (e.g., CMOS cameras). At the distribution center 24 of the delivery service, through these cameras, all sides 20a, 20b of the packages 14a, 14b, 14c, 14d (see Figure 2 and Figure 3 ) are recorded (see Figure 3), i.e., images of all six sides 20a, 20b of the package having a cubic basic shape, and temporarily storing them as a reference image data set RDS in each case. For this purpose, the distribution center camera array 10 in the present exemplary embodiment has six cameras, one camera corresponding to each side 20a, 20b. Different from the present exemplary embodiment, fewer cameras can also be used. The processing device for the packages 14a, 14b, 14c, 14d can be associated with the distribution center camera array 10 in the distribution center 24, and through this processing device, the orientations of the packages 14a, 14b, 14c, 14d can be determined to record images. Instead of a CMOS camera, an IR camera, a stereo camera, or a LIDAR system and combinations thereof can also be used.

[0024] The handheld device 12 is configured for human-machine interaction. The handheld device 12 is very small and lightweight and can be held by a delivery agent with one hand. The handheld device 12 has an interface for wireless data transmission with the control unit 8 and has a battery for providing operating power. The handheld device 12 can be in the form of a mobile phone, a tablet computer, a laptop computer, or other portable devices, and can also be in the form of a barcode reader, for example.

[0025] Different from the present exemplary embodiment, different devices or devices of the delivery vehicle 4 (for example, an optical projector and / or an activatable lamp on a rack in the delivery vehicle 4) can also be provided to replace the handheld device 12 for human-machine interaction.

[0026] Now, reference will be made additionally to Figure 2 .

[0027] The loading space of the delivery vehicle 4 belonging to the system 2 is shown.

[0028] In the present exemplary embodiment, the delivery vehicle 4 is in the form of a van, a land vehicle that does not travel on a track. However, different from the present exemplary embodiment, the delivery vehicle 4 can be in the form of a land vehicle that travels on a track, or can also be in the form of a ship or an aircraft.

[0029] Inside the loading space of the delivery vehicle 4, a delivery vehicle camera array 16 such as the distribution center camera array 10 is arranged. In the present exemplary embodiment, the delivery vehicle camera array 16 includes a plurality of cameras 18a, 18b, 18c, 18d, 18e, 18f (such as CMOS cameras). In the loading space of the delivery vehicle 4, through these cameras, images of different sides 20a, 20b of the packages 14a, 14b, 14c, 14d are recorded and temporarily stored as a partial image data set TBD. Instead of a CMOS camera, an IR camera, a stereo camera, or a LIDAR system and combinations thereof can also be used.

[0030] For the tasks and functions described, System 2 and its components can have correspondingly configured hardware and / or software components. To this end, System 2 can also have one or more artificial neural networks.

[0031] An artificial neural network (ANN) has multiple artificial neurons, which, in the case of a deep neural network, are arranged in a number of hidden layers between an input layer and an output layer. Compared to a feedforward neural network (FFN), a recurrent neural network (RNN) is a neural network distinguished by the connection of neurons in one layer to neurons in the same or a previous layer. Thus, neurons in the same or different layers are fed back.

[0032] In this exemplary embodiment, training is performed through supervised learning. Different from this exemplary embodiment, training can also be performed through unsupervised learning, reinforcement learning, or stochastic learning.

[0033] During operation, in a first operation phase, packages 14a, 14b, 14c, 14d are sorted in the distribution center 24 and combined into groups according to the delivery vehicle 4. Images of each side 20a, 20b of the packages 14a, 14b, 14c, 14d are recorded to obtain image data. Then, the image data forms a reference image data set RDS in each case. For each of these images, an identification data set IDS is prepared through a deep learning algorithm (e.g., a trained siamese neural network). The identification data set ID is a compression of the reference image data set RDS or the image, where the identification data set IDS is a machine-readable binary string and thus cannot be interpreted by a human.

[0034] For example, multiple identification data sets IDS of each package 14a, 14b, 14c, 14d form a database (gallery).

[0035] In addition, symbol data (cartons, trademarks, labels, tapes) of the packages 14a, 14b, 14c, 14d are extracted from the reference image data set RDS in the distribution center 24 and temporarily stored as a reference symbol data set RSD, which later helps to re-identify the packages 14a, 14b, 14c, 14d, as described in detail below.

[0036] In a second operating phase, a spatial positioning data set RLD of the packages 14a, 14b, 14c, 14d is determined for each of the packages 14a, 14b, 14c, 14d during the loading and picking phase. For this purpose, a trained convolutional neural network (CNN) is used in the present exemplary embodiment. By means of the determined spatial positioning data set RLD, the sides 20a, 20b of the packages 14a, 14b, 14c, 14d are extracted. For example, in each case, one, two or three sides 20a, 20b form a partial image data set TBD, which is used to determine an assignment data set ZDS by means of the same trained siamese neural network.

[0037] In the present exemplary embodiment, a 3D envelope is then determined, by means of which each side 20a, 20b of the packages 14a, 14b, 14c, 14d is determined. Another advantage is that the dimensions of the packages 14a, 14b, 14c, 14d can thus be determined. This information helps to re-identify the packages 14a, 14b, 14c, 14d. In the present exemplary embodiment, using the dimensions and eight vertices, the spatial positioning data set RLD of the packages 14a, 14b, 14c, 14d can be determined by means of the PnP (Perspective n Point, see Li, Shiqi, Xu, Chi and Xie, Ming, A Robust O(n) Solution to the Perspective-n-Point Problem, 2012) algorithm.

[0038] For this purpose, a trained convolutional neural network can be used, which determines the midpoints as well as the eight corner points and the dimensions of the packages 14a, 14b, 14c, 14d and the 3D envelope. It is capable of detecting multiple packages 14a, 14b, 14c, 14d and is robust with respect to masks, background changes, changes in lighting conditions, etc. In the present exemplary embodiment, CenterPose (see Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A. Vela and Stan Birchfield, Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB image, 2021) will be used for this purpose.

[0039] Reference will also be made to Figure 3 。

[0040] After determining the spatial positioning data set RLD, based on the widths and heights of the packages 14a, 14b, 14c, 14d, each side 20a, 20b of the packages 14a, 14b, 14c, 14d is extracted and deformed into a rectangle. In this exemplary embodiment, an open-source tool using the opencv integrated function "WarpPerfective" is used for this purpose.

[0041] Now refer additionally to Figure 4 .

[0042] The reference image data set RDS and the partial image data set TBD are further analyzed to extract symbol data, and a reference symbol data set RSD is formed based on the RDS in the reference image data set, and a symbol data set SDS is formed based on the TBD in the partial image data set.

[0043] By means of intelligent alignment technology, the assignment data set ZDS, the identification data set IDS, the reference symbol data set RSD, and the symbol data set SDS are analyzed to identify the packages 14a, 14b, 14c, 14d.

[0044] For this purpose, a deep learning algorithm can be used, such as a trained neural feedforward network. The assignment data set ZDS and the identification data set IDS are fed into the trained neural feedforward network on the input side, and the assignment data set ZDS is provided by the trained neural feedforward network on the output side.

[0045] A static-based algorithm or a deterministic algorithm can also be used, which checks the maximum interference of the combined probability of the image data and the symbol data. Thus, for example, the product of the probabilities associated with the similarity between the image data and the symbol data of multiple sides 20a, 20b is compared.

[0046] Now refer additionally to Figure 5 .

[0047] In this exemplary embodiment, another trained neural network 22 is used to re-identify packages 14a, 14b, 14c, 14d. This neural network 22 converts images of sides 20a, 20b (i.e., the reference image dataset RDS) into an identification dataset IDS (see Zheng, Zhedong, Zheng, Liang, and Yang, Yi, A Discriminatively Learned CNN Embedding for Person Reidentification, 2018) and uses a neural network with a backbone architecture (e.g., ResNet or MobileNet) to extract features. The artificial neural network 22 provides an output vector with a vector size of 128 nodes on the output side as output. Different from this exemplary embodiment, the vector size of the output vector can also be different, for example, another value with a power of 2 (e.g., 1024) so that the output vector has sufficient information content for re-identification.

[0048] The output vector includes all important visual information of sides 20a, 20b of packages 14a, 14b, 14c, 14d. Images representing the same sides 20a, 20b of packages 14a, 14b, 14c, 14d should generate similar identification datasets IDS. In contrast, images representing different packages 14a, 14b, 14c, 14d should generate very different identification datasets IDS.

[0049] The similarity of the identification dataset IDS can be determined by means of a standard distance metric (e.g., cosine similarity or Euclidean distance). With this metric, the database can be searched to determine the best-matching assignment dataset ZDS.

[0050] In this exemplary embodiment, additional data is obtained by additionally using symbolic data determined by analyzing the reference image dataset RDS and a partial image dataset TBD. For this purpose, a trained neural network is used in this exemplary embodiment, which provides symbolic data for each side 20a, 20b to limit the search space. For example, in cases where a package logo, barcode, label, color, strap, shape, or trademark can be determined, the search space can be limited to the relevant packages 14a, 14b, 14c, 14d.

[0051] In this exemplary embodiment, Yolo is used for this purpose, but other tools, such as Faster R-CNN, can also be used.

[0052] Now, with additional reference to Figure 6 the sequence of the operating method of system 2 will be explained.

[0053] In an initial step, during the training phase, in this exemplary embodiment, the neural network mentioned so far is trained with training data by supervised learning. In this exemplary embodiment, 725 images of packages 14a, 14b, 14c, 14d are used as a base, all of which have a uniform green image background. To expand the base, the image background is changed, that is, the green background is replaced with a random background.

[0054] In a first step S100, packages 14a, 14b, 14c, 14d are sorted in the distribution center 24.

[0055] In a further step S200, the packages 14a, 14b, 14c, 14d of the delivery vehicle 4 are combined into groups.

[0056] In a further step S300, a reference image data set RDS of the packages 14a, 14b, 14c, 14d is generated.

[0057] In a further step S400, the reference image data set RDS is analyzed to determine an identification data set IDS of the packages 14a, 14b, 14c, 14d, and the identification data set IDS is temporarily stored in the cloud service 6.

[0058] In a further step S500, the symbol data of the reference image data set RDS is extracted.

[0059] In a further step, the packages 14a, 14b, 14c, 14d are loaded into the delivery vehicle 4.

[0060] In a further step S600, a spatial location data set RLD of the packages 14a, 14b, 14c, 14d is determined.

[0061] In a further step S700, the sides 20a, 20b of the packages 14a, 14b, 14c, 14d are determined.

[0062] In a further step S800, a partial image data set TBD of the packages 14a, 14b, 14c, 14d in the delivery vehicle 4 is generated in the cloud service 6.

[0063] In a further step S900, the partial image data set TBD is evaluated in response to a request signal AFS, for example, from a delivery agent, in order to determine an allocation data set ZDS of the packages 14a, 14b, 14c, 14d.

[0064] In a further step S1000, the partial image data set TBD is evaluated to extract symbol data.

[0065] In a further step S1100, the allocated data set ZDS, the identification data set IDS, the reference symbol data set RSD, and the symbol data set SDS are compared with each other to identify the packages 14a, 14b, 14c, 14d.

[0066] In a further step S1200, a position data set PDS related to the current positions of the packages 14a, 14b, 14c, 14d in the delivery vehicle 4 is generated and the position data set PDS is transmitted to the handheld device 12 or another output device, which subsequently provides a corresponding acoustic and / or visual output in order to inform the delivery agent of the current positions of the packages 14a, 14b, 14c, 14d.

[0067] Different from this exemplary embodiment, the order of the steps can also be different. In addition, multiple steps can also be carried out simultaneously or in parallel. In addition, different from this exemplary embodiment, individual steps can also be skipped or omitted.

[0068] The method allows for the re-identification of the packages 14a, 14b, 14c, 14d in question based on the reference image data set RDS by comparison with a partial image data set TBD. In other words, the fact is exploited that only a part of the data used for the initial identification is necessary for the re-identification of the packages 14a, 14b, 14c, 14d.

[0069] List of reference signs 2 System 4 Delivery vehicle 6 Cloud service 8 Control unit 10 Allocation center camera array 12 Handheld device 14a Package 14b Package 14c Package 14d Package 16 Delivery vehicle camera array 18a Camera 18b Camera 18c Camera 18d Camera 18e Camera 18f Camera 20a Side 20b Side 22 Neural network 24 Allocation center AFS request signal IDS Identification data set PDS Position data set RDS Reference image data set RLD Spatial Location Dataset RSD Reference Symbol Dataset SDS Symbol Dataset TBD Partial Image Dataset ZDS Allocation Dataset S100 Step S200 Step S300 Step S400 Step S500 Step S600 Step S700 Step S800 Step S900 Step S1000 Step S1100 Step S1200 Step

Claims

1. A method for operating a delivery vehicle, comprising the steps of: Generate a reference image dataset of the package; analyzing the reference image dataset to determine an identification dataset of the package, and temporarily storing the identification dataset; generating a partial image dataset of the package in the delivery vehicle; determining a spatial location data set indicating the spatial location of the package; analyzing at least a portion of the image dataset using the spatial positioning dataset to determine an allocation dataset for the package; as well as The allocation data set is compared to the identification data set to re-identify the package.

2. The method according to claim 1, further comprising the steps of: The package is identified and, upon identification, a location data set is generated that is associated with a current location of the package in the delivery vehicle.

3. The method according to claim 1, wherein the identification data set and the allocation data set are based on image data. 4 . The method according to claim 1 , further comprising determining symbol data and analyzing the symbol data by analyzing the reference image data set and the partial image data set.

5. A system for operating a delivery vehicle, wherein the system is configured to generate a reference image dataset of a package, comprising: analyzing the reference image dataset to determine an identification dataset of the package, and temporarily storing the identification dataset; generating a partial image dataset of the package in the delivery vehicle; determining a spatial location data set indicating the spatial location of the package; analyzing at least a portion of the image dataset using the spatial positioning dataset; determining a distribution data set for the package; as well as The allocation data set is compared to the identification data set to re-identify the package.

6. The system of claim 5, wherein the system is configured to, upon identifying the package, generate a location data set associated with the current location of the package in the delivery vehicle.

7. The system of claim 5, wherein the identification data set and the allocation data set are based on image data.

8. The system of claim 5, wherein symbol data is also determined and analyzed by analyzing the reference image data set and the partial image data set.

9. The system of claim 5, wherein the reference image dataset includes images of all sides of the package.

10. A vehicle comprising: one or more processors; as well as A memory storing executable instructions that, when executed by the one or more processors, cause the system to: Generate a reference image dataset of the package; analyzing the reference image dataset to determine an identification dataset of the package, and temporarily storing the identification dataset; generating a partial image dataset of the package in the delivery vehicle; determining a spatial location data set indicating the spatial location of the package; analyzing the partial image dataset using the spatial positioning dataset to determine an allocation dataset for the package; as well as The allocation data set is compared to the identification data set to re-identify the package.

11. The vehicle of claim 10, wherein the system is configured to, upon identifying the package, generate a location data set associated with a current location of the package in the delivery vehicle.

12. The vehicle of claim 10, wherein the recognition data set and the allocation data set are based on image data.

13. The vehicle of claim 10, wherein symbol data are also determined and analyzed by analyzing the reference image data set and the partial image data set.

14. The vehicle of claim 10, wherein the reference image dataset includes images of all sides of the package.

15. The vehicle of claim 10, further comprising transmitting the spatial location of the package to a handheld device associated with the vehicle.

Images