Determination of allocation and / or sorting information for automated distribution and / or sorting of shipments

Abstract

In particular, a method is disclosed, which is performed by at least one device (1), wherein the method includes: - acquiring (20) image data, wherein the image data represents an image (3) of a surface of a shipment captured by an image sensor; - determining (21) metadata associated with the image data, at least in part based on the image data, wherein the metadata represents a plurality of image features of the image (3) represented by the image data, wherein each of the image features represented by the metadata is an image component of the image represented by the image data; and - determining (23) allocation information and / or sorting information for automatically allocating and / or sorting shipments, at least in part based on the image features represented by the metadata.

Description

Technical Field

[0001] Exemplary embodiments of the present invention relate to a method, an apparatus, a system, and a computer program associated with determining allocation information and / or sorting information for the automated distribution and / or sorting of shipments. Background Technology

[0002] Methods are known in the prior art for automatically identifying and analyzing characters contained in the address area of ​​a mail item in order to enable automatic sorting and distribution of the mail item. Automatic identification and analysis are typically not possible when characters contained in the address area cannot be captured or are not fully captured (e.g., because the address area is partially obscured or pasted around a corner of the mail item) or when these characters are, for example, Arabic, Cyrillic, Greek, or Asian characters (e.g., Thai, Chinese, or Japanese characters), thus necessitating manual analysis (e.g., using a translator) in these cases. Summary of the Invention

[0003] Therefore, the present invention specifically aims to overcome the aforementioned disadvantages of the prior art.

[0004] According to the present invention, a method is disclosed, which is performed by at least one device, wherein the method includes:

[0005] - Obtain image data, which represents images of the surface of the shipment captured by an image sensor.

[0006] - Based at least in part on these image data, determine the metadata associated with these image data, wherein these metadata represent multiple image features of the image represented by these image data, wherein each of the image features represented by these metadata is an image component of the image represented by these image data, and

[0007] - The allocation and / or sorting information for the automatic allocation and / or sorting of shipments is determined based at least in part on the image features represented by these metadata.

[0008] The idea that the method is performed by at least one device should be understood, for example, to mean that all the steps of the method are performed either by the same device (e.g., one of the devices disclosed below) or at least in part by different devices (e.g., multiple devices of the devices disclosed below).

[0009] According to the present invention, an apparatus is disclosed, wherein the apparatus includes a mechanism configured to perform the disclosed method, or the apparatus includes a corresponding mechanism for performing steps of the disclosed method. Here, one or more steps may be performed by the same mechanism, or different steps may be performed by different mechanisms. The mechanism of the disclosed apparatus may include hardware components and / or software components.

[0010] The apparatus may, for example, include at least one memory having program instructions of a computer program (such as the computer program disclosed below) and at least one processor designed to execute the program instructions from the at least one memory. Accordingly, it should also be understood that an apparatus comprising at least one processor and at least one memory having program instructions, wherein the at least one memory and the program instructions are configured, together with the at least one processor, to cause the apparatus to perform and / or control the disclosed methods.

[0011] Furthermore, the device may include, for example, a communication interface (e.g., a wireless and / or wired network adapter) and / or a user interface (e.g., a keyboard, mouse, screen, touchscreen, speaker, microphone, camera, etc.) and / or an image sensor (e.g., a CMOS sensor, CCD sensor, or camera). It should be understood that, alternatively or additionally, the device may include other mechanisms.

[0012] For example, the device is a server. Such a server can be either a physical server (i.e., a server with hardware and / or software components) or a virtual server. Here, a virtual server should be understood, for example, as server functionality provided by the hardware and / or software components of one or more physical servers (e.g., multiple servers in a so-called server cloud). Multiple physical servers can, for example, work together to provide the server functionality of such a virtual server.

[0013] Furthermore, according to the present invention, a system is disclosed comprising a plurality of devices configured to jointly perform the disclosed method. One of these devices is, for example, one of the devices disclosed above (e.g., a server), and another of these devices is, for example, an automated sorting and / or distributing device.

[0014] Furthermore, according to the present invention, a computer program is disclosed, wherein the computer program comprises program instructions designed to cause a device (e.g., the disclosed device) to perform the disclosed method when executed by at least one processor.

[0015] The disclosed computer program may be contained in and / or stored on a computer-readable storage medium. A computer-readable storage medium should be understood, for example, as a physical and / or tactile storage medium.

[0016] The disclosed methods, apparatus, systems, and computer programs are used, for example, to determine allocation information and / or sorting information for the automated distribution and / or sorting of shipments.

[0017] The disclosed methods, apparatus, system and computer program will be described below (partly by way of example).

[0018] The items being sent are, for example, single parcels, parcels (e.g., packages), and / or letters (e.g., letters). An address area is typically present on the surface of such items, containing characters identifying the recipient and / or sender of the item. Here, the surface of the item should be understood as the externally visible surface of the item, particularly its packaging (e.g., parcels).

[0019] For example, image data acquired by at least one apparatus performing the disclosed method should be understood as follows: such image data is received by a remote apparatus (e.g., via a wireless or wired communication connection) or generated by an image sensor (which is part of the apparatus performing the disclosed method). Accordingly, the image sensor may, for example, be part of at least one apparatus performing the method, or may be part of an apparatus remote from the at least one apparatus performing the method.

[0020] These image data represent, for example, images of the surface of the object captured by the image sensor in coded and / or digital form (e.g., as grid images and / or bitmaps). These image data can, in particular, represent images of the surface of the object captured by the image sensor according to standardized image data formats. Examples of standardized image data formats for grid images and / or bitmaps are the JPEG File Exchange Format (JFIF) or the Tag Image File Format (TIFF). The JFIF image data format is specifically specified by ITU-T Recommendation T.871, which is currently available at https: / / www.itu.int / rec / T-REC-T.871; and the specification for the TIFF data image format is currently available in version 6.0 at https: / / www.adobe.io / open / standards / TIFF.html.

[0021] This image data is generated by the image sensor, for example, as a result of capturing the surface of the shipment, in such a way that the image data represents an image of the surface of the shipment captured by the image sensor. It should be understood that the image sensor can capture the entire surface or a portion of the surface of the shipment. Accordingly, the image represented by the image data can show the entire surface or a portion of the surface of the shipment.

[0022] The metadata associated with these image data is determined at least in part based on these image data. In other words, these image data are taken into account when determining the metadata associated with them. Here, determining the metadata associated with these image data includes, for example, applying image processing methods and / or image processing algorithms (e.g., segmentation methods and / or segmentation algorithms as detailed below) to these image data or the image represented by these image data.

[0023] Here, this metadata is associated with the image data in such a way that it represents multiple image features of the image represented by the image data. The metadata associated with the image data may, for example, be added to the image data and / or stored together with it. It should be understood that this metadata may also be separate from the image data and / or may be stored separately from it.

[0024] Image components of an image represented by these image data are, for example, each element (e.g., one or more pixels) and / or segment (e.g., a set of pixels) contained in the image, such as edges, corners, regions, prominent locations, or patterns. Image features should be understood, for example, as image components characterizing the image represented by the image data and / or a (e.g., a pre-given) image locality of the image represented by the image data. For example, such image features (e.g., each of these image features) are segments contained in the image and / or a (e.g., a pre-given) image locality of the image, such as edges, corners, regions, prominent locations, or patterns. Accordingly, determining metadata includes, for example, determining image features based at least in part on the image data. These image features may be obtained, for example, as a result of applying segmentation methods and / or segmentation algorithms to these image data and / or as an image represented by these image data. Examples of such segmentation methods and / or algorithms include pixel-based segmentation methods (e.g., thresholding methods), edge-based segmentation methods (e.g., edge detection methods such as the Sobel operator, Scharr operator, or Laplace filter), region-based segmentation methods (e.g., region growing methods), model-based segmentation methods (e.g., Hough transform), and detection methods for detected salient locations (e.g., detection methods based on SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Feature) algorithms). It should be understood that the present invention is not limited to these segmentation methods and / or algorithms.

[0025] Furthermore, determining the metadata includes, for example, determining representations of image features. This metadata represents multiple image features of an image represented by image data, and should be understood, for example, that for each of these image features, the metadata represents a corresponding characteristic of the corresponding image feature (e.g., the location, shape, size, color, or orientation of an image component) or a characteristic of the corresponding image feature's corresponding surrounding environment. For this purpose, for each of these image features, the metadata can be represented in vector form, which contains and / or describes one or more characteristics of the corresponding image feature. For example, such vectors can be obtained as the result of applying methods and / or algorithms to these image data and / or the image represented by these image data. Examples of such methods and / or algorithms are methods and / or algorithms for describing prominent locations, such as the SIFT (Scale Invariant Feature Transform) algorithm or the SURF (Speed-Up Robust Feature Transform) algorithm. Accordingly, these image features can be represented by the metadata in vector form as SIFT image features or SURF image features. It should be understood that the invention is not limited to these methods and / or algorithms.

[0026] Subsequently, allocation information and / or sorting information for the automated allocation and / or sorting of shipments are determined, at least in part, based on the image features represented by these metadata. In other words, the image features represented by these metadata are taken into account when determining the allocation information and / or sorting information for the automated allocation and / or sorting of shipments. Here, the allocation information and / or sorting information for the automated allocation and / or sorting of shipments is determined according to an algorithm and / or data model (e.g., pre-given). Such an algorithm and / or such a data model, for example, takes the image features represented by these metadata as input parameters. The algorithm and / or the data model may, for example, be pre-given: what allocation information and / or sorting information should be determined for the image features represented by these metadata.

[0027] For this purpose, the algorithm (e.g., a classifier and / or classification method) and / or the data model (e.g., a database or artificial neural network) may, for example, take into account image features of allocation and / or sorting information determined for other shipments, as well as images of the surfaces of these other shipments. In other words, the data model may be based on image features of allocation and / or sorting information determined for other shipments, as well as images of the surfaces of these other shipments. In particular, the data model may not include information relating to the shipment for which allocation and / or sorting information is currently determined for automatic allocation and / or sorting.

[0028] For example, allocation and / or sorting information determined for other shipments can be stored in database entries along with image features of images of the surfaces of these other shipments, thereby enabling comparison of image features represented by the database with image features represented by metadata.

[0029] When a comparison is made where the image features represented by the database for a given entry are at least substantially similar to or correspond to the image features represented by the metadata (e.g., based on similarity and / or a pre-given similarity threshold), it can be proposed, for example, that allocation and / or sorting information stored in the entries of the database along with the representations of these image features is assigned to the image features represented by the metadata and is determined as allocation and / or sorting information for the automated allocation and / or sorting of shipments. Such a comparison can be made, for example, according to comparison methods used to compare image features or their representations (e.g., comparison methods for comparing image features in vector form (e.g., SIFT or SURF image features) where the distance between corresponding vectors (e.g., Euclidean distance) is used as a similarity measure (i.e., the smaller the distance between two vectors, the more similar the image features represented by those vectors).

[0030] Accordingly, based on the image features represented by these metadata, allocation information and / or sorting information for the automated distribution and / or sorting of mail can be determined. For example, as a result of the comparison, image features stored in the database that are at least substantially similar to or correspond to the image features represented by these metadata can be obtained. Because these image features are determined independently of the identification and analysis of characters contained in the address area, this comparison result is independent of whether characters contained in the address area cannot be captured by the image sensor or cannot be fully captured by the image sensor (e.g., because the address area is partially obscured or pasted around the corner of the mail) or whether these characters are, for example, Arabic, Cyrillic, Greek, or Asian characters (e.g., Thai, Chinese, or Japanese characters), thereby overcoming the aforementioned disadvantages described in conjunction with the prior art.

[0031] Allocation and / or sorting information for automating the allocation and / or sorting of mail items is configured, for example, to influence (e.g., control) the allocation and / or sorting of mail items through automated allocation and / or sorting devices, such as to sort the mail item to another mail item that should be delivered in the same delivery area as the mail item. This allocation and / or sorting information may be provided (e.g., printed) on the surface of the mail item, for example, in text form and / or in the form of a QR code. Examples of such QR codes are alphanumeric codes, barcodes, or QR codes. An example of providing information in text form is an address description in a pre-given language (e.g., the official language of the delivery area).

[0032] In one exemplary embodiment of the present invention, at least one of these image features is at least one of the following image components:

[0033] -The corresponding region contained in this image,

[0034] -The corresponding corners contained in this image,

[0035] -The corresponding edges contained in the image,

[0036] - The corresponding prominent position contained in the image, and / or

[0037] - The corresponding pattern contained in the image.

[0038] It should be understood that multiple or each of these image features may be at least one of these image components.

[0039] As disclosed above, each of these image components is a corresponding segment contained in the image, which can be obtained as a result of a segmentation method and / or segmentation algorithm applied to the image.

[0040] For example, the corresponding areas, corners, edges, prominent positions, and / or patterns contained in the image may be the address area of ​​the mailing or a part of the address area of ​​the mailing. For example, when determining metadata (e.g., when determining image features within the scope of determining metadata), only the address area of ​​the mailing or components that are part of the address area of ​​the mailing are considered. It should be understood that, instead of the address area of ​​the mailing, or in addition, other image parts of the image may also be considered when determining metadata (e.g., when determining image features within the scope of determining metadata).

[0041] In one exemplary embodiment of the present invention, for at least one of these image features, the metadata represents at least one of the following characteristics of the corresponding image feature or its surrounding environment:

[0042] - The size of the corresponding image features (e.g., in the form of area description and / or pixel count, which are part of the image features),

[0043] - The shape and / or contour of the corresponding image features (e.g., in the form of a length description of the surrounding environment and / or the number of pixels adjacent to the image feature, and / or in the form of a Fourier descriptor),

[0044] -The location of the corresponding image features

[0045] - The color of the corresponding image feature or the color in the surrounding environment of the corresponding image feature.

[0046] - The orientation of the corresponding image features or their surrounding environment,

[0047] - The geometric centroid (Schwerpunkt) of the corresponding image features.

[0048] - The momentum of the corresponding image feature or its surroundings (e.g., mass (e.g., in the form of the sum of gray values ​​of pixels that are part of the image feature), or the centroid of the image feature (e.g., in the form of the average mass and / or the average product of position and mass).

[0049] When image features are, for example, an address region of a mail item or a portion thereof, these metadata can represent, for example, the geometric and / or topological characteristics of the characters contained in that address region, such as:

[0050] - The area used by the character (e.g., in the form of an area description and / or the number of pixels),

[0051] - The quality of the character (e.g., in the form of the sum of all grayscale values ​​of the pixels that make up a part of the character).

[0052] - The surrounding environment of the characters (e.g., in the form of the number of all pixels adjacent to these characters),

[0053] - The centroid of a character (e.g., in the form of the average product of position and mass).

[0054] Here, for example, it is assumed that all patterns, corners, and / or edges (which are part of the address area) correspond to characters, such that these metadata represent, for example, the geometric and / or topological characteristics of all patterns, corners, and / or edges contained in the address area, in order to represent the geometric and / or topological characteristics of the characters contained in the address area.

[0055] It should be understood that the present invention is not limited to the disclosed characteristics, and additionally or alternatively, these metadata may also represent other characteristics (especially geometric and / or topological characteristics) of the corresponding image features or their surrounding environment. Preferably, these characteristics are rotation-invariant and / or mirror-invariant and / or translation-invariant.

[0056] It should be understood that, for one or more of these image features, these metadata may represent at least one of these characteristics of the corresponding image feature or its surrounding environment.

[0057] Accordingly, determining the metadata includes determining the characteristics of the corresponding image features. This determination of the characteristics of the corresponding image features is performed, for example, within the scope of the disclosed methods and / or algorithms used to describe prominent locations, such as the SIFT (Scale Invariant Feature Transform) algorithm or the SURF (Speed-Up Robust Feature Transform) algorithm. Accordingly, these image features can be represented by metadata as SIFT image features or SURF image features.

[0058] In one exemplary embodiment of the invention, these metadata represent one or more image features in vector form. For example, these metadata represent each of these image features in the form of a corresponding vector.

[0059] As disclosed above, such vectors can contain and / or describe one or more properties of the corresponding image features. For example, such vectors can be obtained as the result of applying methods and / or algorithms to these image data and / or as an image represented by these image data. Examples of such methods and / or algorithms are those used to describe prominent locations, such as the SIFT (Scale Invariant Feature Transform) algorithm or the SURF (Speed-Up Robust Feature Transform) algorithm. Accordingly, these image features can be represented by metadata in vector form as SIFT image features or SURF image features.

[0060] Furthermore, in one exemplary embodiment of the present invention, the method includes:

[0061] - The allocation information and / or sorting information are provided in such a way that it enables and / or facilitates the automated sorting and / or allocation of the shipment, at least in part, based on the allocation information and / or sorting information.

[0062] When allocation and / or sorting information can be obtained (e.g., captured and / or received) by automated allocation and / or sorting devices for distributing and / or sorting shipments, this allocation and / or sorting information should be understood, for example, to be provided in a way that enables and / or facilitates automated sorting and / or allocation of shipments, at least in part, based on this allocation and / or sorting information. For this purpose, the allocation and / or sorting information may be provided, for example, in text form (e.g., as an address description in a pre-given language) and / or in the form of a QR code (e.g., printed) on the surface of the shipment. Examples of such QR codes are alphanumeric codes, barcodes, or QR codes. Alternatively or additionally, the allocation and / or sorting information may be sent to automated allocation and / or sorting devices.

[0063] Therefore, providing the allocation information and / or sorting information includes, for example, printing the allocation information and / or sorting information on the surface of the shipment, and / or sending the allocation information and / or sorting information to a remote device (e.g., an automated allocation device and / or sorting device).

[0064] Furthermore, in one exemplary embodiment of the present invention, the method includes:

[0065] - To automatically allocate and / or sort the shipment, at least in part, based on the allocation and / or sorting information.

[0066] For example, automated distribution and / or sorting of shipments can be performed by automated distribution and / or sorting devices.

[0067] This should be understood, for example, as taking into account allocation and / or sorting information when automating the allocation and / or sorting of mail items (e.g., via automated allocation and / or sorting devices), thereby influencing (e.g., controlling) the allocation and / or sorting of mail items (e.g., via automated allocation and / or sorting devices). As disclosed above, the allocation and / or sorting information for automating the allocation and / or sorting of mail items is configured, for example, to influence (e.g., control) the allocation and / or sorting of mail items via automated allocation and / or sorting devices, for example, to sort the mail item to another mail item that should be delivered in the same delivery area as the mail item.

[0068] Furthermore, in one exemplary embodiment of the present invention, the method includes:

[0069] - Provide a data model for determining allocation and / or sorting information for the automated distribution and / or sorting of the shipment, or

[0070] - Access the data model used to determine the allocation and / or sorting information for the automated distribution and / or sorting of the shipment.

[0071] Here, providing a data model should be understood, for example, as storing the data model in the memory of the at least one device performing the method; and accessing a data model should be understood, for example, as the at least one device performing the method (e.g., via a wireless and / or wired communication connection) accessing the data model stored by a device remote from the at least one device performing the method.

[0072] For example, the allocation information and / or sorting information used for the automated distribution and / or sorting of the shipment is determined based at least in part on the data model. In other words, the data model can be taken into account when determining the allocation information and / or sorting information used for the automated distribution and / or sorting of the shipment.

[0073] For example, the data model will assign allocation and / or sorting information for the shipment to image features represented by these metadata for the purpose of automatically allocating and / or sorting the shipment.

[0074] The data model is, for example, a database. As disclosed above, allocation and / or sorting information determined for other shipments can be stored in entries of the database along with representations of image features of the surfaces of these other shipments, thereby enabling comparison of the image features represented by the database with those represented by metadata. When a comparison is made such that the image features represented by the database in an entry in the database are at least substantially similar to or correspond to the image features represented by the metadata (e.g., based on similarity and / or a pre-given similarity threshold), it can be proposed, for example, that the allocation and / or sorting information stored in the entries of the database along with the representations of these image features is assigned to the image features represented by the metadata and is determined as allocation and / or sorting information for the automated allocation and / or sorting of shipments.

[0075] Alternatively or additionally, this data model can be obtained through machine learning. An example of such a data model obtained through machine learning is an artificial neural network. The data model is obtained, for example, as a result of a learning phase based on an algorithm used for machine learning (e.g., an algorithm based on deep learning methods). In this learning phase, the data model is trained, for example, by means of image features obtained from determining allocation and / or sorting information of other shipments and images of the surfaces of these other shipments.

[0076] Such a data model obtained through machine learning (e.g., an artificial neural network) takes image features represented by these metadata as input parameters and outputs allocation information and / or sorting information assigned to these image features by the data model as output parameters for the automated allocation and / or sorting of shipments.

[0077] In one exemplary embodiment of the present invention, the metadata is determined independently of identifying characters contained in the image represented by these image data and / or independently of the meaning of the characters contained in the image represented by these image data. Attached Figure Description

[0078] Further advantageous exemplary embodiments of the invention can be derived from the following detailed description of some exemplary embodiments of the invention, particularly in conjunction with the accompanying drawings. However, the drawings are for illustrative purposes only and are not intended to define the scope of the invention. These drawings are not necessarily to scale and should only exemplarily reflect the overall concept of the invention. In particular, features included in the drawings should in no way be considered essential components of the invention.

[0079] In the attached diagram:

[0080] Figure 1 A schematic illustration showing an exemplary embodiment of the device according to the invention;

[0081] Figure 2 A flowchart illustrating an exemplary embodiment of the method according to the present invention is shown;

[0082] Figure 3a , 3b A schematic illustration of an image represented by image data according to an exemplary embodiment of the present invention;

[0083] Figure 4 A schematic diagram illustrating an exemplary embodiment of the database according to the present invention is shown;

[0084] Figure 5 An exemplary implementation of the storage medium is shown. Detailed Implementation

[0085] Figure 1 This is a schematic illustration of an exemplary embodiment of the device 1 according to the present invention.

[0086] The device 1 exemplarily includes a processor 10, and includes a first memory as a program memory 11, a second memory as a main memory 12, and an image sensor 13 connected to the processor 10. Additionally, the device 1 may optionally include a communication interface 14.

[0087] The processor (e.g., processor 10) should be understood as, for example, a microprocessor, microcontroller, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA). It should be understood that the device 1 may also include multiple processors 10.

[0088] Processor 10 executes program instructions stored in program memory 11 and, for example, stores intermediate results in main memory 12. Program memory 11 contains program instructions, such as those of a computer program, which, when executed by processor 10, cause processor 10 to perform and / or control disclosed methods (e.g., according to...). Figure 2 The process shown in Figure 2 (Method). Furthermore, program memory 11 may be included in... Figure 4 Database 4 is shown in the image.

[0089] The program memory 11 also contains, for example, the operating system of device 1, which is at least partially loaded into main memory 12 and executed by processor 10 when device 1 is started. Specifically, when device 1 is started, at least a portion of the kernel of the operating system is loaded into main memory 12 and executed by processor 10. Examples of operating systems are Windows, UNIX, Linux, Android, Apple iOS, and / or macOS. The operating system, in particular, enables data processing using device 1. This operating system, for example, manages operating mechanisms (e.g., main memory and program memory), provides functionality based on other computer programs, particularly through a programming interface, and controls the execution of computer programs.

[0090] Program memory (e.g., program memory 11) is, for example, non-volatile memory, such as flash memory, magnetic memory, EEPROM (electrically erasable, programmable read-only memory), and / or optical memory. Main memory (e.g., main memory 12) is, for example, volatile or non-volatile memory, especially memory with random access (RAM), such as static RAM (SRAM), dynamic RAM (DRAM), ferroelectric RAM (FeRAM), and / or magnetic RAM (MRAM).

[0091] Main memory 12 and program memory 11 can also be designed as a single memory. Alternatively, main memory 12 and / or program memory 11 can each be composed of multiple memories. Furthermore, main memory 12 and / or program memory 11 can also be part of processor 10.

[0092] Image sensor 13 is configured, for example, to capture the surface of the mailed item and to generate image data representing an image of the surface of the mailed item captured by image sensor 13 (e.g., Figure 3a Image 3 is shown. With the aid of image sensor 13, device 1 can acquire image data accordingly. Examples of such image sensors are CMOS sensors, CCD sensors, or cameras.

[0093] The communication interface 14 of device 1 can be a wireless or wired communication interface. The communication interface 14 is configured, for example, to communicate with a remote device via a wireless and / or wired communication connection. A wired communication connection should preferably be understood as a communication connection via a wired communication network (e.g., an Ethernet communication network). Ethernet is specified, for example, in the IEEE-802.3 family of standards. Examples of wireless communication connections are those based on wireless communication technologies such as Wireless Local Area Networks (WLANs) or 2G / 3G / 4G / 5G. WLANs are specified, for example, in the IEEE-802.11 family of standards. 2G / 3G / 4G / 5G radio specifications are managed and developed by the 3rd Generation Partnership Project (3GPP) and are currently available on the Internet, particularly at www.3gpp.com.

[0094] With the aid of communication interface 14, device 1 can communicate with a remote device, for example, the image data is received (i.e., acquired) by the remote device and / or access a database stored in the memory of the remote device.

[0095] The components 10 to 14 of device 1 are interconnected in communication and / or operatively, for example, via one or more bus systems (e.g., one or more serial and / or parallel bus connections).

[0096] It should be understood that, in addition to components 10 to 14, device 1 may also include other components, such as a user interface.

[0097] Figure 2 A flow diagram illustrating an embodiment of the method according to the present invention is shown. Figure 2 The following exemplarily illustrates method steps 20 to 24. Here, it is assumed that, based on the above combination... Figure 1 The disclosed apparatus 1 is used to perform method steps 20 to 24.

[0098] In step 20, device 1 acquires image data, wherein the image data represents an image of the surface of the object captured by image sensor 13.

[0099] For example, image sensor 13 captures the surface of the object and generates image data in step 20 representing an image of the surface of the object captured by image sensor 13. The image data generated by image sensor 13 can represent the image of the surface of the object captured by image sensor 13 according to a standardized image data format. Examples of standardized image data formats for grid images and / or bitmaps, as disclosed above, are JPEG File Exchange Format (JFIF) or Tag Image File Format (TIFF).

[0100] Figure 3a This is a schematic illustration of image 3, representing the image data obtained in step 20, according to an exemplary embodiment of the present invention. It is assumed that: Figure 3a Image 3 shows the surface of the mail item captured by image sensor 13 in step 20. Image 3 specifically includes the address area 30 of the mail item. The address area 30 contains characters and markings.

[0101] In step 21, device 1 determines metadata associated with the image data based at least in part on the image data obtained in step 20. Here, the metadata represents multiple image features of the image 3 represented by the image data; and each of the image features represented by the metadata is an image component of the image 3 represented by the image data.

[0102] As disclosed above, image features should be understood, for example, as image components that characterize image 3 represented by image data and / or a (e.g., pre-given) image portion of image 3 represented by image data. For example, such image features (e.g., each of these image features) are segments, such as edges, corners, regions, prominent locations, or patterns, contained in the image and / or in a (e.g., pre-given) image portion of the image.

[0103] Accordingly, determining the metadata in step 21 includes, for example, determining image features based at least in part on the image data. These image features may be obtained as a result of applying segmentation methods and / or segmentation algorithms to the image data and / or as an image represented by the image data. The segmentation methods and / or segmentation algorithms disclosed above are exemplary segmentation methods for pixels (e.g., thresholding methods), segmentation methods for edges (e.g., edge detection methods, such as the Sober operator, Schar operator, or Laplacian filter), segmentation methods for regions (e.g., region growing methods), model-based segmentation methods (e.g., Hough transform), and detection methods for detected salient locations (e.g., detection methods based on SIFT (Scale Invariant Feature Transform) or SURF (Speed-Up Robust Feature Transform) algorithms).

[0104] exist Figure 3bThe diagram schematically illustrates the result of applying such a segmentation method and / or segmentation algorithm to the image data obtained in step 20 and / or the image 3 represented by such image data. Figure 3b A magnified portion of image 3 with address region 30 is displayed. For example, in Figure 3b The circle encloses the image features obtained as a result of applying segmentation methods and / or segmentation algorithms to image data and / or image 3 represented by image data. Figure 3b The image features circled in the image are a portion of address region 30. This is, for example, where, in step 21, when determining image features within the scope of determining metadata, only the address region of the mail item or an image component that is part of address region 30 is considered, for example, by applying the segmentation method and / or the segmentation algorithm only to the portion of the image data obtained in step 20 representing address region 30, or to a portion of image 3 representing address region 30. Figure 3b The image features circled in the image are, by way of example, the edges, corners, and other prominent locations of the image region representing address area 30 in image 3. It should be understood that these image features are merely exemplary, and other image features of image 3 may also be obtained as a result of applying the segmentation method and / or segmentation algorithm to the image data obtained in step 20 and / or as a representation of image 3 by the image data.

[0105] These metadata represent multiple image features of image 3 represented by image data. For example, they should be understood as representing, for each of these image features, a corresponding characteristic of the corresponding image feature (e.g., the location, shape, size, color, or orientation of an image component) or a characteristic of the corresponding image feature's surrounding environment. Accordingly, determining the metadata in step 21 includes, for example, determining a representation of the image features or each of these image features.

[0106] An example of such a representation of the corresponding image features is a vector containing and / or describing one or more properties of the corresponding image features. For example, such a vector can be obtained as the result of applying methods and / or algorithms to these image data and / or as an image represented by these image data. Examples of such methods and / or algorithms disclosed above are methods and / or algorithms for describing prominent locations, such as the SIFT (Scale Invariant Feature Transform) algorithm or the SURF (Speed-Up Robust Feature Transform) algorithm. Accordingly, these image features can be represented by metadata in vector form as SIFT image features or SURF image features. It should be understood that the present invention is not limited to these methods and / or algorithms.

[0107] The following is an example assumption: the metadata determined in step 21 is for image 3 in Figure 3bEach image feature enclosed in the image features represents the corresponding location of the corresponding image feature. Here, in Figure 3b The locations of the image features enclosed in the image, represented by metadata, can be, for example, relative locations, which are described in... Figure 3b The image features circled in the image are positioned relative to each other.

[0108] In optional step 22, device 1 provides a data model for determining allocation information and / or sorting information for the automated distribution and / or sorting of shipments. This should be understood, for example, as the data model being stored in the program memory 11 of device 1.

[0109] As disclosed above, examples of such data models are databases and / or artificial neural networks obtained through machine learning. Such databases and artificial neural networks can take into account image features of allocation and / or sorting information determined for other shipments, as well as images of the surfaces of these other shipments. These other shipments may, for example, involve previous shipments for which corresponding allocation and / or sorting information has previously been determined.

[0110] Figure 4 This is a schematic illustration of an exemplary embodiment of database 4 according to the present invention. Database 4 includes two columns 40 and 41, and two rows 42 and 43. Here, each row corresponds to an entry in database 4 for a previous shipment. For each entry in the database, column 40 correspondingly contains a representation of image features of the surface of the previous corresponding shipment; and for each entry in the database, column 41 correspondingly contains sorting information and / or allocation information determined for the previous corresponding shipment.

[0111] exist Figure 4 The image features represented in column 40 are circled. It should be understood that this is merely a schematic illustration, and the image features may be represented in other ways (e.g., in vector form) by the database 4 in column 40. The circled image features are also merely exemplary. Preferably, each entry in column 40 contains a representation of the image features of an image of the surface of a previously corresponding shipment, which corresponds to the form of the image features represented by the metadata determined in step 21. For example, the representation contained in column 40 is determined in the same manner as the metadata determined in step 21. Therefore, it is exemplarily assumed below that the database 4 in column 40 is for... Figure 4 Each image feature enclosed in the image features represents the corresponding location of the corresponding image feature. Here, in Figure 4 The positions of the image features enclosed in the image, represented by database 4 in column 40, can be, for example, relative positions, which describe the location of the features. Figure 4The image features circled in the image are positioned relative to each other.

[0112] Furthermore, allocation and / or sorting information is shown in column 41 of database 4, for better understanding as address descriptions. It should be understood that the allocation and / or sorting information in column 41 may alternatively represent, for example, a QR code, such as an alphanumeric code, barcode, or QR code.

[0113] Database 4 can have additional columns and / or rows.

[0114] In step 23, device 1 determines allocation information and / or sorting information for automatically allocating and / or sorting shipments, based at least in part on image features represented by metadata. Furthermore, the determination in step 23 is based, for example, at least in part on database 4.

[0115] For example, the image features represented by the metadata determined in step 21 can be compared with the image features represented by database 4 for previous shipments. This comparison can be performed, for example, according to a comparison method used to compare image features or their representations (e.g., a comparison method for comparing SIFT or SURF image features in vector form), in which the distance between corresponding vectors (e.g., Euclidean distance) is used as a similarity measure (i.e., the smaller the distance between two vectors, the more similar the image features represented by those vectors). When a comparison is made such that the image features represented by database 4 for previous shipments are at least substantially similar to or correspond to the image features represented by the metadata (e.g., based on similarity and / or a pre-given similarity threshold), it can be proposed, for example, that allocation and / or sorting information stored in the database along with the representations of these image features is assigned to the image features represented by the metadata and is determined in step 23 as allocation and / or sorting information for the automated allocation and / or sorting of shipments.

[0116] Here, as publicly assumed above: this metadata and the database only represent the corresponding locations of image features. Accordingly, the comparison can be achieved, for example, by comparing locations. Here, in Figure 3b The location of the image features circled in the image and in Figure 4 A comparison of the positions of the image features circled in the image features shows that the image features represented by the entries in row 42 deviate less from the image features represented by the metadata compared to the image features represented by the entries in row 43. Accordingly, the allocation information and / or sorting information contained in the entries in row 42, column 41, for example, is determined in step 23 as allocation information and / or sorting information for the automated allocation and / or sorting of shipments.

[0117] In optional step 24, device 1 provides the allocation information and / or sorting information determined in step 23, i.e., enabling and / or facilitating automated sorting and / or allocation of shipments, at least in part, based on the allocation information and / or sorting information. As disclosed above, this can be achieved, for example, by printing the allocation information and / or sorting information on the surface of the shipment.

[0118] For example, in combination with the basis Figure 3a and Figure 3b as well as Figure 4 The illustration becomes clear, and this method can be performed even if automatic character recognition and analysis cannot be achieved in the address area. Therefore, since address area 30 is pasted at the corner of the displayed mail item, the address area... Figure 3b Only a portion of the text is visible, meaning not all characters contained within can be recognized and analyzed. However, because it has a process... Figure 2 The method of the steps takes image features (and unrecognized characters) into account when determining allocation and sorting information, so the method according to the invention can still determine allocation and sorting information (provided that these image features are, for example, sufficient for comparison with the image features contained in database 4). Therefore, according to the invention, additional structured data in the form of metadata can be provided for determination (e.g., based on data models (such as artificial neural networks and / or databases) and / or algorithms (such as classifiers and / or classification methods)), thereby enabling allocation and sorting information to be determined at the meta-level and / or image level (compared to the text level). Here, there is no need to interpret or analyze the characters contained in the address area or translate this textual information (e.g., translating the language of the address description in address area 30 (here: English) into a pre-given language (e.g., the official language of the delivery area, here: German)). This method can also achieve the joint processing of various image components (such as address areas and barcodes) as image features in a system and / or by utilizing a method. Furthermore, language barriers can be overcome entirely automatically, as shown by the allocation and sorting information in the English address description in address area 30 and the German (i.e., the official language of the delivery area) in database 4.

[0119] at last, Figure 5 Exemplary embodiments of storage media are shown, on which embodiments of a computer program according to the invention can be stored. The storage media may be, for example, magnetic, electrical, optical, and / or other types of storage media. The storage media may, for example, be a processor (e.g., a processor). Figure 1 A portion of the processor 10 of the device 1 shown, such as the processor's (non-volatile or volatile) program memory or a portion thereof (e.g. Figure 1The device 1 shown has a memory 11. Examples of storage media include flash memory 50, SSD hard disk 51, magnetic hard disk 52, memory card 53, memory stick 55 (e.g., USB stick), CD-ROM or DVD 55, or floppy disk 56.

[0120] The exemplary embodiments of the invention described in this specification should also be understood as being disclosed in all combinations thereof. In particular, the description of features included in the embodiments—unless explicitly stated otherwise—should not be construed as meaning that such features are necessary or essential to the functionality of the embodiment. The order of the method steps listed in the various flowcharts in this specification is not mandatory, and alternative orders of these method steps are contemplated unless otherwise stated. These method steps can be implemented in different ways and therefore implementation in software (by program instructions), hardware, or a combination of both is contemplated.

[0121] Terms used in the patent claims, such as "comprising," "having," "including," and "containing," do not exclude other elements or steps. The expression "at least partially" falls into both the categories of "partially" and "completely." The expression "and / or" should be understood to disclose both alternatives and combinations; that is, "A and / or B" means "(A) or (B) or (A and B)." Multiple numbers of units, persons, etc., in the context of this specification mean multiple units, persons, etc. The use of indefinite articles does not exclude majority. A single device can perform the functions of multiple units or devices mentioned in the patent claims. Reference numerals provided in the patent claims should not be construed as limitations on the mechanisms and steps used.

Claims

1. A method, the method being performed by at least one device (1), the method comprising: - obtaining (20) image data, wherein the image data represent an image (3) of a surface of a postal item captured by an image sensor, - determining (21) metadata associated with the image data based at least partly on the image data, wherein the metadata represent a plurality of image features of the image (3) represented by the image data, wherein each of the image features represented by the metadata is an image constituent of the image represented by the image data, wherein the metadata are determined independently of recognizing characters contained in the image (3) represented by the image data and / or independently of characters and / or their meaning contained in the image (3) represented by the image data, - providing (22) or accessing a data model for determining allocation information and / or sorting information for automatically allocating and / or sorting the postal item, wherein the data model takes into account allocation information and / or sorting information determined for other postal items and image features of images of surfaces of the other postal items, and - determining (23) the allocation information and / or sorting information for automatically allocating and / or sorting the postal item based at least partly on the image features represented by the metadata and based at least partly on the data model, wherein the image features represented by the data model are compared to the image features represented by the metadata, wherein the allocation information and / or sorting information are configured to influence the allocation and / or sorting of the postal item by an automated allocation and / or sorting device in order to sort the postal item to other postal items that shall be delivered in the same delivery area as the postal item.

2. The method according to claim 1, wherein at least one of the image features is at least one of the following image constituents: - a respective area contained in the image (3), - a respective corner contained in the image (3), - a respective edge contained in the image (3), - a respective salient position contained in the image (3), and / or - a respective pattern contained in the image (3).

3. The method according to claim 2, wherein the area, corner, edge, salient position contained in the image (3) and / or the pattern contained in the image represents an address area (30) of the postal item or a part of the address area (30) of the postal item.

4. The method according to one of claims 1 to 3, wherein for at least one of the image features, the metadata represent at least one of the following characteristics of the respective image feature or its surroundings: - a size of the respective image feature, - a shape and / or contour of the respective image feature, - a position of the respective image feature, - a color of the respective image feature or in the surroundings of the respective image feature, - an orientation of the respective image feature or its surroundings, - a geometric centroid of the respective image feature, - a momentum of the respective image feature or its surroundings.

5. The method according to claim 1, wherein the metadata represent one or more of the image features in the form of vectors.

6. The method according to claim 1, wherein the data model is a database (4).

7. The method according to claim 1, wherein the data model is obtained by machine learning.

8. A computer program product comprising program instructions designed to cause an apparatus (1) to perform the method according to one of claims 1 to 7 when executed by at least one processor (10).

9. An apparatus (1) comprising: - means (10-14) configured to perform the method according to one of claims 1 to 7, or comprising respective means (10-14) for performing the steps of the method according to one of claims 1 to 7.

Images