Information processing device, luggage detection system, and calibration method

Abstract

To improve the efficiency of calibration of a baggage detection system.SOLUTION: An information processor acquires a captured image in which a board having a plurality of marks is imaged from an imaging camera, acquires an infrared image in which the board is imaged from an infrared camera, and detects the marks from the captured image. The information processor calculates a first index related to the calibration of the captured image based on the number of detected marks, detects the marks from the infrared image, calculates a second index related to the calibration of the infrared image based on the number of detected marks, and outputs information indicating the first index and the second index.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a luggage detection system, and a calibration method. [Background technology]

[0002] With the recent rise in economic activity, the volume of parcels being distributed has been steadily increasing. In the parcel distribution process, the sorting work of sorting parcels by destination is a time-consuming process that has traditionally been done manually, but technologies have been proposed to automate at least part of the sorting work.

[0003] Patent document 1 discloses a system that tracks luggage moving along a transport route, determines an image to display based on information about the luggage read from the luggage and information about the luggage's location, and projects the image onto the luggage using a projector to display the image on the luggage. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 7,090,134 Summary of the Invention [Problem to be solved by the invention]

[0005] To ensure that the projector properly projects images onto the luggage, each sensor is calibrated during system installation, and there is a need to improve the efficiency of this calibration.

[0006] The present disclosure has been devised in view of the above-described conventional circumstances, and relates to a technique for efficiently calibrating a luggage detection system. In particular, an object of the present disclosure is to provide an information processing device, a luggage detection system, and a calibration method. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure includes: Within a given range The position of the imaging camera that captures the captured image, and The predetermined range An information processing device for calibrating the position of an infrared camera that captures an infrared image, the information processing device comprising a processor and a memory, wherein the processor acquires an image of a board having a plurality of marks from the imaging camera, acquires an infrared image of the board from the infrared camera, detects the marks from the captured image, calculates a first index related to the calibration of the captured image based on the number of the detected marks, detects the marks from the infrared image, and calibrates the infrared image based on the number of the detected marks. A second index relating to the calibration is calculated, and information indicating the first index and the second index is output.

[0008] A luggage detection system according to one aspect of the present disclosure includes: Within a given range an imaging camera that captures an image; The predetermined range The information processing device includes an infrared camera that captures infrared images, and an information processing device that controls the projection device, the imaging camera, and the infrared camera, and in calibrating the position of the imaging camera and the position of the infrared camera, the information processing device acquires the captured image of a board having a plurality of marks from the imaging camera, acquires the infrared image of the board from the infrared camera, detects the marks from the captured image, calculates a first index for calibration of the captured image based on the number of detected marks, detects the marks from the infrared image, calculates a second index for calibration of the infrared image based on the number of detected marks, and outputs information indicating the first index and the second index.

[0009] A calibration method according to an aspect of the present disclosure includes, by an information processing device, Within a given range The position of the imaging camera that captures the captured image, and The predetermined rangeA calibration method for calibrating the position of an infrared camera that captures an infrared image, comprising the steps of having an installation worker install a board having a plurality of marks, acquiring the captured image of the board from the imaging camera, acquiring the infrared image of the board from the infrared camera, detecting the marks from the captured image, calculating a first index related to the calibration of the captured image based on the number of marks detected, detecting the marks from the infrared image, calculating a second index related to the calibration of the infrared image based on the number of marks detected, and displaying information indicating the first index and the second index.

[0010] Any combination of the above components, and conversion of the expression of the present disclosure into a method, device, system, storage medium, computer program, etc., are also valid aspects of the present disclosure. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to efficiently calibrate a luggage sorting system. [Brief explanation of the drawings]

[0012] [Figure 1] A block diagram showing a configuration example of a luggage sorting system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing a configuration example of an information processing device according to a first embodiment; [Figure 3] FIG. 1 is a schematic diagram illustrating the transport of luggage according to the first embodiment; [Figure 4] FIG. 1 is a schematic diagram illustrating detection and projection of luggage according to the first embodiment; [Figure 5] Processing sequence of the baggage sorting system according to the first embodiment [Figure 6] Processing sequence of the baggage sorting system according to the first embodiment [Figure 7] FIG. 1 is a schematic diagram for explaining recognition of a package during transportation according to the first embodiment; [Figure 8]FIG. 1 is a schematic diagram for explaining calibration of an imaging camera and an infrared camera according to the first embodiment; [Figure 9] 1 is a flowchart showing a calibration procedure according to the first embodiment; [Figure 10] FIG. 10 is a diagram showing an index image related to calibration according to the first embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with appropriate reference to the accompanying drawings, embodiments specifically disclosing a luggage detection system, a luggage detection method, an information processing device, and a control method thereof according to the present disclosure will be described in detail. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0014] <First Embodiment> [System Configuration] FIG. 1 is a block diagram showing an example of the configuration of a luggage sorting system 100 according to an embodiment. The luggage sorting system 100 is a system that supports the work of a worker who sorts multiple luggage conveyed by a transport conveyor (not shown in FIG. 1 ), and also operates as a luggage detection system. The luggage sorting system 100 is installed, for example, in a logistics center owned by a retailer, wholesaler, internet distributor, or the like. The luggage to be sorted generally has a roughly rectangular parallelepiped shape, but there are no particular limitations on its outer shape or size, and there are no particular limitations on the type of luggage. The transport conveyor that transports the luggage may be, for example, a belt conveyor or a roller conveyor. Furthermore, the transport path of the luggage may change the transport direction or have a portion of the transport path branch off by combining multiple transport conveyors.

[0015] The package sorting system 100 includes a sorting management server 101, a link server 102, a label reader 103, a sorting client 104, a projector 105, a distance sensor 106, and an image sensor 107. A plurality of client systems 110 are provided, each of which is a group of the sorting client 104, the projector 105, the distance sensor 106, and the image sensor 107, and corresponds to a plurality of sorting areas provided on a transport conveyor that transports packages. Therefore, a plurality of sorting clients 104, projectors 105, distance sensors 106, and image sensors 107 are provided, each corresponding to the number of sorting areas. In the following description, when an individual device needs to be described, a suffix (a, b, ...) is added, and when the devices are described collectively, the suffix is ​​omitted.

[0016] The sorting management server 101 is an information processing device that controls the entire luggage sorting system 100 according to this embodiment. The sorting management server 101 acquires detection results from the label reader 103 and acquires various luggage information from the linking server 102. The sorting management server 101 also issues various instructions to and collects various information from one or more sorting clients 104. Furthermore, the sorting management server 101 may control the operation of a transport conveyor, which will be described later, and collect and manage operation information.

[0017] The linking server 102 stores and manages label information and parcel information in association with each other. In response to a request from the sorting management server 101, the linking server 102 provides parcel information corresponding to the label information read by the label reader 103. While the example in FIG. 1 shows one sorting management server 101 and one linking server 102, they may be configured with multiple devices for functional redundancy, load balancing, and other reasons. The sorting management server 101 may also manage label information and sorting patterns in association with each other, and perform the sorting process for each parcel, as described below, based on this. In such a case, the linking server 102 may be omitted, and the sorting management server 101 may perform the sorting process for each parcel, as described below, based on the pattern of the acquired label information.

[0018] The label reader 103 is a device for reading labels attached to packages being transported by a transport conveyor, and is configured to include optical components such as lenses and image sensors. By reading the labels attached to the packages, various information indicated on the labels (hereinafter referred to as "label information") is obtained. The format or method of the label is not particularly limited, and for example, a barcode or a two-dimensional code (QR Code (registered trademark), etc.) can be used. In this embodiment, a barcode is used as an example of the label. Furthermore, the label information may include, for example, identification information for uniquely identifying the package. In this case, the type of characters and the number of digits constituting the identification information are not particularly limited.

[0019] The label reader 103 may use, for example, a general-purpose imaging camera, extract a label area from a captured image using image recognition to read various information, or recognize characters and the like on the label using OCR (Optical Character Recognition / Reader). The label reader 103 may be a dedicated device for reading labels, or may be configured as an integrated device with other devices. The label reader 103 may also be configured to detect the three-dimensional coordinates of the label. Multiple label readers 103 may be provided to accommodate various sizes of packages, or may be configured to photograph packages from multiple directions to accommodate various label attachment positions.

[0020] Examples of the package information according to this embodiment include a package identification number (e.g., identification information for uniquely identifying a package) individually assigned to each package, the name, address, and telephone number of the sender of the package, the name, address, and telephone number of the destination (e.g., recipient), and the type of package. The information is not limited to these, and the number of items included in the package information may be increased or decreased as necessary. In this embodiment, the sorting management server 101 acquires label information from the label reader 103 and uses this label information to query the linking server 102, thereby acquiring the corresponding package information. As described above, in a configuration in which linking with the linking server 102 is omitted, the sorting management server 101 may store the label information in association with the sorting pattern information.

[0021] The sorting client 104 is an information processing device that controls projection processing and the like for the packages being transported based on instructions from the sorting management server 101. The sorting client 104 also acquires the image capture results from the distance sensor 106 and the image sensor 107, and detects and manages the packages based on the image capture results. The sorting client 104 also provides the sorting management server 101 with information such as the location and size of the detected packages and the time of detection.

[0022] The projector 105 is a projection device that projects a projection image including a predetermined projection pattern onto packages on the conveyance route based on instructions from the sorting client 104. In this embodiment, the term "projection image" refers to the entire image projected by the projector 105 and superimposed on the entire sorting area. The term "projection pattern" refers to an image that is superimposed and displayed corresponding to each package detected in the sorting area. Therefore, when a package is detected in the sorting area, the projection image will include one or more projection patterns depending on the package. Examples of projection patterns will be described later.

[0023] The distance sensor 106 is a sensor for detecting distance information between the distance sensor 106 and an imaged object (for example, the surface of a package), and acquires the detection results as a distance image. The distance image may include an image that cannot be recognized by the human eye, such as a table listing numerical values ​​indicating distance. In other words, the distance image may be information indicating the relationship between coordinates and distance within the imaged area, and its data structure is not particularly limited. For example, an infrared sensor (IR sensor) or a stereo camera consisting of multiple cameras may be used as the distance sensor 106.

[0024] The image sensor 107 is an imaging device including optical components such as a lens and an image sensor. The image sensor 107 is configured, for example, with an imaging camera. The imaging camera may be a three-dimensional camera or multiple two-dimensional cameras. The image sensor 107 captures an image of the conveyor including the luggage being conveyed by the conveyor, and generates a color image. The color image here refers to an image in which the color of the surface of the object is expressed in a predetermined gradation. The gradation may include 256 gradations of RGB (Red, Green, Blue), as well as grayscale, etc.

[0025] The number of client systems 110 varies depending on the length of the transport conveyor, the detection range of each sensor, and the like. The sorting client 104 may be configured as a single unit, and this single sorting client 104 may control multiple projectors 105, multiple distance sensors 106, and multiple image sensors 107 corresponding to multiple sorting areas. While the present embodiment illustrates an example in which the distance sensor 106 and the image sensor 107 are used as sensors, a sensor that combines these may be used, or a configuration in which only one of them is used may be used. Other types of sensors may also be used. The transport format of the transport conveyor is not particularly limited, and may be configured to transport packages in a single file or multiple files, for example.

[0026] The devices are connected to each other so that they can communicate with each other via wired or wireless communication. The communication method and connection method are not particularly limited, and the devices may be connected using a predetermined network cable or may be connected using a combination of multiple methods.

[0027] (Information processing device) 2 is a diagram showing an example of the hardware configuration of an information processing device that can be used as the sorting management server 101, the linkage server 102, and the sorting client 104 according to this embodiment. Here, the description is given assuming that each device has the same configuration, but each device may have a different configuration.

[0028] The information processing device 200 includes a CPU (Central Processing Unit) 201, a memory 202, a storage device 203, an input / output unit 204, a communication unit 205, and an external I / F 206. The CPU 201 realizes various functions by reading various programs and data stored in the memory 202 and the storage device 203 and executing the processes. The CPU 201 may be another arithmetic circuit such as a GPU (Graphics Processing Unit), or may be used in combination with another arithmetic circuit. The memory 202 is a storage area for storing and holding various pieces of information, and is configured, for example, by a ROM (Read Only Memory), which is a non-volatile storage area, or a RAM (Random Access Memory), which is a volatile storage area. The storage device 203 is a storage area for storing and holding various pieces of information, and is configured, for example, by a HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory.

[0029] The input / output unit 204 receives user instructions from a mouse or keyboard (not shown), for example, and outputs various types of information to a display (not shown). The communication unit 205 communicates with external devices via a wired or wireless network, transmitting and receiving various types of data and signals. The communication method used by the communication unit 205 is not particularly limited, and may be compatible with a plurality of communication methods. For example, a wide area network (WAN), a local area network (LAN), power line communication, or short-range wireless communication (e.g., Bluetooth (registered trademark)) may be used. The external I / F 206 is an interface for transmitting and receiving data to and from external devices. The components of the information processing device 200 are communicably connected via an internal bus (not shown) or the like.

[0030] [Transportation example] FIG. 3 is a schematic diagram for explaining the transportation of luggage according to this embodiment. A label 301 is attached to each luggage 300 to be transported. The label 301 is read during transportation by a label reader 103 installed at an arbitrary position on the transportation route. In the example of FIG. 3, the label 301 is attached to the top surface of the luggage 300 and is read by the label reader 103 from above, but this is not limiting. For example, the label 301 may be attached to the side of the luggage 300, and the label reader 103 may be able to read the label from the side.

[0031] The package 300 is transported at a desired speed by the transport conveyor 302. A rotary encoder 303 is installed on the transport conveyor 302, which detects the transport distance as needed and provides the detected distance to the sorting management server 101. The rotary encoder 303 is attached to a rotation mechanism (not shown) that rotates as the transport conveyor 302 moves, and outputs the number of rotations of the rotation mechanism. If the transport conveyor 302 is a belt conveyor, this rotation mechanism is a roller in contact with the belt or a motor that moves the belt. If the transport conveyor 302 is a roller conveyor, it is a roller that constitutes the transport conveyor 302. Here, it is assumed that the package is transported in the direction indicated by the arrow. The transport conveyor 302 changes its transport speed or stops transport in response to a user instruction. Transport in the reverse direction may also be performed as necessary. The transport speed of the transport conveyor 302 may also change due to factors other than user instructions, such as the physical characteristics of each mechanism of the transport conveyor 302. However, since the rotary encoder 303 accurately detects the number of rotations of the rotation mechanism, it is less susceptible to changes in the conveying speed, etc. The conveyance by the conveyor 302 may be controlled, for example, by the sorting management server 101 or by a separately provided control device (not shown).

[0032] Downstream of the label reader 103 in the conveying direction, various sensors (in this example, a distance sensor 106 and an image sensor 107) and a plurality of projectors 105 are installed corresponding to the plurality of sorting areas. FIG. 3 shows sorting area A, the most upstream of the plurality of sorting areas. The various sensors corresponding to sorting area A capture images of sorting area A to obtain various images, and the sorting client 104a uses these images to detect the parcel being conveyed. Specifically, when the parcel 300 reaches the start position 305a of sorting area A, the parcel 300 enters the capture ranges of the various sensors. The capture ranges 304 of the distance sensor 106a and the image sensor 107a include at least the range of sorting area A. Furthermore, within the range of sorting area A, the projector 105a projects a predetermined projection image based on instructions from the sorting management server 101 or the sorting client 104. The projection range 306 of the projector 105a includes at least the sorting area A. The distance from the reading position of the label reader 103 to the start position 305a may be determined according to the processing speed of the processing sequence described below, the reading speed of various sensors, and the like.

[0033] Around the transport conveyor 302, workers M who work on the packages 300 are positioned, and perform various tasks as needed. At this time, the workers M can perform the tasks based on the projection image (projection pattern, etc.) projected by the projector 105.

[0034] 4 is a diagram illustrating multiple sorting areas corresponding to multiple client systems 110, respectively. Here, two sorting areas A and B are taken as an example, and the description will be given assuming that they are monitored by client systems 110a and 110b, respectively. The number of sorting areas and their extent (e.g., length and width in the conveying direction) may be determined according to the capabilities of client system 110. Therefore, one or more additional sorting areas may be provided further downstream in the conveying direction from sorting area B.

[0035] The sorting area A is photographed by the distance sensor 106a and the image sensor 107a, and the image is provided to the sorting client 104a. In addition, in the sorting area A, a projection image including an arbitrary projection pattern is projected by the projector 105a. Similarly, the sorting area B adjacent to the sorting area A on the downstream side in the conveying direction is photographed by the distance sensor 106b and the image sensor 107b, and the image is provided to the sorting client 104b. In addition, in the sorting area B, a projection image including an arbitrary projection pattern is projected by the projector 105b. In the sorting area B, monitoring is performed by taking over monitoring of the parcels conveyed from the sorting area A. At this time, as shown in FIG. 4, the sorting areas of the client systems 110 partially overlap, and hereinafter, this area is also referred to as a "handover area." The start position 305b indicates the start position of monitoring of the sorting area B and is different from the end position (not shown) of monitoring of the sorting area A adjacent to the upstream side of the sorting area B. The handover area will be described in detail later.

[0036] [Processing Sequence] The processing sequence of the luggage sorting system 100 according to this embodiment will be described with reference to Figures 5 and 6. The processing of each device may be realized by the execution of a program corresponding to each function by a processing entity (such as a CPU) of each process. When the processing shown in Figures 5 and 6 starts, the operation of the transport conveyor 302 begins, thereby transporting luggage sequentially. Note that the entity of each process in the processing sequence shown below is an example, and some of the processes may be executed by other devices. For example, a configuration may be adopted in which some of the processes of the sorting management server 101 shown below are executed by the sorting client 104. Alternatively, each server and sorting client shown in Figure 1 may be configured as a single device, and the following processing sequence may be executed by this single device.

[0037] The label reader 103 detects the label attached to the package being transported on the transport conveyor 302 and reads the label information (step S501).

[0038] The label reader 103 transmits the label information read in S501 to the sorting management server 101 (step S502). The information transmitted at this time may include not only the information indicated on the label, but also the time of reading and position information detected when reading the label (e.g., three-dimensional information).

[0039] The sorting management server 101 acquires the label information transmitted from the label reader 103 (step S503). At this time, the sorting management server 101 acquires a detection value from the rotary encoder 303 provided on the transport conveyor 302, associates it, and stores it. The detection value from the rotary encoder 303 corresponds to the travel distance of the transport conveyor 302 based on its rotation speed. Furthermore, the sorting management server 101 uses the acquired label information to inquire about package information from the link server 102.

[0040] The linking server 102 searches a predetermined DB (database) for and identifies package information corresponding to the label information indicated in the inquiry from the sorting management server 101 (step S504). The DB may be provided internally in the linking server 102, or may be configured in an external device (not shown) connected to the linking server 102 via a network.

[0041] The linkage server 102 transmits the package information identified in step S504 to the sorting management server 101 as a response to the inquiry (step S505).

[0042] The sorting management server 101 determines a projection pattern based on the package information acquired from the link server 102 (step S506). The projection patterns according to this embodiment are defined and stored in advance as a plurality of patterns. The projection pattern may be determined, for example, according to the destination (e.g., delivery address) indicated in the package information, or according to the sorting area.

[0043] The sorting management server 101 transmits a projection instruction including the projection pattern determined in step S506 to the sorting client 104 (step S507). Here, as shown in FIG. 3, the description will be given assuming that the projection instruction is transmitted to the sorting client 104a of the client system 110a corresponding to the sorting area A located at the most upstream of the conveyance path. That is, each package is first conveyed toward the sorting area A. Note that the projection instruction may include, in addition to the projection pattern, package information acquired in step S506, label information read by the label reader 103 in step S501, and information on the time of reading. The projection instruction here may be transmitted from the sorting management server 101 to the sorting client 104a in response to an inquiry about the package that is made by the sorting client 104a detecting the package using sensors. Alternatively, the projection instruction may be transmitted to the sorting client 104a in response to the inquiry about the package that is made by the sorting management server 101 receiving label information from the label reader 103.

[0044] Step S520 (steps S508 to S515) is a process that is repeatedly performed by client system 110 (here, client system 110a) to monitor each package being transported within the sorting area (here, sorting area A). For convenience, distance sensor 106 and image sensor 107 (here, distance sensor 106a and image sensor 107a) are collectively referred to as "sensors."

[0045] The sensors capture images of the sorting area A and acquire the images (step S508). In this embodiment, the sorting client 104a detects packages using images captured by the sensors, but the sensors may be configured to perform part of the process. For example, the sensors may be configured to identify areas corresponding to packages (whether or not a package is present), and to detect the three-dimensional coordinates (position) and size (length, width, height) of each package.

[0046] The sensors transmit the acquired results to the assortment client 104a (step S509). At this time, the sensors may perform predetermined processing on the acquired images before transmitting them to the assortment client 104a. For example, in the case of the image sensor 107a, any image processing (e.g., image compression or correction processing) may be performed on the acquired image information.

[0047] The sorting client 104a performs a package detection process based on the results of the acquisition transmitted from the sensors in step S509 (step S510). Specifically, the sorting client 104a identifies one or more packages being transported by the transport conveyor 302 within the sorting area A based on the results of the acquisition from the sensors. The process for detecting packages may use a known method, and is not particularly limited. For example, areas corresponding to packages may be detected by area segmentation using machine learning. Alternatively, packages may be detected by pattern matching using a predefined pattern.

[0048] The sorting client 104a associates each piece of information based on the projection instruction sent from the sorting management server 101 in step S507 and the detection result of step S510 (step S511). As a result, each of the detected one or more packages is associated with various pieces of information (projection pattern, etc.) indicated in the projection instruction sent from the sorting management server 101.

[0049] The projection pattern is, for example, an image of a number surrounded by a circular frame in a color that indicates the sorting location corresponding to the delivery address of the package. Here, the number corresponds, for example, to the number of the truck carrying the sorted packages (such as the number of the truck itself or the parking lot number) or the number of the shelf or box to be delivered to the truck. Furthermore, the image of the number may not directly correspond to the number of the shelf or box, but may correspond to the number of a chute (not shown) that moves the picked-up package to another location or truck. Because the parking location of trucks and the like frequently changes depending on traffic conditions, it may be difficult to constantly match the sorting destination as seen from around the conveyor 302. For this reason, by inserting a chute between the conveyor 302 and the conveyor truck, etc. and projecting the chute number around the conveyor 302, it is possible to respond to changes in the sorting destination by, for example, rearranging the chute exit, without constantly changing the configuration around the conveyor 302. In this case, the content of the projection pattern may be switched depending on the change in situation.

[0050] Other examples of numbers displayed as projection patterns include a postal code corresponding to the delivery address and the number of the worker who is to pick up the package P. Examples of information other than numbers include arrows indicating the sorting direction (e.g., right or left relative to the conveying direction of the transport conveyor 302) or letters (e.g., "left" or "right"). Furthermore, the display format is not limited to numbers enclosed in a circular frame, and various configurations are possible, such as numbers enclosed in a square frame (e.g., "3," "359," "24735"). Furthermore, the projection pattern is not limited to numbers or letters enclosed in a frame, and may be numbers or letters in white against a solid background. The shape of the numbers or letters displayed may be switched between circles, triangles, squares, etc., depending on the information to be displayed. Alternatively, a picture that can be individually associated with each piece of information to be displayed may be displayed.

[0051] Furthermore, the projection pattern is not limited to a still image, but may be an animation. Examples of animations include flashing, scaling, or changing the color of the above examples. An animation reflecting the sorting direction may also be projected. Examples of animations reflecting the sorting direction include moving a light beam or light spot in the sorting direction, forming or changing the color of the entire or part of the projection pattern in the sorting direction, or displaying an arrow moving in the sorting direction. When only a portion of the projection pattern is animated, parts that have a large impact on the worker's sorting destination, such as numbers and arrows, may remain unchanged, while parts that have less impact on the sorting destination, such as the frame lines, may be changed. However, in situations where it is more efficient to intuitively convey the sorting direction rather than relying on the meaning of numbers projected within the frame lines, such as when there are few sorting destination options, numbers and arrows may be moved in the sorting direction within a fixed frame line. The animation may also be projected repeatedly or only once.

[0052] Furthermore, the projection pattern is not limited to one indicating the sorting destination, but may also be a notification of an error occurring in the baggage sorting system 100 or a notification that a worker has overlooked a bag.

[0053] Furthermore, instead of projecting the projection pattern directly onto the package, the worker may be made to perceive the projection pattern as if it were projected onto the package through glasses capable of displaying an image. In other words, if the worker is wearing special glasses capable of displaying an image, the projection pattern may be superimposed on the image of the package that is visible through the glasses.

[0054] The projection pattern may also be projected onto a location other than luggage. For example, general notification content for the worker may be projected onto an area of ​​the transport conveyor 302 where no luggage is present. Examples of such notifications include advance notice of an increase or decrease in the amount of arriving luggage, the remaining time for work, and greetings to the worker.

[0055] The sorting client 104a performs processing in three-dimensional space on one or more parcels detected in step S510 (step S512). In this processing in three-dimensional space, each parcel is represented as a three-dimensional shape in a three-dimensional coordinate system on the conveyance route. Depending on the number and arrangement of sensors, information on surfaces located in the blind spots of the sensors may not be available. In this case, the three-dimensional shape of each parcel may be interpolated to match a general parcel shape (e.g., a rectangular parallelepiped). Furthermore, since instructions can be projected as long as the three-dimensional shape of the surface that constitutes the parcel onto which the projection image is to be projected can be recognized, the three-dimensional shape of the parcel may be represented as the surface to be projected, or a set of surfaces including the surface to be projected. In this embodiment, the surface to be projected is the top surface of the parcel, but it may also be a side surface, etc. In this embodiment, the sensors are positioned above the parcel so that at least information on the top surface can be obtained. However, if the surface to be projected is a side surface, etc., the parcel is photographed from a position aligned with that surface.

[0056] It is assumed that the origin and each coordinate axis of the three-dimensional coordinate system on the conveyance path are predefined within the client system 110a. Furthermore, it is assumed that the correspondence between the three-dimensional coordinate systems of the label reader 103 and each of the multiple client systems 110 is predefined and adjusted as necessary. The multiple three-dimensional coordinate systems may be associated with each other using an absolute coordinate system shared by the entire baggage sorting system 100. Alternatively, the multiple three-dimensional coordinate systems may be associated with each other using a relative coordinate system. When using a relative coordinate system, for example, a client system 110 located upstream of the baggage sorting system 100 may be used as the reference, and the coordinate systems used by other client systems 110 may be defined as coordinate systems relative to the reference client system 110. The reference coordinate system for the relative coordinate system may be any coordinate system within the baggage sorting system 100. For example, it may be another coordinate system, such as a coordinate system used by another client system or a group of sensors. Note that this process may be performed in a two-dimensional space, that is, a flat space, if adjustment in the height direction is not required due to the shape of the cargo to be transported or the configuration of the projector 105.

[0057] The sorting client 104a renders a projection image including a projection pattern associated with each of the one or more packages having a three-dimensional shape processed in step S512 (step S513). Specifically, the sorting client 104a calculates the projection position of the projection pattern for each package and determines a projection image for the entire sorting area A. As an example, this projection image includes a projection pattern for each package in an area corresponding to each package, and a black image in an area where no package exists. By using such a projection image, it is possible to simultaneously project onto a large number of packages while reducing the number of projectors. Note that with a typical projector, the closer the distance between the projector and the surface onto which the projection is to be performed, the smaller the image, and the farther the distance, the larger the image. Therefore, the size of the projection pattern for each package in the projection image rendered in step S512 may be adjusted so that the closer the distance between the projector and each package is, the larger the image.

[0058] The assortment client 104a instructs the projector 105a to project the image determined in step S513 (step S514).

[0059] Based on the projection instruction from the sorting client 104a in step S514, the projector 105a projects a projection image onto the sorting area A (step S515). Since each package is continuously transported by the transport conveyor 302, the process shown in step S520 may be continuously executed while the transport continues.

[0060] (Baggage handover processing) 5, the processing has been explained focusing on the first (most upstream) sorting area A downstream in the conveying direction from the reading position of the label reader 103. As described above, multiple sorting areas are provided on the conveying path of the conveyor 302, and the monitoring of each package between the sorting areas is handed over (hereinafter also referred to as "handover") between the client systems 110.

[0061] The processing sequence for handover processing will be explained using Figure 6. Here, the handover processing between client system 110a corresponding to sorting area A on the conveying path and client system 110b corresponding to sorting area B adjacent to sorting area A on the downstream side in the conveying direction will be explained as an example. Since the number of sorting areas can vary depending on the length of the conveying path, handover processing is performed as appropriate between client systems 110 corresponding to adjacent sorting areas. Note that client system 110a is assumed to be simultaneously performing the processing sequence explained in Figure 5.

[0062] Step S620 (steps S601 to S603) is a process that is repeatedly performed by client system 110a to monitor each piece of luggage being transported within sorting area A. Here, for convenience, distance sensor 106a and image sensor 107a will be collectively referred to as sensors A.

[0063] Sensors A capture images of the inside of sorting area A and acquire the images (step S601). Here, as shown in FIG. 5, the explanation will be given assuming that the sorting client 104a detects packages using the various images captured by sensors A. If multiple packages are being transported to sorting area A, each package is detected. In the case of client system 110a, this process corresponds to step S508 in FIG. 5.

[0064] The sensors A transmit the acquired images to the sorting client 104a (step S602). At this time, the sensors A may perform predetermined processing on the acquired images before transmitting them to the sorting client 104a. In the case of the client system 110a, this processing corresponds to step S509 in FIG. 5.

[0065] The sorting client 104a detects packages based on the results of the acquisition transmitted from sensors A in step S602 (step S603). Furthermore, the sorting client 104a transmits transport information of the detected packages to the sorting management server 101 (step S603). Specifically, based on the results of the processing in the three-dimensional space already performed in the processing sequence of FIG. 5, the sorting client 104a may identify, among the tracked packages, packages approaching the boundary with the downstream sorting area (sorting area B in this case), and transmit information about the packages as transport information. The processing for detecting packages may use a known method, and is not particularly limited. For example, a region corresponding to a package may be detected by area segmentation using machine learning. Alternatively, a package may be detected by pattern matching using a predefined pattern.

[0066] Step S630 (steps S604 to S606, steps S610 to S614) is a process that is repeatedly performed by client system 110b to monitor each package being transported within sorting area B. For convenience, distance sensor 106b and image sensor 107b will be collectively referred to as sensors B here.

[0067] Sensors B capture images of the inside of sorting area B and acquire the images (step S604). Here, as in sorting area A, the explanation will be given assuming that the sorting client 104b detects packages using the various images captured by sensors B. If multiple packages are being transported to sorting area B, each package is detected.

[0068] The sensors B transmit the acquired results to the assorting client 104b (step S605). At this time, the sensors B may perform a predetermined process on the acquired information before transmitting it to the assorting client 104b.

[0069] The sorting client 104b detects packages based on the acquisition results transmitted from sensors B in step S605 (step S606). Furthermore, the sorting client 104b transmits transport information of the detected packages to the sorting management server 101. For example, the sorting client 104b may identify, among the detected packages, a package located on the boundary with the upstream sorting area (here, sorting area A) (i.e., a newly transported package), and transmit information about that package (such as its position and size) as transport information.

[0070] Based on the transport information transmitted from sorting client 104a in step S603 and the transport information transmitted from sorting client 104b in step S606, sorting management server 101 identifies packages to be handed over between sorting areas (here, between sorting area A and sorting area B) (step S607). In other words, in order to hand over the monitoring entity for a package from client system 110a to client system 110b as the package is transported, packages located on the boundary between sorting areas detected by each client system 110 are associated as the same package. For example, sorting management server 101 may associate packages across sorting areas by identifying which of the package information it manages corresponds to the package.

[0071] The sorting management server 101 determines the projection pattern for the downstream sorting area (sorting area B in this case) based on the association in step S607 and the package information used in the upstream sorting area (sorting area A in this case) (step S608). The projection patterns according to this embodiment are defined and stored in advance as a plurality of patterns. The projection pattern may be the same as that in the upstream sorting area (sorting area A in this case), or may be configured to switch depending on the sorting area being transported, even for the same package.

[0072] The sorting management server 101 transmits a projection instruction including the projection pattern determined in step S608 to the sorting client 104 (step S609). Here, the projection instruction is transmitted to the sorting client 104b of the client system 110b to which the monitoring of the package is handed over. Note that the projection instruction may include package information of the corresponding package in addition to the projection pattern.

[0073] The sorting client 104b associates each piece of information based on the projection instruction transmitted from the sorting management server 101 in step S609 and the detection result of step S606 (step S610). Specifically, the sorting client 104b identifies one or more packages being transported by the transport conveyor 302 within the sorting area B based on the detection result of the sensors B. Furthermore, the sorting client 104b associates various pieces of information included in the projection instruction transmitted from the sorting management server 101 with each of the identified one or more packages. As a result, each of the one or more packages detected by the sorting client 104b is associated with various pieces of information (projection pattern, etc.) indicated in the projection instruction transmitted from the sorting management server 101.

[0074] The sorting client 104b performs processing in three-dimensional space for one or more packages identified in step S610 (step S611). In this processing in three-dimensional space, each package is represented as a three-dimensional shape in a three-dimensional coordinate system on the conveyance route. This processing may be the same as the processing in step S512 of FIG. 5.

[0075] The sorting client 104b renders a projection image including a projection pattern associated with each of the one or more packages having a three-dimensional shape processed in step S611 (step S612). Specifically, the sorting client 104b calculates the projection position of the projection pattern for each package and determines a projection image for the entire sorting area B. The projection image here may have the same configuration as that of step S513 described in FIG. 5.

[0076] Note that multiple projectors may be installed in association with one sorting area. For example, one sorting area may be further subdivided to set areas each covered by multiple projectors. In this case, the sorting client 104b generates a projection image corresponding to the area covered by each of the multiple projectors. An example of a projection image in this case is an image in which a projection pattern for each package is included at the location of packages present within the area covered, and a black image is included at locations within the area where no packages are present. In this way, even if it is difficult for a single projector to project onto the entire sorting area due to projector performance or limitations on the height at which the projector can be installed, it is possible to project onto the entire sorting area.

[0077] Alternatively, each projector may be assigned to one piece of luggage and project only onto that piece of luggage. In this case, each of the multiple projectors changes the angle of irradiation according to the movement of the luggage it is responsible for, or projects a black image at a position other than the location of the luggage it is responsible for, thereby projecting only onto the luggage it is responsible for. In this way, each of the multiple projectors can project using parameters optimal for the luggage it is responsible for. For example, the focal length of each of the multiple projectors can be automatically adjusted, so that even if there is variation in the height of the luggage, it is possible to project an image that is in focus on each piece of luggage.

[0078] The assortment client 104b instructs the projector 105b to project the image determined in step S612 (step S613).

[0079] Based on the projection instruction from the sorting client 104b in step S613, the projector 105b projects the projection image onto the sorting area B (step S614). Since each package is continuously transported by the transport conveyor 302, the client systems 110 responsible for monitoring the sorting area B and beyond may continue to execute the process shown in step S630 while the transport continues.

[0080] [Baggage Recognition] 7 is a diagram for explaining the recognition of packages in a sorting area according to this embodiment. Here, a sorting area 701 on the conveyance path of the transport conveyor 302 and adjacent sorting areas 702 and 703 on the upstream and downstream sides of the transport direction are taken as an example. It is also assumed that three packages 706, 707, and 708 are being transported by the transport conveyor 302.

[0081] A handover area 704 is provided between sorting area 701 and sorting area 702. Handover area 704 is an area where detection is performed by sensors in both the client system corresponding to sorting area 701 and the client system corresponding to sorting area 702. Similarly, handover area 705 is provided between sorting area 701 and sorting area 703. Handover area 705 is an area where detection is performed by sensors in both the client system 110 corresponding to sorting area 701 and the client system 110 corresponding to sorting area 703.

[0082] In each sorting area according to this embodiment, the range of each package being transported is identified as needed. By identifying the range of the package, the size and shape of the package are recognized. The range here does not need to exactly match the shape of the package, and a predetermined margin may be provided. Then, the size, shape, and position of the projection pattern to be projected onto each package are adjusted according to the range of each package. The adjustment method is not particularly limited, and may be performed taking into consideration, for example, the visibility of the projection pattern and work efficiency.

[0083] In the present embodiment, the label reader 103 and the image sensor 107 are described as being different devices. However, the label reader 103 may be used in place of the image sensor 107. Specifically, the position of the label read by the label reader 103 is treated as the location of the package, and a projection image is projected onto the label position. Because package identification information can be obtained from the label, adopting such a configuration eliminates the need for processing to link identical packages between client systems 110. However, because the area recognizable by the label reader 103 is limited to the label area, the degree of freedom in the size of the projected image and the position at which the projected image is projected is reduced compared to tracking using the image sensor 107, which can track the entire package. Similarly, the image sensor 107 may be used in place of the label reader 103. Specifically, label information may be obtained from an image of a label included in an image acquired by the image sensor 107. Unless otherwise specified, the following describes an example of package tracking using an image captured by the image sensor 107. However, the label reader 103 may be used in place of the image sensor 107.

[0084] (Calibration of imaging camera and infrared camera) FIG. 8 is a schematic diagram for explaining the calibration of the imaging camera 107 and the infrared camera 106 according to the first embodiment.

[0085] As described above, the image sensor 107 may be configured by the imaging camera 107. An image captured by the imaging camera 107 is referred to as a captured image. As described above, the distance sensor 106 may be configured by the infrared camera 106. An image captured by the infrared camera 106 is referred to as an infrared image.

[0086] 8, the installation position of the imaging camera 107 may differ from the installation position of the infrared camera 106. Therefore, when the imaging camera 107 and the infrared camera 106 are installed, calibration is performed to calculate correction parameters for mapping the image captured from the position of the imaging camera 107 onto three-dimensional coordinates and correction parameters for mapping the infrared image captured from the position of the infrared camera 106 onto three-dimensional coordinates.

[0087] Calibration may be performed using a board 820 as shown in FIG. 8. The board 820 may have a configuration including a flat surface 821 that is aligned with the surface of the transport conveyor 302 and an inclined surface 822 that extends obliquely in the height direction from one side of the flat surface 821. Although the board 820 in FIG. 8 has one inclined surface 822, the board 820 may have two or more inclined surfaces 822. The inclined surface 822 is for performing calibration in a direction perpendicular to the surface of the transport conveyor 302. Therefore, when performing calibration only in a direction along the surface of the transport conveyor 302, the inclined surface 822 may be omitted. The configuration using the board 820 for calibration is one example. Calibration may be performed using a box or the like with a mark or a two-dimensional code attached thereto instead of the board 820.

[0088] 8, a plurality of marks 823 may be drawn on the surfaces of the flat surface 821 and the inclined surface 822 of the board 820. The shape of the marks 823 may be a circle. However, the shape of the marks 823 is not limited to a circle and may be a polygon, a predetermined pattern, or the like.

[0089] In calibration, the information processing device 104 used as the sorting client 104 during operation receives an image of the board 820 from the imaging camera 107, detects marks 823 from the image, and calculates correction parameters for the image based on the detected marks 823. Similarly, in calibration, the information processing device 104 receives an infrared image of the board 820 from the infrared camera 106, detects marks 823 from the infrared image, and calculates correction parameters for the infrared image based on the detected marks 823.

[0090] The correction parameters described above may be calculated using known techniques. For example, the information processing device 104 calculates the tilt of the captured image with respect to a horizontal plane based on the positions and shapes of multiple (e.g., three) marks 823 detected from a flat surface 821 of the board 820 in the captured image, and calculates the tilt of the captured image with respect to a vertical plane based on the positions and shapes of multiple (e.g., three) marks 823 detected from a sloped surface 822 of the board 820 in the captured image, thereby calculating the correction parameters for mapping the captured image to three-dimensional coordinates. Similarly, the information processing device 104 calculates the tilt of the infrared image with respect to a horizontal plane based on the positions and shapes of multiple (e.g., three) marks 823 detected from the flat surface 821 of the board 820 in the infrared image, and calculates the tilt of the infrared image with respect to a vertical plane based on the positions and shapes of multiple (e.g., three) marks 823 detected from a sloped surface 822 of the board 820 in the infrared image, thereby calculating the correction parameters for mapping the infrared image to three-dimensional coordinates.

[0091] As a result, during operation, the sorting client 104 can accurately detect the three-dimensional coordinates (position) and size (length, width, height) of the luggage 300 flowing on the transport conveyor 302 using the captured image to which the correction parameters have been applied and the infrared image to which the correction parameters have been applied.

[0092] Here, when the number of marks 823 detected from the board 820 is small (for example, three), the accuracy of the correction parameters is relatively low, and as the number of marks 823 detected from the board 820 increases, the accuracy of the correction parameters tends to become relatively high.

[0093] An installation worker performing calibration places the board 820 on the transport conveyor 302, has the imaging camera 107 and the infrared camera 106 capture an image of the board 820, has the information processing device 104 calculate correction parameters, and if the calculated correction parameters are not accurate, changes the position of the board 820, has the imaging camera 107 and the infrared camera 106 capture an image of the board 820 again, and has the information processing device 104 calculate the correction parameters again. This calibration method, however, is time-consuming and labor-intensive, resulting in low work efficiency. In particular, it is difficult to place the board 820 in a position where the accuracy of both the correction parameters of the imaging camera 107 and the correction parameters of the infrared camera 106 is sufficient. Therefore, a method for improving the work efficiency of this calibration will be described below.

[0094] Fig. 9 is a flowchart showing the procedure of calibration according to embodiment 1. Fig. 10 is a diagram showing an index image related to calibration according to embodiment 1. Next, a calibration method according to this embodiment will be described with reference to Figs. 9 and 10.

[0095] An installer performing calibration places the board 820 within a range in which the projector 105 on the transport conveyor 302 can project a projection image (step S901). At this time, the information processing device 104 may cause the projector 105 to project a projection image that indicates the maximum range that can be projected. In this case, the installer only needs to place the board 820 so that it fits within the range of the projected image (projection range 306).

[0096] The information processing device 104 receives a captured image of the board 820 from the imaging camera 107, and receives an infrared image of the board 820 from the infrared camera 106 (step S902).

[0097] The information processing device 104 detects the marks 823 from the captured image, and counts the number of marks 823 on the flat surface 821 of the board 820 and the number of marks 823 on the inclined surface 822 of the board 820 (step S903).

[0098] The information processing device 104 detects the marks 823 from the infrared image, and counts the number of marks 823 on the flat surface 821 of the board 820 and the number of marks 823 on the inclined surface 822 of the board 820 (step S904).

[0099] The information processing device 104 calculates an index (hereinafter referred to as a first index 911) related to the calibration of the captured image based on the number of marks 823 on the flat surface 821 and the number of marks 823 on the inclined surface 822 counted in step S903 (step S905). The first index 911 may tend to become larger as the number of counted marks 823 increases. In general, the more marks 823 that can be counted, the more likely it is that the accuracy of the calibration will increase, and therefore, it can be expected that the larger the first index is, the higher the accuracy of the calibration related to the captured image will also increase.

[0100] The information processing device 104 calculates an index (hereinafter referred to as second index 921) related to the calibration of the infrared image based on the number of marks 823 on the flat surface 821 and the number of marks 823 on the inclined surface 822 counted in step S904 (step S906). The second index 921 may tend to become larger as the number of counted marks 823 increases. In general, the more marks 823 that can be counted, the more likely it is that the accuracy of the calibration will increase, so it can be expected that the accuracy of the calibration related to the infrared image will also increase as the second index becomes larger.

[0101] 10, the information processing device 104 outputs a first index image 910 indicating a first index 911 related to the calibration of the imaging camera 107 and a second index image 920 indicating a second index 921 related to the calibration of the infrared camera 106 to a display device 900 (e.g., a liquid crystal display) connected to the information processing device 104 (step S907). As shown in Fig. 10, a first threshold value 912 may be displayed on the first index image 910, and a second threshold value 922 may be displayed on the second index image 920.

[0102] The first threshold 912 may be set to a value greater than the first index 911 calculated when the number of marks 823 detected from at least one of the flat surface 821 or the inclined surface 822 of the board 820 (e.g., both the flat surface 821 and the inclined surface 822) is less than three. In other words, the first threshold 912 may be set to a value greater than the first index 911 calculated when the number of marks 823 detected from at least one of the flat surface 821 or the inclined surface 822 of the board 820 is a predetermined number equal to or greater than three.

[0103] The second threshold 922 may be set to a value greater than the second index 921 calculated when the number of marks 823 detected from at least one of the flat surface 821 or the inclined surface 822 of the board 820 (e.g., both the flat surface 821 and the inclined surface 822) is less than three. In other words, the second threshold 922 may be set to a value greater than the second index 921 calculated when the number of marks 823 detected from at least one of the flat surface 821 or the inclined surface 822 of the board 820 is a predetermined number equal to or greater than three.

[0104] In this way, by displaying the first threshold value 912 on the first index image 910, the installer can check at a glance whether the first index 911 exceeds the first threshold value 912. Similarly, by displaying the second threshold value 922 on the second index image 920, the installer can check at a glance whether the second index 921 exceeds the second threshold value 922. Note that when the first index 911 exceeds the first threshold value 912 and the second index 921 exceeds the second threshold value 922, an additional display may be displayed to more clearly notify that each index has exceeded its respective threshold value. This additional display may be displayed, for example, as an image different from the first index image 910 and the second index image 920, or may be displayed by changing the display mode, such as the color, of the first index image 910 and the second index image 920. In other words, any display mode may be used as long as the displayed content changes between when either index does not exceed the threshold and when both indexes exceed the threshold.

[0105] The information processing device 104 determines whether or not the installer has instructed that preparation for calibration is complete (step S908).

[0106] If the installer has not issued an instruction that preparation for calibration is complete (step S908: NO), the information processing device 104 returns the process to step S901. For example, the installer looks at the first index image 910 and the second index image 920 displayed in step S907, and if the first index 911 is less than the first threshold 912 or the second index 921 is less than the second threshold 922, the installer changes the position of the board 820 without issuing an instruction that preparation for calibration is complete. In this case, the processes of steps S901 to S907 are performed for the changed position of the board 820.

[0107] When the installer issues an instruction that preparation for calibration is complete (step S908: YES), the information processing device 104 proceeds to the next step S909. For example, the installer looks at the first index image 910 and the second index image 920 displayed in step S907 while changing the position of the board 820, and when the first index 911 is equal to or greater than the first threshold value 912 and the second index 921 is equal to or greater than the second threshold value 922, the installer issues an instruction that preparation for calibration is complete. Note that the instruction that preparation for calibration is complete may be issued automatically or manually by the installer.

[0108] The information processing device 104 calculates correction parameters for the captured image based on the mark 823 detected in step S903, and stores the correction parameters in the storage device 203 (step S909).

[0109] The information processing device 104 calculates correction parameters for the infrared image based on the mark 823 detected in step S904, and stores the correction parameters in the storage device 203 (step S910), and then ends this process.

[0110] This allows the installer performing the calibration to change the position of the board 820 while looking at the first index image 910 and the second index image 920, thereby placing the board 820 at a position where the accuracy of both the correction parameters of the imaging camera 107 and the correction parameters of the infrared camera 106 is sufficient. In other words, the efficiency of the calibration work is improved.

[0111] <Other embodiments> In the above-described embodiment, the projected image from the projector 105 may be used to calculate the correction parameters. For example, a predetermined image (e.g., a light and dark pattern) may be projected from the projector 105 onto the board 820, and correction parameters for aligning the coordinate systems of the projector 105 and each camera may be calculated by analyzing how the image is captured by the imaging camera 107 or the infrared camera 106. In this way, calibration between the projector 105 and the imaging camera 107, or between the projector 105 and the infrared camera 106, may be performed.

[0112] As an example of calibration using a light and dark pattern projected from the projector 105, a spatial coding method can be used. In the spatial coding method, light and dark patterns are projected onto an object while switching in a predetermined order, and the changes in light and dark are recorded as a code. For example, a change in the order of "light, dark, light" is associated with "101," and a change in the order of "light, light, dark" is associated with "110." This allows the three-dimensional shape of the object to be estimated by analyzing the deviation between the coordinates corresponding to "101" and the coordinates corresponding to "101" in the captured image, assuming the object has a predetermined shape (e.g., a flat surface). Furthermore, by projecting a sufficiently long pattern so that the changes in the light and dark projection patterns do not coincide with each other, the coordinates corresponding to each code can be uniquely set. Since various techniques are known for the procedure of the spatial coding method, a detailed description thereof will be omitted in this embodiment. Accurate calibration can be performed by using an object, such as the board 820 in the above-described embodiment, that has a known three-dimensional shape and allows the user to estimate which part of each three-dimensional shape is being observed based on the marks. Specifically, the marks allow estimation of which part of the three-dimensional shape is being observed by each sensor, and the deviation between the observation results estimated by the spatial encoding method and the actual observation results can be estimated. Then, by calculating correction parameters to correct this deviation, calibration of each sensor can be performed. As described in the above-described embodiment, the board 820 may be an object of other shapes. That is, it may be of other shapes, such as a cube or a rectangular parallelepiped, as long as the shape of the surface on which the marks are attached can be recorded. Similarly, when only calibration is performed in the direction along the surface of the transport conveyor 302, the inclined surface 822 may be omitted.

[0113] When the board 820 is provided with an inclined surface 822 or another three-dimensional shape is used instead of the board 820, it is preferable to provide a mark 823 on the side of the inclined surface 822 or the three-dimensional shape so that the projector 105 can project onto it and the sensors can observe it. In this way, the projector 105 can project a light and dark pattern onto the location of the mark 823, and the sensors can observe the mark 823 onto which the light and dark pattern is projected, so that calibration in the height direction can be performed with high precision.

[0114] In the above-described embodiment, the completion of the calibration preparation is confirmed by an input from the installer. However, the information processing device 104 may automatically determine the completion of the calibration preparation. For example, the information processing device 104 may automatically determine that the calibration preparation is complete when the first index 911 is equal to or greater than the first threshold 912 and the second index 921 is equal to or greater than the second threshold 922. In this case, the information processing device 104 may notify the installer of the completion of the calibration preparation by voice, image, or the like, to prevent the installer from further moving the board 820. In this case, the information processing device 104 may also receive an instruction from the installer indicating whether or not to resume the calibration. This is because changing the position of the board 820 may further improve the accuracy of the correction parameters.

[0115] In the above-described embodiment, calibration was performed regarding the positions of the infrared camera 106 and the imaging camera 107, but this embodiment may also be applied to calibration regarding the positions of cameras of the same type, i.e., between infrared cameras or between imaging cameras.

[0116] In addition, the programs and applications for realizing the functions of one or more of the above-described embodiments can be supplied to a system or device using a network or storage medium, etc., and one or more processors in the computer of the system or device can read and execute the programs.

[0117] Alternatively, it may be realized by a circuit that realizes one or more functions (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array)).

[0118] Summary of the Disclosure The contents of this disclosure can be expressed as follows:

[0119] [Expression 1] The information processing device (104, 200) disclosed herein is an information processing device that calibrates the position of an imaging camera 107 that captures an image used to detect a baggage 300 being transported, and the position of an infrared camera 106 that captures an infrared image used to detect the baggage 300, and is equipped with a processor 201 and a memory 202, and the processor 201 acquires an image of a board 820 having a plurality of marks 823 from the imaging camera 107, acquires an infrared image of the board 820 from the infrared camera 106, detects the marks 823 from the image, calculates a first index 911 related to the calibration of the image based on the number of detected marks 823, detects the marks 823 from the infrared image, calculates a second index 921 related to the calibration of the infrared image based on the number of detected marks 823, and outputs information indicating the first index 911 and the second index 921. This allows the installer performing the calibration work to place the board 820 at a position where the accuracy of the correction parameters for the captured image and the correction parameters for the infrared image is sufficient, based on the output first index 911 and second index 921. In other words, the efficiency of the calibration work is improved.

[0120] [Expression 2] In the information processing device described in Expression 1, the processor 201 may set a predetermined first threshold 912 to be compared with the first index 911 and a predetermined second threshold 922 to be compared with the second index 921, and output a first index image 910 indicating whether the first index 911 is greater than or equal to the first threshold 912 and a second index image 920 indicating whether the second index 921 is greater than or equal to the second threshold 922. This allows the installer to look at the first index image 910 and the second index image 920 and recognize at a glance whether the first index 911 is equal to or greater than the first threshold value 912 and whether the second index 921 is equal to or greater than the second threshold value 922 at the placement position of the board 820. This allows the installer to more easily place the board 820 at a position where the accuracy of the correction parameters for the captured image and the correction parameters for the infrared image is sufficient.

[0121] [Expression 3] In the information processing device described in Expression 1, the processor 201 may output an image of a different form from the image output when the first index 911 is greater than or equal to the first threshold 912 and the second index 921 is greater than or equal to the second threshold 922, when the first index 911 is less than the first threshold 912, or when the second index 921 is less than the second threshold 922. This allows the installer to recognize at a glance from the output image whether the first index 911 is equal to or greater than the first threshold 912 and whether the second index 921 is equal to or greater than the second threshold 922.

[0122] [Expression 4] In the information processing device described in Expression 2 or 3, the processor 201 may set a first threshold 912 so that the first index 911 falls below the first threshold 912 when the number of marks 823 detected from the captured image is less than a predetermined number, and may set a second threshold 922 so that the second index 921 falls below the second threshold 922 when the number of marks 823 detected from the infrared image is less than the predetermined number. This allows the installation worker to recognize that if the first index 911 is below the first threshold 912, or if the second index 921 is below the second threshold 922, the number of detected marks 823 is less than the specified number and the accuracy of the correction parameters is insufficient.

[0123] [Expression 5] In an information processing device described in any one of Expressions 1 to 4, the board 820 includes a plane 821 along the surface on which the luggage is transported and a slope 822 extending at an angle in the height direction from one side of the plane 821, and the processor 201 may calculate a first index 911 based on the number of marks 823 detected from the plane 821 in the captured image and the number of marks 823 detected from the slope 822 in the captured image, and calculate a second index 921 based on the number of marks 823 detected from the plane 821 in the infrared image and the number of marks 823 detected from the slope 822 in the captured image. As a result, the first index 911 or the second index 921 is calculated based on the number of marks 823 detected from the flat surface 821 and the inclined surface 822, respectively, so that it is possible to more accurately indicate whether the accuracy of the correction parameters is sufficient.

[0124] [Expression 6] In the information processing device described in Expression 1, the luggage is luggage onto which a projection image is projected from a projection device (e.g., projector 105), and the processor 201 may project a range within which the projection device can project a projection image, taking the range within which the board 820 can be installed as the range within which the projection device can project a projection image. This allows the installer to easily place the board 820 within a range where the projection image can be projected.

[0125] [Expression 7] In the information processing device described in Expression 1, the luggage is luggage onto which a projected image is projected from a projection device (e.g., projector 105), the projection device projects a predetermined image onto board 820, and processor 201 may calculate correction parameters to be used for calibrating the projection device and imaging camera 107 based on an image captured of board 820 onto which the predetermined image is projected, and may calculate correction parameters to be used for calibrating the projection device and infrared camera 106 based on an infrared image of board 820 onto which the predetermined image is projected. This makes it possible to calculate the correction parameters used for calibrating the projection device and the imaging camera 107, and the correction parameters used for calibrating the projection device and the infrared camera 106.

[0126] [Expression 8] The baggage sorting system 100 of the present disclosure includes a projection device (e.g., a projector 105) that projects a projected image onto the baggage 300 being conveyed, an imaging camera 107 that captures an image used to detect the baggage 300, an infrared camera 106 that captures an infrared image used to detect the baggage 300, and an information processing device (e.g., a sorting client 104) that controls the projection device, the imaging camera 107, and the infrared camera 106. The information processing device performs calibration for the position of the imaging camera 107 and the position of the infrared camera 106 by performing a plurality of An image of a board 820 having marks 823 is obtained from an imaging camera 107, an infrared image of the board 820 is obtained from an infrared camera 106, the marks 823 are detected from the image, a first index 911 relating to the calibration of the image is calculated based on the number of detected marks 823, the marks 823 are detected from the infrared image, a second index 921 relating to the calibration of the infrared image is calculated based on the number of detected marks 823, and information indicating the first index 911 and the second index 921 is output. This allows the installer performing the calibration work to place the board 820 at a position where the accuracy of the correction parameters for the captured image and the correction parameters for the infrared image is sufficient, based on the output first index 911 and second index 921. In other words, the efficiency of the calibration work is improved.

[0127] [Expression 9] A calibration method of the present disclosure, in which an information processing device (e.g., a sorting client 104) calibrates the position of an imaging camera 107 that captures an image used to detect transported luggage 300, and the position of an infrared camera 106 that captures an infrared image used to detect luggage 300, involves having an installation worker install a board 820 having a plurality of marks 823, obtaining an image of the board 820 from the imaging camera 107, obtaining an infrared image of the board 820 from the infrared camera 106, detecting marks 823 from the image, calculating a first index 911 related to the calibration of the image based on the number of detected marks 823, detecting marks 823 from the infrared image, calculating a second index 921 related to the calibration of the infrared image based on the number of detected marks 823, and displaying information indicating the first index 911 and the second index 921. This allows the installer performing the calibration work to place the board 820 at a position where the accuracy of the correction parameters for the captured image and the correction parameters for the infrared image is sufficient, based on the output first index 911 and second index 921. In other words, the efficiency of the calibration work is improved.

[0128] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to these examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents may be made within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0129] The present disclosure is useful for improving the efficiency of baggage sorting work. [Explanation of symbols]

[0130] 100...Baggage sorting system 101...Sorting management server 102... Collaboration server 103...Label reader 104...Sorting client 105...Projector 106...Distance sensor (infrared camera) 107...Image sensor (imaging camera) 110...Client system 300…Luggage 301...Label 302...Transport conveyor 820...Board 821…Plane 822...Slope 823...Mark 900...Display device 910...First indicator image 911…the first indicator 912...First threshold 920...Second indicator image 921...Second indicator 922...Second threshold

Claims

1. An information processing device that performs calibration regarding the position of an imaging camera that captures an image of a predetermined range and the position of an infrared camera that captures an infrared image of the predetermined range, a processor and a memory, The processor: Acquiring the captured image of a board having a plurality of marks from the imaging camera; The infrared image of the board is acquired from the infrared camera; Detecting the marks from the captured image, and calculating a first index related to calibration of the captured image based on the number of the detected marks; Detecting the marks from the infrared image and calculating a second index related to the calibration of the infrared image based on the number of the detected marks; outputting information indicating the first index and the second index; Information processing device.

2. The processor: setting a predetermined first threshold to be compared with the first indicator and a predetermined second threshold to be compared with the second indicator; outputting a first index image indicating whether the first index is equal to or greater than the first threshold value and a second index image indicating whether the second index is equal to or greater than the second threshold value; The information processing device according to claim 1 .

3. The processor: outputting an image in a manner different from an image output when the first index is equal to or greater than the first threshold and the second index is equal to or greater than the second threshold, when the first index is less than the first threshold, or when the second index is less than the second threshold; The information processing device according to claim 2 .

4. The processor: setting the first threshold value such that the first index falls below the first threshold value when the number of the marks detected from the captured image is less than a predetermined number; setting the second threshold value so that the second indicator falls below the second threshold value when the number of the marks detected from the infrared image is less than a predetermined number; 4. The information processing device according to claim 2 or 3.

5. The imaging camera captures an image used to detect the baggage being transported, and the infrared camera is used to detect the baggage. The information processing device according to claim 1 .

6. the board includes a flat surface along a surface on which the luggage is transported and an inclined surface extending obliquely in a height direction from one side of the flat surface, The processor: calculating the first index based on the number of the marks detected from the flat surface of the captured image and the number of the marks detected from the inclined surface of the captured image; calculating the second index based on the number of the marks detected from the flat surface of the infrared image and the number of the marks detected from the inclined surface of the captured image; The information processing device according to claim 5 .

7. the luggage is luggage onto which a projection image is projected from a projection device, the processor projects a range in which the projection device can project a projection image, the range being an area in which the board can be installed; The information processing device according to claim 5 .

8. the luggage is luggage onto which a projection image is projected from a projection device, the projection device projects a predetermined image onto the board; the processor calculates correction parameters to be used for calibrating the projection device and the imaging camera based on the captured image of the board onto which the predetermined image is projected; calculating correction parameters to be used for calibrating the projection device and the infrared camera based on the infrared image of the board onto which the predetermined image is projected; The information processing device according to claim 5 .

9. An imaging camera that captures an image of a predetermined range; an infrared camera that captures an infrared image of the predetermined range; an information processing device that controls the imaging camera and the infrared camera, In the calibration of the position of the imaging camera and the position of the infrared camera, the information processing device Acquiring the captured image of a board having a plurality of marks from the imaging camera; The infrared image of the board is acquired from the infrared camera; Detecting the marks from the captured image, and calculating a first index related to calibration of the captured image based on the number of the detected marks; Detecting the marks from the infrared image and calculating a second index related to the calibration of the infrared image based on the number of the detected marks; outputting information indicating the first index and the second index; Baggage detection system.

10. A calibration method for performing calibration, by an information processing device, for a position of an imaging camera that captures an image of a predetermined range and a position of an infrared camera that captures an infrared image of the predetermined range, the method comprising: having an installer install a board having a plurality of marks; The captured image of the board is acquired from the imaging camera; The infrared image of the board is acquired from the infrared camera; Detecting the marks from the captured image, and calculating a first index related to calibration of the captured image based on the number of the detected marks; Detecting the marks from the infrared image and calculating a second index related to the calibration of the infrared image based on the number of the detected marks; displaying information indicating the first index and the second index; Calibration method.

Images