Information Processing Apparatus, Information Processing System, Information Processing Method, and Program

Abstract

To provide an information processor that can easily process positional information of an object that is moved by a mobile body.SOLUTION: The information processor according to an embodiment of the present invention is for processing positional information of an object that is moved by a mobile body, and includes an output unit for outputting positional information of the object acquired on the basis of the positional information of the mobile body acquired on the basis of an image taken by an imaging unit and holding information showing one of a holding state and a not-holding state of the object by the mobile body.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present application relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, an information processing apparatus that processes position information of an object such as a cargo or a pallet moved by a moving body such as a forklift is known.

[0003] Further, a processing unit that classifies the operating state of a transport machine is provided using a first signal transmitted from a first detector that directly detects the movement of the transport machine and a second signal transmitted from a second detector that detects the presence or absence of an article in the transport machine, and information indicating the position of the transport machine is used to output information indicating the movement path of the transport machine for each classified operating state (see, for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the apparatus of Patent Document 1, since signals from a plurality of detectors are processed, the processing of the position information of the object may be complicated.

[0005] An object of the present invention is to provide an information processing apparatus that can easily process the position information of an object moved by a moving body.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present invention is an information processing apparatus that processes position information of an object moved by a moving body, and outputs the position information of the object acquired based on the position information of the moving body acquired based on an imaging image by an imaging unit and holding information indicating either a state of holding or not holding the object by the moving body. The position information of the object is shown in association with an ID number of a section indicating a predetermined position range, position information of the section, and a barcode indicating the section.An operator working in the warehouse registers information indicating either the incoming or outgoing of the object together with information regarding the initial position of the object, and causes a screen for confirming the ID numbers of the object and the section to be displayed on a mobile terminal carried by the operator. It follows.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide an information processing apparatus capable of easily processing the position information of an object moved by a moving body.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Mode for Carrying Out the Invention

[0009] Hereinafter, modes for carrying out the invention will be described with reference to the drawings. In each drawing, the same reference numerals are given to the same components, and redundant descriptions may be omitted.

[0010] The information processing apparatus according to the embodiment processes the position information of an object moved by a moving body. For example, the moving body is a forklift, and the object includes a pallet. The information processing apparatus according to the embodiment processes the position information of an object such as a pallet moved by a forklift, recognizes the movement of the object, and also tracks it.

[0011] Here, FIG. 1 is a diagram for explaining an example of the position information of an object in a warehouse. FIG. 1 shows the inside of the warehouse 100 and the periphery of the warehouse 100 as viewed from above (ceiling side).

[0012] The warehouse 100 is a terminal warehouse (a warehouse installed at a transportation relay point). This terminal warehouse has a warehouse form called a cross-docking type. In a cross-docking type warehouse, a plurality of pallets for each product are received from a factory or a wholesaler, etc., and temporarily placed in the warehouse. Then, at the time of shipment, a plurality of types of pallets are combined with the same pallet in the same loading state and shipped to different retail stores.

[0013] In FIG. 1, there is a truck yard 200 around the warehouse 100. FIG. 1 shows the state where the container 300 carried by the trailer and separated, and the truck bed is connected to the warehouse in the truck yard 200. The forklift 10 takes out the pallet 31 from at least one of the truck bed of the truck that has arrived at the truck yard 200 or the container 300 carried by the trailer.

[0014] After that, the forklift 10 transports the pallet 31 to the temporary storage location 40 and temporarily stores it. Then, at the time of shipment, the forklift 10 transports the pallet 31 to a location near the truck yard 200 in the warehouse for sorting, and then loads the pallet 31 onto the truck bed or the container 300.

[0015] In the temporary storage location 40, in order to flexibly secure space against the daily fluctuations in the types and quantities of incoming and outgoing goods, there are often no partitions for each product. On the other hand, since a plurality of workers temporarily store the pallet 31 at arbitrary locations, it is necessary to search for the desired pallet from among the plurality of temporarily stored pallets at the time of shipment.

[0016] In order to perform this search operation efficiently, an information processing system that visualizes by recognizing and tracking the movement of the pallet 31 in the warehouse 100 while effectively utilizing the space by not defining the temporary storage location of the pallet 31 is required. The information processing device according to the embodiment is, as an example, used in such an information processing system.

[0017] Hereinafter, an information processing system having the information processing device according to the embodiment will be described.

[0018] [First Embodiment] (Overall Configuration Example of Information Processing System 1) FIG. 2 is a diagram showing an example of the overall configuration of the information processing system 1 according to the first embodiment. As shown in FIG. 2, the information processing system 1 includes a forklift 10, an omnidirectional camera 20, and an on-premises server 50. These are communicably connected via a network 400 such as a LAN (Local Area Network). Note that devices other than the above, such as an external server or an image forming apparatus, may be communicably connected to the network 400.

[0019] The forklift 10 is an example of a moving body that holds the cargo 32 placed on the pallet 31 and transports the pallet 31 and the cargo 32 in a held state. Note that the transportation by the moving body is an example of the movement by the moving body. Also, the pallet 31 and the cargo 32 are each an example of an object. In the following, when the pallet 31 and the cargo 32 are not particularly distinguished, they are collectively referred to as the object 30. Also, the forklift 10 is a generic notation for a plurality of forklifts, the pallet 31 is a generic notation for a plurality of pallets, and the cargo 32 is a generic notation for a plurality of cargos.

[0020] The forklift 10 may transport the object 30 in response to the driving operation of an operator, or may transport the object 30 by autonomous driving without going through an operator.

[0021] The omnidirectional camera 20 is an example of an imaging unit provided on the forklift 10. The omnidirectional camera 20 is a camera capable of imaging 360 degrees in all directions around the omnidirectional camera 20. The azimuth 20a indicates the azimuth that the omnidirectional camera 20 can image.

[0022] The omnidirectional image (all-round image) captured by the omnidirectional camera 20 is an example of a captured image. However, the imaging unit is not limited to the omnidirectional camera 20, and any device that can image the surroundings of the forklift 10 may be used. Also, the captured image does not necessarily have to be an omnidirectional image.

[0023] The omnidirectional image includes an image of the scenery on the conveyance direction 11 side of the object 30 viewed from the forklift 10 and an image of the scenery on the vertically upward direction 12 side viewed from the forklift 10. In other words, the conveyance direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the omnidirectional camera 20 can image the entire range, it can image the front and above of the forklift 10 in a single image. Note that the conveyance direction 11 is an example of the moving direction.

[0024] The omnidirectional camera 20 is preferably attached on the roof of the forklift 10 or on the support member 22 that supports the fork 21. Thereby, a suitable view for imaging the front and above of the forklift 10 can be ensured.

[0025] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the on-premises server 50 via the network 400.

[0026] The cargo 32 is provided with a barcode 33 which is an example of identification information indicating the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information indicating the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information which is the result of the reading is transmitted to the on-premises server 50 via the network 400. Note that the identification information is not limited to the barcode and may be a QR code (registered trademark) or an ID (Identifier) number or the like.

[0027] The on-premises server 50 is installed inside the warehouse 100 and is an example of an information processing device that processes the position information of the object 30 conveyed by the forklift 10.

[0028] The on-premises server 50 processes the position information of the object 30 based on the omnidirectional image received via the network 400 and the identification information indicating the object 30. Also, by using the omnidirectional images captured by the plurality of omnidirectional cameras 20 respectively, the position information of the objects 30 held by the plurality of forklifts 10 can be obtained.

[0029] (Hardware configuration example of on-premises server 50) Next, FIG. 3 is a block diagram showing an example of the hardware configuration of the on-premises server 50. The on-premises server 50 is constructed by a computer.

[0030] As shown in FIG. 3, the on-premises server 50 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, and a display 506. The on-premises server 50 also includes an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0031] Among these, the CPU 501 controls the operation of the entire on-premises server 50. The ROM 502 stores programs used for driving the CPU 501 such as IPL. The RAM 503 is used as a work area for the CPU 501.

[0032] The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images.

[0033] The external device connection I / F 508 is an interface for connecting various external devices. Examples of external devices in this case include a USB (Universal Serial Bus) memory, a printer, and the like. The network I / F 509 is an interface for data communication using the network 400. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 3.

[0034] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 514 controls reading or writing of various data with respect to the DVD-RW 513 as an example of a removable recording medium. Note that it is not limited to DVD-RW, and it may be DVD-R or the like. The media I / F 516 controls reading or writing (storage) of data with respect to the recording medium 515 such as a flash memory.

[0035] (Functional configuration example of the information processing system 1) FIG. 4 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in FIG. 4, the on-premises server 50 includes a receiving unit 51, a mobile body position acquisition unit 52, a held information acquisition unit 53, an identification information acquisition unit 54, a time acquisition unit 55, an object position acquisition unit 56, an output unit 57, and a storage unit 58.

[0036] Each of these units is a function or means realized by operating any of the components shown in FIG. 3 according to an instruction from the CPU 501 according to a program expanded from the HD 504 onto the RAM 503.

[0037] In addition, the forklift 10 is provided with an omnidirectional camera 20 and a transmission unit 101. The function of the transmission unit 101 can be realized by an electric circuit provided in either the forklift 10 or the omnidirectional camera 20, or can also be realized by software (CPU). It may also be realized by a plurality of circuits or a plurality of software.

[0038] The on-premises server 50 acquires the position information of the object 30 based on the position information of the forklift 10 obtained based on the omnidirectional image by the omnidirectional camera 20 and the holding information indicating either the state of holding or non-holding of the object 30 by the forklift 10. Then, the acquired position information of the object 30 can be output to the outside via the output unit 57.

[0039] The receiving unit 51 receives the omnidirectional image captured by the omnidirectional camera 20 and transmitted via the transmission unit 101 via the network 400, and outputs it to each of the moving body position acquisition unit 52 and the holding information acquisition unit 53. The receiving unit 51 also receives the identification information read by a reader such as a barcode reader via the network 400, and outputs it to the identification information acquisition unit 54.

[0040] The moving body position acquisition unit 52 acquires the position information of the forklift 10 by calculation based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0041] The holding information acquisition unit 53 acquires, by calculation, the holding information indicating either the state of holding or non-holding of the object 30 by the forklift 10 based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0042] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the reception unit 51, and outputs it to the object position acquisition unit 56. However, the acquisition of the identification information by the identification information acquisition unit 54 is not limited to that via the network 400. For example, the identification information acquisition unit 54 may acquire the identification information input by a user such as an administrator using the keyboard 511 or the pointing device 512 in FIG. 3, or may acquire the identification information stored in advance in the storage unit 58, or may acquire the identification information via the external device connection I / F 508. Note that the administrator is the administrator of the information processing system 1 or the warehouse 100.

[0043] The time acquisition unit 55 acquires information indicating the time when the reception unit 51 receives the omnidirectional image and the identification information, and outputs it to the object position acquisition unit 56.

[0044] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information. Further, the object position acquisition unit 56 associates the position information of the object 30, the identification information indicating the object 30, and the time information with each other, and outputs them via the output unit 57. The output destination of the output unit 57 is an external device such as a PC (Personal Computer), a display device such as the display 506, or a storage device such as the HD 504.

[0045] The storage unit 58 can store the identification information indicating the object 30 such as the pallet 31 or the cargo 32.

[0046] (Example of processing by the on-premises server 50) FIG. 5 is a flowchart showing an example of the processing by the on-premises server 50. FIG. 5 shows the processing triggered by the timing when the on-premises server 50 receives an operation to start acquiring the position information of the object 30. The operation to start acquiring the position information of the object 30 is performed by a user such as an administrator using the pointing device 512 in FIG. 3.

[0047] First, in step S51, the reception unit 51 receives the omnidirectional image and the identification information via the network 400.

[0048] Subsequently, in step S52, the mobile body position acquisition unit 52 acquires, by calculation, the position information of the forklift 10 based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0049] Subsequently, in step S53, the holding information acquisition unit 53 acquires, by calculation, holding information indicating either the holding or non-holding state of the object 30 by the forklift 10 based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0050] Subsequently, in step S54, the identification information acquisition unit 54 acquires the identification information indicating the object 30 by inputting it, and outputs it to the object position acquisition unit 56.

[0051] Subsequently, in step S55, the time acquisition unit 55 acquires information indicating the time when the reception unit 51 received the omnidirectional image and the identification information, and outputs it to the object position acquisition unit 56.

[0052] Note that the processes in steps S52 to S55 may be appropriately rearranged in order, or each may be performed in parallel.

[0053] Subsequently, in step S56, the object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information.

[0054] Subsequently, in step S57, the object position acquisition unit 56 associates the position information of the object 30, the identification information indicating the object 30, and the time information with each other, and outputs them via the output unit 57.

[0055] Subsequently, in step S58, the on-premises server 50 determines whether to end the process. The on-premises server 50 determines whether to end the process based on an end operation by a user such as an administrator via a pointing device 512 or the like.

[0056] In step S58, if it is determined that the process ends (step S58, Yes), the on-premises server 50 ends the process. On the other hand, if it is determined that the process does not end (step S58, No), the on-premises server 50 performs the processes after step S51 again.

[0057] In this way, the on-premises server 50 can process the position information of the object 30 and obtain the position information of the object 30.

[0058] (Example of acquisition result of object position information) FIG. 6 is a diagram showing a first example of the acquisition result of object position information. FIG. 6(a) is a diagram showing a position map, and FIG. 6(b) is a diagram showing position information and a time stamp.

[0059] The position map 61 shown in FIG. 6(a) is created by the moving body position acquisition unit 52 based on the omnidirectional image captured by the omnidirectional camera 20. The moving body position acquisition unit 52 acquires a point group including three-dimensional coordinate information, and creates the position map 61 by projecting this point group onto a two-dimensional plane.

[0060] The position map 61 is created, for example, according to the movement of one forklift 10 and is used in the acquisition (self-position recognition) of the position information of all the forklifts 10. Thereby, each position of a plurality of forklifts 10 can be expressed in the same coordinate system.

[0061] As a method for creating the position map 61 based on the captured image such as the omnidirectional image, for example, the simultaneous execution technology of map creation and self-position recognition (SLAM: Simultaneous Localization and Mapping) can be applied (for example, refer to "Explanation of the Current Situation and Future Prospects of SLAM," Masahiro Tomonou, Yoshiaki Hara, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ). Such self-position recognition based on the captured image can acquire position information with higher accuracy than other position acquisition methods using GPS or an acceleration sensor when the captured image can be stably acquired, such as in an indoor environment.

[0062] In a flat-type warehouse, which is common in a cross-docking type warehouse, the position map 61 is likely to change according to the position change of the object 30 accompanying the incoming and outgoing of the pallet 31. The change in the position map 61 may complicate the acquisition process of the position information. Note that the flat-type warehouse refers to a type of warehouse with few structures such as shelves and places the pallet 31 or the cargo 32 on the floor surface.

[0063] On the other hand, warehouses often have a high ceiling height of 5 [m] or more, and it is easy to secure an upper field of view on the ceiling side. Therefore, in the embodiment, the position map 61 is created using the image indicating the upward direction among the omnidirectional images captured by the omnidirectional camera 20, and the position information of the forklift 10 is acquired. Thereby, using the position map 61 with suppressed time change, the position information of the forklift 10 can be acquired without complicating the process.

[0064] In addition, the holding information acquisition unit 53 uses the image indicating the front among the omnidirectional images in order to acquire the holding information of the object 30 by the forklift 10. Thereby, using one omnidirectional image captured by the omnidirectional camera 20, it is possible to acquire the position information of the forklift 10 and the holding information of the object 30.

[0065] Note that the holding information acquisition unit 53 can also use information other than the omnidirectional image to acquire the holding information. Examples of information other than the omnidirectional image include detection information from a contact sensor, an infrared sensor, an ultrasonic sensor, a distance measurement sensor, a load sensor, or the like. The holding information acquisition unit 53 can also acquire the holding information by combining the omnidirectional image or the detection information from each sensor. However, from the viewpoint of simplifying the processing, it is more preferable to use the omnidirectional camera 20 capable of acquiring the position information of the forklift 10 and the holding information from one omnidirectional image.

[0066] The object position acquisition unit 56 can acquire a position map 62 indicating the position of the object 30 by using the position at which the holding information changes from being held to not being held among the position information of the forklift 10 shown in the position map 61. Since the position map 61 shown in FIG. 6(a) can also be said to indicate the position information of the object 30, the position map 62 is shown in parentheses in FIG. 6(a).

[0067] In addition, the position table 63 shown in FIG. 6(b) is a table including the three-dimensional coordinates of the point group indicating the position of the object 30 and a time stamp 64 indicating the time when the three-dimensional coordinates were acquired. The position table 63 is created by the object position acquisition unit 56. The time stamp 64 is an example of the time information acquired by the time acquisition unit 55. The object position acquisition unit 56 can output the position table 63 in which the position information of the object 30 and the time information are associated with each other via the output unit 57.

[0068] The object position acquisition unit 56 excludes the information of the point cloud after a predetermined time has elapsed, and updates the position table 63 using the information of the new point cloud. As a result, even when it is difficult to secure the upper field of view and it is necessary to rely on the horizontal field of view, the position information of the forklift 10 and the position information of the object 30 can be suitably acquired in response to changes in the surrounding state of the forklift 10. However, when the movement of the forklift 10 is small, since the number of newly acquired point clouds is small, the number of points may decrease. In this regard, in order to prevent the information of the point cloud shown in the position table 63 from decreasing too much, it is preferable to lower the update frequency of the table 63 when the movement amount is small according to the movement amount of the forklift 10, so that the number of points does not decrease too much.

[0069] In order to track the positions of the plurality of objects 30 in the warehouse 100, identification information such as barcodes attached to the objects 30 is used. The identification information indicating the object 30 is stored in the storage unit 58 together with the initial position information of the object 30.

[0070] As the identification information indicating the object 30, when the forklift 10 takes out the pallet 31 from the container 300 or the like and temporarily places the pallet 31, the barcode or the like attached to the cargo 32 or the pallet 31 is read and stored in the storage unit 58. Such storage is called initial registration. In the present embodiment, in addition to the identification information indicating the object 30 that is initially registered, the position information of the object 30 acquired by the object position acquisition unit 56 can be stored in the storage unit 58 in association therewith.

[0071] Here, an example of a simpler method for tracking the positions of the plurality of objects 30 in the warehouse 100 will be described with reference to FIG. 7. FIG. 7 is a diagram showing a second example of object position information. FIG. 7(a) is a diagram showing sections, and FIG. 7(b) is a diagram showing a barcode indicating position information and sections.

[0072] The sections A, B, C, and D shown in FIG. 7(a) indicate position ranges predetermined in the warehouse 100. Operations for initial registration of the object 30 are performed within each of the sections A, B, C, and D.

[0073] By reading the barcodes indicating compartments A, B, C, and D at either the timing before or after reading the barcode indicating the object 30 using a reader, the position coordinates indicating the object 30 and each compartment can be associated. Thereafter, when the object 30 is transported by the forklift 10, the position of the object 30 can be tracked including the identification information of which object 30 is being transported. Table 71 shown in FIG. 7(b) shows the association of the ID number indicating the compartment, the position information of the compartment, and the barcode indicating the compartment.

[0074] The barcodes indicating each compartment are printed and carried by the operator so that they can be read by the operator using a reader for initial registration. Alternatively, they are attached to the floor, pillars in the warehouse 100, or the forklift 10. Thereby, the barcodes of each compartment can be easily read at the timing before and after performing the operation of reading the barcode indicating the object 30.

[0075] FIG. 8 is a diagram showing an example of a screen for an operator to register the initial position of the object 30. For example, a mobile terminal for reading the barcode indicating the object 30 is attached on the forklift 10, and the operator performs the registration operation using this mobile terminal. The registration screen 81 shown in FIG. 8 is a screen displayed on the mobile terminal. Also, the information read by the simple method shown in FIG. 7 can be displayed on the mobile terminal, and the operator can be made to select information indicating either the incoming or outgoing of the cargo 32, or only the confirmation of the ID numbers of the object 30 and the compartments can be requested from the operator.

[0076] FIG. 9 is a diagram showing another example of a screen for an operator to register the initial position of the object 30. The registration screen 91 shown in FIG. 9 is a display screen of the mobile terminal and displays the barcode 92 indicating the compartment. The registration screen 91 displays the state of either the incoming or outgoing of the cargo 32 and the barcode indicating the compartment that is a candidate for the initial position, and the initial position can also be registered by the operator reading the appropriate barcode using a reader.

[0077] FIG. 10 is a diagram showing an example of a display screen of the destination of the object 30 being transported by the forklift 10. The display screen 93 displays a position map 94. The position map 94 includes a source position 95 and a destination position 96. The source position 95 indicates the position of the source, and the destination position 96 indicates the position of the destination. By displaying the positions such as the pre-registered temporary storage locations on the screen, more user-friendly information can be provided.

[0078] FIG. 11 is a diagram showing an example of a screen for displaying the position map of the object 30. The display screen 111 may be a screen displayed on a mobile terminal provided in the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0079] After the object 30 is initially registered in the warehouse 100, the on-premises server 50 displays the ID numbers 112 or 113, etc. indicating all the objects 30 on the map based on their respective position coordinate information during the period until the barcode is read at the time of shipment.

[0080] FIG. 12 is a diagram showing an example of a screen displayed when searching for the object 30. The display screen 121 may be a screen displayed on a mobile terminal provided in the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0081] When searching for the object 30, when the operator performing the transportation enters the ID number of the pallet 31 that the operator wants to search for, the display screen 121 displays a mark 122 indicating the position of the corresponding pallet 31 on the position map 123. At this time, the pallet 31 for which the position is displayed is not limited to one, and the ID numbers of a plurality of pallets 31 may be input in sequence so that the positions of the plurality of pallets 31 are displayed collectively.

[0082] (Function and Effect of the On-Premises Server 50) As described above, the on-premises server 50 (information processing apparatus) included in the information processing system 1 according to the present embodiment processes the position information of the object 30 transported (moved) by the forklift 10 (mobile object).

[0083] The on-premises server 50 outputs the position information of the object 30 obtained based on the position information of the forklift 10 obtained based on the omnidirectional image (captured image) by the omnidirectional camera 20 (imaging unit) and the holding information indicating either the state of holding or non-holding of the object 30 by the forklift 10.

[0084] In order to obtain the position information of the forklift 10 based on the image, it can be realized with a minimum device configuration as compared with, for example, a method of installing a plurality of wireless communication tags indoors and installing a device such as a receiver on the mobile object. Thereby, an on-premises server 50 that can easily process the position information of the object 30 transported by the forklift 10 can be provided.

[0085] Also, in the present embodiment, the holding information acquisition unit 53 acquires the holding information based on the omnidirectional image. Thereby, since the position information of the forklift 10 and the holding information can be acquired using one omnidirectional image, the position information of the object 30 can be processed more easily.

[0086] Also, in the present embodiment, the omnidirectional image is an image captured by the omnidirectional camera 20 provided on the forklift 10. As a result, the imaging range by the omnidirectional camera 20 changes according to the movement of the forklift 10 inside or outside the warehouse 100, so that a wider range inside or outside the warehouse 100 can be imaged.

[0087] In this embodiment, the omnidirectional image includes an image of the scenery on the conveyance direction 11 side of the object 30 viewed from the forklift 10 and an image of the scenery on the vertically upward direction 12 side viewed from the forklift 10. From the image of the scenery on the ceiling side of the warehouse where it is easy to secure the field of view, the position information of the forklift 10 can be obtained by simple processing. Also, holding information indicating whether the object 30 is held or not held in the scenery on the conveyance direction 11 side of the forklift 10 can be obtained. Thereby, with the configuration of the simple information processing system 1, the position of the object 30 can be accurately recognized and tracked.

[0088] In this embodiment, the on-premises server 50 has a receiving unit 51 that receives at least one of the holding information or the position information of the forklift 10. Using at least one of the received holding information or the position information of the forklift 10, the position information of the object 30 can be obtained. When the receiving unit 51 receives this information by wireless communication, complex wiring can be omitted, which is more preferable.

[0089] In this embodiment, a time stamp 64 (information indicating time) corresponding to the position of the object 30 is further output. Thereby, the position of the object 30 becomes easier to recognize and track.

[0090] In this embodiment, the on-premises server 50 acquires identification information indicating the object 30 and outputs the identification information in association with the position information of the object 30. Thereby, the positions of the plurality of objects 30 become easier to recognize and track.

[0091] [Second Embodiment] Next, the information processing system 1a according to the second embodiment will be described. Note that the same component numbers as those described in the first embodiment are assigned to the same components, and redundant descriptions are omitted as appropriate.

[0092] FIG. 13 is a diagram showing an example of the overall configuration of the information processing system 1a. As shown in FIG. 13, the information processing system 1a includes a fixed camera 60 and an on-premises server 50a.

[0093] Here, in the warehouse, in addition to the forklift 10, a pallet handling device called a hand pallet truck that can be easily handled by a person may be used. In the present embodiment, even when the position of the object 30 is changed by a hand pallet truck not provided with the omnidirectional camera 20, the position of the object 30 is recognized based on the captured image by the fixed camera 60 and made trackable.

[0094] The fixed camera 60 is a camera provided near the ceiling of the warehouse 100 or the like and photographs the inside of the warehouse 100 from the ceiling side toward the floor side. There is no particular limitation on the type of the fixed camera 60, but it is preferably one that can image a wide range, and a hemispherical (fisheye camera) with an angle of view of about 180 degrees is more preferable. The number of fixed cameras 60 installed may be one or a plurality. In the present embodiment, a plurality of fixed cameras 60 are provided, and the fixed camera 60 is a general term for a plurality of fixed cameras.

[0095] (Functional configuration example of the information processing system 1a) FIG. 14 is a block diagram showing an example of the functional configuration of the information processing system 1a. As shown in FIG. 14, on the ceiling 500, a fixed camera 60, a ceiling object position acquisition unit 501, and a transmission unit 502 are provided.

[0096] Each function of the ceiling object position acquisition unit 501 and the transmission unit 502 is realized by an electric circuit provided in either the ceiling 500 or the fixed camera 60, and a part of these functions can also be realized by software (CPU). Also, these functions may be realized by a plurality of circuits or a plurality of software.

[0097] Here, the fixed camera 60 is an example of a second imaging unit provided outside the forklift 10 and imaging the periphery of either the forklift 10 or the object 30. The image captured by the fixed camera 60 is an example of a second captured image.

[0098] The ceiling object position acquisition unit 501 acquires the position information of the object 30 transported by means other than the forklift 10 based on the image captured by the fixed camera 60, and transmits it to the on-premises server 50a via the transmission unit 502. Based on the image captured by the fixed camera 60, the position information of the object 30 acquired by the ceiling object position acquisition unit 501 is an example of the second position information.

[0099] The forklift 10 is provided with a moving body position acquisition unit 102 and a holding information acquisition unit 103. The forklift 10 is equipped with a single-board computer and is wired-connected to the omnidirectional camera 20. Each function of the moving body position acquisition unit 102 and the holding information acquisition unit 103 is realized by this single-board computer.

[0100] Here, the omnidirectional camera 20 is an example of a first imaging unit provided on the forklift 10. The omnidirectional image captured by the omnidirectional camera 20 is an example of the first captured image.

[0101] The moving body position acquisition unit 102 acquires the position information of the forklift 10 based on the image captured by the omnidirectional camera 20, and transmits it to the on-premises server 50a via the transmission unit 101.

[0102] The holding information acquisition unit 103 acquires holding information indicating either the holding or non-holding state of the object 30 by the forklift 10 based on the image captured by the omnidirectional camera 20, and transmits it to the on-premises server 50a via the transmission unit 101.

[0103] The on-premises server 50a has an object position acquisition unit 56a. The object position acquisition unit 56a acquires the position information of the object 30 transported by the forklift 10 based on the position information of the forklift 10 received via the reception unit 51 and the holding information. The position information of the object 30 acquired based on the omnidirectional image captured by the omnidirectional camera 20 is an example of the first position information.

[0104] In addition, the object position acquisition unit 56a can acquire the position information of the object 30 conveyed by entities other than the forklift 10 via the reception unit 51. The position information of the object 30 acquired based on the image captured by the fixed camera 60 is an example of the second position information. The object position acquisition unit 56a can output the acquired position information of the object 30 via the output unit 57.

[0105] In order to recognize and track the position of the object 30, it is preferable to align the three-dimensional coordinate systems between the position of the fixed camera 60 and the position of the object 30. As a method for this, a method of aligning the three-dimensional coordinate systems by attaching markers with known three-dimensional coordinate information to the floor of the warehouse 100 and recognizing them with the fixed camera 60 can be mentioned. There is also a method of aligning the three-dimensional coordinate systems by initially registering the position of the fixed camera 60 using the three-dimensional coordinate information of the forklift 10 that recognizes its own position.

[0106] FIG. 15 is a diagram showing the forklift 10 as viewed from the fixed camera 60. As shown in FIG. 15, the forklift 10 is provided with an identification marker 15. The identification marker 15 preferably has a two-dimensional code that is easy to recognize from the ceiling 500. The fixed camera 60 can align the three-dimensional coordinate systems by recognizing the identification marker 15 and initially registering the position of the fixed camera 60.

[0107] (Example of acquisition result of position information of object 30) FIG. 16 is a diagram showing an example of a screen for displaying the tracking result of the object 30 by the fixed camera 60. The display screen 161 may be a screen displayed on a mobile terminal provided on the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0108] As shown in FIG. 16, the display screen 161 includes an imaging image screen 162 and a camera position map screen 163. The imaging image screen 162 displays an image captured by the fixed camera 60, and can display not only still images but also moving images. The camera position map screen 163 shows the position of the fixed camera 60. The camera marks 164 included in the camera position map screen 163 indicate the positions of a plurality of fixed cameras 60.

[0109] The imaging image screen 162 can be scrolled in the X direction and the Y direction indicated by the arrows in FIG. 16. An operator who conveys the object 30 can select a fixed camera 60 for which an imaging image is to be displayed from among a plurality of fixed cameras 60 while viewing the camera position map screen 163.

[0110] Also, 165 in FIG. 16 is a fast forward button, and 166 is a scroll bar. For the ID number of the object 30 to be searched, a cue function can also be provided that can jump to the playback of a moving image at the time when it was last placed at that location. With the cue function, even when the object 30 is moved by something other than the forklift 10 afterwards, the playback for tracking can be performed efficiently.

[0111] (Function and effect of the information processing system 1a) As described above, the information processing system 1a according to the present embodiment has a fixed camera 60. Thereby, even when the position of the object 30 is changed by a hand pallet truck or the like where the omnidirectional camera 20 is not provided, the position of the object 30 can be recognized and tracked based on the imaging image by the fixed camera 60.

[0112] [Modification example] FIG. 17 is a block diagram showing an example of the functional configuration of an information processing system 1b according to the first modification example. As shown in FIG. 17, the information processing system 1b includes a cloud server 50b, an input / output terminal 600, and a network switch 700. The input / output terminal 600 and the cloud server 50b are communicably connected via the Internet 800.

[0113] The cloud server 50b is an external server installed outside the warehouse 100. The cloud server 50b has the same hardware configuration as the on-premises server 50 shown in FIG. 3.

[0114] The input / output terminal 600 has an input / output unit 601 and a transmission / reception unit 602. The input / output terminal 600 is, for example, a mobile terminal attached to the forklift 10 described in the first embodiment. The network switch 700 is a switch that can switch between the network 400 and the Internet 800.

[0115] The information processing system according to the embodiment can also be configured as shown in FIG. 17.

[0116] FIG. 18 is a block diagram showing an example of the functional configuration of the information processing system 1c according to the second modification. As shown in FIG. 18, the information processing system 1c has a forklift 10c. The forklift 10c has a sensor 104 and a holding information acquisition unit 103b.

[0117] The sensor 104 is at least one of a contact sensor, an infrared sensor, an ultrasonic sensor, a distance measuring sensor, or a load sensor. The sensor 104 detects data or signals for acquiring holding information.

[0118] The holding information acquisition unit 103b can acquire holding information based on the data or signals detected by the sensor 104.

[0119] The information processing system according to the embodiment can also be configured as shown in FIG. 18.

[0120] Although the embodiments have been described above, the present invention is not limited to the specifically disclosed embodiments, and various modifications and changes are possible without departing from the scope of the claims.

[0121] Although a forklift has been shown as an example of the moving body, the present invention is not limited thereto. For example, it may be an automated guided vehicle or a drone.

[0122] The embodiment also includes an information processing method. For example, the information processing method is an information processing method by an information processing device that processes position information of an object moved by a moving body, and includes a step of outputting the position information of the object obtained based on the position information of the moving body acquired based on an imaging image by an imaging device and holding information indicating either a state of holding or not holding the object by the moving body. By such an object detection method, the same effects as those of the information processing system described above can be obtained.

[0123] The embodiment also includes a program. For example, the program is a program that operates on an information processing device that processes position information of an object moved by a moving body, and causes a computer to execute a process of outputting the position information of the object obtained based on the position information of the moving body acquired based on an imaging image by an imaging device and holding information indicating either a state of holding or not holding the object by the moving body. By such a program, the same effects as those of the information processing system described above can be obtained.

[0124] Note that all the numbers such as ordinal numbers and quantities used in the description of the embodiment are for exemplification to specifically describe the technology of the present invention, and the present invention is not limited to the exemplified numbers. Also, the connection relationship between components is for exemplification to specifically describe the technology of the present invention, and the present invention is not limited to this connection relationship for realizing the functions of the present invention.

[0125] Also, the division of blocks in the functional block diagram is an example, and a plurality of blocks may be realized as one block, one block may be divided into a plurality, and / or some functions may be transferred to other blocks. Also, the functions of a plurality of blocks having similar functions may be processed by a single piece of hardware or software in parallel or in time division. Also, some or all of the functions may be distributed among a plurality of computers.

Description of Symbols

[0126] 1 Information processing system 10 Forklift (an example of a moving body) 11 Conveying direction (an example of a moving direction) 12 Vertically upward direction 20 Omnidirectional camera (an example of an imaging unit, an example of a first imaging unit) 20a Azimuth 21 Fork 22 Support member 30 Object 31 Pallet 32 Cargo 33 Barcode (an example of identification information) 40 Temporary storage location 50 On-premises server (an example of an information processing device) 50b Cloud server (an example of an information processing device) 51 Receiving unit 52 Moving body position acquisition unit 53 Holding information acquisition unit 54 Identification information acquisition unit 55 Time acquisition unit 56 Object position acquisition unit 57 Output unit 58 Storage unit 60 Fixed camera (an example of a second imaging unit) 61 Position map (an example of the position information of a moving body) 62 Position map (an example of the position information of an object) 63 Position table (an example of the position information of an object) 64 Timestamp (an example of time information) 81, 91 Registration screen 100 Warehouse 200 Truck yard 300 Container 400 Network

Prior Art Documents

Patent Documents

[0127]

Patent Document 1

Claims

1. An information processing apparatus for processing position information of an object moved by a moving body, having an output unit that outputs the position information of the object obtained based on the position information of the moving body obtained based on a captured image by an imaging unit and the holding information indicating either the holding or non-holding state of the object by the moving body; The position information of the object is shown in association with an ID number of a section indicating a predetermined position range, the position information of the section, and a barcode indicating the section; An information processing apparatus that causes an operator working in a warehouse to register information indicating either the incoming or outgoing of the object together with information regarding the initial position of the object, and displays a screen for the operator to confirm the ID numbers of the object and the section on a mobile terminal carried by the operator.

2. The information processing apparatus according to claim 1, further comprising a holding information acquisition unit that acquires the holding information based on the captured image.

3. The information processing apparatus according to claim 1 or 2, further comprising an object position acquisition unit that acquires the position information of the object based on the position information of the moving body and the holding information.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the captured image is an image captured by the imaging unit provided on the moving body.

5. The information processing apparatus according to any one of claims 1 to 4, wherein the captured image includes an image of the scenery on the moving direction side of the object viewed from the moving body and an image of the scenery in the vertically upward direction viewed from the moving body.

6. The information processing apparatus according to any one of claims 1 to 3, wherein the captured image includes a first captured image captured by a first imaging unit provided on the moving body and a second captured image captured by a second imaging unit provided outside the moving body and imaging at least one of the moving body or the object.

7. The position information of the object includes the first position information of the object obtained based on the first captured image and the second position information of the object obtained based on the second captured image. The information processing apparatus according to claim 6.

8. The information processing apparatus according to any one of claims 1 to 7, further comprising a receiving unit configured to receive at least one of the holding information or the position information of the moving body.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the output unit further outputs information indicating a time corresponding to the position of the object.

10. The information processing apparatus according to any one of claims 1 to 9, further comprising an identification information acquisition unit configured to acquire identification information indicating the object. The output unit further outputs the identification information in association with the position information of the object.

11. The moving body, The imaging unit, An information processing system comprising the information processing apparatus according to any one of claims 1 to 10.

12. The imaging unit is an omnidirectional camera configured to image the surroundings of the moving body. The information processing system according to claim 11.

13. An information processing method by an information processing apparatus for processing position information of an object moved by a moving body, The method includes a step of outputting position information of the object obtained based on position information of the moving body obtained based on a captured image by an imaging device and holding information indicating either a state of holding or non-holding of the object by the moving body. The position information of the object is shown in association with an ID number of a section indicating a predetermined position range, position information of the section, and a barcode indicating the section. An information processing method for causing an operator working in a warehouse to register information indicating either the incoming or outgoing of the object together with information regarding the initial position of the object, and to display on a mobile terminal carried by the operator a screen for confirming the ID numbers of the object and the section.

14. A program that operates in an information processing apparatus for processing position information of an object moved by a moving body, outputting the position information of the object obtained based on the position information of the moving body obtained based on a captured image by an imaging device and holding information indicating either a state of holding or non-holding of the object by the moving body; causing a computer to execute the processing, the position information of the object is shown in association with an ID number of a section indicating a predetermined position range, position information of the section, and a barcode indicating the section, A program for causing an operator working in a warehouse to register information indicating either the incoming or outgoing of the object together with information regarding the initial position of the object, and to display on a mobile terminal carried by the operator a screen for confirming the ID numbers of the object and the section.

Images