Information processing apparatus and information processing system

Abstract

To provide an information processor that can easily process positional information of an object that a mobile body carries.SOLUTION: The information processor according to an embodiment of the present invention has an output unit for outputting positional information of an object acquired on the basis of information on the position of a mobile body and holding information showing one of a held state or a non-held state of the object by a holding unit in the mobile body. The holding information is acquired on the basis of an image taken by an imaging unit.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present application relates to an information processing apparatus and an information processing system.

Background Art

[0002] Conventionally, an information processing apparatus that processes position information of an object such as a cargo or a pallet carried by a moving body such as a forklift is known.

[0003] Further, a processing unit that classifies the operating state of a transport machine is provided using a first signal transmitted from a first detector that directly detects the movement of the transport machine and a second signal transmitted from a second detector that detects the presence or absence of an article in the transport machine, and information indicating the position of the transport machine is used to output information indicating the movement path of the transport machine for each classified operating state (see, for example, Patent Document 1).

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the apparatus of Patent Document 1, since signals from a plurality of detectors are processed, the processing of the position information of the object may be complicated.

[0005] An object of the present invention is to provide an information processing apparatus capable of easily processing the position information of an object moved by a moving body.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present invention includes an output unit that outputs the position information of the object acquired based on information related to the position of the moving body and holding information indicating either a state of holding or non-holding of the object by a holding unit provided in the moving body, and the holding information is obtained based on an image obtained by an imaging unit ri imaging the entire celestial sphere image among which The imaging range including the holding unit or the object is 、 acquired based on an image that has been deformed in advance into a perspective transformation image.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide an information processing apparatus capable of easily processing the position information of an object moved by a moving body.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In each drawing, the same reference numerals are given to the same components, and redundant descriptions may be omitted.

[0010] The information processing apparatus according to the embodiment processes the position information of an object moved by a moving body. For example, the moving body is a forklift, and the object includes a pallet. The information processing apparatus according to the embodiment processes the position information of an object such as a pallet moved by a forklift, recognizes the movement of the object, and also tracks it.

[0011] Here, FIG. 1 is a diagram for explaining an example of the position information of an object in a warehouse. FIG. 1 shows the inside of the warehouse 100 and the periphery of the warehouse 100 as viewed from above (ceiling side).

[0012] The warehouse 100 is a terminal warehouse (a warehouse installed at a transportation relay point). This terminal warehouse has a form of a warehouse called a cross-docking type. In a cross-docking type warehouse, a plurality of pallets for each product are received from a factory or a wholesaler, etc., and temporarily placed in the warehouse. Then, at the time of shipment, while maintaining the same loading state of the same pallet, a plurality of types of pallets are combined and shipped to different retail stores.

[0013] In FIG. 1, there is a truck yard 200 around the warehouse 100. FIG. 1 shows a state where a container 300 carried by a trailer and separated, and the cargo bed of a truck is connected to the warehouse in the truck yard 200. The forklift 10 takes out the pallet 31 from at least one of the cargo bed of the truck that has arrived at the truck yard 200 or the container 300 carried by the trailer.

[0014] After that, the forklift 10 transports the pallet 31 to the temporary storage location 40 and temporarily stores it. Then, at the time of shipment, the forklift 10 transports the pallet 31 to a location near the truck yard 200 in the warehouse 100 for loading alignment, and then loads the pallet 31 onto the cargo bed of the truck or the container 300.

[0015] In the temporary storage location 40, since the types and quantities of goods entering and leaving vary daily, there is often no partition for each type of product in order to flexibly secure space. On the other hand, since multiple workers temporarily place the pallets 31 at arbitrary locations, it is necessary to search for the desired pallet from among the multiple temporarily placed pallets at the time of shipment.

[0016]

[0015] In order to perform this search operation efficiently, there is a need for an information processing system that visualizes by recognizing and tracking the movement of the pallets 31 within the warehouse 100 while effectively utilizing the space by not specifying the temporary placement locations of the pallets 31. The information processing apparatus according to the embodiment is, as an example, used in such an information processing system.

[0017] Hereinafter, an information processing system having the information processing apparatus according to the embodiment will be described.

[0018] [First Embodiment] (Example of the overall configuration of the information processing system 1) FIG. 2 is a diagram showing an example of the overall configuration of the information processing system 1 according to the first embodiment. As shown in FIG. 2, the information processing system 1 includes a forklift 10, an omnidirectional camera 20, and an on-premises server 50. These are communicably connected via a network 400 such as a LAN (Local Area Network). Note that other devices such as an external server or an image forming apparatus may be communicably connected to the network 400.

[0019] The forklift 10 is an example of a moving body that holds the goods 32 placed on the pallet 31 and transports them while holding, and thereby transports the pallet 31 and the goods 32. Note that the transportation by the moving body is an example of the movement by the moving body. Also, the pallet 31 and the goods 32 are each an example of an object. Note that hereinafter, when the pallet 31 and the goods 32 are not particularly distinguished, they are collectively referred to as the object 30. Also, the forklift 10 is a general term notation for a plurality of forklifts, the pallet 31 is a general term notation for a plurality of pallets, and the goods 32 is a general term notation for a plurality of goods.

[0020] The forklift 10 may transport the object 30 according to the driving operation of the operator, or may transport the object 30 by autonomous driving without going through the operator.

[0021] The omnidirectional camera 20 is an example of an imaging unit provided on the forklift 10. The omnidirectional camera 20 is a camera capable of imaging the 360-degree omnidirectional view around the omnidirectional camera 20. The azimuth 20a indicates the azimuth that the omnidirectional camera 20 can image.

[0022] The omnidirectional image (omnidirectional view image) captured by the omnidirectional camera 20 is an example of an imaged image. However, the imaging unit is not limited to the omnidirectional camera 20, and any device that can image the surroundings of the forklift 10 may be used. Also, the imaged image does not necessarily have to be an omnidirectional image.

[0023] The omnidirectional image includes an image of the scenery on the conveyance direction 11 side of the object 30 viewed from the forklift 10 and an image of the scenery on the vertically upward direction 12 side viewed from the forklift 10. In other words, the conveyance direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the omnidirectional camera 20 can image the omnidirectional view, it can image the front and above of the forklift 10 in a single image. Note that the conveyance direction 11 is an example of the moving direction.

[0024] The omnidirectional camera 20 is preferably attached on the roof of the forklift 10 or on the support member 22 that supports the fork 21. Thereby, a view for imaging the front and upper sides of the forklift 10 can be suitably secured. Here, the fork 21 is an example of a holding part provided on the forklift 10.

[0025] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the on-premises server 50 via the network 400.

[0026] The cargo 32 is provided with a barcode 33 which is an example of identification information indicating the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information indicating the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information which is the result of the reading is transmitted to the on-premises server 50 via the network 400. Note that the identification information is not limited to a barcode and may be a QR code (registered trademark) or an ID (Identifier) number or the like.

[0027] The on-premises server 50 is installed in the warehouse 100 and is an example of an information processing device that processes the position information of the object 30 carried by the forklift 10.

[0028] The on-premises server 50 processes the position information of the object 30 based on the omnidirectional image received via the network 400 and the identification information indicating the object 30. Also, by using the omnidirectional images captured by the plurality of omnidirectional cameras 20 respectively, the position information of the objects 30 held by the plurality of forklifts 10 can be acquired.

[0029] (Example of the hardware configuration of the on-premises server 50) Next, FIG. 3 is a block diagram showing an example of the hardware configuration of the on-premises server 50. The on-premises server 50 is constructed by a computer.

[0030] As shown in FIG. 3, the on-premises server 50 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, and a display 506. The on-premises server 50 also includes an external device connection I / F (Interface) 508, a network I / F 509, a data bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0031] Among these, the CPU 501 controls the operation of the entire on-premises server 50. The ROM 502 stores programs used for driving the CPU 501 such as the IPL. The RAM 503 is used as a work area for the CPU 501.

[0032] The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images.

[0033] The external device connection I / F 508 is an interface for connecting various external devices. Examples of external devices in this case include a USB (Universal Serial Bus) memory and a printer. The network I / F 509 is an interface for performing data communication using the network 400. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG. 3.

[0034] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading or writing of various data with respect to the DVD-RW 513 as an example of a removable recording medium. Note that it is not limited to DVD-RW, and it may be DVD-R or the like. The media I / F 516 controls reading or writing (storage) of data with respect to the recording medium 515 such as a flash memory.

[0035] (Functional configuration example of the information processing system 1) FIG. 4 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in FIG. 4, the on-premises server 50 includes a receiving unit 51, a mobile body position acquisition unit 52, a held information acquisition unit 53, an identification information acquisition unit 54, a time acquisition unit 55, an object position acquisition unit 56, an output unit 57, and a storage unit 58.

[0036] Each of these units is a function or means realized by operating any of the components shown in FIG. 3 according to an instruction from the CPU 501 based on a program expanded from the HD 504 onto the RAM 503.

[0037] In addition, the forklift 10 is provided with an omnidirectional camera 20 and a transmission unit 101. The function of the transmission unit 101 can be realized by an electric circuit provided in either the forklift 10 or the omnidirectional camera 20, or can also be realized by software (CPU). It may also be realized by a plurality of circuits or a plurality of software.

[0038] The on-premises server 50 acquires the position information of the object 30 based on the position information of the forklift 10 acquired based on the omnidirectional image by the omnidirectional camera 20 and the held information indicating either the state of holding or non-holding of the object 30 by the forklift 10. Then, the acquired position information of the object 30 can be output to the outside via the output unit 57.

[0039] The receiving unit 51 receives the omnidirectional image captured by the omnidirectional camera 20 and transmitted via the transmitting unit 101 via the network 400, and outputs it to each of the moving body position acquisition unit 52 and the held information acquisition unit 53. Further, the receiving unit 51 receives the identification information read by a reader such as a barcode reader via the network 400, and outputs it to the identification information acquisition unit 54.

[0040] Based on the input omnidirectional image, the moving body position acquisition unit 52 acquires the position information of the forklift 10 by calculation, and outputs it to the object position acquisition unit 56. For the acquisition process (self-position estimation process) of the position information of the forklift 10, the simultaneous execution technology of map creation and self-position recognition (SLAM: Simultaneous Localization and Mapping) can be applied (for example, refer to "Explanation of the Current Situation and Future Prospects of SLAM," Masahiro Tomonou, Shoo Harada, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ).

[0041] Based on the input omnidirectional image, the held information acquisition unit 53 acquires, by calculation, held information indicating either the held or non-held state of the object 30 by the forklift 10, and outputs it to the object position acquisition unit 56.

[0042] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the receiving unit 51, and outputs it to the object position acquisition unit 56. However, the acquisition of the identification information by the identification information acquisition unit 54 is not limited to that via the network 400. For example, the identification information acquisition unit 54 may acquire the identification information input by a user such as an administrator using the keyboard 511 or the pointing device 512 in FIG. 3, may acquire the identification information pre-stored in the storage unit 58, or may acquire the identification information via the external device connection I / F 508. Note that the administrator is the administrator of the information processing system 1 or the warehouse 100.

[0043] The time acquisition unit 55 acquires information indicating the time when the reception unit 51 receives the omnidirectional image and the identification information, and outputs it to the object position acquisition unit 56.

[0044] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information. Further, the object position acquisition unit 56 mutually associates the position information of the object 30, the identification information indicating the object 30, and the time information, and outputs them via the output unit 57. The output destination of the output unit 57 is an external device such as a PC (Personal Computer), a display device such as a display 506, or a storage device such as an HD 504.

[0045] The storage unit 58 can store the identification information indicating the object 30 such as the pallet 31 or the cargo 32.

[0046] (Example of processing by the on-premises server 50) FIG. 5 is a flowchart showing an example of processing by the on-premises server 50. FIG. 5 shows processing triggered by the timing when the on-premises server 50 receives an operation to start acquiring the position information of the object 30. The operation to start acquiring the position information of the object 30 is performed by a user such as an administrator using the pointing device 512 in FIG. 3.

[0047] First, in step S51, the reception unit 51 receives the omnidirectional image and the identification information via the network 400.

[0048] Subsequently, in step S52, the moving body position acquisition unit 52 acquires the position information of the forklift 10 by calculation based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0049] Subsequently, in step S53, the holding information acquisition unit 53 acquires, by calculation, the holding information indicating either the holding or non-holding state of the object 30 by the forklift 10 based on the input omnidirectional image, and outputs it to the object position acquisition unit 56.

[0050] Subsequently, in step S54, the identification information acquisition unit 54 acquires the identification information indicating the object 30 by inputting it, and outputs it to the object position acquisition unit 56.

[0051] Subsequently, in step S55, the time acquisition unit 55 acquires information indicating the time when the reception unit 51 received the omnidirectional image and the identification information, and outputs it to the object position acquisition unit 56.

[0052] Note that the processing in steps S52 to S55 may be appropriately reordered, or each may be performed in parallel.

[0053] Subsequently, in step S56, the object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information.

[0054] Subsequently, in step S57, the object position acquisition unit 56 associates the position information of the object 30, the identification information indicating the object 30, and the time information with each other, and outputs them via the output unit 57.

[0055] Subsequently, in step S58, the on-premises server 50 determines whether to end the process. The on-premises server 50 determines whether to end the process based on an end operation by a user such as an administrator via the pointing device 512 or the like.

[0056] If it is determined in step S58 that the process is to end (step S58, Yes), the on-premises server 50 ends the process. On the other hand, if it is determined that the process is not to end (step S58, No), the on-premises server 50 performs the processes after step S51 again.

[0057] In this way, the on-premises server 50 processes the position information of the object 30 and can acquire the position information of the object 30.

[0058] (Example of method for acquiring holding information) Next, the method for acquiring the holding information by the holding information acquisition unit 53 will be described in detail.

[0059] In this embodiment, the holding information acquisition unit 53 acquires the holding information of the object 30 by the forklift 10 and the absolute position information of the object 30 at the time of attachment and detachment. In other words, the holding information of the object 30 is the attachment / detachment timing information of loading and unloading the object 30. Further, the holding information acquisition unit 53 acquires the attachment / detachment timing information of the object 30 based on the omnidirectional image by the omnidirectional camera 20.

[0060] Note that, for the acquisition of the attachment / detachment timing information of the object 30, it is also possible to use a proximity sensor using infrared rays, ultrasonic waves, etc. Further, sensors built in the forklift 10 such as a load sensor for detecting the weight of the cargo 32 and a fork height sensor, or sensors externally attached to the forklift 10 can also be used.

[0061] <Regarding the input image> The omnidirectional camera 20 attached to the fork 21 can capture the fork 21, the object 30, etc. within one omnidirectional image (see FIG. 2). Here, FIG. 6 is a diagram for explaining an example of image conversion of the fork 21. FIG. 6(a) is a diagram showing a part of the omnidirectional image, FIG. 6(b) is a diagram showing a perspective conversion camera, and FIG. 6(c) is a diagram showing a perspective conversion image.

[0062] The omnidirectional camera 20 projects a wide-angle view onto a plane by a projection method such as Equirectangular or equidistant projection. However, in these projection methods, as shown in the fork image 262 in FIG. 6(a), since a straight line in the three-dimensional space is projected onto a curve, the fork 21, which is mostly composed of straight line parts, is also projected as a curve, which is not preferable for the acquisition process of the attachment / detachment timing information. Therefore, in this embodiment, only the imaging range including the fork 21 or the object 30, etc. in the omnidirectional image 261 is used as an image deformed in advance into a perspective conversion image. Thereby, it becomes possible to perform image recognition using a general-purpose algorithm such as straight line detection.

[0063] As shown in FIG. 6(b), as the captured image plane 264 of the virtual perspective transformation image, a suitable range that is parallel to the floor surface and includes the entire fork 21 with the vertical axis aligned with the longitudinal direction of the fork 21 is set. The projection center 263 is the projection center of the omnidirectional camera 20.

[0064] As a result, as shown in the perspective transformation image 266 in FIG. 6(c), a fork image 267 in which two forks 21 are arranged substantially in parallel can be obtained. This transformation can be realized based on the projection method of the original omnidirectional image 261 and the installation direction of the omnidirectional camera 20. Therefore, when attaching the omnidirectional camera 20 to the forklift 10, it is only necessary to measure the installation direction of the omnidirectional camera 20. Hereinafter, the process of obtaining the attachment / detachment timing information of the object 30 using this perspective transformation image 266 as the input image will be described.

[0065] In addition, a camera can be provided separately from the omnidirectional camera 20, and the attachment / detachment timing information can be obtained using this camera. However, even in this case, since it is difficult to install the camera directly above the center of the fork 21, the perspective transformation process is effective.

[0066] <Regarding the marker method> It is preferable to previously provide a marker such as a barcode or an AR marker on the surface of the fork 21 imaged by the omnidirectional camera because it facilitates image recognition. For the image recognition process of the AR marker, for example, the AR marker image recognition function of the open-source software OpenCV can be used. Here, FIG. 7 is a diagram for explaining a marker method, which is a method of obtaining holding information using an AR marker. FIG. 7(a) is a diagram showing an example of a marker, FIG. 7(b) is a diagram showing a state where the marker is shielded by the cargo 32, and FIG. 7(c) is a diagram showing the horizontal distance estimation using a plurality of markers.

[0067] The AR marker image 272 shown in FIG. 7 shows the image of the AR marker provided on the fork 21. This AR marker is an example of an identification marker. Based on the AR marker image 272, the position information of the AR marker and the identification information indicated by the AR marker can be obtained at high speed.

[0068] When the forklift 10 is not holding the object 30, as shown in Fig. 7(a), the AR marker image 272 is detected in the perspective-transformed image 266. However, when the forklift 10 is holding the object 30, as shown in Fig. 7(b), the AR marker image 272 is blocked by the object 30 and cannot be detected in the perspective-transformed image 266.

[0069] In this embodiment, this is utilized to obtain the attachment / detachment timing information of the object 30 from the detection result of the AR marker image 272 in the perspective-transformed image 266. In Fig. 7(b), the fork image 267 is shown through the cargo image 273. However, in the perspective-transformed image 266, the AR marker cannot be seen at all because it is blocked by the cargo image 273 and cannot be detected.

[0070] <Acquisition of Attachment / Detachment Timing Information> The holding information acquisition unit 53 acquires the attachment / detachment timing information that changes from the state where the AR marker image 272 is not detected to the state where the AR marker image 272 is detected in the perspective-transformed image 266, taking the pallet loading time as the attachment timing and the reverse change as the unloading time. However, since there is an error in this acquisition, it is preferable to perform noise reduction processing such as adopting the acquisition result only when multiple frames are continuous. The acquisition error is, for example, the error that the AR marker image 272 is not detected although it is included in the perspective-transformed image 266, or the error that the AR marker image 272 is detected although it is not included.

[0071] <Determination of the Position of Object 30> Next, a method for determining the position of the detachable object 30 based on the acquisition result of the position information of the forklift 10 (self-position estimation result) by VisualSLAM and the perspective-transformed image 266 for acquiring the attachment / detachment timing information will be described.

[0072] ≪Horizontal Position Determination≫ If the mounting position of the omnidirectional camera 20 on the forklift 10 is fixed and the size of the object 30 to be held is substantially constant, it is considered that the relative position between the omnidirectional camera 20 and the object 30 in the horizontal direction is substantially constant. For example, it is assumed that the forklift 10 holds the object 30 such that the center of the pallet 31 comes to a position about 1 [m] away along the transport direction 11 of the forklift 10 from the omnidirectional camera 20.

[0073] The transport direction 11 of the forklift 10 can be measured by VisualSLAM. The holding information acquisition unit 53 can determine, as the absolute position of the object 30 at the time of attachment / detachment, a position, for example, 1 [m] away along the transport direction 11 of the forklift 10 from the absolute position of the omnidirectional camera 20 at the time of attachment / detachment of the object 30.

[0074] Also, if a plurality of AR markers with different identification information are installed on the fork 21 and the object 30 is held at the tip of the fork 21 according to which AR marker is detected in the perspective transformation image 266, it is possible to cope even when the horizontal distance between the omnidirectional camera 20 and the object 30 changes.

[0075] For example, three AR markers are provided on each of the two forks 21. The AR marker images 272a1, 272b1, and 272c1 and the AR marker images 272a2, 272b2, and 272c2 shown in FIG. 7(c) are images of a plurality of AR markers in the perspective transformation image 266.

[0076] Here, it is assumed that the AR marker images 272c1 and 272c2 are detected in the perspective transformation image 266 and the AR marker images 272a1, 272a2, 272b1, and 272b2 are not detected in the perspective transformation image 266. In this case, it is determined that the object 30 is held at a position separated from the base of the fork 21 by a distance E. Then, the horizontal distance between the omnidirectional camera 20 and the object 30 can be determined to be 1 + E [m], which is E meters farther than when measured up to the base of the fork 21 (for example, 1 [m]).

[0077] ≪Position determination in the height direction≫ The forklift 10 can generally move the forks 21 vertically up and down, and can perform operations such as loading not only the goods 32 placed on the floor but also the pallets 31 placed on top of other goods 32, and conversely, operations such as installing the pallets 31 on top of other goods 32. The holding information acquisition unit 53 acquires the position information of the object 30 in the height direction during such attachment and detachment based on the omnidirectional image.

[0078] In the perspective-transformed image 266, the size within the perspective-transformed image 266 changes in inverse proportion to the distance from the omnidirectional camera 20 to the object 30. Therefore, based on the size of the object 30 within the perspective-transformed image 266, the actual size of the object 30, and the focal length of the lens included in the omnidirectional camera 20, the holding information acquisition unit 53 can estimate the distance to the object 30.

[0079] FIG. 8 is a diagram showing an example of estimating the height of the forks 21. FIG. 8(a) is a diagram showing the distance between the AR marker images 272 within the perspective-transformed image 266, and FIG. 8(b) is a diagram showing the height relationship between the omnidirectional camera 20 and the forks 21.

[0080] As shown in FIG. 8(a), the interval w between the two fork images 267 within the perspective-transformed image 266 is measured from the positions of the AR marker images 272 provided on the two fork images 267 detected within the perspective-transformed image 266.

[0081] While the forklift 10 is holding the object 30, the AR marker image 272 cannot be detected. However, during the attachment and detachment operation of the object 30 by the forklift 10, the height of the forks 21 is fixed. Therefore, as shown in FIG. 8(b), from the interval w between the AR marker images 272 on the forks 21 immediately before the loading time or immediately after the unloading time, the height direction distance d between the omnidirectional camera 20 and the AR marker, that is, between the omnidirectional camera 20 and the forks 21, at the time of attachment and detachment of the object 30 can be measured.

[0082] In the perspective-transformed image 266, the size S of the object 30 on a plane parallel to the captured image plane 264 is related to the size s on the perspective-transformed image 266, the distance d between the omnidirectional camera 20 and the object 30, and the focal length f of the lens included in the omnidirectional camera 20 by the following equation (1). d = S × f / s ···(1)

[0083] If the interval W between the AR markers on the actual fork 21 is measured in advance, the distance d from the omnidirectional camera 20 to the object 30 can be obtained based on the interval w between the AR marker images 272 in the perspective-transformed image 266. That is, by substituting W for S and w for s in equation (1), the distance d can be obtained through calculation.

[0084] Also, if the height Hc from the floor 281 at the position where the omnidirectional camera 20 is attached is measured in advance, the height H of the fork 21 from the floor 281 can be calculated by the following equation (2). H = Hc - d ···(2)

[0085] When the omnidirectional camera 20 is attached at a position that moves vertically along the vertical direction on the forklift 10, the height Hc of the omnidirectional camera 20 can be determined using the results of the VisualSLAM process.

[0086] ≪Fork Detection Method≫ Since the upper surface of the fork 21 comes into contact with the pallet 31 and there is intense friction, if an AR marker is provided, it may be damaged or contaminated. Therefore, the holding information acquisition unit 53 can also acquire the attachment / detachment timing information by a fork detection method that does not use an AR marker.

[0087] In the fork detection method, the fork image 267 itself is detected from the perspective-transformed image 266. The fork 21 moves vertically and its position changes. However, since the thickness and interval of the two fork images 267 remain constant while the distance from the omnidirectional camera 20 changes, in the perspective-transformed image 266, the position and size change, but it is a similar transformation.

[0088] Therefore, the ratio of the thickness wf to the interval wg of the fork image 267 does not change. Here, FIG. 9 is a diagram for explaining an example of the fork detection method. FIG. 9(a) is a diagram showing the case where the fork 21 is at a low position, and FIG. 9(b) is a diagram showing the case where the fork 21 is at a high position. In FIG. 9, since the ratio of the thickness wf to the interval wg of the fork image 267 does not change, wf0 / wg0 = wf1 / wg1. Paying attention to this ratio, the outer shape of the fork image 267 is detected. It is assumed that the thicknesses of the two forks 21 are the same.

[0089] Here, FIG. 10 is a diagram for explaining an example of the combination selection of edge line segments. FIG. 10(a) is a diagram showing the extracted straight line, FIG. 10(b) is a diagram showing the first rejection example, and FIG. 10(c) is a diagram showing the second rejection example.

[0090] First, straight line segments are detected from the input image by means of a Canny filter, a Hough transform, or the like.

[0091] Next, among the detected straight line segments, only those whose directions are close to the Y direction are extracted. At this point, since it is highly likely that straight line segments other than the contour of the fork image 267 due to dirt on the fork 21 or floor lines are also included, the straight line segments are selected by the following procedure.

[0092] Next, any four straight lines 301 are selected from the detected straight lines along the Y direction, and the three intervals (w0, w1, w2) between them are measured. Note that the intervals w0 and w1 at both ends correspond to the thickness of the fork 21, and the central w1 corresponds to the interval of the fork 21. Note that the straight line 301 is a general notation for four straight lines.

[0093] Although the detected straight lines 301 are not strictly parallel lines due to the inclination of the fork image 267 and detection errors, since they are lines substantially along the Y direction, as shown in FIG. 10(a), the intervals between the intersections with a predetermined horizontal line 302 along the X direction can be regarded as widths.

[0094] Next, among the combinations of edges having the thickness and interval of the fork image 267 in the perspective transformation image 266 within the movement range of the fork 21 measured in advance, the four combinations with the ratio closest to the measured value are adopted as the fork edges.

[0095] For example, FIG. 10(b) is rejected when the captured image of the fork 21 in the movement range of the fork 21 is too thick or the interval is too narrow. Further, in FIG. 10(c), since the ratios of the thickness and the interval are far from the ratio of the actual size compared to FIG. 10(a), FIG. 10(a) is selected.

[0096] In this way, by detecting the fork image 267 focusing on the edge, stable detection is possible without being affected by changes in the appearance due to the illumination conditions and dirt on the surface of the fork 21. And even if an AR marker is not provided on the fork 21, the position of the fork 21 moving up and down along the vertical direction in the perspective transformation image 266 can be determined. As a result, similar to the AR marker method, the attachment / detachment timing information of the object 30 can be acquired.

[0097] Specifically, in the perspective transformation image 266, the presence or absence of detection of the fork image 267 is used instead of the presence or absence of detection of the AR marker image 272, and the interval of the fork image 267 in the perspective transformation image 266 is used instead of the distance between the AR marker images 272 in the perspective transformation image 266. Thereby, the attachment / detachment timing information and the attachment / detachment position of the object 30 can be determined.

[0098] Also, by using the vertical length of the edge of the detected fork image 267 instead of the detection results of a plurality of AR marker images 272, it is possible to measure the horizontal distance between the omnidirectional camera 20 and the object 30.

[0099] ≪Image movement amount determination method≫ The detection of the fork 21 based on the edge functions well on a flat floor, but may be misrecognized when there are vertical lines with a shape similar to the fork 21 on the cargo 32 or the pallet 31. As a detection method that does not depend on the outer shape of the cargo 32, the following image movement amount determination method can also be used.

[0100] Looking down at the fork 21 from directly above, when the cargo 32 is not being held, the floor surface can be seen between the two forks 21. On the other hand, when the cargo 32 is being held, the floor surface cannot be seen regardless of the outer shape as long as the cargo 32 and the pallet 31 are not transparent.

[0101] Also, when the cargo 32 is not being held, the pattern of the floor surface changes its position within the perspective-transformed image 266 due to the movement of the forklift 10 itself. However, when the image of the cargo 32 held by the forklift 10 is included in the perspective-transformed image 266, the position within the perspective-transformed image 266 does not change significantly even when the forklift 10 moves.

[0102] Therefore, in the image movement amount determination method, within the perspective-transformed image 266, when the cargo 32 is not being held, the area where the floor surface should be visible is monitored, and based on whether the temporal change corresponds to the movement of the forklift 10, the holding information of the object 30 is acquired.

[0103] Here, FIG. 11 is a diagram for explaining an example of the image movement amount determination method. FIG. 11(a) is a diagram showing an example of the monitoring area 313 within the perspective-transformed image 266, and FIG. 11(b) is a diagram showing an example of a plurality of monitoring areas 314.

[0104] First, a monitoring area 313 is set between the fork images 267 within the perspective-transformed image 266 (see FIG. 11(a)). This monitoring area 313 includes the image of a subject other than the fork 21 such as the floor if the cargo 32 is not being held, and includes the image of the cargo 32 if the cargo 32 is being held. If the monitoring area 313 is too small, the features of the included image are reduced, making tracking difficult. However, if the monitoring area 313 is too large, both the floor and the cargo 32 are included, making it difficult to distinguish. In this method, it is a square with one side included in the interval between the forks 21. If the shape of the interval between the forks 21 is elongated, as shown in FIG. 11(b), a plurality of monitoring areas 314 can be arranged, and the results of independent tracking for each area can be comprehensively determined.

[0105] The result of obtaining the position information of the forklift 10 by VisualSLAM will be described for the case where there is a position change at a certain distance from the previous image frame.

[0106] First, it is determined whether there are trackable feature points within the monitoring area 313. For example, the Harris operator or the like can be used.

[0107] If there are feature points, the displacement amount within the image from the previous image frame is measured. For the measurement, template matching technology or the like can be used.

[0108] Next, the displacement amount on the image when the feature points are fixed to the floor is predicted from the position information of the previous image frame and the current image frame.

[0109] It is determined whether the displacement amount of the tracking result is closer to the predicted self-movement amount or the case of no image movement and no displacement. Note that even when the object 30 is held by the forklift 10, it may be displaced slightly due to shaking or the like. Also, when the feature points are at a position higher (closer to the camera) than the floor, they move more than the predicted movement amount on the floor surface. Therefore, for example, when there is a displacement of more than half in the same direction as the image movement due to the change in the self-position, the feature points are determined to be floor-fixed, and when the displacement is below a predetermined amount in any direction, the omnidirectional camera 20 can be fixed (rejected as a mismatch if it is neither).

[0110] In this way, when the image between the forks is determined to be floor-fixed, it can be determined that the object 30 is not being held (non-holding state), and when the omnidirectional camera 20 is determined to be fixed, it can be determined that the object 30 is being held (holding state).

[0111] In addition, when the floor surface is flat and lacks luminance variation, there are no features, so displacement cannot be measured. However, during the period when the fork 21 is inserted into and removed from the pallet 31, the end of the pallet 31 is likely to be a feature point and can be tracked by image. Therefore, around the loading timing of the object 30, changes from floor-fixed feature point detection to omnidirectional camera 20-fixed feature point detection are likely to occur in the opposite way around the loading and unloading timing. Thus, this period can be determined as the attachment / detachment timing.

[0112] Furthermore, when the object 30 is attached or detached, for safety reasons, the forklift 10 often stops or moves at an extremely low speed. Therefore, by setting the timing when the moving speed of the forklift 10 is minimized near these change timings as the attachment / detachment timing, higher accuracy can also be achieved.

[0113] It is difficult to measure the height of the object 30 at the time of attachment / detachment only by this image movement amount determination. Therefore, the height is detected by using fork edge detection before and after the attachment / detachment timing.

[0114] ≪Variations of the Image Movement Amount Determination Method≫ When the omnidirectional camera 20 is fixed above along the vertical direction at the base of the fork 21, and the fork 21 only moves up and down, the monitoring area 313 for movement amount determination can be fixed within the perspective-transformed image 266. However, there may be cases where it is better to change the monitoring area 313 according to the state of the fork 21.

[0115] Here, FIG. 12 is a diagram showing another example of the image movement amount determination method. FIG. 12(a) is a diagram showing an example of the monitoring area in the image, FIG. 12(b) is a diagram when the fork 21 is low, and FIG. 12(c) is a diagram when the fork 21 is high.

[0116] For example, when omnidirectional camera 20 is attached to support member 22 or the like, depending on the up-down position of fork 21, fixed member 321 at the base of fork 21 may be interposed between omnidirectional camera 20 and fork 21 as shown in Fig. 12(a) and may be included in perspective transformation image 266. Since fixed member 321 moves together with forklift 10, even when object 30 is not being held, a fixed feature point of omnidirectional camera 20 may be detected, which may cause erroneous recognition.

[0117] Also, when the omnidirectional camera 20 is fixed at a position that is not linked to the forward / backward tilt of the support member 22, the position of the fork image 267 moves up and down along the vertical direction, and the monitored area also moves. To deal with such cases, an AR marker can be placed on a member that is linked to the monitored area, and the position and size of the monitored area can be changed according to the detected position of the AR marker.

[0118] For example, as shown in Fig. 12(b), when an AR marker on an obstacle is detected at position m0, the monitoring area is set to three areas a0, a1, and a2. Also, as shown in Fig. 12(c), when an AR marker is detected at position m1, area a2 is excluded from the monitoring area, and only a0 and a1 are monitored. In addition, the same can be done when the fork 21 moves and the monitoring area shifts.

[0119] In these cases, it is believed that long-term operation is possible because the possibility of wear of the AR marker is low compared to the surface of the fork 21. Also, if the shape of the part to be detected is known and stable, it is possible to detect the shape of the part itself without an AR marker, just like the fork detection method.

[0120] <Feature point 3D position measurement method> In the image movement amount determination method, it was assumed that the omnidirectional camera 20 was installed at a position where the floor could be seen between the forks 21. Depending on the installation position of the omnidirectional camera 20, there may be a case where the fork 21 itself is blocked by an obstacle and is not included in the omnidirectional image captured by the omnidirectional camera 20. For example, when the omnidirectional camera 20 is attached to the support member 22, if an object 30 of a certain height is held, it will be included in the omnidirectional image, but the fork 21 itself may not be included.

[0121] In this case, when the object 30 is not held, the wall or the like existing on the conveyance direction 11 side of the forklift 10 is imaged in the omnidirectional image. However, for a distant subject, since the change in the image position is small even when the forklift 10 moves, there is a high possibility that it will be determined as a fixed feature point of the omnidirectional camera 20 in the image movement amount determination method. In order to cope with such an installation position of the omnidirectional camera 20, there is a method of using three-dimensional position measurement similar to VisualSLAM.

[0122] In VisualSLALM, the self-position and the subject position are simultaneously determined based on the correspondence relationship between the images of the feature points on the stationary subject. Since the object 30 before being held by the forklift 10 is also stationary with respect to the floor, the three-dimensional position of the feature points on the object 30 can also be measured by VisualSLAM processing.

[0123] On the other hand, after the object 30 is held by the forklift 10, even if it is the same feature point, it is not stationary with respect to the floor and is almost stationary with respect to the omnidirectional camera 20, so it does not become a feature point effective for three-dimensional measurement. Specifically, it is removed from the three-dimensional calculation processing target as an outlier.

[0124] Therefore, it is possible to track the feature points in the image area including the object 30 and determine whether the target feature point is stationary with respect to the floor or stationary with respect to the camera based on whether the feature point is a target for three-dimensional calculation processing.

[0125] If this result is utilized, even when the fork 21 itself or the floor between the forks 21 is not included in the omnidirectional image, the state of holding or not holding the object 30 can be determined in the same manner as the image movement amount determination method, and attachment and detachment can be detected.

[0126] In this case, since the fork 21 itself is not included in the omnidirectional image, the attachment and detachment height of the object 30 by edge detection or AR marker detection of the fork 21 cannot be measured. Therefore, it is necessary to apply it to a lift of a type that can only be attached and detached from the floor surface without raising the fork 21 to a high position, or to use a fork height measurement sensor or the like separately.

[0127] (Example of acquisition result of object position information) FIG. 13 is a diagram showing a first example of the acquisition result of object position information. FIG. 13(a) is a diagram showing a position map, and FIG. 13(b) is a diagram showing position information and a time stamp.

[0128] The position map 61 shown in FIG. 13(a) is created by the moving body position acquisition unit 52 based on the omnidirectional image captured by the omnidirectional camera 20. The moving body position acquisition unit 52 acquires a point group including three-dimensional coordinate information, and creates the position map 61 by projecting this point group onto a two-dimensional plane.

[0129] The position map 61 is created, for example, in accordance with the movement of one forklift 10 and is used in the acquisition (self-position recognition) of the position information of all the forklifts 10. Thereby, each position of a plurality of forklifts 10 can be expressed in the same coordinate system.

[0130] As a method for creating the position map 61 based on an imaging image such as an omnidirectional image, for example, the above-described SLAM technology can be applied.

[0131] In a flat-type warehouse, which is common in cross-docking type warehouses, the position map 61 is likely to change in accordance with the change in the position of the object 30 accompanying the loading and unloading of the pallet 31. The change in the position map 61 may complicate the acquisition process of the position information. The flat-type warehouse refers to a type of warehouse with few structures such as shelves and places the pallet 31 or the cargo 32 on the floor surface.

[0132] On the other hand, warehouses often have a high ceiling height of 5 m or more, making it easy to secure an upward field of view on the ceiling side. Therefore, in the embodiment, a position map 61 is created using an image showing the upward direction among the omnidirectional images captured by the omnidirectional camera 20, and the position information of the forklift 10 is acquired. As a result, using the position map 61 with suppressed temporal changes, the position information of the forklift 10 can be acquired without the process becoming complicated.

[0133] In addition, the holding information acquisition unit 53 uses an image showing the front among the omnidirectional images in order to acquire the holding information of the object 30 by the forklift 10. As a result, using one omnidirectional image captured by the omnidirectional camera 20, it becomes possible to acquire the position information of the forklift 10 and the holding information of the object 30.

[0134] Note that the holding information acquisition unit 53 can also use information other than the omnidirectional image in order to acquire the holding information. Examples of the information other than the omnidirectional image include detection information by a contact sensor, an infrared sensor, an ultrasonic sensor, a distance measurement sensor, a load sensor, or the like. The holding information acquisition unit 53 can also acquire the holding information by combining the omnidirectional image or the detection information by each sensor. However, from the viewpoint of performing the process simply, it is more preferable to use the omnidirectional camera 20 capable of acquiring the position information of the forklift 10 and the holding information from one omnidirectional image.

[0135] The object position acquisition unit 56 can acquire, as the position information of the object 30, a position map 62 showing the position of the object 30 by deleting, from the position information of the forklift 10 shown in the position map 61, those indicating non-holding of the holding information. Since the position map 61 shown in FIG. 13(a) can also be said to show the position information of the object 30, in FIG. 13(a), the position map 62 is displayed in parentheses.

[0136] In addition, the position table 63 shown in FIG. 13(b) is a table including the three-dimensional coordinates of the point group indicating the position of the object 30 and a time stamp 64 indicating the time when the three-dimensional coordinates were acquired. The position table 63 is created by the object position acquisition unit 56. The time stamp 64 is an example of the time information acquired by the time acquisition unit 55. The object position acquisition unit 56 can output the position table 63 in which the position information and time information of the object 30 are associated via the output unit 57.

[0137] The object position acquisition unit 56 excludes the information of the point group after a predetermined time has elapsed and updates the position table 63 using the information of the new point group. When the movement of the forklift 10 is small, it is preferable to reduce the update frequency so that the information of the point group shown in the position table 63 does not decrease too much. Thereby, even when there is a state change around the forklift 10 and it is difficult to secure the field of view above the forklift 10, the position information of the object 30 can be suitably acquired by updating the position map 62 and the position table 63 indicating the position of the object.

[0138] In order to track the positions of the plurality of objects 30 in the warehouse 100, identification information such as a barcode attached to the object 30 is used. The identification information indicating the object 30 is stored in the storage unit 58 together with the initial position information of the object 30.

[0139] As the identification information indicating the object 30, when the forklift 10 takes out the pallet 31 from the container 300 or the like and temporarily places the pallet 31, the barcode or the like attached to the cargo 32 or the pallet 31 that has been read is stored in the storage unit 58. Such storage is called initial registration. In the present embodiment, in addition to the identification information indicating the object 30 that is initially registered, the position information of the object 30 acquired by the object position acquisition unit 56 can be stored in the storage unit 58 in association therewith.

[0140] Here, an example of a simpler method for tracking the positions of multiple objects 30 within the warehouse 100 will be described with reference to FIG. 14. FIG. 14 is a diagram showing a second example of object position information. FIG. 14(a) is a diagram showing compartments, and FIG. 14(b) is a diagram showing a barcode indicating position information and compartments.

[0141] The compartments A, B, C, and D shown in FIG. 14(a) indicate predetermined position ranges within the warehouse 100. Operations for initially registering the objects 30 are performed within each of the compartments A, B, C, and D.

[0142] By reading the barcodes indicating the compartments A, B, C, and D at either the timing before or after reading the barcode indicating the object 30 using a reader, the position coordinates indicating the object 30 and each compartment can be associated. Thereafter, when the object 30 is transported by the forklift 10, the position of the object 30 can be tracked including the identification information of which object 30 is being transported. The table 71 shown in FIG. 14(b) shows the ID numbers indicating the compartments, the position information of the compartments, and the barcodes indicating the compartments in association.

[0143] The barcodes indicating each compartment are printed and carried by the operator so that they can be read by the operator using a reader. Alternatively, they are attached to the floor, pillars, or the forklift 10 within the warehouse 100. Thereby, the barcodes of each compartment can be easily read at the timing before and after performing the operation of reading the barcode indicating the object 30.

[0144] FIG. 15 is a diagram showing an example of a screen for an operator to register the initial position of the object 30. For example, a mobile terminal for reading the barcode indicating the object 30 is attached on the forklift 10, and the operator performs the registration operation using this mobile terminal. The registration screen 81 shown in FIG. 15 is the screen displayed by the mobile terminal. Also, the information read by the simple method shown in FIG. 14 can be displayed on the mobile terminal, and the operator can be made to select information indicating either the incoming or outgoing of the cargo 32, or the operator can be simply required to confirm the ID numbers of the object 30 and the compartments.

[0145] FIG. 16 is a diagram showing another example of a screen for an operator to register the initial position of the object 30. The registration screen 91 shown in FIG. 16 is a display screen of the mobile terminal and displays a barcode 92 indicating a section. On the registration screen 91, a barcode indicating either the incoming or outgoing state of the cargo 32 or a section that is a candidate for the initial position is displayed, and the operator can read an appropriate barcode from among them using a reader to register the initial position.

[0146] FIG. 17 is a diagram showing an example of a display screen of the destination of the object 30 by the forklift 10. The display screen 93 displays a position map 94. The position map 94 includes a transport source position 95 and a transport destination position 96. The transport source position 95 indicates the position of the transport source, and the transport destination position 96 indicates the position of the transport destination. By displaying the positions such as the pre-registered temporary storage locations on the screen, more convenient information for work can be provided.

[0147] FIG. 18 is a diagram showing an example of a screen for displaying the position map of the object 30. The display screen 111 may be a screen displayed on a mobile terminal provided in the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0148] After the object 30 is initially registered in the warehouse 100, the on-premises server 50 displays the ID numbers 112 or 113, etc. indicating all the objects 30 on the map based on their respective position coordinate information during the period until the barcode is read at the time of shipment.

[0149] FIG. 19 is a diagram showing an example of a screen displayed when searching for the object 30. The display screen 121 may be a screen displayed on a mobile terminal provided in the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0150] When searching for the object 30, when an operator who performs transportation inputs the ID number of the pallet 31 that the operator wants to search for, the display screen 121 displays a mark 122 indicating the position of the corresponding pallet 31 on the position map 123. At this time, the number of pallets 31 for which the position is to be displayed is not limited to one, and the ID numbers of a plurality of pallets 31 may be input in order so that the positions of the plurality of pallets 31 are displayed all at once.

[0151] (Operation and effect of the on-premises server 50) As described above, the on-premises server 50 (information processing device) included in the information processing system 1 according to the present embodiment processes the position information of the object 30 that is transported (moved) by the forklift 10 (mobile body).

[0152] The on-premises server 50 has an output unit 57 that outputs the position information of the object 30 acquired based on the information related to the position of the forklift 10 (mobile body) and the holding information indicating either the state of holding or non-holding of the object 30 by the fork 21 (holding unit) provided on the forklift 10.

[0153] Also, the holding information acquisition unit 53 acquires the holding information based on the captured image of the fork 21 captured by the omnidirectional camera 20 (imaging unit) attached to the forklift 10. In other words, the holding information is acquired based on the captured image captured by the omnidirectional camera 20.

[0154] In order to acquire the holding information of the object 30 by the fork 21 based on the omnidirectional image, the position information of the object 30 can be acquired with a simpler process compared to the case of processing signals from a plurality of detectors. Thereby, an on-premises server 50 that can easily process the position information of the object 30 transported by the forklift 10 can be provided.

[0155] In this embodiment, the omnidirectional image (captured image) includes an image of the fork 21 and the floor on which the forklift 10 moves, and the holding information acquisition unit 53 acquires holding information based on the amount of movement of the floor or the object 30 within the monitoring area 313 included in the omnidirectional image and the amount of movement of the forklift 10 acquired based on information related to the position of the forklift 10.

[0156] Thereby, even when there are vertical lines similar in shape to the fork 21 on the cargo 32 or the pallet 31, the object 30 can be recognized without misrecognizing the fork 21, and the position information of the object 30 can be accurately acquired.

[0157] In this embodiment, the omnidirectional image includes an AR marker (identification marker) indicating the fork 21, and the holding information acquisition unit 53 acquires holding information based on the AR marker. Since the AR marker is easily recognized in an image, holding information can be acquired with simpler processing.

[0158] In this embodiment, the holding information acquisition unit 53 corrects the position information of the object 30 based on the AR marker. Thereby, even when the horizontal distance between the omnidirectional camera 20 and the object 30 changes, the position information of the object 30 can be accurately acquired.

[0159] In this embodiment, the holding information acquisition unit 53 detects the height of the fork 21 based on the omnidirectional image, and the output unit 57 outputs the position information of the object 30 acquired based on the height of the fork 21, the holding information, and the information related to the position of the forklift 10. Thereby, when performing operations such as loading only the pallet 31 placed on another cargo 32 or installing the pallet 31 on another cargo 32, the position information in the height direction of the object 30 at the time of attachment and detachment can be provided.

[0160] In this embodiment, a time stamp 64 (information indicating time) corresponding to the position of the object 30 is further output. Thereby, the position of the object 30 becomes easier to recognize and easier to track.

[0161] [Second Embodiment] Next, the information processing system 1a according to the second embodiment will be described. Note that the same components described in the first embodiment are denoted by the same part numbers, and redundant descriptions will be omitted as appropriate.

[0162] FIG. 20 is a diagram showing an example of the overall configuration of the information processing system 1a. As shown in FIG. 20, the information processing system 1a includes a fixed camera 60 and an on-premises server 50a.

[0163] Here, in the warehouse, in addition to the forklift 10, a pallet handling device called a hand pallet truck that can be easily handled by a person may be used. In this embodiment, even when the position of the object 30 is changed by a hand pallet truck in which the omnidirectional camera 20 is not provided, the position of the object 30 is recognized and made trackable based on the captured image by the fixed camera 60.

[0164] The fixed camera 60 is provided near the ceiling of the warehouse 100 or the like, and is a camera that photographs the inside of the warehouse 100 from the ceiling side toward the floor side. The fixed camera 60 is provided other than the forklift 10, and images the periphery of either the forklift 10 or the object 30.

[0165] There is no particular limitation on the type of the fixed camera 60, but those that can image a wide range are preferable, and an omnidirectional camera is more preferable. The number of fixed cameras 60 installed may be one or a plurality. In this embodiment, it is assumed that a plurality of fixed cameras 60 are provided, and the fixed camera 60 is a general term for a plurality of fixed cameras.

[0166] (Functional Configuration Example of Information Processing System 1a) FIG. 21 is a block diagram showing an example of the functional configuration of the information processing system 1a. As shown in FIG. 21, on the ceiling 500, a fixed camera 60, a ceiling object position acquisition unit 501, and a transmission unit 502 are provided.

[0167] The functions of the object position acquisition unit 501 for the ceiling and the transmission unit 502 are realized by an electric circuit provided in either the ceiling 500 or the fixed camera 60. Alternatively, a part of these functions can also be realized by software (CPU). Further, these functions may be realized by a plurality of circuits or a plurality of software.

[0168] Based on the image captured by the fixed camera 60, the object position acquisition unit 501 for the ceiling acquires the position information of the object 30 carried by means other than the forklift 10 and transmits it to the on-premises server 50a via the transmission unit 502.

[0169] The forklift 10 is provided with a moving body position acquisition unit 102 and a holding information acquisition unit 103. The forklift 10 is equipped with a single-board computer and is wired-connected to the omnidirectional camera 20. The functions of the moving body position acquisition unit 102 and the holding information acquisition unit 103 are realized by this single-board computer.

[0170] Based on the image captured by the omnidirectional camera 20, the moving body position acquisition unit 102 acquires the position information of the forklift 10 and transmits it to the on-premises server 50a via the transmission unit 101.

[0171] Based on the image captured by the omnidirectional camera 20, the holding information acquisition unit 103 acquires holding information indicating either the state of holding or non-holding of the object 30 by the forklift 10 and transmits it to the on-premises server 50a via the transmission unit 101.

[0172] The on-premises server 50a has an object position acquisition unit 56a. Based on the position information of the forklift 10 received via the reception unit 51 and the holding information, the object position acquisition unit 56a acquires the position information of the object 30 carried by the forklift 10.

[0173] In addition, the object position acquisition unit 56a can acquire the position information of the object 30 carried by other than the forklift 10 via the reception unit 51. The object position acquisition unit 56a can output the acquired position information of the object 30 via the output unit 57.

[0174] In order to recognize and track the position of the object 30, it is preferable to match the three-dimensional coordinate systems between the position of the fixed camera 60 and the position of the object 30. As this method, there is a method of matching the three-dimensional coordinate systems by attaching a marker whose three-dimensional coordinate information is known to the floor of the warehouse 100 and recognizing it with the fixed camera 60. Also, there is a method of matching the three-dimensional coordinate systems by initially registering the position of the fixed camera 60 using the three-dimensional coordinate information of the forklift 10 that recognizes its own position.

[0175] FIG. 22 is a diagram showing the forklift 10 as viewed from the fixed camera 60. As shown in FIG. 22, the forklift 10 is provided with an AR marker 15. The AR marker 15 preferably has a two-dimensional code that is easy to recognize from the ceiling 500. The fixed camera 60 can match the three-dimensional coordinate systems by recognizing the AR marker 15 and initially registering the position of the fixed camera 60.

[0176] (Example of acquisition result of position information of object 30) FIG. 23 is a diagram showing an example of a screen for displaying the tracking result of the object 30 by the fixed camera 60. The display screen 161 may be a screen displayed on a mobile terminal provided on the forklift 10, or may be a screen displayed on the display 506 of the on-premises server 50.

[0177] As shown in FIG. 23, the display screen 161 includes an imaging image screen 162 and a camera position map screen 163. The imaging image screen 162 displays the image captured by the fixed camera 60, and can display not only still images but also moving images. The camera position map screen 163 shows the position of the fixed camera 60. The camera marks 164 included in the camera position map screen 163 show the positions of a plurality of fixed cameras 60.

[0178] The captured image screen 162 can be scrolled in the X and Y directions respectively indicated by the arrows in FIG. 23. An operator who conveys the object 30 can select a fixed camera 60 for which a captured image is to be displayed from among the plurality of fixed cameras 60 while viewing the camera position map screen 163.

[0179] Also, 165 in FIG. 23 is a fast-forward button, and 166 is a scroll bar. Regarding the ID number of the object 30 to be searched, a cue function can also be provided that enables jumping the playback of the video to the time when it was last placed at that location. With the cue function, even when the object 30 is moved by something other than the forklift 10 afterwards, efficient playback for tracking can be performed.

[0180] (Function and Effect of Information Processing System 1a) As described above, the information processing system 1a according to the present embodiment has the fixed cameras 60. Thereby, even when the position of the object 30 is changed by a hand pallet truck or the like in which the omnidirectional camera 20 is not provided, the position of the object 30 can be recognized and tracked based on the captured image by the fixed cameras 60.

[0181] Also, in the present embodiment, the object position acquisition unit 56a acquires the holding information calculated by the single board computer provided in the forklift 10 via the reception unit 51. Thereby, the position information of the object 30 can be acquired by simple processing without performing the acquisition process of the holding information in the on-premises server 50a.

[0182] [Modification Example] FIG. 24 is a block diagram showing an example of the functional configuration of an information processing system 1b according to a first modification example. As shown in FIG. 24, the information processing system 1b includes a cloud server 50b, an input / output terminal 600, and a network switch 700. The input / output terminal 600 and the cloud server 50b are communicably connected via the Internet 800.

[0183] The cloud server 50b is an external server installed outside the warehouse 100. The cloud server 50b has the same hardware configuration as the on-premises server 50 shown in FIG. 3.

[0184] The input / output terminal 600 has an input / output unit 601 and a transmission / reception unit 602. The input / output terminal 600 is, for example, a mobile terminal attached to the forklift 10 described in the first embodiment. The network switch 700 is a switch capable of switching between the network 400 and the Internet 800.

[0185] The information processing system according to the embodiment can also be configured as shown in FIG. 24.

[0186] FIG. 25 is a block diagram showing an example of the functional configuration of the information processing system 1c according to the second modification. As shown in FIG. 25, the information processing system 1c has a forklift 10c. The forklift 10c has a sensor 104 and a holding information acquisition unit 103b.

[0187] The sensor 104 is at least one sensor among a contact sensor, an infrared sensor, an ultrasonic sensor, a distance measuring sensor, or a load sensor. The sensor 104 detects data or signals for acquiring holding information.

[0188] The holding information acquisition unit 103b can acquire holding information based on the data or signals detected by the sensor 104.

[0189] The information processing system according to the embodiment can also be configured as shown in FIG. 25.

[0190] Although the embodiments have been described above, the present invention is not limited to the specifically disclosed embodiments, and various modifications and changes can be made without departing from the scope of the claims.

[0191] Although a forklift has been shown as an example of the moving body, it is not limited thereto. For example, an automated guided vehicle or a drone may be used.

[0192] Note that the numbers such as ordinal numbers and quantities used in the description of the embodiments are all exemplified for specifically describing the technology of the present invention, and the present invention is not limited to the exemplified numbers. Also, the connection relationships between components are exemplified for specifically describing the technology of the present invention, and the connection relationships for realizing the functions of the present invention are not limited thereto.

[0193] Also, the division of blocks in the functional block diagram is an example, and it is also possible to implement a plurality of blocks as one block, divide one block into a plurality, and / or transfer some functions to other blocks. Further, the functions of a plurality of blocks having similar functions may be processed by a single piece of hardware or software in parallel or in a time-sharing manner. Also, some or all of the functions may be distributed among a plurality of computers.

Explanation of Reference Numerals

[0194] 1 Information processing system 10 Forklift (an example of a moving body) 11 Conveying direction (an example of a moving direction) 12 Vertically upward direction 15 AR marker (an example of an identification marker) 20 Omnidirectional camera (an example of an imaging unit) 20a Azimuth 21 Fork (an example of a holding unit) 22 Support member 30 Object 31 Pallet 32 Cargo 33 Barcode 40 Temporary storage location 50 On-premises server (an example of an information processing device) 50b Cloud server (an example of an information processing device) 51 Receiving unit 52 Moving body position acquisition unit 53 Holding information acquisition unit 54 Identification information acquisition unit 55 Time acquisition unit 56 Object position acquisition unit 57 Output unit 58 Storage unit 60 Fixed camera 61 Position map (an example of the position information of the moving object) 62 Position map (an example of the position information of the object) 63 Position table (an example of the position information of the object) 64 Timestamp (an example of the time information) 81, 91 Registration screen 100 Warehouse 200 Truck yard 266 Perspective transformation image 267 Fork image 272 AR marker image 300 Container 313 Monitoring area 314 Multiple monitoring areas 400 Network H Height

Prior art documents

Patent documents

[0195]

Patent Document 1

Claims

1. An output unit that outputs position information of the object acquired based on information related to the position of the moving body and holding information indicating either a state of holding or non-holding of the object by a holding unit provided on the moving body. The holding information is information obtained based on an image in which an imaging range including the holding unit or the object is deformed in advance into a perspective conversion image among omnidirectional images captured by an imaging unit, in an information processing apparatus.

2. The information processing apparatus according to claim 1, further comprising an object position acquisition unit that acquires the position information of the object based on the information related to the position of the moving body and the holding information.

3. An information processing apparatus having a holding information acquisition unit that acquires the holding information based on the omnidirectional image. The omnidirectional image includes an image in which the holding unit is imaged by the imaging unit attached to the moving body, in the information processing apparatus according to claim 1 or 2.

4. The omnidirectional image includes an image in which the holding unit and the floor on which the moving body moves are imaged. The holding information acquisition unit acquires the holding information based on the amount of movement of the floor or the object within a monitoring area included in the omnidirectional image and the amount of movement of the moving body acquired based on information related to the position of the moving body, in the information processing apparatus according to claim 3.

5. The omnidirectional image includes an identification marker indicating the holding unit. The holding information acquisition unit acquires the holding information based on the identification marker, in the information processing apparatus according to claim 3.

6. The information processing apparatus according to claim 5, wherein the holding information acquisition unit corrects the position information of the object based on the identification marker.

7. The holding information acquisition unit detects the height of the holding unit based on the omnidirectional image. The output unit outputs the position information of the object acquired based on the height of the holding unit, the holding information, and information related to the position of the moving body, in the information processing apparatus according to any one of claims 3 to 6.

8. The information processing apparatus according to claim 1 or 2, further comprising a reception unit that receives the holding information.

9. The output unit outputs the position information of the object in association with time information indicating the time when the position information of the object is acquired, in the information processing apparatus according to any one of claims 1 to 8.

10. The moving body An imaging unit provided on the moving body that images at least the holding unit. An information processing system comprising the information processing apparatus according to any one of claims 1 to 9.

11. The information processing system according to claim 10, wherein the imaging unit is attached to the moving body so as to be able to image the scenery on the moving direction side of the moving body as viewed from the moving body and the holding unit.

Images