Information processing device, information processing system and information processing method

Abstract

To recognize conveyance states (holding / non-holding) of a plurality of objects, in making a moving body convey the plurality of objects at the same time.SOLUTION: An information processing device, which processes position information on objects that are moved by a moving body that has a photographing part that photographs an image and a holding part that can hold the plurality of objects, comprises: a moving body position obtaining part that obtains position information on the moving body, on the basis of an image of the circumference of the moving body photographed by the photographing part; a holding information obtaining part that obtains either of state of holding and non-holding of the objects by the moving body, on the basis of an image of the holding part photographed by the photographing part; and an object position obtaining part that obtains position information on the objects, on the basis of the position information on the moving body obtained by the moving body position obtaining part and the state obtained by the holding information obtaining part. The holding information obtaining part calculates distances of the objects with respect to the holding part, at the time when the objects are held and at the time before and after the objects are held, and fixes either of the states of holding and non-holding of the second object held by the moving body and thereafter.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, and an information processing method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known an information processing device that processes position information of an object such as cargo or a pallet transported by a mobile object such as a forklift.

[0003] Patent document 1 discloses a technology for estimating the position information of a moving body from an image captured by a camera attached to the moving body, and further acquiring the position information of an object by recognizing the transport state (held / unheld) of the object (cargo or pallet) being moved by the moving body. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, when a mobile body transports two objects held at the base and tip of a fork, which is a holding part attached to the mobile body that holds the objects, the object located at the tip of the fork is hidden by the object located at the base of the fork, making it impossible to capture an image of the object at the tip of the fork from the camera attached to the mobile body, and making it impossible to recognize the transport status (held / not held) of the object at the tip of the fork.

[0005] The present invention has been made in view of the above, and has an object to recognize the transport state (held / not held) of a plurality of objects when the plurality of objects are transported simultaneously by a moving body. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an information processing device that processes position information of objects moved by a moving body having an imaging unit that captures images and a holding unit that can hold multiple objects, and is equipped with a moving body position acquisition unit that acquires position information of the moving body based on an image of the surroundings of the moving body captured by the imaging unit, a holding information acquisition unit that acquires either a holding or non-holding state of the object by the moving body based on an image of the holding unit captured by the imaging unit, and an object position acquisition unit that acquires position information of the object based on the position information of the moving body acquired by the moving body position acquisition unit and the state acquired by the holding information acquisition unit, and is characterized in that the holding information acquisition unit calculates the distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines either a holding or non-holding state of the second or subsequent objects by the moving body. [Effects of the Invention]

[0007] According to the present invention, when a plurality of objects are simultaneously transported by a moving body, it is possible to recognize the transport state (held / not held) of the plurality of objects. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of position information of an object in a warehouse. [Figure 2] FIG. 2 is a diagram illustrating an example of the overall configuration of an information processing system according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of an on-premise server. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the information processing system. [Figure 5] FIG. 5 is a flowchart illustrating an example of processing by the on-premise server. [Figure 6] FIG. 6 is a diagram showing a first example of the result of acquiring object position information. [Figure 7]FIG. 7 is a diagram showing a second example of the object position information acquisition result. [Figure 8] FIG. 8 is a diagram illustrating an example of object detection in the retained information acquisition unit. [Figure 9-1] FIG. 9-1 is a diagram illustrating an operation procedure until the holding information acquisition unit determines the holding state of the second palette. [Figure 9-2] FIG. 9-2 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the holding state of the second palette. [Figure 9-3] FIG. 9-3 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the holding state of the second palette. [Figure 9-4] FIG. 9-4 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the holding state of the second palette. [Figure 10-1] FIG. 10-1 is a diagram illustrating an operation procedure until the holding information acquisition unit determines the holding state of only the first palette. [Figure 10-2] FIG. 10-2 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the holding state of only the first palette. [Figure 10-3] FIG. 10-3 is a diagram illustrating an operation procedure until the holding information acquisition unit determines the holding state of only the first palette. [Figure 10-4] FIG. 10-4 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the holding state of only the first palette. [Figure 11-1] FIG. 11-1 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the non-holding state of the second palette. [Figure 11-2] FIG. 11-2 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the non-holding state of the second palette. [Figure 11-3] FIG. 11-3 is a diagram illustrating the operation procedure until the holding information acquisition unit determines the non-holding state of the second palette. [Figure 12-1]FIG. 12A is a diagram illustrating an operation procedure until the holding information acquisition unit determines the number of stacked pallets. [Figure 12-2] FIG. 12B is a diagram illustrating an operation procedure until the holding information acquisition unit determines the number of stacked pallets. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an information processing device, an information processing system, and an information processing method will be described in detail with reference to the accompanying drawings. In the drawings, the same components are denoted by the same reference numerals, and duplicated descriptions may be omitted.

[0010] An information processing device according to an embodiment processes position information of an object moved by a moving body. For example, the moving body is a forklift, and the object includes a pallet. The information processing device according to an embodiment processes position information of the object, such as the pallet, moved by the forklift, and recognizes and tracks the movement of the object.

[0011] Here, Fig. 1 is a diagram for explaining an example of position information of an object in a warehouse. Fig. 1 shows the interior of a warehouse 100 and the periphery of the warehouse 100 as viewed from above (the ceiling side).

[0012] Warehouse 100 is a terminal warehouse (a warehouse established at a transit point in transportation). This terminal warehouse is a type of warehouse known as a cross-docking type. In a cross-docking type warehouse, multiple pallets for each product are received from a factory or wholesaler, and temporarily stored in the warehouse. Then, at the time of shipment, multiple types of pallets are combined while still packed on the same pallet, and shipped to the respective retail stores.

[0013] In Fig. 1, a truck yard 200 is located around a warehouse 100. Fig. 1 shows that the truck yard 200 has detached containers 300 transported by trailers and truck beds connected to the warehouse. A forklift 10 removes a pallet 31 from at least one of the beds of trucks that have arrived at the truck yard 200 or the containers 300 transported by trailers.

[0014] Thereafter, the forklift 10 carries the pallet 31 to the temporary storage location 40 and temporarily stores it there. Thereafter, at the time of shipping, the forklift 10 carries the pallet 31 to a location close to the truck yard 200 in the warehouse 100, arranges the items, and then loads the pallet 31 onto the bed of a truck or a container 300.

[0015] In order to ensure flexible space for the daily changes in the types and quantities of goods coming in and out, temporary storage locations 40 often do not have designated sections for each product. However, because multiple workers temporarily store pallets 31 in arbitrary locations, when shipping, it is necessary to search for the desired pallet from among the multiple temporarily stored pallets.

[0016] To efficiently perform this search work, there is a need for an information processing system that can effectively utilize space by not specifying temporary storage locations for the pallets 31, while recognizing and tracking the movement of the pallets 31 within the warehouse 100 to visualize it. As an example, the information processing device according to the embodiment is used in such an information processing system.

[0017] An information processing system including an information processing device according to an embodiment will be described below.

[0018] (Example of overall configuration of information processing system 1) Fig. 2 is a diagram showing an example of the overall configuration of an information processing system 1 according to an embodiment. As shown in Fig. 2, the information processing system 1 includes a forklift 10, a spherical camera 20, and an on-premise server 50. These are communicably connected via a network 400 such as a local area network (LAN). Note that devices other than those described above, such as an external server or an image forming device, may also be communicably connected to the network 400. For example, a reader 700 such as a barcode reader, a smartphone 600 which is a mobile terminal carried by the operator of the forklift 10, and the like are communicably connected to the network 400.

[0019] The forklift 10 is an example of a mobile body that transports the pallet 31 and the cargo 32 by holding the cargo 32 placed on the pallet 31 and transporting it while holding it. Transport by a mobile body is an example of movement by a mobile body. The pallet 31 and the cargo 32 are each an example of an object. In the following, the pallets 31 and the cargo 32 will be collectively referred to as the object 30 unless they are particularly distinguished from one another. The forklift 10 is a generic term for multiple forklifts, the pallet 31 is a generic term for multiple pallets, and the cargo 32 is a generic term for multiple cargoes.

[0020] The forklift 10 may transport the object 30 in response to the driving operation of an operator, or may transport the object 30 by automatic driving without the intervention of an operator.

[0021] The omnidirectional camera 20 is an example of an imaging device provided on the forklift 10. The omnidirectional camera 20 is a camera that can capture images in all directions of 360 degrees around the omnidirectional camera 20. The direction 20a indicates the direction in which the omnidirectional camera 20 can capture images.

[0022] The spherical image (omnidirectional image) captured by spherical camera 20 is one example of a captured image. However, the imaging device is not limited to spherical camera 20, and any device that can capture an image of the area around forklift 10 may be used. Furthermore, the captured image does not necessarily have to be a spherical image.

[0023] The spherical image includes an image capturing a scene in the conveying direction 11 of the object 30 as seen from the forklift 10, and a scene in the vertically upward direction 12 as seen from the forklift 10. In other words, the conveying direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the spherical camera 20 can capture images in all directions, it can capture an image including both the front and the above of the forklift 10 in a single image. The conveying direction 11 is an example of a moving direction.

[0024] The spherical camera 20 is preferably mounted on the roof of the forklift 10 or on a support member 22 that supports the forks 21. This ensures a good field of view for capturing images in front of and above the forklift 10. Here, the forks 21 are an example of a holding unit that can hold multiple objects 30 (pallets 31 and cargo 32).

[0025] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the on-premise server 50 via the network 400.

[0026] The cargo 32 is provided with a barcode 33, which is an example of identification information identifying the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information identifying the pallet 31. The barcode 33 is read by a reader 700 such as a barcode reader, and the identification information resulting from the reading is transmitted to the on-premise server 50 via the network 400. Note that the identification information is not limited to a barcode, and may be a QR code (registered trademark), an ID (identifier) ​​number, or the like.

[0027] The on-premise server 50 is an example of an information processing device that is installed in the warehouse 100 and processes the position information of the object 30 transported by the forklift 10. The on-premise server 50 can be replaced with a cloud server (a computing device installed outside the warehouse environment). This can reduce the introduction cost and running cost.

[0028] The on-premise server 50 processes the position information of the object 30 based on the omnidirectional image received via the network 400 and the identification information indicating the object 30. In addition, by using the omnidirectional images captured by the multiple omnidirectional cameras 20, the position information of the object 30 held by the multiple forklifts 10 can be obtained.

[0029] Furthermore, the on-premise server 50 performs a calculation for recognizing the self-position of the forklift 10 based on the omnidirectional image received via the network 400. The on-premise server 50 performs the calculation process for recognizing the self-position on the multiple omnidirectional cameras 20, thereby grasping the current positions of the multiple forklifts 10.

[0030] The forklift 10 includes, for example, a single-board computer. The forklift 10 has the omnidirectional camera 20 connected to the single-board computer via a wired connection. Calculation processing such as self-position recognition may be performed on the single-board computer, and the results may be transmitted and aggregated by wireless communication to the on-premise server 50 via the network 400. This allows the on-premise server 50 to grasp the current position of each forklift 10. In this case, the single-board computer of the forklift 10 functions as part of the information processing device.

[0031] The on-premise server 50 reads the ID, location information, video image information, etc. from the HD 504 (see Figure 3) within the on-premise server 50, identifies the final location information of the forklift 10 and the pallet 31, and visualizes it by outputting it to the display of the forklift 10 (single-board computer), the display of the smartphone 600, etc.

[0032] (Example of hardware configuration of on-premise server 50) 3 is a block diagram showing an example of the hardware configuration of the on-premise server 50. The on-premise server 50 is constructed by a computer. Note that the single-board computer of the forklift 10 is also constructed by a computer and has substantially the same configuration as the on-premise server 50, so a description thereof will be omitted.

[0033] 3 , the on-premise server 50 includes a central processing unit (CPU) 501, a read-only memory (ROM) 502, a random access memory (RAM) 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, and a display 506. The on-premise server 50 also includes an external device connection interface (I / F) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a digital versatile disk rewritable (DVD-RW) drive 514, and a media I / F 516.

[0034] Of these, the CPU 501 controls the overall operation of the on-premise server 50. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0035] The HD 504 stores various data such as programs, etc. The HDD controller 505 controls reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, text, or image.

[0036] The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network 400. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 3.

[0037] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0038] (Example of functional configuration of information processing system 1) Fig. 4 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in Fig. 4, the on-premise server 50 includes a receiving unit 51, a mobile object position acquiring unit 52, a retained information acquiring unit 53, an identification information acquiring unit 54, a time acquiring unit 55, an object position acquiring unit 56, an output unit 57, and a storage unit 58.

[0039] Each of these units is a function or means for performing a function that is realized when any of the components shown in FIG. 3 operates in response to an instruction from CPU 501 in accordance with a program loaded from HD 504 onto RAM 503.

[0040] The forklift 10 is also provided with an omnidirectional camera 20 and a transmitter 101. The function of the transmitter 101 can be realized by an electric circuit provided in either the forklift 10 or the omnidirectional camera 20, or by software (CPU). Alternatively, the function may be realized by a plurality of circuits or a plurality of pieces of software.

[0041] The on-premise server 50 acquires the position information of the object 30 based on the position information of the forklift 10 acquired based on the omnidirectional image captured by the omnidirectional camera 20 and the holding information indicating whether the object 30 is being held or not being held by the forklift 10. The acquired position information of the object 30 can then be output to the outside via the output unit 57.

[0042] Receiving unit 51 receives, via network 400, the omnidirectional image captured by omnidirectional camera 20 and transmitted via transmitting unit 101, and outputs the omnidirectional image to moving object position acquiring unit 52 and retained information acquiring unit 53. Receiving unit 51 also receives, via network 400, identification information read by reader 700 such as a barcode reader, and outputs the identification information to identification information acquiring unit 54.

[0043] The mobile object position acquisition unit 52 acquires position information of the forklift 10 by calculation based on the input spherical image, and outputs the acquired information to the object position acquisition unit 56. The process of acquiring the position information of the forklift 10 (self-position estimation process) can apply a technology for simultaneously creating a map and recognizing its own position (SLAM: Simultaneous Localization and Mapping) (see, for example, "Commentary: The Current State and Future Prospects of SLAM," by Tomono Masahiro and Hara Yoshitaka, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ).

[0044] The holding information acquisition unit 53 acquires, by calculation, holding information indicating whether the object 30 is being held or not held by the forklift 10 based on the input spherical image, and outputs the information to the object position acquisition unit 56.

[0045] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the receiving unit 51, and outputs the identification information to the object position acquisition unit 56. However, acquisition of the identification information by the identification information acquisition unit 54 is not limited to via the network 400. For example, the identification information acquisition unit 54 may acquire identification information input by a user such as an administrator using the keyboard 511 or the pointing device 512 in FIG. 3, or may acquire identification information stored in advance in the storage unit 58, or may acquire identification information via the external device connection I / F 508. The administrator is the administrator of the information processing system 1 or the warehouse 100.

[0046] The time acquisition unit 55 acquires information indicating the time when the receiving unit 51 received the omnidirectional image and the identification information, and outputs the information to the object position acquisition unit 56 .

[0047] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the retained information acquired by the retained information acquisition unit 53. The object position acquisition unit 56 also associates the position information of the object 30 with the identification information indicating the object 30 and the time information, and outputs the information via the output unit 57. The output destination of the output unit 57 is an external device such as a PC (Personal Computer), a display device such as the display 506, a storage device such as the HD 504, or the like.

[0048] The storage section 58 can store identifying information indicative of an object 30 such as a pallet 31 or cargo 32 .

[0049] (Example of processing by on-premise server 50) 5 is a flowchart showing an example of processing by the on-premise server 50. Fig. 5 shows processing triggered by the timing at which the on-premise server 50 receives an operation to start acquiring location information of the object 30. The operation to start acquiring location information of the object 30 is performed by a user such as an administrator using the pointing device 512 in Fig. 3 or the like.

[0050] First, in step S51, the receiving unit 51 receives the spherical image and the identification information via the network 400.

[0051] Subsequently, in step S52, the moving object position acquisition unit 52 acquires position information of the forklift 10 by calculation based on the input spherical image, and outputs the acquired information to the object position acquisition unit 56.

[0052] Next, in step S53, the holding information acquisition unit 53 acquires holding information indicating whether the object 30 is being held or not by the forklift 10 based on the input spherical image, and outputs the information to the object position acquisition unit 56.

[0053] Subsequently, in step S 54 , the identification information acquisition unit 54 acquires the identification information indicating the object 30 by inputting it, and outputs it to the object position acquisition unit 56 .

[0054] Subsequently, in step S55, the time acquisition unit 55 acquires information indicating the time when the receiving unit 51 received the omnidirectional image and the identification information, and outputs the information to the object position acquisition unit 56.

[0055] The order of the processes in steps S52 to S55 may be changed as appropriate, or they may be performed in parallel.

[0056] Subsequently, in step S56 , the object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the retained information acquired by the retained information acquisition unit 53 .

[0057] Subsequently, in step S57, the object position acquisition unit 56 associates the position information of the object 30, the identification information indicating the object 30, and the time information with each other, and outputs the result via the output unit 57.

[0058] Subsequently, in step S58, the on-premise server 50 determines whether to end the process. The on-premise server 50 determines whether to end the process based on an end operation by a user such as an administrator via the pointing device 512 or the like.

[0059] If it is determined in step S58 that the process is to be ended (step S58, Yes), the on-premise server 50 ends the process. On the other hand, if it is determined that the process is not to be ended (step S58, No), the on-premise server 50 performs the processes from step S51 onwards again.

[0060] In this way, the on-premise server 50 can process the location information of the object 30 and obtain the location information of the object 30 .

[0061] (Example of object location information acquisition results) 6A and 6B are diagrams showing a first example of the object position information acquisition results, where Fig. 6A is a diagram showing a position map, and Fig. 6B is a diagram showing position information and timestamps.

[0062] 6(a) is created by the moving object position acquisition unit 52 based on the omnidirectional image captured by the omnidirectional camera 20. The moving object position acquisition unit 52 acquires a point cloud including three-dimensional coordinate information, and creates the position map 61 by projecting the point cloud onto a two-dimensional plane.

[0063] The position map 61 is created, for example, in response to the movement of one forklift 10, and is used to acquire position information (self-position recognition) of all forklifts 10. This makes it possible to express the positions of multiple forklifts 10 in the same coordinate system.

[0064] The above-mentioned SLAM technique, for example, can be applied to a method of creating the position map 61 based on captured images such as a spherical image.

[0065] In flat-bed warehouses, which are common among cross-docking warehouses, the position map 61 is prone to change in response to changes in the position of the objects 30 due to the inbound and outbound shipment of pallets 31. Changes in the position map 61 can complicate the process of acquiring position information. A flat-bed warehouse is a type of warehouse that has few structures such as shelves and in which pallets 31 or cargo 32 are placed on the floor.

[0066] On the other hand, warehouses often have high ceilings, such as 5 m, making it easy to ensure visibility upward from the ceiling. Therefore, in the embodiment, position map 61 is created using an image showing the upward direction from among the omnidirectional images captured by omnidirectional camera 20, and position information of forklift 10 is acquired. In this way, position map 61 in which time change is suppressed can be used to acquire position information of forklift 10 without complicating processing.

[0067] Furthermore, holding information acquisition unit 53 uses the image showing the front of the omnidirectional image to acquire holding information of object 30 held by forklift 10. This makes it possible to acquire position information of forklift 10 and holding information of object 30 using one omnidirectional image captured by omnidirectional camera 20.

[0068] Note that the retained information acquisition unit 53 can also use information other than the spherical image to acquire the retained information. Examples of information other than the spherical image include information detected by a contact sensor, an infrared sensor, an ultrasonic sensor, a distance sensor, or a load sensor. The retained information acquisition unit 53 can also acquire the retained information by combining the spherical image or information detected by each sensor. However, from the perspective of simplifying the processing, it is more preferable to use the spherical camera 20 that can acquire the position information of the forklift 10 and the retained information from one spherical image.

[0069] The object position acquisition unit 56 can acquire a position map 62 indicating the position of the object 30 as the position information of the object 30 by deleting the position information of the forklift 10 shown in the position map 61 for which the holding information indicates non-holding. The position map 61 shown in Fig. 6(a) can also be said to indicate the position information of the object 30, and therefore the position map 62 is shown in parentheses in Fig. 6(a).

[0070] 6(b) is a table including three-dimensional coordinates of a point cloud indicating the position of the object 30 and a timestamp 64 indicating the time when the three-dimensional coordinates were acquired. The position table 63 is created by the object position acquisition unit 56. The timestamp 64 is an example of time information acquired by the time acquisition unit 55. The object position acquisition unit 56 can output the position table 63, which associates the position information of the object 30 with time information, via the output unit 57.

[0071] The object position acquisition unit 56 excludes point cloud information after a predetermined time has elapsed and updates the position table 63 using the new point cloud information. When the forklift 10 is not moving much, it is preferable to reduce the update frequency so that the point cloud information shown in the position table 63 does not decrease too much. As a result, even when there is a change in the conditions around the forklift 10 or it is difficult to ensure visibility above the forklift 10, by updating the position map 61 indicating the position of the forklift 10, the position map 62 indicating the positions of objects, and the position table 63, it becomes possible to preferably acquire the position information of the forklift 10 and the position information of the object 30.

[0072] Identification information such as a barcode attached to each object 30 is used to track the location of each of the multiple objects 30 within the warehouse 100. The identification information indicating each object 30 is stored in the storage unit 58 together with the initial location information of each object 30.

[0073] When the forklift 10 removes the pallet 31 from the container 300 or the like and temporarily places the pallet 31, a barcode or the like attached to the cargo 32 or the pallet 31 is read and stored in the storage unit 58 as identification information indicating the object 30. This storage is called initial registration. In this embodiment, in addition to the identification information indicating the object 30 that is initially registered, the position information of the object 30 acquired by the object position acquisition unit 56 can be associated and stored in the storage unit 58.

[0074] Here, an example of a simpler method for tracking the positions of multiple objects 30 within the warehouse 100 will be described with reference to Fig. 7. Fig. 7 is a diagram showing a second example of the results of acquiring object position information. Fig. 7(a) is a diagram showing sections, Fig. 7(b) is a diagram showing position information and barcodes indicating sections, and Fig. 7(c) is a diagram showing a map of the warehouse.

[0075] In order to track each object 30 within a warehouse, it is necessary to register the ID and initial position (ID or coordinates linked to the ID) of the pallet 31. Regarding the ID of the pallet 31, when the pallet 31 is removed from a container or the like using a forklift 10 and temporarily placed thereon, a barcode or other ID attached to the object 30 or the pallet 31 is generally scanned. By combining this conventional work with the aforementioned position recognition of the forklift 10 and information on the attachment / detachment recognition of the pallet 31, the initial position of the pallet 31 can be ascertained and registered using the position information of the forklift 10.

[0076] 7(a) indicate predetermined positional ranges within the warehouse 100. Work is carried out to initially register the object 30 within each of the sections 1, 2, 3 and 4.

[0077] By reading the barcodes indicating sections 1, 2, 3, and 4 using reader 700 either before or after reading the barcode indicating object 30, it is possible to associate object 30 with position information (position coordinates) indicating each section. Thereafter, when object 30 is transported by forklift 10, it becomes possible to track the position of object 30, including identification information indicating which object 30 is being transported.

[0078] Table 71 shown in FIG. 7(b) shows ID numbers indicating sections, location information of the sections, and barcodes indicating the sections in association with each other.

[0079] The barcodes indicating each section are printed and carried by the worker performing the initial registration so that they can be read by the reader 700. Alternatively, they are affixed to the floor or pillars in the warehouse 100, or to the forklift 10. This makes it possible to easily read the barcodes of each section before and after the work of reading the barcodes indicating the objects 30 is performed.

[0080] 7(c) shows a warehouse map in the application of the on-premise server 50, linking the ID (identification information) and location information of the object 30 and showing the registered pallet 31. According to the display shown in FIG. 7(c), once the pallet 31 is registered, it can be tracked even if it is moved by the forklift 10 without having to be scanned again.

[0081] FIG. 7(c) shows the operation section of the application on the on-premise server 50. It displays the time when the ID of the pallet 31 specified on the map was updated. According to the display shown in FIG. 7(c), by importing a list of incoming IDs and outgoing IDs of the pallets 31, it is possible to display a list of the incoming and outgoing status of the pallets 31 listed. Furthermore, according to the display shown in FIG. 7(c), by entering an ID in the text box, the corresponding pallet on the map is highlighted.

[0082] This pallet information (customer ID, coordinates, etc.) is managed as a table (pallet list) on the on-premise server 50, so it can be read out and displayed on a map at any time.

[0083] Next, calculation of the position information of the object 30 based on the image captured by the omnidirectional camera 20 installed on the forklift 10 will be described.

[0084] FIG. 8 is a diagram illustrating an example of detection of the object 30 by the retained information acquisition unit 53. In FIG.

[0085] The example shown in FIG. 8(a) shows a case where the cargo 32 on the second pallet 31 is visible from the base of the forks 21 to the spherical camera 20. Here, we consider determining whether the cargo 32 on the first pallet 31 and the cargo 32 on the second pallet 31 are stacked or unloaded based on the distance from the base of the forks 21 of the forklift 10 to the cargo 32 on the first pallet 31. In the example shown in FIG. 8(a), the cargo 32 on the first pallet 31 is always visible from the spherical camera 20, so the holding information acquisition unit 53 can reliably determine whether the cargo 32 is stacked or unloaded. Note that "reliably" here does not mean that it can be recognized in all cases.

[0086] The example shown in Fig. 8(b) illustrates a case where cargo 32 on the second pallet 31 is not visible from the base of fork 21 to spherical camera 20. As shown in Fig. 8(b), cargo 32 on the second pallet 31 may be hidden by cargo 32 on the first pallet 31 and may not be visible to spherical camera 20, so holding information acquisition unit 53 may not be able to accurately determine whether cargo 32 has been stacked or unloaded. However, in the example shown in Fig. 8(b), based on the position of cargo 32 on the first pallet 31, holding information acquisition unit 53 can estimate that cargo 32 on the second pallet 31 has been stacked or unloaded.

[0087] The example shown in Fig. 8(c) is an example in which position information of an object 30 is calculated using one-dimensional sensors 25a and 25b instead of the spherical camera 20. In the example shown in Fig. 8(c), two one-dimensional sensors (ToF in this case) 25a and 25b are attached to the head guard 10a of a forklift 10. One-dimensional sensor 25a is adjusted so that the laser light strikes the floor at the tips of the forks 21. One-dimensional sensor 25b is adjusted so that the laser light strikes the floor at the center of the forks 21.

[0088] One-dimensional sensor 25a detects the attachment or detachment of first pallet 31 at the tips of forks 21, and one-dimensional sensor 25b detects that first pallet 31 is inserted in the center of forks 21 (that there is a possibility that second pallet 31 is inserted at the tips of forks 21), thereby making it possible to recognize the transport state (held / not held) of pallet 31 in the same way as omnidirectional camera 20. In this case, although devices for one-dimensional sensors 25a and 25b are added in addition to omnidirectional camera 20, calculation of position information of pallet 31 based on the omnidirectional image captured by omnidirectional camera 20 can be reduced, and therefore the calculation resources of the single-board computer of forklift 10 can be used effectively.

[0089] The example shown in Figure 8(d) illustrates a case where cargo 32 on the second pallet 31 is not visible from the base of the fork 21 to the one-dimensional sensors 25a, 25b. As shown in Figure 8(d), cargo 32 on the second pallet 31 may be hidden by cargo 32 on the first pallet 31 and may not be visible to the one-dimensional sensors 25a, 25b, so the holding information acquisition unit 53 may not be able to accurately determine whether cargo 32 has been stacked or unloaded. However, in the example shown in Figure 8(d), the holding information acquisition unit 53 can estimate that cargo 32 on the second pallet 31 has been stacked or unloaded based on the position of the cargo 32 on the first pallet 31.

[0090] The following describes the process in which the holding information acquisition unit 53 estimates that the cargo 32 on the second pallet 31 has been loaded / unloaded.

[0091] There are four patterns for transporting the pallet 31 as described above, and the following parameters are used to determine the transport state (held / unheld) of the cargo 32 on the second pallet 31. The following parameters show an example when the width b of the pallet 31 is 1.1 m. Note that the setting values ​​(specific numerical values) described below are for reference only and are determined based on the size of the pallet 31 and the length of the forks 21 of the forklift 10. a: Distance from the base of the fork to the pallet [m] (variable) b: Pallet width [m] (fixed) c: Pallet width / 2 [m] (fixed)

[0092] 9-1 to 9-4 are diagrams illustrating the operation procedure up to when the holding information acquisition unit 53 determines the holding state of the second pallet 31. FIG.

[0093] Figure 9-1 shows the procedure for stacking two pallets separately.

[0094] Step 1: Insert the forks into the first pallet. When the distance from the base of the forks to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0095] Step 2: Stack the first pallet and move it.

[0096] Step 3: Insert the forks into the second pallet. When the distance from the base of the forks to the pallet satisfies a≦0.5 (the forks fit more than halfway into the second pallet), it is detected that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of pallet 2 has already been registered in storage unit 58: The location information of the pallet as the source of movement is stored in storage unit 58. The location information of the pallet is calculated from the relative distance (a+b+c) from the fork. Note that since it is not known whether the second pallet (pallet 2) is actually being stacked, the location information of the source of movement is not updated.

[0097] Step 4: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Forklift 10 (single board computer) display: displays one stack Display of smartphone 600: The first palette (palette 1) is displayed as a circle (solid line) to indicate that it has been stacked. : The second pallet (pallet 2) is displayed as a circle (dotted line or flashing) as it is uncertain whether it has been stacked.

[0098] Step 5: Stack two pallets and move them.

[0099] Step 6: Confirm that two pallets have been stacked. The operator of the forklift 10 presses a button on the holding information input unit provided on the forklift 10 to confirm that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 transmits the detection result to the on-premise server 50.

[0100] Step 7: The displays on the forklift 10 (single-board computer) and smartphone 600 are updated. The displays on the forklift 10 (single-board computer) and smartphone 600 indicate that two pallets are being held. The forklift operator can confirm that the button on the holding information input unit has been pressed correctly. If the identification information of palette 2 is not registered in the storage unit 58: it is displayed as a new palette. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet that was stored in step 3 is updated to the latest position information.

[0101] Figure 9-2 shows the procedure for unloading two pallets separately.

[0102] Step 1: Place both pallets on the ground.

[0103] Step 2: The forks are removed from the second pallet (pallet 2). (They are moved back the distance of one pallet.) When the distance from the base of the forks 21 to the pallet satisfies a≧(0.5 to 1.0), it is detected that the second pallet (pallet 2) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b+c) from the fork.

[0104] Step 3: The display on the display unit of the forklift 10 (single board computer) and the display unit of the smartphone 600 are updated. Since the second pallet (pallet 2) has been unloaded, the display shows that one pallet has been stacked.

[0105] Step 4: Stack and move only the first pallet (Pallet 1). If you want to stack a second pallet after this, proceed to Step 3 in Figure 9-1.

[0106] Step 5: Remove the forks from the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≧(1.5 to 2.0), it is detected that the first pallet (pallet 1) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0107] Step 6: The display on the display unit of the forklift 10 (single-board computer) and the display unit of the smartphone 600 are updated. Since the first pallet (pallet 1) has been unloaded, the display shows that there is no pallet.

[0108] Figure 9-3 shows the operating procedure for stacking two pallets at the same time.

[0109] Step 1: Insert the forks into the first pallet. When the distance from the base of the forks to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0110] Step 2: Insert the forks into the second pallet. When the distance from the base of the forks 21 to the pallet satisfies a≦0.5 (the forks can fit more than halfway into the second pallet), it is detected that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of pallet 2 has already been registered in storage unit 58: The location information of the pallet as the source of movement is stored in storage unit 58. The location information of the pallet is calculated from the relative distance (a+b+c) from the fork. Since it is not known whether the second pallet (pallet 2) is actually being stacked, the location information of the source of movement is not updated.

[0111] Step 3: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Forklift 10 (single board computer) display: displays one stack Display of smartphone 600: The first palette (palette 1) is displayed as a circle (solid line) to indicate that it has been stacked. : The second pallet (pallet 2) is displayed as a circle (dotted line or flashing) as it is uncertain whether it has been stacked.

[0112] Step 4: Stack two pallets and move them.

[0113] Step 5: Confirm that two pallets have been stacked. The operator of the forklift 10 presses a button on the holding information input unit provided on the forklift 10 to confirm that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 transmits the detection result to the on-premise server 50.

[0114] Step 6: The display on the forklift 10 (single-board computer) and the display on the smartphone 600 are updated. The Edge output unit and the display on the smartphone 600 display a message indicating that two pallets are being held. The forklift operator can confirm that the button on the holding information input unit has been pressed correctly. If the identification information of palette 2 is not registered in the storage unit 58: it is displayed as a new palette. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet that was stored in step 2 is updated to the latest position information.

[0115] Figure 9-4 shows the operating procedure for unloading two pallets at the same time.

[0116] Step 1: Place both pallets on the ground.

[0117] Step 2: The forks are removed from the second pallet (pallet 2). (They are moved back the distance of one pallet.) When the distance from the base of the forks 21 to the pallet satisfies a≧(0.5 to 1.0), it is detected that the second pallet (pallet 2) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b+c) from the fork.

[0118] Step 3: The display on the display unit of the forklift 10 (single board computer) and the display unit of the smartphone 600 are updated. Since the second pallet (pallet 2) has been unloaded, the display shows that one pallet has been stacked.

[0119] Step 4: Remove the forks from the first pallet (pallet 1). When the distance from the base of the forks 21 to the pallet satisfies a≧(1.5 to 2.0), it is detected that the first pallet (pallet 1) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 transmits the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0120] Step 5: The display on the display of the forklift 10 (single board computer) and the display on the display of the smartphone 600 are updated. Since the first pallet (pallet 1) has been unloaded, the display shows that there is no pallet.

[0121] 10-1 to 10-4 are diagrams illustrating the operation procedure up to when the holding information acquisition unit 53 determines the holding state of only the first palette.

[0122] Figure 10-1 shows the operating procedure when stacking only one pallet on the tip of the fork.

[0123] Step 1: Insert the forks into the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0124] Step 2: Stack the first pallet (pallet 1) and move it. If you want to stack the second pallet (pallet 2) after this, proceed to step 3 in Figure 9-1.

[0125] Step 3: The display on the display unit of the forklift 10 (single-board computer) and the display unit of the smartphone 600 are updated. The display shows the first pallet (pallet 1) stacked one by one.

[0126] Figure 10-2 shows the operating procedure when only one pallet is unloaded onto the fork tips.

[0127] Step 1: Place the first pallet (pallet 1) on the ground.

[0128] Step 2: Remove the forks from the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≧(1.5 to 2.0), it is detected that the first pallet (pallet 1) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0129] Step 3: The display on the display of the forklift 10 (single board computer) and the display on the display of the smartphone 600 are updated. Since the first pallet (pallet 1) has been unloaded, the display shows that there is no pallet.

[0130] Figure 10-3 shows the operating procedure when stacking only one pallet at the base of the forks.

[0131] Step 1: Insert the forks into the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0132] Step 2: Insert the forks all the way into the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≦0.5 (the forks can fit more than halfway into the second pallet), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50.

[0133] At this point, stacking a second pallet is prohibited. Even if the forks are inserted all the way into the first pallet (pallet 1) and then the second pallet is inserted into the forks, the location information when the second pallet is inserted cannot be obtained, making it impossible to track, so this is prohibited.

[0134] Step 3: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Forklift 10 (single board computer) display: displays one stack Display of smartphone 600: The first palette (palette 1) is displayed as a circle (solid line) to indicate that it has been stacked. : The second pallet (pallet 2) is displayed as a circle (dotted line or flashing) as it is uncertain whether it has been stacked.

[0135] Step 4: Stack one pallet and move it.

[0136] Figure 10-4 shows the operating procedure when only one pallet is unloaded onto the base end of the forks.

[0137] Step 1: Place the first pallet (pallet 1) on the ground.

[0138] Step 2: The forks are removed from the first pallet (pallet 1). (They move back the distance of one pallet.) When the distance from the base of the forks 21 to the pallet satisfies a≧(0.5 to 1.0), it is detected that the second pallet (pallet 2) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. However, since the second pallet (pallet 2) does not exist, nothing happens.

[0139] Step 3: The display on the display unit of the forklift 10 (single board computer) and the display unit of the smartphone 600 are updated. The display becomes one stack of sheets.

[0140] Step 4: Remove the forks from the first pallet (pallet 1). When the distance from the base of the forks 21 to the pallet satisfies a≧(1.5 to 2.0), it is detected that the first pallet (pallet 1) has been unloaded. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 transmits the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the destination. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0141] Step 5: The display on the display of the forklift 10 (single board computer) and the display on the display of the smartphone 600 are updated. Since the first pallet (pallet 1) has been unloaded, the display shows that there is no pallet.

[0142] 11-1 to 11-3 are diagrams illustrating the operation procedure up to the point where the holding information acquisition unit 53 determines the non-holding state of the second palette.

[0143] Figure 11-1 shows the procedure for stacking two pallets separately.

[0144] Step 1: Insert the forks into the first pallet. When the distance from the base of the forks 21 to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0145] Step 2: Stack and move the first pallet (pallet 1).

[0146] Step 3: Insert the forks into the second pallet (pallet 2). When the distance from the base of the forks 21 to the pallet satisfies a≦0.5 (the forks can fit more than halfway into the second pallet), it is detected that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of pallet 2 has already been registered in storage unit 58: The position information of the pallet is stored in storage unit 58 as the source of movement. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. Although it is not known whether the second pallet (pallet 2) has actually been stacked, once the first pallet (pallet 1) is inserted all the way to the base of the forks 21, the second pallet (pallet 2) is considered to have been stacked.

[0147] Step 4: The displays on the display units of the forklift 10 (single-board computer) and smartphone 600 are updated. The displays on the display units of the forklift 10 (single-board computer) and smartphone 600 indicate that two pallets are being held. If the identification information of palette 2 is not registered in the storage unit 58: it is displayed as a new palette. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet that was stored in step 3 is updated to the latest position information.

[0148] Step 5: Stack two pallets and move them.

[0149] The operating procedure for unloading two pallets separately is the same as that explained in Figure 9-2, so the explanation will be omitted.

[0150] Figure 11-2 shows the procedure for stacking two pallets at the same time.

[0151] Step 1: Insert the forks into the first pallet (pallet 1). When the distance from the base of the forks 21 to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0152] Step 2: Insert the forks into the second pallet (pallet 2). When the distance from the base of the forks 21 to the pallet satisfies a≦0.5 (the forks can fit more than halfway into the second pallet), it is detected that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 2 has not been registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. If the identification information of pallet 2 has already been registered in storage unit 58: The position information of the pallet is stored in storage unit 58 as the source of movement. The position information of the pallet is calculated from the relative distance (a+b+c) from the forks. Although it is not known whether the second pallet (pallet 2) has actually been stacked, once the first pallet (pallet 1) is inserted all the way to the base of the forks 21, the second pallet (pallet 2) is considered to have been stacked.

[0153] Step 3: The display of the forklift 10 (single-board computer) and the display of the smartphone 600 are updated. The output of the edge and the output of the terminal display a message indicating that two pallets are being held. If the identification information of palette 2 is not registered in the storage unit 58: it is displayed as a new palette. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet that was stored in step 2 is updated to the latest position information.

[0154] Step 4: Stack two pallets and move them.

[0155] The operating procedure for unloading two pallets at the same time is the same as that explained in Figure 9-4, so the explanation will be omitted.

[0156] Furthermore, the operating procedure for stacking only one pallet on the tip of the fork is the same as the procedure explained in Figure 10-1, so the explanation will be omitted.

[0157] In addition, the operating procedure for unloading only one pallet onto the tip of the fork is the same as the procedure explained in Figure 10-2, so the explanation will be omitted.

[0158] Figure 11-3 shows the operating procedure when stacking only one pallet at the base of the forks.

[0159] Step 1: Insert the forks into the first pallet (pallet 1). When the distance from the base of the forks to the pallet satisfies a≦2 (approximately the distance of two pallets), it is detected that the first pallet (pallet 1) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. If the identification information of pallet 1 is not registered in storage unit 58: The identification information and position information of a new pallet are registered in storage unit 58. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork. If the identification information of pallet 1 has already been registered in storage unit 58: The position information of the pallet is updated as the movement source. The position information of the pallet is calculated from the relative distance (a+b / 2) from the fork.

[0160] Step 2: Insert the forks 21 all the way into the first pallet (pallet 1). When the distance from the base of the forks 21 to the pallet satisfies a≦0.5 (the forks fit more than halfway into the second pallet), it is detected that the second pallet (pallet 2) has been stacked. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50. Although it is not known whether the second pallet (pallet 2) has actually been stacked, once the forks 21 have been inserted all the way into the first pallet (pallet 1), it is considered that the second pallet (pallet 2) has been stacked.

[0161] At this point, stacking a second pallet is prohibited. Even if the forks are inserted all the way into the first pallet (pallet 1) and then the second pallet is inserted into the forks, the location information when the second pallet is inserted cannot be obtained, making it impossible to track, so this is prohibited.

[0162] Step 3: The displays on the display units of the forklift 10 (single-board computer) and smartphone 600 are updated. The displays on the display units of the forklift 10 (single-board computer) and smartphone 600 indicate that two pallets are being held. If the identification information of palette 2 is not registered in the storage unit 58: it is displayed as a new palette. If the identification information of the pallet 2 has already been registered in the storage unit 58: The position information of the pallet that was stored in step 3 is updated to the latest position information.

[0163] Step 4: Stack and move the first pallet (pallet 1).

[0164] Step 5: Confirm that one pallet has been stacked. The forklift operator presses a button on the holding information input unit provided on the forklift 10 to confirm that the first pallet (pallet 1) has been stacked (the second pallet is not being held). If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 sends the detection result to the on-premise server 50.

[0165] Step 6: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Display of Forklift 10 (single board computer): Displays one stack. Display unit of smartphone 600: The first palette (palette 1) is displayed as a circle (solid line) to indicate that the palette has been stacked.

[0166] The operating procedure for unloading only one pallet onto the base of the forks is the same as that explained in Figure 10-4, so the explanation will be omitted.

[0167] 12-1 and 12-2 are diagrams illustrating the operation procedure up to when the holding information acquisition unit 53 determines the number of stacked pallets.

[0168] Figure 12-1 shows the procedure for stacking / unloading pallets in two-tier stacking.

[0169] Step 1: Insert the fork into the stacked pallet (2 layers).

[0170] Step 2: Stack the stacked pallets (two layers) and move them.

[0171] Step 3: Confirm that the multi-tiered pallets (two tiers) have been stacked. The forklift operator confirms that the multi-tiered pallets (two tiers) have been stacked by pressing a button on the holding information input unit provided on the forklift 10. Note that if three or more tiers are stacked, numbers must also be entered. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the detection results are sent from the holding information acquisition unit 53 to the on-premise server 50.

[0172] Step 4: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Display of the forklift 10 (single board computer): Displays the stacking status or the number of stacked items (here, 2). Display of smartphone 600: When only the first stage is being transported, a gray circle is displayed, but when the second stage is being transported, an orange circle or a number inside a gray circle is displayed.

[0173] Step 5: Unload the stacked pallet (2 layers).

[0174] Step 6: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Display of forklift 10 (single board computer): Indicates that there is no pallet on the forks. Display of smartphone 600: The single-tier palette displays a gray circle, while the double-tier palette displays an orange circle or a number inside a gray circle.

[0175] Figure 12-2 shows the procedure for stacking / unloading pallets in tiers (three or more tiers). Here, we will explain stacking (three tiers). Note that the procedure is the same for tiers of four or more, except for the numbers you enter.

[0176] Step 1: Insert the fork into the stacked pallet (3 layers).

[0177] Step 2: Stack the stacked pallets (3 layers) and move them.

[0178] Step 3: Confirm that the stacked pallets (3 layers) have been stacked. The forklift operator confirms that the stacked pallets (3 layers) have been stacked by inputting the number of layers into a holding information input unit provided on the forklift 10. If the holding information acquisition unit 53 is implemented by the forklift 10 (single-board computer), the holding information acquisition unit 53 transmits the detection result to the on-premise server 50.

[0179] Step 4: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Display unit of the forklift 10 (single board computer): Shows the stacking state or the number of stacked items (here, 3). Display of smartphone 600: When only one level is being transported, a gray circle is displayed, but when three levels are being transported, a number is displayed inside the gray circle.

[0180] Step 5: Unload the stacked pallets (3 layers).

[0181] Step 6: The display of the forklift 10 (single board computer) and the display of the smartphone 600 are updated. Display of forklift 10 (single board computer): Indicates that there is no pallet on the forks. Display of smartphone 600: A single-tier palette is displayed as a gray circle, and a stacked palette is displayed as a gray circle with the number of layers (here, 3) inside.

[0182] Thus, according to this embodiment, when multiple objects are being transported simultaneously, it is possible to recognize not only the transport state (held / not held) of one object being moved by the moving body, but also the transport state (held / not held) of multiple objects.

[0183] More specifically, the holding information acquisition unit 53 calculates the distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines whether the second and subsequent objects are being held or not (held state) by the forklift. Determining the state here means recognizing the state. This makes it possible to recognize the transport state (held / not held) of the second cargo at the tip of the fork when two cargoes are stacked at the same time by the forklift.

[0184] Furthermore, the holding information acquisition unit 53 calculates the distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines whether the second and subsequent objects are being held or not held by the forklift (non-held state). Determining the state here means recognizing the state. This makes it possible to recognize the transport state (held / not held) of the second pallet at the tip of the fork when two pallets are stacked by the forklift.

[0185] Furthermore, the holding information acquisition unit 53 calculates the distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines the stacking state of the second and subsequent objects. Determining the state here means recognizing the state. This makes it possible to recognize the transport state of the pallets (how many layers are being held in the stack) when multiple pallets are stacked by a forklift.

[0186] In the present embodiment, a forklift is used as an example of a moving object, but the moving object is not limited to this. For example, the moving object may be an automated guided vehicle, a drone, or the like.

[0187] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0188] Note that the information processing device is not limited to the on-premise server 50, as long as it is a device with a communication function. The information processing device may be, for example, an image forming device, a PJ (Projector), an IWB (Interactive White Board: a white board with an electronic blackboard function that allows mutual communication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0189] For example, aspects of the present invention are as follows. <1> An information processing device that processes position information of an object moved by a moving body having an imaging unit that captures an image and a holding unit that can hold a plurality of objects, a mobile object position acquisition unit that acquires position information of the mobile object based on an image of the surroundings of the mobile object captured by the imaging unit; a holding information acquisition unit that acquires a state of either holding or not holding the object by the moving body based on an image of the holding unit captured by the imaging unit; an object position acquisition unit that acquires position information of the object based on the position information of the moving object acquired by the moving object position acquisition unit and the state acquired by the held information acquisition unit; Equipped with the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines whether the second or subsequent objects are being held or not held by the moving body. 1. An information processing device comprising: <2> the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and estimates that the second or subsequent objects are being held by the moving body. Characterized by <1> The information processing device described in <3> the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and estimates that the second and subsequent objects are not being held by the moving body. Characterized by <1> or <2> The information processing device described in <4> the holding information acquisition unit calculates a distance of the object from the holder when the object is being held and before and after the object is being held, and determines a stacking state of the second and subsequent objects by the moving body. Characterized by <1> Or <3> 10. The information processing device according to claim 9, wherein <5> A moving object and At least one imaging device attached to the moving body; <1> Or <4> an information processing device according to any one of the above items; An information processing system comprising: <6> 1. An information processing method for an information processing device that processes position information of an object moved by a moving body having an imaging unit that captures an image and a holding unit that can hold a plurality of objects, a moving object position acquisition step of acquiring position information of the moving object based on an image of the surroundings of the moving object captured by the imaging unit; a holding information acquisition step of acquiring a holding or non-holding state of the object by the moving body based on an image of the holding unit captured by the imaging unit; an object position acquisition step of acquiring position information of the object based on the position information of the moving body acquired in the moving body position acquisition step and the state acquired in the held information acquisition step; Including, the holding information acquisition step calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines whether the second or subsequent objects are being held or not held by the moving body; 1. An information processing method comprising: [Explanation of symbols]

[0190] 10 Mobile 20 Imaging device 21 Holding part 30 Object 50 Information processing equipment 52 Mobile object position acquisition section 53 Retained information acquisition unit 56 Object position acquisition section [Prior art documents] [Patent documents]

[0191] [Patent Document 1] Japanese Patent Publication No. 2022-130187

Claims

1. An information processing device that processes position information of an object moved by a moving body having an imaging unit that captures an image and a holding unit that can hold a plurality of objects, a mobile object position acquisition unit that acquires position information of the mobile object based on an image of the surroundings of the mobile object captured by the imaging unit; a holding information acquisition unit that acquires a state of either holding or not holding the object by the moving body based on an image of the holding unit captured by the imaging unit; an object position acquisition unit that acquires position information of the object based on the position information of the moving object acquired by the moving object position acquisition unit and the state acquired by the held information acquisition unit; Equipped with the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and determines whether the second or subsequent objects are being held or not held by the moving body; 1. An information processing device comprising:

2. the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and estimates that the second or subsequent objects are being held by the moving body.

2. The information processing apparatus according to claim 1, wherein:

3. the holding information acquisition unit calculates a distance of the object from the holding unit when the object is being held and before and after the object is being held, and estimates that the second and subsequent objects are not being held by the moving body.

2. The information processing apparatus according to claim 1, wherein:

4. the holding information acquisition unit calculates a distance of the object from the holder when the object is being held and before and after the object is being held, and determines a stacking state of the second and subsequent objects by the moving body.

2. The information processing apparatus according to claim 1, wherein:

5. A moving object and At least one imaging device attached to the moving body; An information processing device according to any one of claims 1 to 4; An information processing system comprising:

6. 1. An information processing method for an information processing device that processes position information of an object moved by a moving body having an imaging unit that captures an image and a holding unit that can hold a plurality of objects, a moving object position acquisition step of acquiring position information of the moving object based on an image of the surroundings of the moving object captured by the imaging unit; a holding information acquisition step of acquiring a holding or non-holding state of the object by the moving body based on an image of the holding unit captured by the imaging unit; an object position acquisition step of acquiring position information of the object based on the position information of the moving body acquired in the moving body position acquisition step and the state acquired in the held information acquisition step; Including, the holding information acquisition step calculates a distance of the object from the holder when the object is being held and before and after the object is being held, and determines whether the second or subsequent objects are being held or not held by the moving body; 1. An information processing method comprising:

Images