Information processing device, information processing system and information processing method

Abstract

To surely photograph an image of a marker installed on a back rest by mounting a camera on a position which is not on an extension line in a direction in which the back rest of a moving body moves upward and downward, so as to estimate a height of an object that is moved by the moving body, from a position of the photographed marker.SOLUTION: An information processing system comprises: a moving body that comprises a holding part that holds an object and can move the object upward and downward and a marker installed on the holding part; at least one photographing device mounted on the moving body, at a position that is not on the same line with respect to a direction in which the object is moved upward and downward, which photographs the circumference of the moving body; and an estimating part that estimates a height of the object, by detecting a position of the marker from an image of the marker photographed by the photographing part.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, and an information processing method. [Background technology]

[0002] Conventionally, there is already known a technology for estimating the position information of a moving object, such as a forklift, that travels mainly indoors in warehouses and factories, based on images captured by a camera attached to the moving object, and further recognizing the transport state (held / unheld) of the object (cargo or pallet) being moved by the moving object, thereby acquiring the position information of the object.

[0003] Patent Document 1 discloses a technology for detecting the height of an object being moved by a forklift, in which a camera attached to a mobile body that moves an object, on an extension of the upward and downward movement direction of a backrest equipped with forks for moving the object, and below the backrest, captures an image of a marker installed on the backrest directly above. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology of Patent Document 1 has a problem in that, depending on the shape of the vehicle body, it may not be possible to capture an image of the marker on the backrest from a camera attached on an extension line of the direction in which the backrest moves up and down. As a result, it may not be possible to detect the position of the marker attached to the backrest, and it may not be possible to estimate the height of an object moved by the vehicle.

[0005] The present invention has been made in consideration of the above, and aims to provide an information processing system, an information processing device, and an information processing method that can reliably capture an image of a marker installed on the backrest by attaching a camera to a location that is not on an extension of the ascending and descending direction of the backrest of a moving body, and can estimate the height of an object moved by the moving body from the position of the captured marker. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention is characterized by comprising a moving body including a holding part that holds an object and allows the object to move upward and downward, and a marker installed on the holding part, at least one imaging device that is attached to the moving body at a location that is not on the same line as the direction of the object's upward and downward movement and that images the surroundings of the moving body, and an estimation part that estimates the height of the object by detecting the position of the marker from an image of the marker captured by the imaging device. [Effects of the Invention]

[0007] According to the present invention, by attaching a camera to a location that is not on an extension of the upward and downward movement direction of the backrest of a moving body, it is possible to reliably capture an image of a marker installed on the backrest and estimate the height of an object being moved by the moving body from the position of the captured marker. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of position information of an object in a warehouse. [Figure 2] FIG. 2 is a diagram illustrating an example of an overall configuration of an information processing system according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing the relationship between the omnidirectional camera of the forklift and the markers. [Figure 4] FIG. 4 is a diagram showing the configuration and installation of the marker. [Figure 5] FIG. 5 is a block diagram illustrating an example of a hardware configuration of an on-premise server. [Figure 6] FIG. 6 is a block diagram illustrating an example of a functional configuration of the information processing system. [Figure 7] FIG. 7 is a flowchart showing an example of a cargo attachment / detachment position / height estimation process performed by the cargo attachment / detachment position / height estimation unit. [Figure 8] FIG. 8 is a flowchart showing the flow of processing by the management unit at the time of attachment. [Figure 9] FIG. 9 is a flowchart showing the flow of processing performed by the management unit at the time of detachment. [Figure 10] FIG. 10 is a diagram showing a configuration of a forklift according to the second embodiment. [Figure 11] FIG. 11 is a flowchart showing the flow of the process of switching between the two cameras. [Figure 12] FIG. 12 is a diagram showing a configuration of a forklift according to the third embodiment. [Figure 13] FIG. 13 is a diagram showing a first modified example of the configuration of the forklift truck according to the third embodiment. [Figure 14] FIG. 14 is a diagram showing a second modified example of the configuration of the forklift truck according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an information processing system, an information processing device, and an information processing method will be described in detail with reference to the accompanying drawings.

[0010] (First embodiment) An information processing device according to a first embodiment processes position information of an object moved by a moving body. For example, the moving body is a counter-type forklift, and the object includes a pallet. The information processing device according to the embodiment processes position information of the object, such as a pallet, moved by the forklift, and recognizes and tracks the movement of the object.

[0011] Here, Fig. 1 is a diagram for explaining an example of position information of an object in a warehouse. Fig. 1 shows the interior of a warehouse 100 and the periphery of the warehouse 100 as viewed from above (the ceiling side).

[0012] Warehouse 100 is a terminal warehouse (a warehouse established at a transit point in transportation). This terminal warehouse is a type of warehouse known as a cross-docking type. In a cross-docking type warehouse, multiple pallets for each product are received from a factory or wholesaler, and temporarily stored in the warehouse. Then, at the time of shipment, multiple types of pallets are combined while still packed on the same pallet, and shipped to the respective retail stores.

[0013] In Fig. 1, a truck yard 200 is located around a warehouse 100. Fig. 1 shows that the truck yard 200 has detached containers 300 transported by trailers and truck beds connected to the warehouse. A forklift 10 removes a pallet 31 from at least one of the beds of trucks that have arrived at the truck yard 200 or the containers 300 transported by trailers.

[0014] Thereafter, the forklift 10 carries the pallet 31 to the temporary storage location 40 and temporarily stores it there. Thereafter, at the time of shipping, the forklift 10 carries the pallet 31 to a location close to the truck yard 200 in the warehouse 100, arranges the items, and then loads the pallet 31 onto the bed of a truck or a container 300.

[0015] In order to ensure flexible space for the daily changes in the types and quantities of goods coming in and out, temporary storage locations 40 often do not have designated sections for each product. However, because multiple workers temporarily store pallets 31 in arbitrary locations, when shipping, it is necessary to search for the desired pallet from among the multiple temporarily stored pallets.

[0016] To efficiently perform this search work, there is a need for an information processing system that can effectively utilize space by not specifying temporary storage locations for the pallets 31, while recognizing and tracking the movement of the pallets 31 within the warehouse 100 to visualize it. As an example, the information processing device according to the embodiment is used in such an information processing system.

[0017] An information processing system including an information processing device according to the first embodiment will be described below.

[0018] (Example of overall configuration of information processing system 1) Fig. 2 is a diagram showing an example of the overall configuration of an information processing system 1 according to the first embodiment. As shown in Fig. 2, the information processing system 1 includes a forklift 10, an omnidirectional camera 20, and an on-premise server 50. These are communicably connected via a network 400 such as a LAN (Local Area Network). Note that devices other than those described above, such as an external server or an image forming device, may also be communicably connected to the network 400.

[0019] The forklift 10 is an example of a mobile body that transports the pallet 31 and the cargo 32 by holding the cargo 32 placed on the pallet 31 and transporting it while holding it. Transport by a mobile body is an example of movement by a mobile body. The pallet 31 and the cargo 32 are each an example of an object. In the following, the pallets 31 and the cargo 32 will be collectively referred to as the object 30 unless they are particularly distinguished from one another. The forklift 10 is a generic term for multiple forklifts, the pallet 31 is a generic term for multiple pallets, and the cargo 32 is a generic term for multiple cargoes.

[0020] The forklift 10 may transport the object 30 in response to the driving operation of an operator, or may transport the object 30 by automatic driving without the intervention of an operator.

[0021] The omnidirectional camera 20 is an example of an imaging device that is attached to the forklift 10 and captures images of the surroundings of the forklift 10. The omnidirectional camera 20 is a camera that can capture images in all directions of 360 degrees around the omnidirectional camera 20 as a viewing angle. The direction 20a indicates the direction in which the omnidirectional camera 20 can capture images.

[0022] The spherical image (omnidirectional image) captured by the spherical camera 20 is one example of a captured image. However, the imaging unit is not limited to the spherical camera 20, and may be any device that has a wide field of view close to a sphere and can capture an image of the area around the forklift 10. Furthermore, the captured image does not necessarily have to be a spherical image.

[0023] The spherical image includes an image capturing a scene in the conveying direction 11 of the object 30 as seen from the forklift 10, and a scene in the vertically upward direction 12 as seen from the forklift 10. In other words, the conveying direction 11 is in front of the forklift 10, and the vertically upward direction 12 is above the forklift 10. Since the spherical camera 20 can capture images in all directions, it can capture an image including both the front and the above of the forklift 10 in a single image. The conveying direction 11 is an example of a moving direction.

[0024] The omnidirectional camera 20 is preferably mounted on the head guard (roof of the driver's seat) 10a of the forklift 10 or on the outer mast 22 that supports the forks 21. This ensures a good field of view for capturing images in front of and above the forklift 10. Here, the forks 21 and backrest 23 are an example of a holding portion that is provided on the forklift 10 and that holds an object and allows the object to move up and down. In this embodiment, the omnidirectional camera 20 is mounted on the outer mast 22 that supports the forks 21.

[0025] The omnidirectional camera 20 has a wireless communication function and transmits the captured omnidirectional image to the on-premise server 50 via the network 400.

[0026] FIG. 3 is a diagram showing the relationship between the omnidirectional camera 20 of the forklift 10 and the marker M1. 4 is a diagram showing the configuration and installation mode of the marker M1. The forklift 10 shown in FIG. 3 is an example in which the omnidirectional camera 20 is attached to an outer mast 22 that supports the forks 21. FIGS. 3(a) to 3(c) show the rising mode of the marker M1 as the forks 21 of the forklift 10 rise.

[0027] The outer mast 22 is a support member that supports the inner mast 24 so that it can move up and down. The inner mast 24 is driven by a chain and moves up and down to raise and lower the forks 21. Generally, when the forks 21 of the forklift 10 are raised, the inner mast 24 rises, and the backrest 23 and the forks 21 rise relative to the inner mast 24.

[0028] 3, the omnidirectional camera 20 is provided on the outer mast 22 at a location that is not on the same line as the upward and downward movement directions (vertical upward direction 12) of the forks 21 of the forklift 10 and the backrest 23 that moves up and down integrally with the forks 21. Because the outer mast 22 does not move up and down, it can be wired to the main body of the forklift 10 by a cable.

[0029] In this embodiment, the marker M1, which serves as a landmark and is attached to the backrest 23, is captured by the omnidirectional camera 20. Then, the type of marker M1 is detected by image recognition of the image captured by the omnidirectional camera 20, and the height of the fork 21 is calculated from the type and position of the detected marker M1.

[0030] 3 and 4, the backrest 23 is provided with a plurality of cylindrical markers M1. The markers M1 are captured by the spherical camera 20 and subjected to image recognition. The positions of the markers M1 are detected from the images of the markers M1 captured by the spherical camera 20, and are used to estimate the heights of the forks 21 of the forklift 10 and the object 30. In this embodiment, three markers M1 are provided at different height positions on the backrest 23.

[0031] As shown in Fig. 4, the marker M1 placed on the backrest 23 has a cylindrical shape. The marker M1 has a different pattern depending on the placement position on the backrest 23. The three markers M1 with different patterns are marker M1a, marker M1b, and marker M1c. By making the marker M1 cylindrical, even if the relative angle between the omnidirectional camera 20 used to detect height and the marker M1 changes, the marker M1 can be seen with approximately the same cylindrical thickness, and the pattern of the marker M1 can be stably detected.

[0032] The cargo 32 is provided with a barcode 33, which is an example of identification information that identifies the cargo 32. Such a barcode may be provided on the pallet 31 and used as identification information that identifies the pallet 31. The barcode 33 is read by a reader such as a barcode reader, and the identification information resulting from the reading is transmitted to the on-premise server 50 via the network 400. Note that the identification information is not limited to a barcode, and may be a QR code (registered trademark), an ID (identifier) ​​number, or the like.

[0033] The on-premise server 50 is an example of an information processing device that is installed in the warehouse 100 and processes the position information of the object 30 transported by the forklift 10. The on-premise server 50 can be replaced with a cloud server (a computing device installed outside the warehouse environment). This can reduce the introduction cost and running cost.

[0034] The on-premise server 50 processes the position information of the object 30 based on the omnidirectional image received via the network 400 and the identification information indicating the object 30. In addition, by using the omnidirectional images captured by the multiple omnidirectional cameras 20, the position information of the object 30 held by the multiple forklifts 10 can be acquired.

[0035] Furthermore, the on-premise server 50 executes a self-position recognition calculation process for the forklift 10 based on the omnidirectional image received via the network 400. The on-premise server 50 performs the self-position recognition calculation process for the multiple omnidirectional cameras 20, thereby grasping the current positions of the multiple forklifts 10.

[0036] The forklift 10 includes, for example, a single-board computer (vehicle-mounted edge). The forklift 10 has the omnidirectional camera 20 connected to the single-board computer via a wired connection. The self-position recognition calculation process is performed on the single-board computer, and the results are transmitted and aggregated by wireless communication to the on-premise server 50 via the network 400. This allows the on-premise server 50 to grasp the current position of each forklift 10.

[0037] The on-premise server 50 reads the ID, location information, video image information, etc. from HD504 (see Figure 5) within the on-premise server 50, identifies the final location information of the forklift 10 and pallet 31, and visualizes it by outputting it to a terminal (smartphone, etc.).

[0038] As described above, the information processing system 1 includes the on-premise server 50 outside the forklift 10, and the processing unit is provided in the on-premise server 50. This makes it possible to reduce the power consumption of the single-board computer included in the forklift 10. Specifically, what previously required approximately 40 W can now be reduced to approximately 10 W.

[0039] However, since the forklift 10 moves around, communication between the forklift 10 (or the on-board edge) and the on-premise server 50 may be interrupted. Therefore, as will be described in detail later, in this embodiment, the forklift 10 (or the on-board edge) and the on-premise server 50 each have a communication status confirmation unit. This allows the connection status on the network 400 to be monitored. When communication is interrupted, the data that was not transmitted at the time of the interruption will be sent all at once after the connection is re-established.

[0040] (Example of hardware configuration of on-premise server 50) 5 is a block diagram showing an example of a hardware configuration of the on-premise server 50. The on-premise server 50 is constructed by a computer.

[0041] 5 , the on-premise server 50 includes a central processing unit (CPU) 501, a read-only memory (ROM) 502, a random access memory (RAM) 503, a hard disk (HD) 504, a hard disk drive (HDD) controller 505, and a display 506. The on-premise server 50 also includes an external device connection interface (I / F) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a digital versatile disk rewritable (DVD-RW) drive 514, and a media I / F 516.

[0042] Of these, the CPU 501 controls the overall operation of the on-premise server 50. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0043] The HD 504 stores various data such as programs, etc. The HDD controller 505 controls reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, menus, windows, characters, or images.

[0044] The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using the network 400. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 5.

[0045] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0046] (Example of functional configuration of information processing system 1) 6 is a block diagram showing an example of the functional configuration of the information processing system 1. As shown in FIG. 6, the on-premise server 50 includes a receiving unit 51, a mobile object position acquiring unit 52, a retained information acquiring unit 53, an identification information acquiring unit 54, a time acquiring unit 55, an object position acquiring unit 56, an output unit 57, a storage unit 58, a cargo attachment / detachment position / height estimating unit 59, and a management unit 60.

[0047] Each of these units is a function or a means for performing a function that is realized when any of the components shown in Figure 5 operates in accordance with instructions from CPU 501 in accordance with a program expanded from HD 504 onto RAM 503.

[0048] The forklift 10 also includes an omnidirectional camera 20, a transmitter 101, and a recording unit 102. The omnidirectional camera 20 acquires a video of the surroundings (a omnidirectional image). The recording unit 102 records the omnidirectional image acquired by the omnidirectional camera 20 on a recording medium. The transmitter 101 transmits the omnidirectional image acquired by the omnidirectional camera 20 to the on-premise server 50 via the network 400.

[0049] The functions of the transmitting unit 101 and the recording unit 102 can be realized by an electric circuit provided in either the forklift 10 or the omnidirectional camera 20, or by software (CPU). Alternatively, they may be realized by a plurality of circuits or a plurality of pieces of software.

[0050] The on-premise server 50 acquires the position information of the object 30 based on the position information of the forklift 10 acquired based on the omnidirectional image captured by the omnidirectional camera 20 and the holding information indicating whether the object 30 is being held or not being held by the forklift 10. The acquired position information of the object 30 can then be output to the outside via the output unit 57.

[0051] Receiving unit 51 receives, via network 400, the omnidirectional image captured by omnidirectional camera 20 and transmitted via transmitting unit 101, and outputs the omnidirectional image to moving object position acquiring unit 52 and retained information acquiring unit 53. Receiving unit 51 also receives, via network 400, identification information read by a reader such as a barcode reader, and outputs the identification information to identification information acquiring unit 54.

[0052] The mobile object position acquisition unit 52 acquires the position information of the forklift 10 by calculation based on the input spherical image, and outputs the acquired information to the object position acquisition unit 56. The process of acquiring the position information of the forklift 10 (self-position estimation process) can apply a technology for simultaneously creating a map and recognizing its own position (SLAM: Simultaneous Localization and Mapping) (see, for example, "Commentary: The Current State and Future Prospects of SLAM," by Tomono Masahiro and Hara Yoshitaka, Systems / Control / Information, Vol. 64, No. 2, 2020, pp. 45-50, https: / / www.jstage.jst.go.jp / article / isciesci / 64 / 2 / 64_45 / _article / -char / ja / ).

[0053] The holding information acquisition unit 53 acquires, by calculation, holding information indicating whether the object 30 is being held or not held by the forklift 10 based on the input spherical image, and outputs the information to the object position acquisition unit 56.

[0054] The identification information acquisition unit 54 acquires the identification information by inputting the identification information from the receiving unit 51, and outputs the identification information to the object position acquisition unit 56. However, acquisition of the identification information by the identification information acquisition unit 54 is not limited to via the network 400. For example, the identification information acquisition unit 54 may acquire identification information input by a user such as an administrator using the keyboard 511 or the pointing device 512 in FIG. 5, or may acquire identification information stored in advance in the storage unit 58, or may acquire identification information via the external device connection I / F 508. The administrator is the administrator of the information processing system 1 or the warehouse 100.

[0055] The time acquisition unit 55 acquires information indicating the time when the receiving unit 51 received the omnidirectional image and the identification information, and outputs the information to the object position acquisition unit 56 .

[0056] The object position acquisition unit 56 acquires the position information of the object 30 based on the position information of the forklift 10 and the holding information. The object position acquisition unit 56 also associates the position information of the object 30 with the identification information indicating the object 30 and time information, and outputs the information via the output unit 57. The output destination of the output unit 57 is an external device such as a PC (Personal Computer), a display device such as the display 506, a storage device such as the HD 504, or the like.

[0057] The cargo attachment / detachment position / height estimation unit 59 acquires a vehicle position estimation result, which is position information of the forklift 10, from the mobile object position acquisition unit 52, and also acquires a height estimation result, which is height position information of the fork 21 estimated based on the pattern of the marker M1 on the backrest 23 included in the input omnidirectional image, and estimates the attachment / detachment position and height when the forklift 10 attaches or detaches the object 30. The cargo attachment / detachment position / height estimation unit 59 is an example of an estimation unit that estimates the height of the object 30 by detecting the position of the marker M1 from the image of the marker M1 captured by the omnidirectional camera 20.

[0058] The storage section 58 can store identifying information indicative of an object 30 such as a pallet 31 or cargo 32 .

[0059] The management unit 60 manages the attachment state of the object 30 (cargo 32) on the forklift 10 and the detachment state of the object 30 (cargo 32).

[0060] (Example of processing by on-premise server 50) 7 is a flowchart showing an example of a cargo attachment / detachment position / height estimation process performed by the cargo attachment / detachment position / height estimation unit 59. Fig. 7 shows a process triggered by the timing at which the on-premise server 50 receives an operation to start acquiring the position information of the object 30. The operation to start acquiring the position information of the object 30 is performed by a user such as an administrator using the pointing device 512 in Fig. 5 or the like.

[0061] First, the cargo loading / unloading position / height estimation unit 59 acquires a vehicle position estimation result, which is position information of the forklift 10, from the moving body position acquisition unit 52, and also acquires a height estimation result, which is height position information of the fork 21 estimated based on the pattern of the marker M1 on the backrest 23 included in the input omnidirectional image (step S1).

[0062] Next, the cargo attachment / detachment position / height estimation unit 59 judges whether the height of the forks 21 of the forklift 10 is near the ground by comparing it with a preset threshold value (step S2).

[0063] When the cargo attachment / detachment position / height estimation unit 59 determines that the height of the forks 21 of the forklift 10 is near the ground (Yes in step S2), it determines whether the cargo status of the forks 21 has changed (whether the object 30 is still being held or the object 30 is still not being held) (step S3).

[0064] When the cargo attachment / detachment position / height estimation unit 59 determines that there is no change in the cargo status through continuous image recognition of the input spherical image for a certain period of time (Yes in step S3), it determines that the object 30 is still being held by the forks 21 or that the object 30 is still not being held by the forks 21, and terminates the processing.

[0065] On the other hand, if the cargo attachment / detachment position / height estimation unit 59 determines that there is a change in the cargo status through continuous image recognition of the input spherical image for a certain period of time (No in step S3), it determines that an attach (attaching the object 30 to the fork 21) or detach (detaching the object 30 from the fork 21) event has occurred.

[0066] When the cargo attachment / detachment position / height estimation unit 59 determines whether to attach or detach the object 30, it calculates the attachment / detachment position of the object 30 from the vehicle position (temporarily recorded and current position) and the set parameters (step S4).

[0067] Next, the cargo attachment / detachment position / height estimation unit 59 transmits the calculated attachment / detachment position / height of the object 30 to the management unit 60 (step S5).

[0068] Thereafter, the cargo attachment / detachment position / height estimation unit 59 resets the temporary record of the maximum height of the forks 21 and the vehicle position of the forklift 10 (step S6).

[0069] On the other hand, if the cargo loading / unloading position / height estimation unit 59 determines that the height of the forks 21 of the forklift 10 is not near the ground (No in step S2), it determines whether the height of the forks 21 is greater than the temporarily recorded value (step S7).

[0070] If the cargo loading / unloading position / height estimation unit 59 determines that the height of the forks 21 is greater than the temporarily recorded value (Yes in step S7), it updates the temporary record of the maximum height of the forks 21 and the vehicle position of the forklift 10 (step S8).

[0071] On the other hand, if the cargo attachment / detachment position / height estimation unit 59 determines that the height of the forks 21 is not greater than the temporarily recorded value (No in step S7), it ends the process.

[0072] Next, a description will be given of the processing in the management unit 60. Note that there are two ways of stacking the objects 30 (cargo 32): a rack stacking mode in which the objects 30 (cargo 32) are stacked on a shelf, and a direct stacking mode in which the objects 30 (cargo 32) are directly stacked, and it is assumed that the operator of the forklift 10 can manually switch between these modes, for example.

[0073] First, a description will be given of the flow of processing performed by the management unit 60 during attachment. Fig. 8 is a flowchart showing the flow of processing performed by the management unit 60 during attachment.

[0074] When the management unit 60 receives the attachment position and height of the object 30 (cargo 32) calculated by the cargo attachment / detachment position / height estimation unit 59 (step S11), it determines whether the attachment position and height of the object 30 (cargo 32) match those of an existing object 30 (cargo 32) within a threshold value (step S12).

[0075] If the management unit 60 determines that the attachment position and height of the object 30 (cargo 32) match those of an existing object 30 (cargo 32) within the threshold (Yes in step S12), it determines whether the mode is rack stacking, in which the object 30 (cargo 32) is stacked on a shelf (step S13).

[0076] If the management unit 60 determines that the mode is not rack stacking mode (No in step S13), it determines that the mode is direct stacking mode in which objects 30 (cargo 32) are stacked directly, and recognizes the attached existing objects 30 (cargo 32) and all existing objects 30 (cargo 32) stacked at a higher position than the attached existing objects 30 (cargo 32) as being taken out (step S14), and terminates the processing.

[0077] On the other hand, if the management unit 60 determines that the attachment position and height of the object 30 (cargo 32) do not match those of an existing object 30 (cargo 32) within the threshold value (No in step S12), it determines that the object 30 (cargo 32) is a new object 30 (cargo 32) (step S15) and terminates the processing.

[0078] On the other hand, if the management unit 60 determines that the mode is rack stacking mode (Yes in step S13), it determines that the mode is for stacking the object 30 (cargo 32) on a rack (shelf), determines that the existing object 30 (cargo 32) is being removed (step S16), and terminates the processing.

[0079] Next, a description will be given of the flow of processing performed by the management unit 60 at the time of detachment. Fig. 9 is a flowchart showing the flow of processing performed by the management unit 60 at the time of detachment.

[0080] When the management unit 60 receives the detachment position and height of the object 30 (cargo 32) calculated by the cargo attachment / detachment position / height estimation unit 59 (step S21), it determines whether the detachment height of the object 30 (cargo 32) is near the ground by comparing it with a preset threshold value (step S22).

[0081] When the management unit 60 determines that the detachment height of the object 30 (cargo 32) is not near the ground (No in step S22), it determines whether the mode is rack stacking, in which the object 30 (cargo 32) is stacked on a shelf (step S23).

[0082] If the management unit 60 determines that the mode is not rack stacking mode (No in step S23), it determines that the mode is direct stacking mode in which the object 30 (cargo 32) is directly stacked, and determines whether the detachment position and height of the object 30 (cargo 32) match the position of an existing object 30 (cargo 32) laid flat within a threshold value (step S24).

[0083] If the management unit 60 determines that the detachment position and height of the object 30 (cargo 32) matches that of an existing object 30 (cargo 32) within the threshold (Yes in step S24), it records it as a stack (step S25) and terminates the processing.

[0084] On the other hand, if the management unit 60 determines that the detachment height of the object 30 (cargo 32) is near the ground (Yes in step S22), it records it as flat cargo (step S26) and ends the process.

[0085] Furthermore, if the management unit 60 determines that the mode is rack stacking mode (Yes in step S23), it determines that the mode is for stacking the object 30 (cargo 32) on a rack (shelf), records the detachment position and height of the object 30 (cargo 32) (step S27), and terminates the processing.

[0086] Furthermore, if the management unit 60 determines that the detachment position and height of the object 30 (cargo 32) do not match those of an existing object 30 (cargo 32) within the threshold value (No in step S24), it records the detachment position and height of the object 30 (cargo 32) with an unknown flag indicating that they are unknown (step S28), and terminates the processing.

[0087] As described above, when placing another object 30 (cargo 32) on top of an object 30 (cargo 32) already placed on the ground, or when removing an object 30 (cargo 32) from an object 30 (cargo 32) already placed on the ground, the position information of the stacked (or removed) object 30 (cargo 32) is estimated using the position information of the forklift 10, the height of the detected fork 21, and the position information of the object 30 (cargo 32) placed on the ground.

[0088] As described above, according to this embodiment, markers are installed on the backrest of a forklift, which moves up and down together with the forks, and a spherical camera is attached to a location that is not on the same line as the direction of the marker's upward and downward movement, and images are acquired. The height of the fork can then be estimated not only from the size of the marker reflected in the image, but also from information about the marker's position in the image. This improves the accuracy of detecting the height of an object moved by a moving body.

[0089] According to this embodiment, the cargo attachment / detachment position / height estimation unit 59 performs image recognition on images acquired by a single omnidirectional camera 20 having a wide field of view that is spherical or close to spherical, thereby simplifying the configuration of the imaging device.

[0090] (Second embodiment) Next, a second embodiment will be described.

[0091] The second embodiment differs from the first embodiment that includes omnidirectional camera 20 in that it includes a sub-camera provided on the upper part of outer mast 22 and a main camera provided on head guard (roof of driver's seat) 10a. In the following description of the second embodiment, descriptions of the same parts as in the first embodiment will be omitted, and only differences from the first embodiment will be described.

[0092] FIG. 10 is a diagram showing the configuration of a forklift 10 according to the second embodiment.

[0093] 10, the forklift 10 of this embodiment includes a main camera 25 that captures images in an upward direction and a sub-camera 26 that captures images in a downward direction, instead of the omnidirectional camera 20 of the first embodiment. The main camera 25 and the sub-camera 26 are examples of an imaging unit provided in the forklift 10.

[0094] 10, the main camera 25 is provided on the head guard 10a of the forklift 10. The main camera 25 is an imaging device capable of capturing images with a viewing angle of 180 degrees upward around the main camera 25. The orientation 25a indicates the orientation in which the main camera 25 can capture images.

[0095] The images captured by main camera 25 are primarily used to determine the position of forklift 10. When determining the position of forklift 10 indoors using captured images, it is desirable to use an upward-facing main camera 25 capable of capturing images with a viewing angle of 180 degrees upward, so that the ceiling, pillars, and other building structures are easily captured.

[0096] 10, the sub-camera 26 is provided on the outer mast 22 at a location that is not on the same line as the upward and downward movement direction (vertical upward direction 12) of the forks 21 of the forklift 10 or the backrest 23 that moves up and down integrally with the forks 21. The sub-camera 26 is an imaging device that can capture images with a viewing angle of 180 degrees downward around the sub-camera 26. An orientation 26a indicates the orientation in which the sub-camera 26 can capture images.

[0097] The state of holding the object 30 by the forks 21 of the forklift 10 is detected using an image captured by the sub-camera 26 when the forks 21 are near the ground.

[0098] Furthermore, as will be described in detail later, the images captured by the main camera 25 and the images captured by the sub-camera 26 are used by the cargo attachment / detachment position / height estimation unit 59 to estimate the attachment / detachment position and height when the forklift 10 attaches or detaches the object 30. In general, when estimating the attachment / detachment position and height when the forklift 10 attaches or detaches the object 30, the cargo attachment / detachment position / height estimation unit 59 uses images captured by the sub-camera 26 up to a certain height from the ground, and images captured by the main camera 25 above the certain height. This method is advantageous for a counter-mounted forklift 10 because there is little obstruction of the marker M1 from the sub-camera 26 and the main camera 25 is located far from the axis of ascent of the marker M1 when detecting the marker M1. When the object 30 is attached or detached at a high position that is beyond the range of the sub-camera 26, the attachment / detachment position of the object 30 cannot be seen directly from the main camera 25, but the attachment / detachment position and height of the object 30 can be estimated by the cargo attachment / detachment position / height estimation unit 59.

[0099] In this embodiment, a marker M1, which serves as a landmark and is attached to the backrest 23, is captured by either the main camera 25 or the sub-camera 26. The image captured by either the main camera 25 or the sub-camera 26 is then recognized to detect the type of marker M1, and the height of the fork 21 is calculated from the type and position of the detected marker M1.

[0100] Next, switching between the two cameras (main camera 25 and sub-camera 26) by cargo attachment / detachment position / height estimation unit 59 will be described.

[0101] 11 is a flowchart showing the process of switching between the two cameras. As shown in FIG. 11, if the cargo attachment / detachment position / height estimation unit 59 can detect the marker M1 by performing image recognition on the image captured by the sub-camera 26 (Yes in step S31), it estimates the attachment / detachment position / height of the object 30 using the image captured by the sub-camera 26 (step S32).

[0102] On the other hand, if the cargo attachment / detachment position / height estimation unit 59 cannot detect the marker M1 through image recognition of the image captured by the sub-camera 26 (No in step S31), it determines whether it can detect the marker M1 through image recognition of the image captured by the main camera 25 (step S33).

[0103] If the cargo attachment / detachment position / height estimation unit 59 can detect the marker M1 by image recognition of the image captured by the main camera 25 (Yes in step S33), it estimates the attachment / detachment position / height of the object 30 using the image captured by the main camera 25 (step S34).

[0104] On the other hand, if the cargo attachment / detachment position / height estimation unit 59 cannot detect the marker M1 by image recognition of the image captured by the main camera 25 (No in step S33), it terminates the processing without estimating the attachment / detachment position / height of the object 30 (step S35).

[0105] As described above, according to this embodiment, markers are attached to the backrest of the forklift, which moves up and down together with the forks. A main camera capable of capturing images with a viewing angle of 180 degrees upward is attached to the head guard of the forklift, and a sub-camera capable of capturing images with a viewing angle of 180 degrees downward is attached in a location that is not on the same line as the direction of the marker's upward and downward movement. Images are acquired, and the height of the fork is estimated by image recognition. This makes it possible to estimate the height not only from the size of the marker reflected in the image, but also from information about the marker's position in the image. This improves the accuracy of detecting the height of an object moved by a moving vehicle.

[0106] In this embodiment, cylindrical markers M1 having different patterns are provided on the backrest 23, but the present invention is not limited to this and, for example, markers M1 having different patterns may be formed using LEDs (Light Emitting Diodes).Furthermore, the structure of the backrest 23 itself may be used as the marker M1.

[0107] (Third embodiment) Next, a third embodiment will be described.

[0108] In the first and second embodiments, a counter type forklift 10 is used as the mobile unit, but the third embodiment differs from the first and second embodiments in that a reach type forklift is used as the mobile unit. In the following description of the third embodiment, the same parts as those in the first and second embodiments will be omitted, and only differences from the first and second embodiments will be described.

[0109] Fig. 12 is a diagram showing the configuration of a forklift according to the third embodiment. As shown in Fig. 12, a forklift 70 according to this embodiment is a reach-type forklift in which forks 81, an outer mast 82 as a support member, a backrest 83, and an inner mast 84 as a member used for raising and lowering the forklift can be extended and retracted (moved forward and backward).

[0110] 12, forklift 70 has omnidirectional camera 80, which is an imaging device, attached to head guard 70a (roof of the driver's seat) of forklift 70. Orientation 80a indicates the orientation in which omnidirectional camera 80 can capture images. Furthermore, forklift 70 of this embodiment has marker M2 attached to outer mast 82, which is separate from marker M1 on backrest 83.

[0111] In this embodiment, the marker M2 is set on the outer mast 82, but this is not limiting and the marker M2 may be formed using, for example, an LED (Light Emitting Diode). Also, the structure of the outer mast 82 itself may be used as the marker M2.

[0112] As shown in FIG. 12, the forklift 70 has a structure in which parts including the fork 81, outer mast 82, backrest 83, inner mast 84, etc., protrude (reach).

[0113] With this configuration, the cargo attachment / detachment position / height estimation unit 59 of this embodiment can determine the height of the fork 81 by performing image recognition on the image captured by the omnidirectional camera 80 attached to the head guard 70a and detecting the type of marker M1 on the backrest 83.

[0114] Additionally, the cargo attachment / detachment position / height estimation unit 59 of this embodiment can calculate the reach of the forks 81 by detecting the markers M2 on the outer mast 82 through image recognition of an image captured by the omnidirectional camera 80 attached to the head guard 70a. By calculating the reach, the relative positional relationship between the forklift 70 and the object 30 being carried, i.e., the attachment / detachment position of the object 30, can be calculated more accurately.

[0115] In this embodiment, the omnidirectional camera 80 is attached to the head guard (roof of the driver's seat) 70a of the forklift 70, but the present invention is not limited to this.

[0116] [Variation 1] 13 is a diagram showing a first modified example of the configuration of the forklift truck according to the third embodiment. In the first modified example shown in FIG. 13, the forklift truck 70 may include a main camera 85 provided on the head guard (roof of the driver's seat) 70a and a sub-camera 86 provided on the upper part of the outer mast 82.

[0117] Main camera 85 is an imaging device capable of capturing images with a viewing angle of 180 degrees upward around main camera 85. Orientation 85a indicates the orientation in which main camera 85 can capture images.

[0118] The sub-camera 86 is an imaging device capable of capturing images with a viewing angle of 180 degrees downward around the sub-camera 86. An orientation 86a indicates an orientation in which the sub-camera 86 can capture images.

[0119] In the first modification shown in FIG. 13, a marker M2, which is separate from the marker M1 on the backrest 83, is attached to the main body 71 of the forklift .

[0120] In this case, the cargo attachment / detachment position / height estimation unit 59 can determine the height of the fork 81 by performing image recognition on the image captured by the main camera 85 attached to the head guard 70a or the sub-camera 86 attached to the outer mast 82 and detecting the type of marker M1 on the backrest 83.

[0121] Additionally, the forklift 70 can determine the reach of the forks 81 by detecting the marker M2 on the body of the forklift 70 through image recognition of an image captured by the sub-camera 86 provided on the upper part of the outer mast 82. Additionally, the cargo attachment / detachment position / height estimation unit 59 can stably detect the height of the forks 81 because the sub-camera 86 can easily recognize the marker M1 on the backrest 23 without any obstruction.

[0122] [Variation 2] 14 is a diagram showing a second modified example of the configuration of the forklift according to the third embodiment. In the second modified example shown in FIG. 14, the omnidirectional camera 80 is provided on the upper part of the outer mast 82, rather than on the head guard (roof of the driver's seat) 70a of the forklift 70.

[0123] In this case, the cargo loading / unloading position / height estimation unit 59 can perform vehicle positioning, cargo presence, and fork 81 height detection all based on the outer mast 82, eliminating the need to detect the amount of protrusion using the marker M2 in a reach forklift.

[0124] In the present embodiment, a forklift is used as an example of a moving object, but the moving object is not limited to this. For example, the moving object may be an automated guided vehicle, a drone, or the like.

[0125] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0126] Note that the information processing device is not limited to the on-premise server 50, as long as it is a device with a communication function. The information processing device may be, for example, an image forming device, a PJ (Projector), an IWB (Interactive White Board: a white board with an electronic blackboard function that allows mutual communication), an output device such as digital signage, a HUD (Head Up Display) device, industrial machinery, an imaging device, a sound collection device, medical equipment, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0127] For example, aspects of the present invention are as follows. <1> a moving body including a holding unit that holds an object and allows the object to move up and down, and a marker installed on the holding unit; at least one imaging device attached to the moving body at a location not on the same line as the upward and downward movement directions of the object, for capturing images of the surroundings of the moving body; an estimation unit that estimates the height of the object by detecting the position of the marker from the image of the marker captured by the imaging device; An information processing system comprising: <2> the estimation unit detects the position of the marker from an image of the marker captured by a single imaging device having a celestial sphere or a wide field of view close to the celestial sphere; Characterized by <1> An information processing system according to claim 1. <3> The estimation unit Detecting the position of the marker from an image of the marker captured by an imaging device capable of capturing an image with a viewing angle of 180 degrees or a direction close to 180 degrees upward, and estimating the height of the object at a certain height or above; Detecting the position of the marker from an image of the marker captured by another imaging device capable of capturing an image with a viewing angle of 180 degrees downward or in a direction close to 180 degrees, and estimating the height of the object up to a certain height from the ground. Characterized by <1> An information processing system according to claim 1. <4> the estimation unit detects the position of the marker from an image of the marker captured by a single imaging device that is attached to a roof of a driver's seat of the moving body and has a celestial sphere or a wide field of view close to the celestial sphere; Characterized by <1> An information processing system according to claim 1. <5> the estimation unit estimates the reach of the holding unit based on an image captured by the at least one imaging device. Characterized by <1> An information processing system according to claim 1. <6> the estimation unit estimates the reach of the holding unit based on an image captured by the at least one imaging device installed in the holding unit. Characterized by <1> An information processing system according to claim 1. <7> the estimation unit estimates the height of the object by detecting a position of the cylindrical marker placed on the holding unit from an image of the marker. Characterized by <1> Or <6> 10. The information processing system according to claim 9, wherein: <8> a management unit that manages the attachment / detachment state of the object on the moving body, Characterized by <1> Or <7> 10. The information processing system according to claim 9, wherein: <9> An information processing device that processes position information of an object moved by a moving body that includes a holding unit that holds the object and allows the object to move up and down, and a marker that is attached to the holding unit, an estimation unit that estimates the height of the object by detecting the position of a marker from an image of the marker attached to the holding unit, the image being captured by at least one imaging device attached to the moving body at a location not on the same line with respect to the upward and downward movement directions of the object; 1. An information processing device comprising: <10> 1. An information processing method by an information processing device for processing position information of an object moved by a moving body including: a holding unit that holds the object and allows the object to move upward and downward; and a marker that is attached to the holding unit, an estimation step of estimating the height of the object by detecting the position of a marker from an image of the marker attached to the holding unit, the marker being captured by at least one imaging device attached to the moving body at a location not on the same line with respect to the upward and downward movement directions of the object; 1. An information processing method comprising: [Explanation of symbols]

[0128] 10, 70 Mobile 10a, 70a Driver's seat roof 20, 25, 26, 80, 85, 86 Imaging device 21, 23 Holding part 30 Object 50 Information processing equipment 59 Estimation part 60 Management Department M1 marker [Prior art documents] [Patent documents]

[0129] [Patent Document 1] Patent No. 7240038

Claims

1. a moving body including a holding unit that holds an object and allows the object to move up and down, and a marker installed on the holding unit; at least one imaging device attached to the moving body at a location not on the same line as the upward and downward movement directions of the object, for capturing images of the surroundings of the moving body; an estimation unit that estimates the height of the object by detecting the position of the marker from the image of the marker captured by the imaging device; An information processing system comprising:

2. the estimation unit detects the position of the marker from an image of the marker captured by a single imaging device having a celestial sphere or a wide field of view close to the celestial sphere; 2. The information processing system according to claim 1, wherein:

3. The estimation unit Detecting the position of the marker from an image of the marker captured by an imaging device capable of capturing an image with a viewing angle of 180 degrees or a direction close to 180 degrees upward, and estimating the height of the object at a certain height or above; Detecting the position of the marker from an image of the marker captured by another imaging device capable of capturing an image with a viewing angle of 180 degrees downward or in a direction close to 180 degrees, and estimating the height of the object up to a certain height from the ground.

2. The information processing system according to claim 1, wherein:

4. the estimation unit detects the position of the marker from an image of the marker captured by a single imaging device that is attached to a roof of a driver's seat of the moving body and has a celestial sphere or a wide field of view close to the celestial sphere; 2. The information processing system according to claim 1, wherein:

5. the estimation unit estimates the reach of the holding unit based on an image captured by the at least one imaging device.

2. The information processing system according to claim 1, wherein:

6. the estimation unit estimates the reach of the holding unit based on an image captured by the at least one imaging device installed in the holding unit.

2. The information processing system according to claim 1, wherein:

7. the estimation unit estimates the height of the object by detecting a position of the cylindrical marker placed on the holding unit from an image of the marker; 2. The information processing system according to claim 1, wherein:

8. a management unit that manages the attachment / detachment state of the object on the moving body, 2. The information processing system according to claim 1, wherein:

9. An information processing device that processes position information of an object moved by a moving body that includes a holding unit that holds the object and allows the object to move up and down, and a marker that is attached to the holding unit, an estimation unit that estimates the height of the object by detecting the position of a marker from an image of the marker attached to the holding unit, the image being captured by at least one imaging device attached to the moving body at a location not on the same line with respect to the upward and downward movement directions of the object; 1. An information processing device comprising:

10. 1. An information processing method for an information processing device that processes position information of an object moved by a moving body that includes a holding unit that holds the object and allows the object to move up and down, and a marker that is attached to the holding unit, the method comprising: an estimation step of estimating the height of the object by detecting the position of a marker from an image of the marker attached to the holding unit, the marker being captured by at least one imaging device attached to the moving body at a location not on the same line with respect to the upward and downward movement directions of the object; 1. An information processing method comprising:

Images